Speech-Based Virtual Assistant for Mental Health Support  Through Natural Interaction

Dimas Rizqi Kurniawan; Donny Avianto

doi:10.29207/resti.v9i6.6585

Authors

Dimas Rizqi Kurniawan Universitas Teknologi Yogyakarta http://orcid.org/0009-0006-5658-4676
Donny Avianto Universitas Teknologi Yogyakarta http://orcid.org/0000-0001-5499-5478

DOI:

https://doi.org/10.29207/resti.v9i6.6585

Keywords:

depression rate, generative ai, speech recognition, tacotron2 waveglow, virtual assistant

Abstract

Mental health is a significant global concern. Indonesia has reported high rates of depression and anxiety, compounded by limited emotional outlets. Although AI virtual assistants are prevalent in e-commerce and education, their application in mental health remains underexplored. Existing solutions are predominantly text-based and transactional, which restricts empathetic and natural interactions. This study develops a voice-based assistant by integrating Automatic Speech Recognition (ASR), a generative AI for empathetic responses, and a Text-to-Speech (TTS) module fine-tuned on an Indonesian dataset to adapt accent and prosody. The system underwent both technical evaluation and human testing to assess its feasibility and user experience. The results showed that the TTS model converged effectively with low loss. Human evaluation indicated 'good' interaction (MS = 3.91, SD = 0.02), 'good' AI responses (MS = 3.83, SD = 0.26), and 'fair' TTS naturalness (MOS = 3.27, SD = 0.05). Most participants found the assistant's responses meaningful, pleasant, and helpful in managing low to moderate anxiety. These results suggest that a voice-based assistant has the potential to support mental health in Indonesia. Future work should enhance speech naturalness and utilize a larger participant pool for evaluation.

Downloads

Download data is not yet available.

References

J. Liu, W. Ning, N. Zhang, B. Zhu, and Y. Mao, “Estimation of the Global Disease Burden of Depression and Anxiety between 1990 and 2044: An Analysis of the Global Burden of Disease Study 2019,” Healthcare, vol. 12, p. 1721, Dec. 2024, doi: 10.3390/healthcare12171721.

Tim Penyusun SKI 2023 Dalam Angka, “SURVEI KESEHATAN INDONESIA 2023 DALAM ANGKA,” 2024. Accessed: Dec. 16, 2024. [Online]. Available: https://www.badankebijakan.kemkes.go.id/ski-2023-dalam-angka/

S. Umadiyan and A. D. Kalifia, “Perbedaan Respon Emosional Antara Remaja Perempuan Dan Laki-Laki Terhadap Faktor-Faktor Penyebab Depresi,” Gudang Jurnal Multidisiplin Ilmu, vol. 2, no. 1, pp. 293–296, 2024.

K. A. Ainunnida and N. Hartini, “Hubungan antara kesepian dengan ide bunuh diri yang dimoderasi oleh depresi pada remaja dengan orang tua bercerai,” Berajah Journal, vol. 2, no. 3, pp. 647–660, 2022.

S. Zhang, Z. Meng, B. Chen, X. Yang, and X. Zhao, “Motivation, social emotion, and the acceptance of artificial intelligence virtual assistants—Trust-based mediating effects,” Front Psychol, vol. 12, p. 728495, 2021.

J. de Freitas, A. K. Uğuralp, Z. Uğuralp, and S. Puntoni, “AI Companions Reduce Loneliness,” no. 24–078, Jun. 2024, doi: 10.2139/ssrn.4893097.

M. D. Rosanti, S. H. Wijoyo, and A. Rachmadi, “Analisis Pengaruh Automated Customer Service Berbasis Artificial Intelligence Pada Aplikasi E-commerce (Studi Kasus Aplikasi Lazada),” Jurnal Pengembangan Teknologi Informasi dan Ilmu Komputer, vol. 9, no. 5, 2025.

T. M. Rahardyan, C. H. Susilo, A. M. N. Iswara, and M. L. Hartono, “ChatGPT: The Future Research Assistant or an Academic Fraud?[A Case Study on a State University Located in Jakarta, Indonesia],” Asia Pacific Fraud Journal, vol. 9, no. 2, pp. 275–293, 2024.

T. Koulouri, R. D. Macredie, and D. Olakitan, “Chatbots to Support Young Adults’ Mental Health: An Exploratory Study of Acceptability,” ACM Trans. Interact. Intell. Syst., vol. 12, no. 2, Jul. 2022, doi: 10.1145/3485874.

R. Hennequin, A. Khlif, F. Voituret, and M. Moussallam, “Spleeter: a fast and efficient music source separation tool with pre-trained models,” J Open Source Softw, vol. 5, no. 50, p. 2154, 2020, doi: 10.21105/joss.02154.

R. Yoneyama, R. Yamamoto, and K. Tachibana, “Nonparallel High-Quality Audio Super Resolution with Domain Adaptation and Resampling CycleGANs,” in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5. doi: 10.1109/ICASSP49357.2023.10097002.

V. C. Govoreanu and M. Neghină, “Speech emotion recognition method using time-stretching in the preprocessing phase and artificial neural network classifiers,” in 2020 IEEE 16th International Conference on Intelligent Computer Communication and Processing (ICCP), 2020, pp. 69–74.

Q. M. M. Zarandah, S. M. Daud, and S. S. Abu-Naser, “Spectrogram flipping: a new technique for audio augmentation,” J Theor Appl Inf Technol, vol. 101, no. 11, 2023.

H. Guo, S. Zhang, F. K. Soong, L. He, and L. Xie, “Conversational End-to-End TTS for Voice Agents,” in 2021 IEEE Spoken Language Technology Workshop (SLT), 2021, pp. 403–409. doi: 10.1109/SLT48900.2021.9383460.

A. Karim, “Text to speech using Mel-Spectrogram with deep learning algorithms,” Periodicals of Engineering and Natural Sciences (PEN), vol. 10, pp. 380–386, Dec. 2022, doi: 10.21533/pen.v10i3.3113.

K. Alexander, “THE ANALYSIS OF THE EFFICIENCY OF GENERATIVE AI ALGORITHMS FOR CREATING A NATURAL DIALOGUE,” The American Journal of Interdisciplinary Innovations and Research, vol. 6, no. 6, pp. 26–34, Jun. 2024, doi: 10.37547/tajiir/Volume06Issue06-08.

G. Team et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” 2024. [Online]. Available: https://arxiv.org/abs/2403.05530

R. Liu, B. Sisman, G. Gao, and H. Li, “Expressive TTS Training with Frame and Style Reconstruction Loss,” IEEE/ACM Trans Audio Speech Lang Process, vol. 29, pp. 1806–1818, 2021, doi: 10.1109/TASLP.2021.3076369.

M. J. Hwang, R. Yamamoto, E. Song, and J. M. Kim, “Tts-by-tts: Tts-driven data augmentation for fast and high-quality speech synthesis,” in ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, Institute of Electrical and Electronics Engineers Inc., 2021, pp. 6598–6602. doi: 10.1109/ICASSP39728.2021.9414408.

M. R. Hasanabadi, “An overview of text-to-speech systems and media applications,” arXiv preprint arXiv:2310.14301, 2023.

A. F. A. Aziz, S. Tiun, and N. Ruslan, “End to end text to speech synthesis for malay language using tacotron and tacotron 2,” International Journal of Advanced Computer Science and Applications, vol. 14, no. 6, 2023.

R. J. Weiss, R. J. Skerry-Ryan, E. Battenberg, S. Mariooryad, and D. P. Kingma, “Wave-Tacotron: Spectrogram-Free End-To-End Text-To-Speech Synthesis,” in ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, Institute of Electrical and Electronics Engineers Inc., 2021, pp. 5679–5683. doi: 10.1109/ICASSP39728.2021.9413851.

Y. Jiao, A. Gabryś, G. Tinchev, B. Putrycz, D. Korzekwa, and V. Klimkov, “Universal neural vocoding with parallel wavenet,” in ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, Institute of Electrical and Electronics Engineers Inc., 2021, pp. 6044–6048. doi: 10.1109/ICASSP39728.2021.9414444.

G. Sanjay, K. C. Sooraj, and D. Mishra, “Natural Text-to-Speech Synthesis by Conditioning Spectrogram Predictions from Transformer Network on WaveGlow Vocoder,” in 2020 7th International Conference on Soft Computing and Machine Intelligence, ISCMI 2020, Institute of Electrical and Electronics Engineers Inc., Nov. 2020, pp. 255–259. doi: 10.1109/ISCMI51676.2020.9311564.

C. Xu et al., “Recent Advances in Direct Speech-to-text Translation,” 2023. [Online]. Available: https://arxiv.org/abs/2306.11646

H. Kheddar, M. Hemis, and Y. Himeur, “Automatic speech recognition using advanced deep learning approaches: A survey,” Information Fusion, vol. 109, p. 102422, 2024, doi: https://doi.org/10.1016/j.inffus.2024.102422.

D. Machacek, R. Dabre, and O. Bojar, “Turning Whisper into Real-Time Transcription System,” 2023. [Online]. Available: https://arxiv.org/abs/2307.14743

A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever, “Robust Speech Recognition via Large-Scale Weak Supervision,” in Proceedings of the 40th International Conference on Machine Learning, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, Eds., in Proceedings of Machine Learning Research, vol. 202. PMLR, Apr. 2023, pp. 28492–28518. [Online]. Available: https://proceedings.mlr.press/v202/radford23a.html

R. Jain, M. Y. Yiwere, D. Bigioi, P. Corcoran, and H. Cucu, “A Text-to-Speech Pipeline, Evaluation Methodology, and Initial Fine-Tuning Results for Child Speech Synthesis,” IEEE Access, vol. 10, pp. 47628–47642, 2022, doi: 10.1109/ACCESS.2022.3170836.

C. Chen, Y. Hu, W. Wu, H. Wang, E. S. Chng, and C. Zhang, “Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback,” 2024. [Online]. Available: https://arxiv.org/abs/2406.00654

S. Wang, E. Székely, and J. Gustafson, “Contextual Interactive Evaluation of TTS Models in Dialogue Systems,” in 25th Interspeech Conferece 2024, Kos Island, Greece, Sep 1 2024-Sep 5 2024, 2024, pp. 2965–2969.

N. M. Johari and P. N. Nohuddin, “Developing a Good Quality Chatbot,” J. Inf. Syst. Technol. Manag, vol. 6, pp. 88–102, 2021.

I. K. F. Haugeland, A. Følstad, C. Taylor, and C. A. Bjørkli, “Understanding the user experience of customer service chatbots: An experimental study of chatbot interaction design,” Int J Hum Comput Stud, vol. 161, p. 102788, 2022, doi: https://doi.org/10.1016/j.ijhcs.2022.102788.