Construct validity and test-retest reliability of the supervised online-administered Indonesian emotion word fluency test

Authors

  • Heni Gerda Pesau Department of Psychology, Faculty of Psychology, Atma Jaya University Makassar, Indonesia
  • Gilles van Luijtelaar Donders Centre for Cognition, Radboud University, The Netherlands; Bachelor Program, Faculty of Psychology, Maranatha Christian University, Indonesia

DOI:

https://doi.org/10.24854/jpu1045

Keywords:

cognition, emotion, neuropsychology, phonemic, semantic, verbal fluency

Abstract

The emotion word fluency test (EWFT), measuring the ability to produce emotional words, may measure a different cognitive construct of verbal flexibility than the commonly used phonemic (PVFT) and semantic (SVFT) verbal fluency tests. This was hypothesized and tested through Principal Component Analyses (PCA) of the test scores of 150 healthy subjects. The test-retest reliability was additionally examined after two weeks in forty subjects. Data was collected through supervised internet-delivered testing. Correlation analyses for construct validity showed that the EWFT correlated only moderately with PVFT and SVFT. PCA showed that a three-factor model explained 68.5% of the total variance. The subscales of each fluency test loaded primarily on separate factors, indicating that the three tests measure different underlying constructs. The test-retest revealed moderate reliability without a clear repetition/practice effect. It can be concluded that the supervised Indonesian online-administered EWFT measures emotion processing abilities. However, it awaits clinical validation in various clinical populations.

Downloads

Download data is not yet available.

References

Abeare, C. A., Freund, S., Kaploun, K., McAuley, T., & Dumitrescu, C. (2017). The Emotion Word Fluency Test (EWFT): Initial psychometric, validation, and physiological evidence in young adults. Journal of Clinical and Experimental Neuropsychology, 39(8), 738–752. https://doi.org/10.1080/13803395.2016.1259396

Abeare, C. A., An, K., Tyson, B., Holcomb, M., May, N., & Erdodi, L. A. (2021). The emotion word fluency test as an embedded performance validity indicator – Alone and in a multivariate validity composite. Applied Neuropsychology: Child, 11(4), 713–724. https://doi.org/10.1080/21622965.2021.1939027

Aita, S. L., Beach, J. D., Taylor, S. E., Borgogna, N. C., Harrell, M. N., & Hill, B. D. (2018). Executive, language, or both? An examination of the construct validity of verbal fluency measures. Applied Neuropsychology: Adult, 26(5), 441–451. https://doi.org/10.1080/23279095.2018.1439830

Altarriba, J. (2003). Does cariño equal “liking”? A theoretical approach to conceptual nonequivalence between languages. International Journal of Bilingualism, 7(3), 305–322. https://doi.org/10.1177/13670069030070030501

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (Eds.). (2014). Standards for educational and psychological testing. American Educational Research Association. https://www.testingstandards.net/uploads/7/6/6/4/76643089/9780935302356.pdf

Auerbach, R. P., Stewart, J. G., Stanton, C. H., Mueller, E. M., Pizzagalli, D. A. (2015). Emotion-processing biases and resting EEG activity in depressed adolescents. Depression and Anxiety, 32(9), 693–701. https://doi.org/10.1002/da.22381

Bevilaqua, C., Carvalho, M. R. De, Ribeiro, P., Palmer, S., Nardi, A. E., & Dias, G. P. (2016). Electroencephalographic findings in patients with major depressive disorder during cognitive or emotional tasks: A systematic review. Revista Brasileira de Psiquiatria 38(4), 338–346. https://doi.org/10.1590/1516-4446-2015-1834

Camodeca, A., Walcott, K., Hosack, A., & Todd, K. Q. (2021). Preliminary Evidence for the Emotion Word Fluency Test as a unique semantic fluency measure. Psychological Assessment, 33(2), 195–200. https://doi.org/10.1037/pas0000965

Capizzi, R., Fisher, M., Biagianti, B., Ghiasi, N., Currie, A., Fitzpatrick, K., Albertini, N., Vinogradov, S. (2021). Testing a novel web-based neurocognitive battery in the general community: Validation and usability study. Journal of Medical Internet Research, 23(5), Artikel e25082. https://doi.org/10.2196/25082

Dasen, P. R. (2022). Culture and cognitive development. Journal of Cross-Cultural Psychology, 53(7–8), 789–816. https://doi.org/10.1177/00220221221092409

Dewi, Z. L., Halim, M. S., & Derksen, J. (2018). Emotional intelligence competencies of three different ethnic groups in Indonesia. Asian Ethnicity, 19(1), 36–58, https://doi.org/10.1080/14631369.2017.1310615

Djiwatampu, L. W. (2009). Happy and shameful experiences among ethnic groups in Indonesia: Cognitive, affective and behavioral dimensions. Psychology: The Journal of Hellenic Psychological Society, 16(2), 175–184. https://doi.org/10.12681/psy_hps.23812

Faul, L., & LaBar, K. S. (2023). Mood-congruent memory revisited. Psychological Review, 130(6), 1421–1456. https://doi.org/10.1037/rev0000394

Frijda, N. H., Markam, S., Sato, K., & Wiers, R. (1995). Emotions and emotion words. In Russell, J. A., Fernández-Dols, J. M., Manstead, A. S. R., Wellenkamp, J. C. (Eds.), Everyday conceptions of emotion. Springer. https://doi.org/10.1007/978-94-015-8484-5_7

Fontaine, J. R. J., Poortinga, Y. H., Setiadi, B., & Markam, S. S. (2002). Cognitive structure of emotion terms in Indonesia and The Netherlands. Cognition and Emotion, 16(1), 61–86. https://doi.org/10.1080/02699933014000130

Hegefeld, H. M., Satpute, A. B., Ochsner, K. N., Davidow, J. Y., & Nook, E. C. (2023). Fluency generating emotion words correlates with verbal measures but not emotion regulation, alexithymia, or depressive symptoms. Emotion, 23(8), 2259–2269. https://doi.org/10.1037/emo0001229

Hendrawan, D., & Hatta, T. (2010). Evaluation of stimuli for development of the Indonesian version of verbal fluency task using ranking method. Psychologia, 53(1), 14–26. https://doi.org/10.2117/psysoc.2010.14

Lezak, M. D., Howieson, D. B., Bigler, E. D., & Tranel, D. (2012). Neuropsychological assessment (5th edition). Oxford University Press.

Li, Y., Li, P., Yang, Q. X., Eslinger, P. J., & Sica, C. T. (2017). Lexical-semantic search under different covert verbal fluency tasks: An fMRI study. Frontiers in Behavioral Neuroscience, 11, Artikel 131. https://doi.org/10.3389/fnbeh.2017.00131

Masturah, A. M. (2019). Which emotion is preferred by Indonesian People? [Conference paper]. 4th ASEAN conference on psychology, counseling, and humanities (ACPCH 2018), Indonesia. https://doi.org/10.2991/acpch-18.2019.10

Mulyadi, Beratha, N. L. S., Oktavianus, Sudipa, I. N. (2012). Emotion verbs in bahasa Indonesia and Asahan Malay language: Cross-language semantics analysis. e-Journal of Linguistics, 6(1), 1–19. https://ojs.unud.ac.id/index.php/eol/article/view/4584

Mousavi, N., Ali, M., Jalil, N., & Ali, B. (2020). Electroencephalographic characteristics of word finding during phonological and semantic verbal fluency tasks. Neuropsychopharmacology Reports, 40(3), 254–261. https://doi.org/10.1002/npr2.12129

Olson, K. E., O'Brien, M. A., Rogers, W. A., Charness, N. (2011). Diffusion of technology: Frequency of use for younger and older adults. Ageing International, 36(1), 123–145. https://doi.org/10.1007/s12126-010-9077-9

Pesau, H. G., Immanuel, A. S., Sulastri, A., Wulanyani, N. M. S., & van Luijtelaar, G. (2022). The influence of ethnicity on language tests: A comparison between Balinese and Banjarese. [Conference paper]. The 2nd International Conference on Biopsychosocial Issues, Semarang, Indonesia. https://conference.unika.ac.id/index.php/ssic/iconbi/paper/view/524

Pesau, H. G., Immanuel, A. S., Sulastri, A., & van Luijtelaar, G. (2023). The role of daily spoken language on the performance of language tests: The Indonesian experience. Bilingualism: Language and Cognition, 26(3), 538–549. https://doi.org/10.1017/S136672892200075X

Pesau, H. G., Utami, M. S. S., Sulastri, A., Suryani, A., Sanjaya, R., Kiling, I. Y., & Damayanti, Y. (2025). Comparison of traditional and tele-neuropsychological testing. Acta Neuropsychologica, 23(2), 189–198. https://doi.org/10.5604/01.3001.0055.1278

Pesau, H. G., & van Luijtelaar, G. (2021). Equivalence of traditional and internet-delivered testing of word fluency tasks. Jurnal Psikologi, 20(1), 35–49. https://doi.org/10.14710/jp.20.1.35-49

Pesau, H. G., Sulastri, A., & van Luijtelaar, G. (2024). Evaluation of Indonesian Version of the Emotion Word Fluency Test. Jurnal Psikologi, 22(2), 157–176. https://doi.org/10.14710/jp.22.2.157-176

Planck, M., Development, H., Masitah, A., & Hills, T. T. (2020). The emotional recall task: Juxtaposing recall and recognition-based affect scales. Journal of Experimental Psychology: Learning, Memory, and Cognition, 46(9), 1782–1794. https://doi.org/10.1037/xlm0000841

Poder, T. G., Coulibaly, L. P., Hassan, A. I., Conombo, B., Laberge, M. (2022). Test–retest reliability of the Cost for Patients Questionnaire. International Journal of Technology Assessment in Health Care, 38(1), Artikel e65. https://doi.org/10.1017/S0266462322000460.

Röttger-Rössler, B., & Markowitsch, H. J. (2009). Introduction. In B. Röttger-Rössler & H. J. Markowitsch (Eds.). Emotions as bio-cultural processes (pp. 1–10). Springer.

Ryan, J. J., Glass, L. A., Hinds, R. M., & Brown, C. N. (2019). Administration order effects on the test of memory malingering. Applied Neuropsychology, 17(4), 246–250. https://doi.org/10.1080/09084282.2010.499802

Schober, P., & Boer, C. (2018). Correlation coefficients: Appropriate use and interpretation. Anesthesia and Analgesia, 126(5), 1763–1768. https://doi.org/10.1213/ANE.0000000000002864

Schreiter, M. L., Chmielewski, W. X., Mückschel, M., Ziemssen, T., & Beste, C. (2019). How the depth of processing modulates emotional interference – evidence from EEG and pupil diameter data. Cognitive, Affective, & Behavioral Neuroscience, 19, 1231–1246. https://doi.org/10.3758/s13415-019-00732-0

Shao, Z., Janse, E., Visser, K., & Meyer, A. S. (2014). What do verbal fluency tasks measure? Predictors of verbal fluency performance in older adults. Frontiers in Psychology, 5, Artikel 772. https://doi.org/10.3389/fpsyg.2014.00772

Villalobos, D., Torres-Simón, L., Pacios, J., Paúl, N., & Del Río, D. (2023). A systematic review of normative data for verbal fluency test in different languages. Neuropsychology Review, 33(4), 733–764. https://doi.org/10.1007/s11065-022-09549-0

Wahyuningrum, S. E., Sulastri, A., Hendriks, M. P. H., & van Luijtelaar, G. (2022). The Indonesian Neuropsychological Test Battery (INTB): Psychometric properties, preliminary normative scores, the underlying cognitive constructs and the effects of age and education. Acta Neuropsychologica, 20(4), 445–470. https://doi.org/10.5604/01.3001.0016.1339

Wauters, L., & Marquardt, T. P. (2018). Category, letter, and emotional verbal fluency in Spanish-English bilingual speakers: A preliminary report. Archives of Clinical Neuropsychology, 33(4), 444–457. https://doi.org/10.1093/arclin/acx063

Wyse, A. E. (2021). How days between tests impacts alternate forms reliability in computerized adaptive tests. Educational and Psychological Measurement, 81(4), 644–667. https://doi.org/10.1177/0013164420979656

Zuhdi, M. S., & Nuqul, F. L. (2022). Konsepsi emosi marah dalam perspektif budaya di Indonesia: sebuah pendekatan indigenous psychology. Jurnal Psikologi, 18(1), 51–62. https://doi.org/10.24014/jp.v18i1.14680

Downloads

Published

18-05-2026

How to Cite

Pesau, H. G., & van Luijtelaar, G. (2026). Construct validity and test-retest reliability of the supervised online-administered Indonesian emotion word fluency test. Jurnal Psikologi Ulayat. https://doi.org/10.24854/jpu1045

Issue

2026: Online first

Section

Original Research

Citation Check

License

Copyright (c) 2026 Heni Gerda Pesau, Gilles van Luijtelaar

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

The authors agree to the following terms:

  1. Authors retain copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
  2. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) for the exposure of their work with an acknowledgement of Jurnal Psikologi Ulayat: Indonesian Journal of Indigenous Psychology (JPU) as an outlet of their published work.

 


This work is licensed under a Creative Commons Attribution 4.0 International License.