Systematic analysis of generative AI tools integration in academic research and peer review

Husain Abdulrasool Salman 1 * , Muhammad Aliif Ahmad 2, Roliana Ibrahim 2, Jamilah Mahmood 2
More Detail
1 University of Bahrain, Zallaq, BAHRAIN
2 Universiti Teknologi Malaysia, Johor Bahru, MALAYSIA
* Corresponding Author
Online Journal of Communication and Media Technologies, Volume 15, Issue 1, Article No: e202502. https://doi.org/10.30935/ojcmt/15832
OPEN ACCESS   70 Views   36 Downloads   Published online: 06 Jan 2025
Download Full Text (PDF)

ABSTRACT

While sparking a big debate among academics, generative artificial intelligence (GAI) tools are becoming integral to academic research, holding the potential to transform traditional research and peer review methods. This systematic literature review investigates the emergent role of GAI tools in academic research workflow and scholarly publications by analyzing 44 articles. The process of identifying the most relevant publications was done following the preferred reporting items for systematic reviews and meta-analyses method. The findings provide a thorough understanding of how GAI is currently being utilized in the various aspects of academic research workflow and peer review process, including concerns, limitations, and proactive measures to better employ these tools effectively. Our review suggests the need for more research to develop appropriate policies and guidelines, enhance researchers’ artificial intelligence literacy through targeted training, and ensure ethical use of these tools to boost research productivity and quality.
Keywords: generative AI, chatbot, ChatGPT, academic research, scientific writing, peer review

CITATION

Salman, H. A., Ahmad, M. A., Ibrahim, R., & Mahmood, J. (2025). Systematic analysis of generative AI tools integration in academic research and peer review. Online Journal of Communication and Media Technologies, 15(1), e202502. https://doi.org/10.30935/ojcmt/15832

REFERENCES

  • Abdelhafiz, A. S., Ali, A., Maaly, A. M., Ziady, H. H., Sultan, E. A., & Mahgoub, M. A. (2024). Knowledge, perceptions and attitude of researchers towards using ChatGPT in research. Journal of Medical Systems, 48, Article 26. https://doi.org/10.1007/s10916-024-02044-4
  • Abuyaman, O. (2023). Strengths and weaknesses of ChatGPT models for scientific writing about medical vitamin B12: Mixed methods study. JMIR Formative Research, 7, e49459. https://doi.org/10.2196/49459
  • Aiumtrakul, N., Thongprayoon, C., Suppadungsuk, S., Krisanapan, P., Miao, J., Qureshi, F., & Cheungpasitporn, W. (2023). Navigating the landscape of personalized medicine: The relevance of ChatGPT, BingChat, and Bard AI in nephrology literature searches. Journal of Personalized Medicine, 13(10), Article 1457. https://doi.org/10.3390/jpm13101457
  • Alkaissi, H., & McFarlane, S. I. (2023). Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus, 15(2), Article e35179. https://doi.org/10.7759/cureus.35179
  • Alshami, A., Elsayed, M., Ali, E., Eltoukhy, A. E., & Zayed, T. (2023). Harnessing the power of ChatGPT for automating systematic review process: Methodology, case study, limitations, and future directions. Systems, 11, Article 351. https://doi.org/10.3390/systems11070351
  • Alyasiri, O. M., Salman, A. M., & Salisu, S. (2024). ChatGPT revisited: Using ChatGPT-4 for finding references and editing language in medical scientific articles. Journal of Stomatology, Oral and Maxillofacial Surgery, 125(5), Article 101842. https://doi.org/10.1016/j.jormas.2024.101842
  • Anghelescu, A., Ciobanu, I., Munteanu, C., Anghelescu, L. A. M., & Onose, G. (2023). ChatGPT: “To be or not to be” … in academic research. the human mind’s analytical rigor and capacity to discriminate between ai bots’ truths and hallucinations. Balneo and PRM Research Journal, 14, Article 4. https://doi.org/10.12680/balneo.2023.614
  • Ariyaratne, S., Iyengar, K. P., Nischal, N., Chitti Babu, N., & Botchu, R. (2023). A comparison of ChatGPT-generated articles with human-written articles. Skeletal Radiology, 52, 1755–1758. https://doi.org/10.1007/s00256-023-04340-5
  • Athaluri, S. A., Manthena, S. V., Kesapragada, V. S. R. K. M., Yarlagadda, V., Dave, T., & Duddumpudi, R. T. S. (2023). Exploring the boundaries of reality: Investigating the phenomenon of artificial intelligence hallucination in scientific writing through ChatGPT references. Cureus, 15(4), Article e37432. https://doi.org/10.7759/cureus.37432
  • Aydin, A., Yürük, S. E., Reisoğlu, İ., & Goktas, Y. (2022). Main barriers and possible enablers of academicians while publishing. Scientometrics, 128, 623–650. https://doi.org/10.1007/s11192-022-04528-x
  • Behera, R. K., Bala, P. K., & Dhir, A. (2019). The emerging role of cognitive computing in healthcare: A systematic literature review. International Journal of Medical Informatics, 129, 154–166. https://doi.org/10.1016/j.ijmedinf.2019.04.024
  • Bekker, M. (2024). Large language models and academic writing: Five tiers of engagement. South African Journal of Science, 120(1/2). https://doi.org/10.17159/sajs.2024/17147
  • Biswas, S., Dobaria, D., & Cohen, H. L. (2023). ChatGPT and the future of journal reviews: A feasibility study. Yale Journal of Biology and Medicine, 96, 415–420. https://doi.org/10.59249/SKDH9286
  • Bond, A., Cilliers, D., Retief, F., Alberts, R., Roos, C., & Moolman, J. (2024). Using an artificial intelligence chatbot to critically review the scientific literature on the use of artificial intelligence in environmental impact assessment. Impact Assessment and Project Appraisal, 42(2), 189–199. https://doi.org/10.1080/14615517.2024.2320591
  • Burger, B., Kanbach, D. K., Kraus, S., Breier, M., & Corvello, V. (2023). On the use of AI-based tools like ChatGPT to support management research. European Journal of Innovation Management, 26, 233–241. https://doi.org/10.1108/EJIM-02-2023-0156
  • Carabantes, D., González-Geraldo, J. L., & Jover, G. (2023). ChatGPT could be the reviewer of your next scientific paper. Evidence on the limits of AI-assisted academic reviews. El Profesional de la Informacion, 32(5). https://doi.org/10.3145/epi.2023.sep.16
  • Checco, A., Bracciale, L., Loreti, P., Pinfield, S., & Bianchi, G. (2021). AI-assisted peer review. Humanities and Social Sciences Communications, 8, Article 25. https://doi.org/10.1057/s41599-020-00703-8
  • Chen, T., Gascó-Hernandez, M., & Esteve, M. (2024). The adoption and implementation of artificial intelligence chatbots in public organizations: Evidence from US state governments. The American Review of Public Administration, 54, 255–270. https://doi.org/10.1177/02750740231200522
  • Crawford, J., Cowling, M., Ashton-Hay, S., Kelder, J.-A., Middleton, R., & Wilson, G. (2023). Artificial intelligence and authorship editor policy: ChatGPT, Bard Bing AI, and beyond. Journal of University Teaching and Learning Practice, 20(5). https://doi.org/10.53761/1.20.5.01
  • Dashti, M., Londono, J., Ghasemi, S., & Moghaddasi, N. (2023). How much can we rely on artificial intelligence chatbots such as the ChatGPT software program to assist with scientific writing? The Journal of Prosthetic Dentistry. https://doi.org/10.1016/j.prosdent.2023.05.023
  • Datt, M., Sharma, H., Aggarwal, N., & Sharma, S. (2023). Role of ChatGPT-4 for medical researchers. Annals of Biomedical Engineering, 52, 1534–1536. https://doi.org/10.1007/s10439-023-03336-5
  • Dengel, A., Gehrlein, R., Fernes, D., Görlich, S., Maurer, J., Pham, H. H., Großmann, G., & Eisermann, N. D. G. (2023). Qualitative research methods for large language models: Conducting semi-structured interviews with ChatGPT and BARD on computer science education. Informatics, 10(4), Article 78. https://doi.org/10.3390/informatics10040078
  • Dergaa, I., Chamari, K., Zmijewski, P., & Ben Saad, H. (2023). From human writing to artificial intelligence generated text: Examining the prospects and potential threats of ChatGPT in academic writing. Biology of Sport, 40, 615–622. https://doi.org/10.5114/biolsport.2023.125623
  • Dhanvijay, A. K. D., Pinjar, M. J., Dhokane, N., Sorte, S. R., Kumari, A., & Mondal, H. (2023). Performance of large language models (ChatGPT, Bing Search, and Google Bard) in solving case vignettes in physiology. Cureus, 15(8), Article e42972. https://doi.org/10.7759/cureus.42972
  • Ebert, C., & Louridas, P. (2023). Generative AI for software practitioners. IEEE Software, 40, 30–38. https://doi.org/10.1109/MS.2023.3265877
  • Eigenmann, P., Akenroye, A., Markovic, M. A., Candotti, F., Ebisawa, M., Genuneit, J., Kalayci, Ö., Kollmann, D., Leung, A. S. Y., & Peters, R. L. (2023). Pediatric allergy and immunology (PAI) is for polishing with artificial intelligence, but careful use. Pediatric Allergy and Immunology, 34(9), Article e14023. https://doi.org/10.1111/pai.14023
  • Firat, M. (2023). What ChatGPT means for universities: Perceptions of scholars and students. Journal of Applied Learning & Teaching, 6(1), 1–22. https://doi.org/10.37074/jalt.2023.6.1.22
  • Gupta, R., Park, J. B., Bisht, C., Herzog, I., Weisberger, J., Chao, J., Chaiyasate, K., & Lee, E. S. (2023). Expanding cosmetic plastic surgery research with ChatGPT. Aesthetic Surgery Journal, 3, 930–937. https://doi.org/10.1093/asj/sjad069
  • Hadi, M. U., Qureshi, R., Shah, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., & Mirjalili, S. (2023). A survey on large language models: Applications, challenges, limitations, and practical usage. TechRxiv. https://doi.org/10.36227/techrxiv.23589741.v1
  • Hake, J., Crowley, M., Coy, A., Shanks, D., Eoff, A., Kirmer-Voss, K., Dhanda, G., & Parente, D. J. (2024). Quality, accuracy, and bias in ChatGPT-based summarization of medical abstracts. Annals of Family Medicine, 22, 113–120. https://doi.org/10.1370/afm.3075
  • Hamamah, Emaliana, I., Hapsari, Y., Degeng, P. D. D., & Fadillah, A. C. (2023). Using nominal group technique to explore publication challenges and the usefulness of AI-based writing technologies: Insights from Indonesian scholars. Theory and Practice in Language Studies, 13(8), 2038–2047. https://doi.org/10.17507/tpls.1308.20
  • Hosseini, M., & Horbach, S. P. J. M. (2023). Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review. Research Integrity and Peer Review, 8, Article 4. https://doi.org/10.1186/s41073-023-00133-5
  • Huang, J., & Tan, M. (2023). The role of ChatGPT in scientific communication: Writing better scientific review articles. American Journal of Cancer Research, 13, Article 1148.
  • Imran, M., & Almusharraf, N. (2023). Analyzing the role of ChatGPT as a writing assistant at higher education level: A systematic review of the literature. Contemporary Educational Technology, 15(4), Article ep464. https://doi.org/10.30935/cedtech/13605
  • Jarrah, A. M., Wardat, Y., & Fidalgo, P. (2023). Using ChatGPT in academic writing is (not) a form of plagiarism: What does the literature say? Online Journal of Communication and Media Technologies, 13(4), Article e202346. https://doi.org/10.30935/ojcmt/13572
  • Jenko, N., Ariyaratne, S., Jeys, L., Evans, S., Iyengar, K., & Botchu, R. (2024). An evaluation of AI generated literature reviews in musculoskeletal radiology. The Surgeon, 22(3), 194–197. https://doi.org/10.1016/j.surge.2023.12.005
  • Keele, S. (2007). Guidelines for performing systematic literature reviews in software engineering. EBSE.
  • Khalifa, A. A., & Ibrahim, M. A. (2024). Artificial intelligence (AI) and ChatGPT involvement in scientific and medical writing, a new concern for researchers. A scoping review. Arab Gulf Journal of Scientific Research, 42(4), 1770–1787. https://doi.org/10.1108/AGJSR-09-2023-0423
  • Khalifa, M., & Albadawy, M. (2024). Using artificial intelligence in academic writing and research: An essential productivity tool. Computer Methods and Programs in Biomedicine Update, 5, Article 100145. https://doi.org/10.1016/j.cmpbup.2024.100145
  • Khlaif, Z. N., Mousa, A., Hattab, M. K., Itmazi, J., Hassan, A. A., Sanmugam, M., & Ayyoub, A. (2023). The potential and concerns of using AI in scientific research: ChatGPT performance evaluation. JMIR Medical Education, 9, Article e47049. https://doi.org/10.2196/47049
  • Kraus, S., Breier, M., & Dasí-Rodríguez, S. (2020). The art of crafting a systematic literature review in entrepreneurship research. International Entrepreneurship and Management Journal, 16, 1023–1042. https://doi.org/10.1007/s11365-020-00635-4
  • Leong, A. P. (2023). Clause complexing in research-article abstracts: Comparing human- and AI-generated texts. ExELL, 11(2), 9–132. https://doi.org/10.2478/exell-2023-0008
  • Livberber, T. (2023). Toward non-human-centered design: Designing an academic article with ChatGPT. Profesional de la Informacion, 32(5). https://doi.org/10.3145/epi.2023.sep.12
  • Lozić, E., & Štular, B. (2023). Fluent but not factual: A comparative analysis of ChatGPT and other AI chatbots’ proficiency and originality in scientific writing for humanities. Future Internet, 15(10), Article 336. https://doi.org/10.3390/fi15100336
  • Mahyoob, M., Algaraady, J., & Alblwi, A. (2023). A proposed framework for human-like language processing of ChatGPT in academic writing. International Journal of Emerging Technologies in Learning, 18(14), 282–293. https://doi.org/10.3991/ijet.v18i14.41725
  • Májovský, M., Černý, M., Kasal, M., Komarc, M., & Netuka, D. (2023). Artificial intelligence can generate fraudulent but authentic-looking scientific medical articles: Pandora’s box has been opened. Journal of Medical Internet Research, 25, Article e46924. https://doi.org/10.2196/46924
  • Margetts, T. J., Karnik, S. J., Wang, H. S., Plotkin, L. I., Oblak, A. L., Fehrenbacher, J. C., Kacena, M. A., & Movila, A. (2024). Use of AI language engine ChatGPT 4.0 to write a scientific review article examining the intersection of Alzheimer’s disease and bone. Current Osteoporosis Reports, 22, 177–181. https://doi.org/10.1007/s11914-023-00853-z
  • Mehta, J. (2023). Growing importance of academic research in education. EPRA International Journal of Research & Development, 8, 261–264.
  • Mohamed Shaffril, H. A., Samsuddin, S. F., & Abu Samah, A. (2021). The ABC of systematic literature review: The basic methodological guidance for beginners. Quality & Quantity, 55, 1319–1346. https://doi.org/10.1007/s11135-020-01059-6
  • Mohammed Alyasiri, O., Salman, A. M., Akhtom, D., & Salisu, S. (2024). ChatGPT revisited: Using ChatGPT-4 for finding references and editing language in medical scientific articles. Journal of Stomatology, Oral and Maxillofacial Surgery, 125(5, Supplement 2), Article 101842. https://doi.org/10.1016/j.jormas.2024.101842
  • Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & Prisma-P Group. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews, 4(1), Article 1. https://doi.org/10.1186/2046-4053-4-1
  • Mollaki, V. (2024). Death of a reviewer or death of peer review integrity? the challenges of using AI tools in peer reviewing and the need to go beyond publishing policies. Research Ethics, 20(2), 239–250. https://doi.org/10.1177/17470161231224552
  • Mondal, H., & Mondal, S. (2023). ChatGPT in academic writing: Maximizing its benefits and minimizing the risks. Indian Journal of Ophthalmology, 71(12), 3600–3606. https://doi.org/10.4103/IJO.IJO_718_23
  • Morocco-Clarke, A., Sodangi, F. A., & Momodu, F. (2023). The implications and effects of ChatGPT on academic scholarship and authorship: A death knell for original academic publications? Information & Communications Technology Law, 33(1), 21–41. https://doi.org/10.1080/13600834.2023.2239623
  • Nakavachara, V., Potipiti, T., & Chaiwat, T. (2024). Experimenting with generative AI: Does ChatGPT really increase everyone’s productivity? ArXiv. https://doi.org/10.2139/ssrn.4746770
  • Nazzal, M. K., Morris, A. J., Parker, R. S., White, F. A., Natoli, R. M., Fehrenbacher, J. C., & Kacena, M. A. (2024). Using AI to write a review article examining the role of the nervous system on skeletal homeostasis and fracture healing. Current Osteoporosis Reports, 22, 217–221. https://doi.org/10.1007/s11914-023-00854-y
  • Nguyen, A., Hong, Y., Dang, B., & Huang, X. (2024). Human-AI collaboration patterns in AI-assisted academic writing. Studies in Higher Education, 49(5), 847–864. https://doi.org/10.1080/03075079.2024.2323593
  • Olujimi, P. A., & Ade-Ibijola, A. (2023). NLP techniques for automating responses to customer queries: A systematic review. Discover Artificial Intelligence, 3, Article 20. https://doi.org/10.1007/s44163-023-00065-5
  • Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hrobjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., …, & Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, Article n71. https://doi.org/10.1136/bmj.n71
  • Patil, N. S., Huang, R. S., van der Pol, C. B., & Larocque, N. (2024). Comparative performance of ChatGPT and bard in a text-based radiology knowledge assessment. Canadian Association of Radiologists Journal, 75(2), 344–350. https://doi.org/10.1177/08465371231193716
  • Pavlik, J. V. (2023). Collaborating with ChatGPT: Considering the implications of generative artificial intelligence for journalism and media education. Journalism & Mass Communication Educator, 78, 84–93. https://doi.org/10.1177/10776958221149577
  • Pérez-Núñez, A. (2023). Exploring the potential of generative AI (ChatGPT) for foreign language instruction: Applications and challenges. Hispania, 106, 355–362. https://doi.org/10.1353/hpn.2023.a906568
  • Perkins, M., & Roe, J. (2023). Academic publisher guidelines on AI usage: A ChatGPT supported thematic analysis. F1000Research, 12, Article 1398. https://doi.org/10.12688/f1000research.142411.2
  • Pesovski, I., Santos, R., Henriques, R., & Trajkovik, V. (2024). Generative AI for customizable learning experiences. Sustainability, 16, 3034. https://doi.org/10.3390/su16073034
  • Praveen, S. V., & Vajrobol, V. (2023). Understanding the perceptions of healthcare researchers regarding ChatGPT: A study based on bidirectional encoder representation from transformers (BERT) sentiment analysis and topic modeling. Annals of Biomedical Engineering, 51, 1654–1656. https://doi.org/10.1007/s10439-023-03222-0
  • Rane, N., Choudhary, S., & Rane, J. (2024). Intelligent manufacturing through generative artificial intelligence, such as ChatGPT or Bard. SSRN. https://doi.org/10.2139/ssrn.4681747
  • Rospigliosi, P. (2023). Artificial intelligence in teaching and learning: What questions should we ask of ChatGPT? Interactive Learning Environments, 31(1), 1–3. https://doi.org/10.1080/10494820.2023.2180191
  • Saad, A., Jenko, N., Ariyaratne, S., Birch, N., Iyengar, K. P., Davies, A. M., Vaishya, R., & Botchu, R. (2024). Exploring the potential of ChatGPT in the peer review process: An observational study. Diabetes & Metabolic Syndrome: Clinical Research & Reviews, 18(2), Article 102946. https://doi.org/10.1016/j.dsx.2024.102946
  • Salimi, A., & Saheb, H. (2023). Large language models in ophthalmology scientific writing: Ethical considerations blurred lines or not at all? American Journal of Ophthalmology, 254, 177–181. https://doi.org/10.1016/j.ajo.2023.06.004
  • Salman, H., Al Mohsin, E., Al Rawi, A., & Shatnawi, S. (2022). Investigating HCI of the LMS blackboard ultra using WAMMI during COVID-19: Usability and design interactivity. In Proceedings of the 2022 International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies (pp. 519–525). IEEE. https://doi.org/10.1109/3ICT56508.2022.9990650
  • Schmidt, L., Piazza, A., & Wiedenhöft, C. (2023). “Augmented brainstorming with AI”–Research approach for identifying design criteria for improved collaborative idea generation between humans and AI. In Frontiers in artificial intelligence and applications (pp. 410–412). IOS Press. https://doi.org/10.3233/FAIA230113
  • Semrl, N., Feigl, S., Taumberger, N., Bracic, T., Fluhr, H., Blockeel, C., & Kollmann, M. (2023). AI language models in human reproduction research: Exploring ChatGPT’s potential to assist academic writing. Human Reproduction, 38(12), 2281–2288. https://doi.org/10.1093/humrep/dead207
  • Seth, I., Sinkjær Kenney, P., Bulloch, G., Hunter-Smith, D. J., Bo Thomsen, J., & Rozen, W. M. (2023). Artificial or augmented authorship? A conversation with a chatbot on base of thumb arthritis. Plastic & Reconstructive Surgery-Global Open, 11(5), Article e4999. https://doi.org/10.1097/GOX.0000000000004999
  • Victor, B. G., Sokol, R. L., Goldkind, L., & Perron, B. E. (2023). Recommendations for social work researchers and journal editors on the use of generative AI and large language models. Journal of the Society for Social Work and Research, 14, 563–577. https://doi.org/10.1086/726021
  • Watermeyer, R., Phipps, L., Lanclos, D., & Knight, C. (2023). Generative AI and the automating of academia. Postdigital Science and Education, 6, 446–466. https://doi.org/10.1007/s42438-023-00440-6
  • Wu, R. T., & Dang, R. R. (2023). ChatGPT in head and neck scientific writing: A precautionary anecdote. American Journal of Otolaryngology, 44(6), Article 103980. https://doi.org/10.1016/j.amjoto.2023.103980
  • Yan, L., Echeverria, V., Nieto, G. F., Jin, Y., Swiecki, Z., Zhao, L., Gašević, D., & Martinez-Maldonado, R. (2023). Human-AI collaboration in thematic analysis using ChatGPT: A user study and design recommendations. ArXiv. https://doi.org/10.1145/3613905.3650732
  • Zhang, P., & Kamel Boulos, M. N. (2023). Generative AI in medicine and healthcare: Promises, opportunities and challenges. Future Internet, 15(9), Article 286. https://doi.org/10.3390/fi15090286
  • Zheng, Z., Zhang, O., Borgs, C., Chayes, J. T., & Yaghi, O. M. (2023). ChatGPT chemistry assistant for text mining and the prediction of MOF synthesis. Journal of the American Chemical Society, 145, 18048–18062. https://doi.org/10.1021/jacs.3c05819
  • Zohouri, M., Sabzali, M., & Golmohammadi, A. (2024). Ethical considerations of ChatGPT-assisted article writing. Synesis, 16(1), 94–113.