1887
  1. Home
  2. A-Z Publications
  3. Qatar Medical Journal
  4. Volume 2024, Issue 4
  5. Article
Volume 2024, Issue 4
  • ISSN: 0253-8253
  • EISSN: 2227-0426

Abstract

Introduction: The inclusion of artificial intelligence (AI) in the healthcare sector has transformed medical practices by introducing innovative techniques for medical education, diagnosis, and treatment strategies. In medical education, the potential of AI to enhance learning and assessment methods is being increasingly recognized. This study aims to evaluate the performance of OpenAI’s Chat Generative Pre-Trained Transformer (ChatGPT) in emergency medicine (EM) residency examinations in Qatar and compare it with the performance of resident physicians.

Methods: A retrospective descriptive study with a mixed-methods design was conducted in August 2023. EM residents’ examination scores were collected and compared with the performance of ChatGPT on the same examinations. The examinations consisted of multiple-choice questions (MCQs) from the same faculty responsible for Qatari Board EM examinations. ChatGPT’s performance on these examinations was analyzed and compared with residents across various postgraduate years (PGY).

Results: The study included 238 emergency department residents from PGY1 to PGY4 and compared their performances with ChatGPT. ChatGPT scored consistently higher than resident groups in all examination categories. However, a notable decline in passing rates was observed among senior residents, indicating a potential misalignment between examination performance and practical competencies. Another likely reason can be the impact of the COVID-19 pandemic on their learning experience, knowledge acquisition, and consolidation.

Conclusion: ChatGPT demonstrated significant proficiency in the theoretical knowledge of EM, outperforming resident physicians in examination settings. This finding suggests the potential of AI as a supplementary tool in medical education.

© 2024 Iftikhar, Anjum, Bhutta, Najam, Bashir, licensee HBKU Press.
Loading

Article metrics loading...

/content/journals/10.5339/qmj.2024.61
2024-11-11
2024-11-20
Loading full text...

Full text loading...

/deliver/fulltext/qmj/2024/4/qmj.2024.61.html?itemId=/content/journals/10.5339/qmj.2024.61&mimeType=html&fmt=ahah

References

  1. Alberts IL, Mercolli L, Pyka T, Prenosil G, Shi K, Rominger A, et al.. Large language models (LLM) and ChatGPT: What will the impact on nuclear medicine be? Eur J Nucl Med Mol Imaging. 2023; 50:(6):1549–52. https://doi.org/10.1007/s00259-023-06172-w
     [Google Scholar]
  2. Bashir K, Azad AM, Hereiz A, Bashir MT, Masood M, Elmoheen A. Current use, perceived barriers, and learning preference of point of care ultrasound (Pocus) in the emergency medicine in qatar – A mixed design. Open Access Emerg Med. 2021; 13:177–82. https://doi.org/10.2147/OAEM.S304153
     [Google Scholar]
  3. Chartier C, Gfrerer L, Knoedler L, Austen WG. Artificial intelligence-enabled evaluation of pain sketches to predict outcomes in headache surgery. Plast Reconstr Surg. 2023; 151:(2):405–11. https://doi.org/10.1097/PRS.0000000000009855
     [Google Scholar]
  4. Moss E. Multiple choice questions: Their value as an assessment tool. Curr Opin Anaesthesiol. 2001; 14:(6):661–6. https://doi.org/10.1097/00001503-200112000-00011
     [Google Scholar]
  5. Deng J, Lin Y. The benefits and challenges of ChatGPT: An overview. Front Comput Intell Syst. 2023; 2:(2):81–3. https://doi.org/10.54097/fcis.v2i2.4465
     [Google Scholar]
  6. Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al.. How does ChatGPT perform on the United States Medical Licensing Examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023; 9:1. https://doi.org/10.2196/45312
     [Google Scholar]
  7. Haque MU, Dharmadasa I, Sworna ZT, Rajapakse RN, Ahmad H. I think this is the most disruptive technology: Exploring sentiments of ChatGPT early adopters using Twitter data. 2022Dec12[cited 2023 Dec 20]. Available from: https://arxiv.org/abs/2212.05856v1
     [Google Scholar]
  8. Jung LB, Gudera JA, Wiegand TLT, Allmendinger S, Dimitriadis K, Koerte IK. ChatGPT passes German state examination in medicine with picture questions omitted. Dtsch Arztebl Int. 2023; 120:(21–22):373–4. https://doi.org/10.3238/arztebl.m2023.0113
     [Google Scholar]
  9. Kasai J, Kasai Y, Sakaguchi K, Yamada Y, Radev D. Evaluating GPT-4 and ChatGPT on Japanese medical licensing examinations. 2023. Available from: http://arxiv.org/abs/2303.18027
     [Google Scholar]
  10. Katz DM, Bommarito MJ, Gao S, Arredondo P. GPT-4 passes the bar exam. SSRN Electr J. 2023;4389233.
    [Google Scholar]
  11. Zielinski C, Winker M, Aggarwal R, Ferris L, Heinemann M, Lapeña JF, et al.. Chatbots, ChatGPT, and Scholarly Manuscripts: WAME recommendations on ChatGPT and Chatbots in relation to scholarly publications. Natl Med J India. 2023Jan1[cited 2023 Dec 20];36(1):1–4. https://doi.org/10.25259/NMJI_365_23
     [Google Scholar]
  12. Wu J, Wu X, Qiu Z, Li M, Zheng Y, Yang J. Qualifying Chinese medical licensing examination with knowledge enhanced generative pre-training model. 2023. Available from: http://arxiv.org/abs/2305.10163
     [Google Scholar]
  13. Naaz S, Asghar A. Artificial intelligence, nano-technology and genomic medicine: The future of anaesthesia. J Anaesthesiol Clin Pharmacol. 2022; 38:(1):11–7. https://doi.org/10.4103/joacp.JOACP_139_20
     [Google Scholar]
  14. Ferrel MN, Ryan JJ. The impact of COVID-19 on medical education. Cureus. 2020; 12:3. https://doi.org/10.7759/cureus.7492
     [Google Scholar]
  15. Harky A, Karimaghaei D, Katmeh H, Hewage S. The impact of COVID-19 on medical examinations. Acta Biomed. 2020; 91:(4):e2020135. https://doi.org/10.23750/abm.v91i4.10487
     [Google Scholar]
  16. Skalidis I, Cagnina A, Luangphiphat W, Mahendiran T, Muller O, Abbe E, et al.. ChatGPT takes on the European Exam in Core Cardiology: An artificial intelligence success story? Eur Heart J Digit Health. 2023; 4:(3):279–81. https://doi.org/10.1093/ehjdh/ztad029
     [Google Scholar]
  17. Jarou ZJ, Dakka A, McGuire D, Bunting L. ChatGPT Versus Human performance on emergency medicine board preparation questions. Ann Emerg Med. 2023; 83:(1):87–88. https://doi.org/10.1016/j.annemergmed.2023.08.010
     [Google Scholar]
  18. Knoedler L, Odenthal J, Prantl L. others. (2023) Artificial intelligence-enabled simulation of gluteal augmentation: A helpful tool in preoperative outcome simulation? J Plast Reconstr Aesthet Surg. 80:94–101. https://doi.org/10.1016/j.bjps.2023.01.039
     [Google Scholar]
  19. King MR. The future of AI in medicine: A perspective from a Chatbot. Ann Biomed Eng. 2023; 51:(2):291–295. https://doi.org/10.1007/s10439-022-03121-w
     [Google Scholar]
  20. Zhai X. ChatGPT user experience: Implications for education. SSRN Electron J. 2023;4312418.
    [Google Scholar]
  21. Gianfrancesco MA, Tamang S, Yazdany J, Schmajuk G. Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med. 2018Nov11[cited 2024 Aug 15];178(11):1544. https://doi.org/10.1001/jamainternmed.2018.3763
     [Google Scholar]
  22. Cabitza F, Rasoini R, Gensini GF. Unintended consequences of machine learning in medicine. JAMA. 2017Aug8; 318:(6):517–8. https://doi.org/10.1001/jama.2017.7797
     [Google Scholar]
  23. Topol EJ. High-performance medicine: The convergence of human and artificial intelligence. Nat Med. 2019Jan; 25:(1):44–56. https://doi.org/10.1038/s41591-018-0300-7
     [Google Scholar]
  24. Morley J, Machado CC, Burr C, Cowls J, Joshi I, Taddeo M, Floridi L. The ethics of AI in health care: A mapping review. Soc Sci Med. 2020Sep1; 260:113172. https://doi.org/10.1016/j.socscimed.2020.113172
     [Google Scholar]
  25. Yu KH, Beam AL, Kohane IS. Artificial intelligence in healthcare. Nat Biomed Eng. 2018Oct; 2:(10):719–31. https://doi.org/10.1038/s41551-018-0305-z
     [Google Scholar]
  26. Stokel-Walker C. ChatGPT listed as author on research papers: Many scientists disapprove. Nature. 2023; 613:(7945):620–1. https://doi.org/10.1038/d41586-023-00107-z
     [Google Scholar]
  27. Majumder MA, Kumar A, Krishnamurthy K, Ojeh N, Adams OP, Sa B. An evaluative study of objective structured clinical examination (OSCE): Students and examiners perspectives. Adv Med Educ Pract. 2019Jun 5:387–97. https://doi.org/10.2147/AMEP.S197275
     [Google Scholar]
  28. Bird JB, Olvet DM, Willey JM, Brenner J. Patients don’t come with multiple choice options: Essay-based assessment in UME. Med Educ Online. 2019Jan1; 24:(1):1649959. https://doi.org/10.1080/10872981.2019.1649959
     [Google Scholar]
  29. Pham H, Trigg M, Wu S, O’Connell A, Harry C, Barnard J, Devitt P. Choosing medical assessments: Does the multiple-choice question make the grade? Educ Health (Abingdon). 2018May1; 31:(2). https://doi.org/10.4103/efh.EfH_229_17
     [Google Scholar]
/content/journals/10.5339/qmj.2024.61
Loading
Performance of ChatGPT in emergency medicine residency exams in Qatar: A comparative analysis with resident physicians
Qatar Medical Journal 2024, 61 (2024); https://doi.org/10.5339/qmj.2024.61
/content/journals/10.5339/qmj.2024.61
/content/journals/10.5339/qmj.2024.61
Loading

Data & Media loading...

  • Article Type: Research Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error