1887
  1. الرئيسية
  2. A-Z Publications
  3. Qatar Medical Journal
  4. المجلد 2024, العدد 4
  5. المقالة
Volume 2024, Issue 4
  • ISSN: 0253-8253
  • E-ISSN: 2227-0426

ملخص

Introduction: The inclusion of artificial intelligence (AI) in the healthcare sector has transformed medical practices by introducing innovative techniques for medical education, diagnosis, and treatment strategies. In medical education, the potential of AI to enhance learning and assessment methods is being increasingly recognized. This study aims to evaluate the performance of OpenAI’s Chat Generative Pre-Trained Transformer (ChatGPT) in emergency medicine (EM) residency examinations in Qatar and compare it with the performance of resident physicians.

Methods: A retrospective descriptive study with a mixed-methods design was conducted in August 2023. EM residents’ examination scores were collected and compared with the performance of ChatGPT on the same examinations. The examinations consisted of multiple-choice questions (MCQs) from the same faculty responsible for Qatari Board EM examinations. ChatGPT’s performance on these examinations was analyzed and compared with residents across various postgraduate years (PGY).

Results: The study included 238 emergency department residents from PGY1 to PGY4 and compared their performances with ChatGPT. ChatGPT scored consistently higher than resident groups in all examination categories. However, a notable decline in passing rates was observed among senior residents, indicating a potential misalignment between examination performance and practical competencies. Another likely reason can be the impact of the COVID-19 pandemic on their learning experience, knowledge acquisition, and consolidation.

Conclusion: ChatGPT demonstrated significant proficiency in the theoretical knowledge of EM, outperforming resident physicians in examination settings. This finding suggests the potential of AI as a supplementary tool in medical education.

© 2024 Iftikhar, Anjum, Bhutta, Najam, Bashir, licensee HBKU Press.
Loading

جارٍ تحميل قياسات المقالة...

/content/journals/10.5339/qmj.2024.61
٢٠٢٤-١١-١١
٢٠٢٤-١١-٢٠
Loading full text...

Full text loading...

/deliver/fulltext/qmj/2024/4/qmj.2024.61.html?itemId=/content/journals/10.5339/qmj.2024.61&mimeType=html&fmt=ahah

References

  1. Alberts IL, Mercolli L, Pyka T, Prenosil G, Shi K, Rominger A, et al.. Large language models (LLM) and ChatGPT: What will the impact on nuclear medicine be? Eur J Nucl Med Mol Imaging. 2023; 50:(6):1549–52. https://doi.org/10.1007/s00259-023-06172-w
     [Google الباحث العلمي]
  2. Bashir K, Azad AM, Hereiz A, Bashir MT, Masood M, Elmoheen A. Current use, perceived barriers, and learning preference of point of care ultrasound (Pocus) in the emergency medicine in qatar – A mixed design. Open Access Emerg Med. 2021; 13:177–82. https://doi.org/10.2147/OAEM.S304153
     [Google الباحث العلمي]
  3. Chartier C, Gfrerer L, Knoedler L, Austen WG. Artificial intelligence-enabled evaluation of pain sketches to predict outcomes in headache surgery. Plast Reconstr Surg. 2023; 151:(2):405–11. https://doi.org/10.1097/PRS.0000000000009855
     [Google الباحث العلمي]
  4. Moss E. Multiple choice questions: Their value as an assessment tool. Curr Opin Anaesthesiol. 2001; 14:(6):661–6. https://doi.org/10.1097/00001503-200112000-00011
     [Google الباحث العلمي]
  5. Deng J, Lin Y. The benefits and challenges of ChatGPT: An overview. Front Comput Intell Syst. 2023; 2:(2):81–3. https://doi.org/10.54097/fcis.v2i2.4465
     [Google الباحث العلمي]
  6. Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al.. How does ChatGPT perform on the United States Medical Licensing Examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023; 9:1. https://doi.org/10.2196/45312
     [Google الباحث العلمي]
  7. Haque MU, Dharmadasa I, Sworna ZT, Rajapakse RN, Ahmad H. I think this is the most disruptive technology: Exploring sentiments of ChatGPT early adopters using Twitter data. 2022Dec12[cited 2023 Dec 20]. Available from: https://arxiv.org/abs/2212.05856v1
     [Google الباحث العلمي]
  8. Jung LB, Gudera JA, Wiegand TLT, Allmendinger S, Dimitriadis K, Koerte IK. ChatGPT passes German state examination in medicine with picture questions omitted. Dtsch Arztebl Int. 2023; 120:(21–22):373–4. https://doi.org/10.3238/arztebl.m2023.0113
     [Google الباحث العلمي]
  9. Kasai J, Kasai Y, Sakaguchi K, Yamada Y, Radev D. Evaluating GPT-4 and ChatGPT on Japanese medical licensing examinations. 2023. Available from: http://arxiv.org/abs/2303.18027
     [Google الباحث العلمي]
  10. Katz DM, Bommarito MJ, Gao S, Arredondo P. GPT-4 passes the bar exam. SSRN Electr J. 2023;4389233.
    [Google الباحث العلمي]
  11. Zielinski C, Winker M, Aggarwal R, Ferris L, Heinemann M, Lapeña JF, et al.. Chatbots, ChatGPT, and Scholarly Manuscripts: WAME recommendations on ChatGPT and Chatbots in relation to scholarly publications. Natl Med J India. 2023Jan1[cited 2023 Dec 20];36(1):1–4. https://doi.org/10.25259/NMJI_365_23
     [Google الباحث العلمي]
  12. Wu J, Wu X, Qiu Z, Li M, Zheng Y, Yang J. Qualifying Chinese medical licensing examination with knowledge enhanced generative pre-training model. 2023. Available from: http://arxiv.org/abs/2305.10163
     [Google الباحث العلمي]
  13. Naaz S, Asghar A. Artificial intelligence, nano-technology and genomic medicine: The future of anaesthesia. J Anaesthesiol Clin Pharmacol. 2022; 38:(1):11–7. https://doi.org/10.4103/joacp.JOACP_139_20
     [Google الباحث العلمي]
  14. Ferrel MN, Ryan JJ. The impact of COVID-19 on medical education. Cureus. 2020; 12:3. https://doi.org/10.7759/cureus.7492
     [Google الباحث العلمي]
  15. Harky A, Karimaghaei D, Katmeh H, Hewage S. The impact of COVID-19 on medical examinations. Acta Biomed. 2020; 91:(4):e2020135. https://doi.org/10.23750/abm.v91i4.10487
     [Google الباحث العلمي]
  16. Skalidis I, Cagnina A, Luangphiphat W, Mahendiran T, Muller O, Abbe E, et al.. ChatGPT takes on the European Exam in Core Cardiology: An artificial intelligence success story? Eur Heart J Digit Health. 2023; 4:(3):279–81. https://doi.org/10.1093/ehjdh/ztad029
     [Google الباحث العلمي]
  17. Jarou ZJ, Dakka A, McGuire D, Bunting L. ChatGPT Versus Human performance on emergency medicine board preparation questions. Ann Emerg Med. 2023; 83:(1):87–88. https://doi.org/10.1016/j.annemergmed.2023.08.010
     [Google الباحث العلمي]
  18. Knoedler L, Odenthal J, Prantl L. others. (2023) Artificial intelligence-enabled simulation of gluteal augmentation: A helpful tool in preoperative outcome simulation? J Plast Reconstr Aesthet Surg. 80:94–101. https://doi.org/10.1016/j.bjps.2023.01.039
     [Google الباحث العلمي]
  19. King MR. The future of AI in medicine: A perspective from a Chatbot. Ann Biomed Eng. 2023; 51:(2):291–295. https://doi.org/10.1007/s10439-022-03121-w
     [Google الباحث العلمي]
  20. Zhai X. ChatGPT user experience: Implications for education. SSRN Electron J. 2023;4312418.
    [Google الباحث العلمي]
  21. Gianfrancesco MA, Tamang S, Yazdany J, Schmajuk G. Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med. 2018Nov11[cited 2024 Aug 15];178(11):1544. https://doi.org/10.1001/jamainternmed.2018.3763
     [Google الباحث العلمي]
  22. Cabitza F, Rasoini R, Gensini GF. Unintended consequences of machine learning in medicine. JAMA. 2017Aug8; 318:(6):517–8. https://doi.org/10.1001/jama.2017.7797
     [Google الباحث العلمي]
  23. Topol EJ. High-performance medicine: The convergence of human and artificial intelligence. Nat Med. 2019Jan; 25:(1):44–56. https://doi.org/10.1038/s41591-018-0300-7
     [Google الباحث العلمي]
  24. Morley J, Machado CC, Burr C, Cowls J, Joshi I, Taddeo M, Floridi L. The ethics of AI in health care: A mapping review. Soc Sci Med. 2020Sep1; 260:113172. https://doi.org/10.1016/j.socscimed.2020.113172
     [Google الباحث العلمي]
  25. Yu KH, Beam AL, Kohane IS. Artificial intelligence in healthcare. Nat Biomed Eng. 2018Oct; 2:(10):719–31. https://doi.org/10.1038/s41551-018-0305-z
     [Google الباحث العلمي]
  26. Stokel-Walker C. ChatGPT listed as author on research papers: Many scientists disapprove. Nature. 2023; 613:(7945):620–1. https://doi.org/10.1038/d41586-023-00107-z
     [Google الباحث العلمي]
  27. Majumder MA, Kumar A, Krishnamurthy K, Ojeh N, Adams OP, Sa B. An evaluative study of objective structured clinical examination (OSCE): Students and examiners perspectives. Adv Med Educ Pract. 2019Jun 5:387–97. https://doi.org/10.2147/AMEP.S197275
     [Google الباحث العلمي]
  28. Bird JB, Olvet DM, Willey JM, Brenner J. Patients don’t come with multiple choice options: Essay-based assessment in UME. Med Educ Online. 2019Jan1; 24:(1):1649959. https://doi.org/10.1080/10872981.2019.1649959
     [Google الباحث العلمي]
  29. Pham H, Trigg M, Wu S, O’Connell A, Harry C, Barnard J, Devitt P. Choosing medical assessments: Does the multiple-choice question make the grade? Educ Health (Abingdon). 2018May1; 31:(2). https://doi.org/10.4103/efh.EfH_229_17
     [Google الباحث العلمي]
/content/journals/10.5339/qmj.2024.61
Loading
Performance of ChatGPT in emergency medicine residency exams in Qatar: A comparative analysis with resident physicians
Qatar Medical Journal 2024, 61 (٢٠٢٤); https://doi.org/10.5339/qmj.2024.61
/content/journals/10.5339/qmj.2024.61
/content/journals/10.5339/qmj.2024.61
Loading

جارٍ تحميل البيانات والوسائط...

  • نوع المستند: Research Article

الأكثر اقتباسًا لهذا الشهر Most Cited RSS feed

هذه الخانة مطلوبة
يُرجى إدخال عنوان بريد إلكتروني صالح
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error