RT Journal Article
T1 Analysis of ChatGPT Performance in Computer Engineering Exams
A1 Rodríguez Echeverría, Roberto
A1 Gutiérrez, Juan D.
A1 Conejero Manzano, José María
A1 Prieto Ramos, Álvaro Enmanuel
K1 Chatbots
K1 Inteligencia Artificial
K1 Educación
K1 Artificial Intelligence
K1 Education
AB The appearance of ChatGPT at the end of 2022 was a milestone in the field of Generative Artificial Intelligence. How- ever, it also caused a shock in the academic world. For the first time, a simple interface allowed anyone to access a large language model and use it to generate text. These capabilities have a relevant impact on teaching-learning methodologies and assessment methods. This work aims to obtain an objective measure of ChatGPT’s possible performance in solving exams related to computer engineering. For this purpose, it has been tested with actual exams of 15 subjects of the Software Engineering branch of a Spanish university. All the questions of these exams have been extracted and adapted to a text format to obtain an answer. Furthermore, the exams have been rewritten to be corrected by the teaching staff. In light of the results, ChatGPT can achieve relevant performance in these exams; it can pass many questions and problems of different natures in multiple subjects. A detailed study of the results by typology of questions and problems is provided as a fundamental contribution, allowing recommendations to be considered in the design of assessment methods. In addition, an analysis of the impact of the non-deterministic aspect of ChatGPT on the answers to test questions is presented, and the need to use a strategy to reduce this effect for performance analysis is concluded.
PB IEEE
SN 2374-0132
YR 2024
FD 2024-03-25
LK http://hdl.handle.net/10347/33875
UL http://hdl.handle.net/10347/33875
LA eng
NO R. Rodriguez-Echeverría, J. D. Gutiérrez, J. M. Conejero and Á. E. Prieto, "Analysis of ChatGPT Performance in Computer Engineering Exams," in IEEE Revista Iberoamericana de Tecnologias del Aprendizaje, vol. 19, pp. 71-80, 2024
NO 10.13039/100006190-Research and Development Project funded by MICIU/AEI/10.13039/501100011033 (Grant Number: ID2021-127412OB-I00)
DS Minerva
RD 29 abr 2026