The world of education has seen significant changes over the years, with technology playing an increasingly important role in the way we learn, teach and do exams. In this blog post, we'll explore this transformation, the challenges it presents, and the potential benefits and drawbacks of what I call closed bot exams.
As artificial intelligence (AI) tools like ChatGPT have become more sophisticated, they pose challenges when it comes to assessing students' knowledge and skills fairly. In the experiment at the end of this blog, ChatGPT has correctly solved 86 of 89 multiple choice questions in the field of Business Analysis. Such performances naturally lead to the idea of closed bot exams, which prohibit the use of AI tools during assessments.
Implementing closed bot exams requires a combination of organizational and technical measures. Organizational measures involve setting clear guidelines and expectations for students, emphasizing academic integrity, and providing training on the responsible use of AI tools. Technical measures include using secure testing environments, implementing plagiarism detection software and ensuring the identity of the test-takers.
From an organizational perspective, teachers (or universities) could simply ask students to sign an honor code:
As an institution, we are committed to maintaining the highest standards of academic integrity and ethical conduct. In order to uphold these values during closed bot exams, we ask all students to adhere to the following honor code:
Closed Bot Exam Honor Code
I, [Student's Name], pledge to uphold the principles of academic integrity and ethical conduct during this closed bot exam by:
- Refraining from using any AI tools, including but not limited to AI chatbots, automated writing tools, or machine learning-based search engines, to generate answers, ideas, or other content for my exam.
- Not accessing unauthorized resources, including textbooks, notes, online materials, or other aids, unless explicitly permitted by the instructor or exam guidelines.
- Working independently on the exam and not collaborating with other students or individuals, unless explicitly permitted by the instructor or exam guidelines.
- Properly citing any sources or references used in my responses, in accordance with the citation style specified by the instructor or exam guidelines.
- Completing the exam within the allotted time frame and submitting my work as required by the instructor or exam guidelines.
- Reporting any violations of the honor code that I become aware of, in order to maintain a fair and just academic environment for all students.
By signing this honor code, I acknowledge that I understand the rules and expectations outlined above, and I commit to upholding them throughout the duration of the closed bot exam. I also understand that any violation of this honor code may result in disciplinary action, including but not limited to grade penalties, academic probation, suspension, or expulsion from the institution.
Student's Signature
Date
Personally, I believe that organizational measures to monitor closed-book exams will remain important in the AI-enhanced learning landscape. Personal checks in the exam room or proctoring at a distance, as we know it from the Covid era, remain effective means to dissuade the use of ChatGPT or other tools when their use is not allowed. For closed bot exams, such existing controls do not need to be replaced, but adapted to the new technical possibilities.
The popular learning management system, Moodle, for example offers a variety of plugins and functionalities that can help implement closed bot exams. These include the Safe Exam Browser for creating a secure testing environment and (at our univiersity) Compilatio for plagiarism detection.
Remember that there are also many other ways to keep AI assistance in reins without imposing a closed bot exam: Moodle's Quiz module offers a variety of question types, such as multiple-choice, true/false, short answer, matching, and essay. Using and mixing these question types, you can design questions that require critical thinking, problem-solving, or creativity to make them more AI-resistant. However, given the rapid development of AI technologies and their already breathtaking capabilities, even such dispositions are nothing more than a race against time.
Moodle also supports randomized question pools. You can create a large question bank and set the quiz to select a certain number of questions randomly for each student. This can be coupled to straight timing to prevent students even to think of switching to other applications if you can not (or do not want to) put in place a closed bot exam. However, such approaches raise pedagogical concerns, as students may be exposed to additional stress, which in turn may reinforce existing inequities. In my opinion, restricting AI assistance or making its use more difficult during exams is always only the second best solution.
The debate around closed bot exams brings forth several pedagogical arguments both for and against their implementation. Supporters of closed bot exams argue that they assess students' true understanding of the subject matter, promote critical thinking, and maintain academic integrity. On the other hand, critics contend that closed bot exams may not accurately reflect real-world conditions where students will use all available resources and might limit opportunities for collaborative learning and the development of digital literacy.
In addition to the practical and pedagogical aspects of implementing closed bot exams, it's crucial to consider the ethical and human aspects as well. This includes ensuring equal access to technology, promoting digital literacy, and nurturing a culture of academic integrity. Educators should also strive to create a supportive environment that reduces anxiety and fosters a growth mindset among students.
In conclusion, while there are valid concerns about the use of AI tools during exams, it's essential to strike a balance between the potential benefits and drawbacks of these technologies. By implementing closed bot exams thoughtfully and ethically, educators can help maintain academic integrity, encourage critical thinking, and prepare students for the increasingly digital world that awaits them.
Of course, it would be best if we all recognized the signs of the times and adapted our exams to the reality of an AI-infused world instead of trying to stall this development for as long as possible...
3 takeaways for multiple choice exams:
- Develop multiple-choice questions that are less susceptible to AI assistance by focusing on critical thinking, problem-solving, or creativity. Avoid questions that can be easily answered by searching the internet or using AI-generated content. Consider incorporating real-world scenarios, case studies, or questions that require students to apply their knowledge in novel ways.
- If really necessary, implement measures to prevent students from using AI tools during the exam. This may include using secure testing environments and proctoring (either in-person or remotely). By ensuring that students adhere to closed bot exam rules, you can maintain the integrity of the assessment.
- Clearly communicate the rules and consequences of using AI tools during the exam and establish an honor code that students must agree to before taking the exam. By fostering a culture of academic integrity, you can encourage students to rely on their own knowledge and skills rather than turning to AI tools for assistance.
3 takeaways for exams with open questions:
- Craft open questions that require students to think critically and apply their knowledge in novel ways, rather than relying on rote memorization. By emphasizing higher-order thinking skills, you can reduce the likelihood that AI tools will provide a significant advantage and encourage students to engage with the material on a deeper level.
- Use the plagiarism check components in your LMS. Compilatio, for example, can be integrated directly into the submission of exams in Moodle.
- Design questions that promote creativity and originality, which are less likely to be replicated by AI tools. By fostering an environment where students feel empowered to express their unique perspectives and ideas, you can better assess their true understanding of the subject matter and minimize the impact of AI-generated responses.
3 takeaways for students:
- Focus on building strong problem-solving and critical thinking skills, as these will be essential for tackling questions that go beyond what AI tools can provide. Engage in active learning techniques, such as self-explanation, elaboration, and concept mapping, to deepen your understanding of the subject matter.
- Commit to upholding the principles of academic integrity during closed bot exams. This means refraining from using AI tools, unauthorized resources, or collaborating with others unless explicitly permitted. Abiding by the honor code ensures a fair testing environment for all students and allows you to take pride in your own accomplishments.
- Familiarize yourself with the rules, guidelines, and technical requirements of closed bot exams. This may involve understanding how to use the secure testing environment, knowing what resources are allowed during the exam, and being prepared for authentication or monitoring procedures. Being comfortable with the exam format and requirements will help reduce anxiety and improve your overall performance.