I am part of a working group of political science faculty who are developing new online courses. I’m planning on teaching “Introduction to Comparative Politics” this summer. Last week we had our first meeting. One of the faculty had volunteered to lead a conversation around ChatGPT, a new artificial intelligence system that can generate text based on the web, books, and other sources. If you are in higher education, you’ve probably heard a lot about ChatGPT over the last month. The New York Times and other media outlets have covered this topic in depth, and with good reason.
In December 2022 Stephen Marche had an excellent article in the Atlantic about how ChatGPT could produce an outstanding college entrance paper. Work by ChatGPT also passed a Wharton MBA exam. Perhaps most chilling, its product has also now passed all three stages of the US medical exam system. So how do we as professors respond?
What has been most interesting to me is to see how my colleagues think about this issue. One senior colleague is literally blowing up his teaching methods. His class will now be hybrid only for the exams, in which students will have to come in and write a pen and ink “green book” exam. Students may also be asked to do oral exams. This represents an immense amount of work for this professor. It also means that this class cannot be a fully online class, because students will have to appear for the testing phase. But this faculty member is deeply committed to their teaching and doesn’t see any alternative.
During this brainstorming session other faculty and I wondered about changing our assessments. One faculty suggested creating a podcast, but then worried that ChatGPT might write the script. I thought about perhaps using other assignments, such as data visualizations. But most faculty felt that this meant entering into an arms race with the soft-ware, one which we were unlikely to win in the long term. I still wonder if my critical reading reflections might be hard for ChatGPT to replicate. Typically I ask the students to pick the three most important readings from the course during the last three weeks, to discuss their strengths and weaknesses, and to select which one was the most important, and to justify their decision. Can ChatGPT really respond to such specific question prompts? The consensus in the discussion group seemed to be that this might not be enough to overcome this new challenge. AKA, “Denial is not a strategy.” But I think many colleagues also felt uncertain about how to respond.
After the meeting, one of my colleagues decided to ask ChatGPT to write a syllabus for the political science course that he is developing. It did so relatively well. It wasn’t a brilliant syllabus, but it was as good as many such syllabi that you could find on the web. More disturbing, ChatGPT also wrote the questions for a class quiz, and an introductory lecture. The lecture was too short, and still needed some work to be used. But it wasn’t a bad first draft, which might save a faculty member some time. The quiz was pretty good.
Personally, I was shocked after receiving this email. But not all of my colleagues felt the same way. I decided to try to testing out some ChatGPT detection tools. One author online wrote that a new website was over 99% effective at detecting fakes: GPT-2 Output Detector Demo. I tested it out using the text that my colleague had generated for his syllabus and quiz. The test found that this text was “99.96%” likely to be real. Based on a single sample with just one such system, I’m not convinced that the new software will be able to work as well as older generation plagiarism software did.
So how do I respond to this new challenge? I am wondering if I could put something in my syllabus to require students to also upload a draft, or to share a link to Google Docs so that I can see their writing process using the history feature? The challenge is that I don’t really want to require students to use one writing system. Pages is still popular with a lot of students who just have an Ipad, not a computer. And perhaps Chat GPT can also write a convincing draft? It’s not a perfect solution. But I wonder if both having very specific writing prompts -and asking for evidence of the writing process- might be able to address plagiarism?
I think that all educators -from middle school to grad programs- are going to face this challenge. I am curious to hear how peers are addressing this challenge at other institutions. This technology is still in its early stages, and new competitors will emerge soon. I predict that all levels of our educational system will have to make major adjustments to respond.
I come from a family of writers. My mom, Phyllis Smallman, wrote a mystery series set in a bar in West Florida, as well as a couple of books set in the Gulf Islands off the West Coast of Canada. Since her death much of her work (the electronic versions for Kindle and Kobo) are available for free online. My sister is also a writer. In ten or fifteen years will we be able to ask Kindle to write a mystery for us, one set in the Roman Republic, with a particular love interest, and a locked room puzzle?
References
Kung, Tiffany H., Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, et al. “Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models.” bioRxiv, December 20, 2022. https://doi.org/10.1101/2022.12.19.22283643.
Marche, Stephen. “The College Essay Is Dead.” The Atlantic, December 6, 2022. https://www.theatlantic.com/technology/archive/2022/12/chatgpt-ai-writing-college-student-essays/672371/.
Mollman, Steve. “ChatGPT Passed a Wharton MBA Exam and It’s Still in Its Infancy. One Professor Is Sounding the Alarm.” Fortune, January 22, 2023. https://fortune.com/2023/01/21/chatgpt-passed-wharton-mba-exam-one-professor-is-sounding-alarm-artificial-intelligence/.