AI-Generated News Quizzes

Asking an LLM to generate multiple-choice quizzes from magazine articles dating back many decades tends to turn up its strength and blind spots in stark relief.

For example: When I asked GPT-3.5 Turbo for a quiz based on this 2014 interview with Taylor Swift, the first pass was impressively cogent and sensible, with one correct answer and three plausible incorrect ones–something any quiz author will tell you is a lot harder than it looks.

Just two problems: First, the correct answer for 7 of the 10 questions was “Taylor Swift.” Second, several of the answers included things that happened several years after this interview was published.

This is all to say that, particularly as the models mature, they can open the door to a lot of projects that would otherwise be prohibitively tedious or labor intensive. (Particularly when all the junior staffers have been laid off.) This is where programmatic use of the LLMs outside of a chat window becomes quite powerful. But it requires practice negotiating with the models and explaining to them–politely!—how to do precisely what you asked. And you still have to check their work.