← All notes · 1 min read

Getting an AI to write 5,500 quiz questions

The AI doesn't know my taste. I have to turn it into rules, every time.

Every question in the game was written by an AI. But not by typing "write me five thousand questions". That would have made a game nobody wants to play.

The first batch was correct and boring. Long sentences, quotes, descriptions that started with "as a famous writer once said". An easy question that is eighteen words long is not easy. I set a rule: easy questions are twelve words at most, one comma. More than a thousand questions got rewritten and the average length dropped by half.

The second problem is that the writer can't check its own work. If you ask the same model "are these correct", it says yes to everything. So now one model writes and a different one checks. The checker isn't always right either. It told me Dolmabahçe is not ten letters. I counted. It is. I overruled 144 of its objections.

The third problem is the hardest one, and it's taste. I had it write multiple choice options for a new mode. The question was "which one is a fruit" and the options were Apple, Bicycle, Soldier, Ayran. Technically fine. But ask an adult that and you've insulted them. Turning that feeling into a rule took me two days: every option has to be plausible, two obviously wrong options are not allowed, a made-up option that sounds real is fine.

What I understood is that the AI doesn't know my taste and doesn't learn it. I have to turn my taste into rules, every time. Saying "this is bad" is not enough. I have to be able to say why. That turned out to be where most of the time goes in making a game: writing down what I like.

Here are the rules I ended up with.

— Kemal