GPT-4 and professional benchmarks: the wrong answer to the wrong question

OpenAI didn’t release much information about GPT-4 — not even the size of the model — but heavily emphasized its performance on professional licensing exams and other standardized tests. For instance, GPT-4 reportedly scored in the 90th percentile on the bar exam. So there’s been much speculation about what this means for professionals such as lawyers. We don’t know the answer, but we hope to inje

[Nyoho]

“As further evidence for this hypothesis, we tested it on Codeforces problems from different times in 2021. We found that it could regularly solve problems in the easy category before September 5, but none of the problems after September 12.”

[ABA]

GPT-4は競技プログラミングコンテストサイトCodeforcesの問題のうち、2021/9/5までの問題は解けるが、2021/9/12以降の問題が全く解けないという話。GPT-4に入っているトレーニングデータは2021/9までのもの

[xef]

GPT-4がtrainでtestしている疑惑。学習データのカットオフ前後で競技プログラミングの成績が全然違うなど。データコンタミへの対策・モデル評価がずさんなのではないかという指摘。