On Violations of LLM Review Policies - ICML Blog

On Violations of LLM Review Policies - ICML Blog

The International Conference of Machine Learning identified 497 papers because 398 reciprocal reviewers used an LLM to write the review when they had promised not to use AI. "At a high level, the LLM detection involved watermarking submission PDFs with hidden LLM instructions, which would subtly influence any review produced via an LLM.... First, we created a dictionary of 170,000 phrases. For each paper, we sampled two phrases randomly from this dictionary. The probability with which a given pair of phrases is picked is thus smaller than one in ten billion. We watermarked the PDF of each paper submitted with instructions, visible only to an LLM, instructing it to include the two selected phrases in the review. (A human reading the PDF would not directly see this watermark.)"