Claude Sonnet 的 Pull Request 描述：diff 中沒有的主張

AI pull request descriptions: claims the diff never made

AI 導讀

Claude Sonnet 為 24 個程式碼差異撰寫 Pull Request 描述，未載入書面規則時有 21 份正確，少數失誤加入差異內容未能支持的主張。加入書面規則後，Sonnet 正確 22 份，Haiku 則由 17 份升至 19 份。

正文

Claude Sonnet described 24 diffs and kept to the code on almost all of them. Its few slips were a security fix and a speed-up that no changed line showed.

Rarely, and the few it adds are the ones a reviewer is most likely to believe. We handed Claude Sonnet 24 diffs and asked it for a pull request description, title included, for each one. From the request alone, 21 of the 24 came out right. Two of its three misses were claims: it called an HTML change a cross-site scripting fix and a set lookup a speed-up, though no changed line said either. With our written rules loaded it got 22 right, so on this test the rules moved Sonnet by one diff. Claude Haiku went from 17 to 19.

Read the full report on AISkills402: https://aiskills402.com/blog/llm-pr-description-claims-diff