GPT-4o gains little from prompting tricks that still help Qwen

A replication study finds GPT-4o gains little from chain-of-thought and few-shot prompting that still boosts Qwen2.5 7B, per a paper posted to arXiv.

The study ran 19,620 generations across 218 context-rich Python functions, testing Zero-Shot, Few-Shot, Chain-of-Thought, Contrastive CoT, and an adapted Program-of-Thought. It ran those five techniques across three model-version pairs: GPT-3.5-Turbo to GPT-4o, Qwen2 7B to Qwen2.5 7B, and Mistral-7B-Instruct to Mistral-Large. Teams carrying the same prompt templates across a model swap can't assume the old lift still applies.

GPT-4o shows diminishing or negative gains from the same structured prompting that reliably boosted GPT-3.5-Turbo, the researchers found. Their read is that the scaffolding gets absorbed during training. The model already reasons in steps without being told to. Adding chain-of-thought instructions on top adds no lift, and sometimes costs some.

Qwen2.5 7B didn't follow that curve. It still gets a real bump from Few-Shot and Contrastive CoT, the same techniques that stopped paying off on GPT-4o.

Mistral's results split the difference. Mistral-7B-Instruct and Mistral-Large moved in mixed directions across the five techniques, with no clean pattern either way.