Reflection 70B, and the 405B that is coming next week

Subject: Reflection 70B / 405B

Claim

"The world's top open-source model." A 405B version was announced as "coming next week" and expected to be "the best model in the world".

Finding

Two days later, independent evaluators reported they were "unable to replicate the eval results claimed" and saw worse performance than the base model it was fine-tuned from. The 405B week has, at the time of this build, not ended.

Timeline

5 Sept 2024
Announcement: "world's top open-source model"; 405B "coming next week".@mattshumer_ · ♥ 9K ↗
7 Sept 2024
Independent evals: unable to replicate; worse than Llama 3.1 70B.@ArtificialAnlys · ♥ 685 ↗
Ongoing
Next week.

Primary sources

The posts this entry is based on, embedded as published.

Matt Shumer

@mattshumer_

I'm excited to announce Reflection 70B, the world’s top open-source model. Trained using Reflection-Tuning, a technique developed to enable LLMs to fix their own mistakes. 405B coming next week - we expect it to be the best model in the world. Built w/ @GlaiveAI. Read on ⬇️:

5 Sept 2024♥ 9KOpen on X ↗

Artificial Analysis

@ArtificialAnlys

Reflection Llama 3.1 70B independent eval results: We have been unable to replicate the eval results claimed in our independent testing and are seeing worse performance than Meta’s Llama 3.1 70B, not better. These evaluations were conducted using our standard methodology,

7 Sept 2024♥ 685Open on X ↗