🔍 Read the full analysis: Can Perplexity Help Measure Intelligence In AI? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI has released some internal measurements of its models’ pretraining perplexity and proposed the metric as a possible common way to track AI progress. Its account connects lower values with past capabilities, but does not establish that a particular score guarantees an ability or predicts when future capabilities will appear.
OpenAI has published some internal measurements of its models’ pretraining perplexity, arguing that the metric could help AI developers compare progress and inform shared safety standards. The release offers a view of how the company tracks predictive performance over time, as discussed in the original analysis of perplexity and intelligence, but it does not establish a common reporting standard or show that any particular score reliably produces a specific capability.
Perplexity measures how well a model predicts the next token in a held-out dataset; lower values indicate better predictions. OpenAI says it has tracked the measure for models dating back to GPT-3.5 and is now sharing some internal results. The company describes the metric as a way to monitor underlying pretraining performance even as downstream benchmarks change or become less useful.
OpenAI reports approximate log-perplexity values of 0.99 for GPT-3.5 and 0.91 for GPT-4, and says later models have reached losses below 0.8. It associates stages of improvement with fluent conversation, productive chain-of-thought reasoning, and mathematics and cybersecurity capabilities. Those are the company’s descriptions of its own models, not evidence that a given value guarantees those abilities.
The post also says a model at about 0.76 log-perplexity produced results on open mathematics problems that its author characterized as groundbreaking. OpenAI forecasts that changes in pretraining research could arrive as soon as 2027 and anticipates capabilities in a range around 0.68 to 0.7. These are projections, not confirmed outcomes; the material provided does not establish when or whether those developments will occur.
Perplexity and Intelligence A shared measure?
OpenAI published internal pretraining measurements and argues they could help developers compare progress and shape shared safety standards. The numbers offer a signal, not a guarantee of future abilities.
What the metric measures
Perplexity measures how well a model predicts the next token in a held-out dataset; lower values mean better predictions.
A model learns from a large dataset.
Perplexity tracks predictive performance on held-out data.
Prompting, fine-tuning, and reinforcement learning can elicit or shape what a model does.
OpenAI’s reported progression
The company associates lower log-perplexity with capabilities observed in its models; the pattern does not prove a value causes or guarantees an ability.
| Reported level | OpenAI’s account |
|---|---|
| ≈0.99 | GPT-3.5; fluent conversation |
| ≈0.91 | GPT-4; productive chain-of-thought reasoning |
| Below 0.8 | Later models; mathematics and cybersecurity capabilities |
| ≈0.68–0.7 | Range where OpenAI anticipates new capabilities |
Why a shared measure could matter
Task benchmarks can change or saturate as models improve. Comparable pretraining measurements could give researchers and policymakers a common reference.
- Track predictive performance over time
- Compare progress if developers use shared methods
- Inform discussion of common safety standards
- What a model can reliably do
- How safely it behaves
- How much oversight development requires
What remains unresolved
The supplied material does not include the underlying chart or enough detail to independently compare the measurements.
- No standardized reporting or audit process is specified.
- No formal industry standard, timetable, or commitments from other developers are announced.
- The possible 2027 research acceleration and future capabilities are projections, not confirmed outcomes.
A Possible Common Measure
If AI developers report comparable measurements, researchers and policymakers could gain a shared reference point for tracking changes across models. That could be useful when task-specific benchmarks are revised, exhausted, or no longer distinguish between increasingly capable systems. OpenAI says it weighed the competitive cost of publishing information about its position against the potential value of industry-wide safety measures.
But a predictive-performance score would be one signal, not a complete safety assessment. It does not, by itself, describe a model’s behavior across applications, reliability, misuse risks, or the level of human oversight it needs. Its value for policy would depend on whether different developers can measure it consistently and whether the scores are shown to relate meaningfully to capabilities and risks.
The proposal therefore matters less as a ready-made standard than as an invitation to test whether one can be built. A metric that is stable across time could complement task evaluations, but using it to guide safety decisions would require shared methods, evidence, and scrutiny.
Products Worth Considering

Jackery Solar Generator 1000 v2 with 200W Solar Panel, 1024Wh LiFePO4 Battery

Logitech MX Keys S Combo Wireless Keyboard and Mouse, Graphite

Samsung 27-Inch Odyssey G5 QHD Gaming Monitor (G51F)
As an affiliate, we earn on qualifying purchases.
From Pretraining to Capabilities
OpenAI describes model development as pretraining on a large dataset, followed by methods such as prompting, fine-tuning, and reinforcement learning that shape or elicit capabilities. It argues that pretraining measurements are worth tracking because the benchmarks used to evaluate downstream tasks can change as models improve.
The company’s post links its reported progression to a history of models including GPT-2, GPT-3, GPT-3.5, and GPT-4. It also acknowledges that the capabilities associated with particular perplexity levels were not predictable in advance. The historical examples explain why OpenAI considers the measure useful; they do not establish that future models will follow the same pattern.
OpenAI further raises the possibility that AI systems could contribute to research on pretraining algorithms. Its suggestion of a possible acceleration around 2027 is based on the author’s expectations about advances in mathematics and deep-learning research. The post recognizes that the fields differ, and the forecast remains speculative.
Limits of the Published Data
The information provided does not include the underlying chart, full dataset details, or enough methodological information to independently compare OpenAI’s measurements with results from other developers. It also does not describe a standardized reporting process, independent auditing, or safety decisions that should follow from particular values.
OpenAI’s examples do not demonstrate that a score causes or guarantees a capability. It remains unclear how much results would vary with dataset choice and measurement methods, or how well the metric corresponds to model behavior and risks. The proposed 2027 research acceleration and the nature and timing of future capabilities are also uncertain.
No formal industry standard, adoption schedule, or commitment by other AI developers is identified in the source material. Until comparable results and methods are available, the published values should be treated as OpenAI’s internal measurements, not an established cross-industry scale.
Whether Others Adopt the Measure
The next concrete test is whether other AI developers publish comparable pretraining measurements and whether researchers can agree on datasets, procedures, and reporting details that make comparisons meaningful. Independent scrutiny could help establish how sensitive scores are to those choices and whether changes in the metric correspond to changes in capabilities or risks.
OpenAI has not announced a timetable for an industry standard, and the source material identifies no formal commitments from other organizations. For now, the proposal remains a call for shared measurement rather than an agreed system. Any use of perplexity in safety standards would depend on evidence beyond the company’s reported examples.
Key Questions
What does perplexity measure?
Perplexity measures how well a model predicts the next token in a held-out dataset. Lower values indicate better predictive performance on that dataset.
Does a lower perplexity score prove a model has a specific capability?
No. OpenAI links its historical measurements with capabilities it observed in its models, but the post does not show that a particular score guarantees a particular ability.
Can the published figures be compared with other AI developers’ models?
Not on the information provided. The source does not supply enough dataset and methods detail to establish a reliable comparison, and it does not identify a shared reporting standard or results from other developers.
Is an AI research acceleration by 2027 confirmed?
No. OpenAI presents the possibility of changes in pretraining research around 2027 as a forecast. The timing and outcome remain uncertain.
Primary source: OpenAI · via ThorstenMeyerAI.com
