Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity
arXiv:2604.24827v2 Announce Type: replace-cross Abstract: Closed-source frontier labs do not disclose parameter counts. Storing F facts requires at least F/(bits per parameter) weights, so factual recall lower-bounds parameter count--an intrinsic, serving-independent signal, though (as we show) a coarse one. We introduce Incompressible Knowledge Probes (IKPs), a benchmark of 1,400 factual questions spanning 7 tiers of obscurity, designed to isolate knowledge that cannot be derived by reasoning or compressed by architectural improvements. We score with no hallucination penalty (lambda = 0: IKP
The increasing prevalence of closed-source frontier LLMs necessitates new methods for assessing their capabilities and underlying scale, driving the development of probe techniques like IKPs.
This research provides a novel, serving-independent metric to estimate the underlying parameter counts of black-box LLMs, offering crucial insights into their factual capacity and potential. It allows for a more objective comparison and understanding of proprietary AI models without direct access to their architectural details.
The ability to estimate LLM parameter counts via factual capacity changes how competitive intelligence in the AI space is gathered and analyzed, enabling better tracking of advancements in closed-source models. It redefines metrics for LLM capability assessment beyond disclosed parameters or task-specific performance benchmarks.
- · AI researchers
- · Competitive intelligence firms
- · Open-source AI movement
- · Regulators
- · Closed-source frontier labs (information asymmetry reduced)
- · Less transparent AI companies
IKPs provide a new benchmark for assessing LLM scale and knowledge retention, offering a non-inferential way to probe model capabilities.
This method could lead to greater transparency and accountability for frontier AI models, influencing how companies disclose or are compelled to disclose model characteristics. The increased transparency could also foster innovation in how models are designed to efficiently store and retrieve information.
Wider adoption of such probes might eventually inform policy discussions around minimum factual recall capabilities or 'know-your-LLM' regulations for critical applications, bridging the gap between proprietary AI and public oversight.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI