Can AI Guess What You Know? Performance Comparison of Large Language Models for Human Domain Knowledge Estimation From Communication Logs
arXiv:2605.22971v1 Announce Type: new Abstract: Employees often struggle to identify ``who knows what,'' leading to organizational productivity losses. We investigate whether Large Language Models (LLMs) can infer individual domain knowledge directly from long-term Slack logs. Analyzing 27,188 messages from 43 users, we evaluated seven models (including Gemini, Claude, and GPT families) by comparing their zero-shot estimates against self-reported skill ratings from 27 participants. Gemini 2.5 Flash achieved the lowest error (MAE 21.13%), while GPT models showed significantly larger discrepanci
Advances in large language models (LLMs) and access to rich internal communication data, like Slack logs, are enabling new applications for AI in organizational intelligence.
This development suggests a near-term capability for AI to autonomously assess human skills and domain knowledge within enterprises, potentially transforming talent management and organizational design.
The ability to accurately infer individual domain knowledge from communication logs alters how enterprises might identify expertise, optimize team formation, and streamline internal knowledge transfer.
- · AI platform providers
- · Enterprise software companies
- · HR tech companies
- · Large organizations
- · Traditional HR consulting
- · Manual skill assessment tools
- · Companies with siloed knowledge
- · Individuals with opaque expertise
Companies will begin deploying AI systems to map internal expertise and knowledge networks.
This granular understanding of human capital will lead to more efficient team composition and project allocation, boosting organizational productivity.
The widespread adoption of such systems could fundamentally alter career paths and the value placed on explicit versus implicit knowledge within organizations, potentially raising privacy concerns regarding internal data.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL