A Theory for Emergence of Complex Skills in Language Models

A major driver of AI products today is the fact that new skills emerge in language models when their parameter set and training corpora are scaled up. This phenomenon is poorly understood, and a mechanistic explanation via mathematical analysis of gradient-based training seems difficult. The current paper takes a different approach, analysing emergence using the famous (and empirical) Scaling Laws

[rawwell]

"A theoretical consideration of how skills (text understanding, background knowledge, etc.,) and their combinations are acquired by LLM training. Assuming that each text fragment is generated by multiple skills, the reduction in excess cross-entropy is directly related to the"