Meta FAIR just released research on Large Concept Models that works completely differently from normal language models. Instead of predicting the next word, LCMs predict entire sentences and concepts at once.
This matters because current AI models get confused when handling long texts. They lose track of what they’re talking about and start contradicting themselves. LCMs solve this by thinking at a higher level – more like how humans plan out what they want to say before saying it.
The model uses something called SONAR, which can handle 200 written languages and 57 spoken ones. This means one model can work across many languages without needing separate training for each one.
I find three things particularly interesting about this research:
1. It’s more efficient than normal language models because it works with whole ideas instead of individual words
2. It can plan out entire paragraphs or sections ahead of time, leading to more coherent writing
3. The same model works for both text and speech, which could enable some powerful applications
The researchers are exploring different ways to generate text, including some techniques borrowed from AI image generation. They’re also working on ways to make the model more efficient by simplifying how it represents concepts.
For more on recent AI developments, check out my analysis of OpenAI’s latest model scoring 88% on a key AGI benchmark: https://adam.holter.com/openai-o3-hits-88-on-alans-agi-countdown-heres-why-that-matters/
The code for LCMs is available on GitHub if you want to try it yourself. Let me know what you think about this new approach to language modeling in the comments.
