OpenMythos: A New Era in AI Architecture

Kai Gomez, a 22-year-old developer, has launched OpenMythos, an open-source initiative aimed at reconstructing the Claude Mythos architecture using a Recurrent-Depth Transformer (RDT) design that focuses on reusing a limited set of layers instead of merely increasing parameters. This innovative approach emphasizes efficiency and modularity, challenging traditional AI model designs that rely on increasing parameters. The RDT model enhances reasoning during inference by iterating the same layers multiple times, refining the internal state, which contrasts with traditional models that add depth through more layers. OpenMythos features a mixture of experts (MOE) system with 384 specialized experts, activating only a subset during processing to improve efficiency and diversify knowledge without redundancy. A 770 million parameter RDT can match the performance of a 1.3 billion parameter standard transformer, challenging traditional scaling assumptions in AI. The RDT operates in latent space, allowing it to represent multiple reasoning paths simultaneously, which enhances its problem-solving capabilities without generating intermediate tokens. Experiments demonstrate that the RDT can perform systematic generalization and extend its reasoning beyond training limits, unlike standard transformers that struggle with unseen knowledge combinations. OpenMythos addresses stability issues in recurrent architectures through linear time invariant injection, ensuring stability of the hidden state during multiple iterations.