Codestral Mamba
Mistral's code model built on the Mamba state-space architecture instead of a transformer
Codestral Mamba is a code generation model Mistral AI released in July 2024, notable mainly for its architecture: instead of the transformer design used by nearly every other code model, it’s built on Mamba, a state-space model architecture. State-space models process sequences with linear rather than quadratic scaling in sequence length, which means inference speed does not degrade as sharply as context grows. Mistral sized it at roughly 7 billion parameters and gave it a 256,000-token context window, with the company reporting reliable in-context retrieval performance even at that length.
On HumanEval, Codestral Mamba scores around 75 percent pass@1, competitive with much larger transformer-based code models. It’s released under the Apache 2.0 license, so the weights can be freely downloaded, fine-tuned, and used commercially, and it’s also available through Mistral’s API at $0.25 per million input and output tokens. Mistral pitched it as a proof of concept that state-space architectures can match transformers on real coding tasks while offering faster and cheaper inference for long-context use cases like reasoning over large repositories.