Architecture
We investigate model architectures that selectively allocate expensive computation where it provides the most value. The question is how to use limited resources well while preserving useful language-model capability.
The research program
Research
Exploring new Pareto frontiers in language-model capability, compute, and memory.
An open question
How much more can a language model do with the resources it already has?
View research status01 / Research philosophyEfficiency has
Efficiency has
more than one axis.
A useful result must hold up beyond the experiment that first revealed it. We study the balance between model quality, computational cost, and memory requirements, then test how that balance changes across settings.
We are working toward better trade-offs. We are not publishing performance comparisons or numerical results at this stage.
Scaling
We evaluate research across increasing model sizes rather than relying on single small-scale wins. Controlled experiments help us decide which ideas warrant a larger investment of compute.
Representation
We validate architectural findings beyond toy byte-level settings using modern subword tokenization and broader text distributions. Transfer to real-world language is a research milestone in its own right.
Memory and inference state
We study how architecture affects the memory required during inference, particularly as sequence length grows. Memory movement and retained state are part of the efficiency question, alongside computational work.
02 / Evaluation methodologyEvidence before
Evidence before
extrapolation.
Our evaluation approach is designed to distinguish promising ideas from findings that hold up under more demanding conditions.
Matched controls
Compare candidates with appropriate baselines under comparable experimental conditions.
Multiple training seeds
Use multiple training seeds for important research promotions to test whether findings are repeatable.
Held-out evaluation
Evaluate on data held out from training to examine generalization.
Model-size scaling
Revisit findings at increasing parameter counts before drawing broader conclusions.
Compute-aware comparison
Consider the computational resources behind a result, alongside model quality.
Recall and long-range diagnostics
Probe information retention and behavior across longer sequences.
03 / Research statusProgress, with
Progress, with
clear boundaries.
Controlled studies have reached the approximately-100M-parameter class. Real-web transfer is complete.
We are preparing the build of our first approximately-300M-parameter generalist model, intended to become our first model candidate for external evaluation and potential public release.
Current focus
Preparing our first ~300M generalist model build.
- Completed
Architecture exploration
- Completed
Controlled small-model studies
- Completed
Subword-token transfer (BPE)
- Completed
~30M-class validation
- Completed
~100M-class controlled scaling
- Completed
Real-web transfer at ~100M class
- Preparing
First ~300M generalist model build
- Future work
Production runtime
Let’s compare notesInterested in efficient
Interested in efficient
AI infrastructure?
We are interested in conversations with researchers, infrastructure teams, hardware companies, potential design partners, and investors working on the economics of language models.