MEETUP · SAN FRANCISCO
Scaling Marin's Open LLM Pretraining
Marin started as an academic project with one full-time researcher, training medium-sized language models on Google’s TPU Research Cloud.
- When
- Tue, Nov 17
6:00 PM to 8:00 PM PT - Where
- South Beach, San Francisco, CA
- Host
- Open Athena
- Entry
- Free to attend
About this event
Marin started as an academic project with one full-time researcher, training medium-sized language models on Google’s TPU Research Cloud. This month, the team finished pre-training a 535B-parameter mixture of experts model on 18T tokens, using GPUs from the Huang Foundation, in the largest live-streamed pre-training run to date.
This required scaling everything: compute, data, focus, and people. We ran thousands of smaller models to understand scaling, built systems for turning trillions of raw tokens into training mixtures, learned how to train sparse models efficiently, built infrastructure for running reliably across large and heterogeneous clusters, and found people and organizations excited to support open development at this scale.
Open Athena is hosting this event at Laude Labs to share the research, engineering, and organizational work that went into scaling Marin, and what we've learned (and are still learning) from the run. Light bites and drinks will be served for the evening.
Agenda: 6:00pm - Door Opens 6:20pm - Welcome Remarks with Percy Liang 6:30pm - Fireside Chat with David Hall and Andy Konwinski 6:45pm - Presentation by Will Held 7:20pm - Freeform Conversations 8:00pm - Event Ends
We welcome the greater academic, engineering, and research community to join us.
Shooting Scaling Marin's Open LLM Pretraining?
We staff meetups across San Francisco with vetted, background-checked professionals. Our studio edits every frame and delivers in 48 to 72 hours, so your recap goes out while the room is still talking about it.
- Photo coverage for meetups from $200
- A vetted professional confirms within 24 hours
- Deposit is only charged once they confirm
- Backup professional on standby for every date
