[Submitted on 25 May 2023] Download a PDF of the paper titled Scaling Data-Constrained Language Models, by Niklas Muennighoff and 8 other authors Download PDF Abstract: The current trend of scaling language models involves increasing both parameter count and training … Read more
Download PDF
There’s now an open letter arguing that the world should impose a six-month moratorium on the further scaling of AI models such as GPT, by government fiat if necessary, to give AI safety and interpretability research a bit more time … Read more
Published: 2023-03-20 Last updated: 2023-03-20 As of March 2023, the Internet Computer repository contains about six hundred thousand lines of Rust code. Last year, we started using Bazel as our primary build system, and we couldn’t have been happier … Read more
Play Constructing the MNIST-1D dataset. As with the original MNIST dataset, the task is to learn to classify the digits 0-9. Unlike the MNIST dataset, which consists of 28×28 images, each of these examples is a one-dimensional sequence of points. … Read more
Lev KokotovNovember 7, 2022 The question “Does it Scale?” has become somewhat of a meme in software engineering. There is a good reason for it though, because most businesses plan for success. If your app, online store, or SaaS becomes … Read more
The past decade of progress in AI can largely be summed up by one word: scale. The era of deep learning that started around 2010 has witnessed a continued increase in the size of state of the art models. This … Read more
At GitHub, we store a lot of Git data: more than 18.6 petabytes of it, to be precise. That’s more than six times the size of the Library of Congress’s digital collections. Most of that data comes from the contents … Read more
Causal is a spreadsheet built for the 21st century to help people work better with numbers. Behind Causal’s innocent web UI is a complex calculation engine — an interpreter that executes formulas on an in-memory, multidimensional database. The engine sends the … Read more
How we survived, relocated a team, and scaled our startup under never-ending missile attacks. The below photo was taken at 2am on Feb 28th. I’m celebrating my 26th birthday in a bunker with my co-founder, Stacy. I hadn’t slept for … Read more