Spaces:

SupraLabs
/

Blog

Running

App Files Files Community

LH-Tech-AI commited on 10 days ago

Commit

f7e5e78

verified ·

1 Parent(s): 518a93f

Create supra-gateddeltanet-300m-tokens.html

Browse files

Files changed (1) hide show

supra-gateddeltanet-300m-tokens.html +266 -0

supra-gateddeltanet-300m-tokens.html ADDED Viewed

	@@ -0,0 +1,266 @@

+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>300M Tokens. Almost SOTA. The Architecture Did That. ⚡ | SupraLabs Blog</title>
+    <style>
+        :root {
+            --bg: #0f0f0f;
+            --surface: #1a1a1a;
+            --border: #333;
+            --text: #e0e0e0;
+            --accent: #536bfe;
+            --muted: #888;
+            --font-mono: 'JetBrains Mono', 'Fira Code', monospace;
+        }
+        * { margin: 0; padding: 0; box-sizing: border-box; }
+        body {
+            background-color: var(--bg);
+            color: var(--text);
+            font-family: 'Inter', -apple-system, sans-serif;
+            line-height: 1.6;
+            padding: 2rem;
+        }
+        code, pre, .mono { font-family: var(--font-mono); }
+        .container { max-width: 900px; margin: 0 auto; }
+        header {
+            border-bottom: 2px solid var(--border);
+            padding-bottom: 2rem;
+            margin-bottom: 3rem;
+            display: flex;
+            justify-content: space-between;
+            align-items: flex-end;
+        }
+        .logo-area h1 {
+            font-size: 1.2rem;
+            text-transform: uppercase;
+            letter-spacing: 2px;
+            color: var(--accent);
+            line-height: 1;
+            display: flex;
+            align-items: center;
+            gap: 10px;
+        }
+        .logo-area a { text-decoration: none; color: inherit; }
+        .logo-area { display: flex; align-items: center; gap: 10px; font-weight: bold; font-size: 1.2rem; }
+        nav a {
+            color: var(--text);
+            text-decoration: none;
+            margin-left: 1.5rem;
+            font-size: 0.9rem;
+            border-bottom: 1px solid transparent;
+        }
+        nav a:hover { border-bottom: 1px solid var(--accent); }
+        .post-header { margin-bottom: 3rem; }
+        .post-header h2 { font-size: 3rem; line-height: 1.1; margin-bottom: 1rem; font-weight: 800; }
+        .post-meta { font-family: var(--font-mono); color: var(--accent); font-size: 0.9rem; margin-bottom: 2rem; }
+        .post-content {
+            background: var(--surface);
+            border: 1px solid var(--border);
+            padding: 3rem;
+            margin-bottom: 4rem;
+        }
+        .post-content h2 { font-size: 1.8rem; margin: 2.5rem 0 1rem 0; color: var(--accent); }
+        .post-content h2:first-child { margin-top: 0; }
+        .post-content p { margin-bottom: 1.5rem; font-size: 1.1rem; color: var(--text); }
+        .post-content ul { margin-bottom: 1.5rem; padding-left: 1.5rem; }
+        .post-content li { margin-bottom: 0.5rem; font-size: 1.1rem; }
+        .post-content strong { color: #fff; }
+        .post-content code {
+            background: #111;
+            border: 1px solid var(--border);
+            padding: 2px 6px;
+            border-radius: 3px;
+            font-size: 0.95em;
+            color: var(--accent);
+        }
+        .callout {
+            border-left: 3px solid var(--accent);
+            background: #111;
+            padding: 1rem 1.5rem;
+            margin: 2rem 0;
+            font-family: var(--font-mono);
+            font-size: 0.95rem;
+            color: #ccc;
+        }
+        .callout span { display: block; color: var(--muted); font-size: 0.8rem; margin-bottom: 0.4rem; }
+        /* IO example pairs */
+        .io-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 1rem; margin: 2rem 0; }
+        .io-pair {
+            background: #111;
+            border: 1px solid var(--border);
+            padding: 1.2rem;
+        }
+        .io-pair .io-label {
+            font-family: var(--font-mono);
+            font-size: 0.72rem;
+            color: var(--muted);
+            margin-bottom: 0.5rem;
+        }
+        .io-pair .io-user { color: #ccc; font-size: 0.95rem; margin-bottom: 0.8rem; line-height: 1.5; }
+        .io-pair .io-result {
+            font-family: var(--font-mono);
+            color: var(--accent);
+            font-size: 0.9rem;
+            font-weight: 700;
+        }
+        /* Flow diagram */
+        .flow {
+            display: flex;
+            align-items: center;
+            justify-content: center;
+            gap: 0.8rem;
+            margin: 2rem 0;
+            flex-wrap: wrap;
+        }
+        .flow-box {
+            background: #111;
+            border: 1px solid var(--border);
+            padding: 0.8rem 1.2rem;
+            font-family: var(--font-mono);
+            font-size: 0.82rem;
+            text-align: center;
+            color: #ccc;
+        }
+        .flow-box.accent { border-color: var(--accent); color: var(--accent); }
+        .flow-arrow { color: var(--accent); font-size: 1.2rem; }
+        .table-wrap { overflow-x: auto; margin: 2rem 0; }
+        table { width: 100%; border-collapse: collapse; font-family: var(--font-mono); font-size: 0.88rem; }
+        th { background: #111; color: var(--accent); padding: 0.75rem 1rem; text-align: left; border: 1px solid var(--border); }
+        td { padding: 0.7rem 1rem; border: 1px solid var(--border); color: var(--text); }
+        tr:nth-child(even) td { background: #111; }
+        .tags { display: flex; gap: 0.5rem; margin-top: 2rem; flex-wrap: wrap; }
+        .tag { font-family: var(--font-mono); font-size: 0.7rem; padding: 2px 8px; border: 1px solid var(--border); border-radius: 4px; color: var(--muted); }
+        footer { margin-top: 6rem; padding-bottom: 2rem; font-size: 0.8rem; color: var(--muted); text-align: center; }
+        @media (max-width: 600px) {
+            .post-header h2 { font-size: 2rem; }
+            .post-content { padding: 1.5rem; }
+            header { flex-direction: column; align-items: flex-start; gap: 1rem; }
+            nav a { margin-left: 0; margin-right: 1rem; }
+            .io-grid { grid-template-columns: 1fr; }
+            .flow { flex-direction: column; }
+        }
+    </style>
+</head>
+<body>
+<div class="container">
+    <header>
+        <div class="logo-area" style="font-size: 1.5em;">
+            <a href="./index.html"><h1><img src="./image.png" style="height: 2em"> SupraLabs_</h1></a>
+        </div>
+        <nav>
+            <a href="./index.html#news">News</a>
+            <a href="https://huggingface.co/SupraLabs" target="blank">HuggingFace</a>
+            <a href="./index.html#hardware">Hardware</a>
+        </nav>
+    </header>
+    <article>
+        <div class="post-header">
+            <div class="post-meta">// 2026-08-30 | Research</div>
+            <h2>300M Tokens. Almost SOTA.<br>The Architecture Did That. ⚡</h2>
+        </div>
+            <div class="post-content">
+        <p>Two days ago, AxionLab built a complete Python Jupyter Notebook for a 5M parameters model and told me to train it on a RTX 5090 on Runpod and it instantly ran - almost like a single-shot of an LLM if you know what I mean 😂</p>
+        <p>Five million parameters. <strong>GatedDeltaNet</strong> under the hood. About <strong>300 million tokens</strong> of training data. That is not a typo. It's real 😭</p>
+        <p>While a lot of models in the community think that you'll need to throw like <strong>several billion tokens</strong> on a 5M just to look competitive, this little thing walked into the hard benchmarks tests - PIQA, HellaSwag, ARC-Easy, ARC-Challenge - and nearly sat down at the same table as <strong>CMA-8M</strong>, <strong>Qana-mini-5M</strong>, and <strong>GPT-S2-5M</strong>.</p>
+        <p>It's not a lie - it's true. Keep reading and you'll find out. :D</p>
+        <h2>The setup, no fluff</h2>
+        <p><strong>Architecture:</strong> GatedDeltaNet<br>
+        <strong>Size:</strong> 5M parameters<br>
+        <strong>Data:</strong> ~300M tokens<br>
+        <strong>Age:</strong> trained two days ago<br>
+        <strong>Mood:</strong> slightly feral 😂</p>
+        <p>Most 5M class models you actually respect were fed some billion tokens. We gave this one a small fraction of that and it still was amazingly competitive!</p>
+        <p>That's the whole thing.</p>
+        <h2>Why GatedDeltaNet hits different</h2>
+        <p>Transformers are the default for a reason. They also spend like it 😭</p>
+        <p>GatedDeltaNet is a linear-ish sequence model with a gated delta rule: memory that updates, forgets on purpose, and does not make you pay quadratic rent for every extra token. At 5M params that is not a nice to have, it's almost obligatory.</p>
+        <p>Fast mixing. Controlled retention. Recurrence that actually remembers what it should. The kind of architecture that makes a small model feel bigger than the spreadsheet says.</p>
+        <p>If you have been waiting for a reason to care about gated delta style sequence models outside of papers: this is one. 🧠</p>
+        <p>Keep reading!</p>
+        <h2>The hard boards</h2>
+        <p>We care about the benches that do not clap for you.</p>
+        <p><strong>PIQA</strong> --> physical commonsense. Does the model know how the world actually behaves?</p>
+        <p><strong>HellaSwag</strong> --> completion that looks easy to humans and eats small models alive.</p>
+        <p><strong>ARC-Easy</strong> and <strong>ARC-Challenge</strong> --> science questions, including the ones that are supposed to hurt.</p>
+        <p>Against CMA-8M, Qana-mini-5M, and GPT-S2-5M - names the community already treats as near the front of this weight class (soon not any longer 😏🔥) - our 5M GatedDeltaNet closed most of the distance.</p>
+        <p>Wait, what? Read that again. Smaller or matched size. Far less data. Same brutal evals. Almost there. Amazing 🤩</p>
+        <p>When compute is scarce, data efficiency is not a footnote. It is the product.</p>
+        <h2>300M vs several billion is not a flex. It is the point.</h2>
+        <p>Training on ~300M tokens is a constraint we chose to respect, not a bug we will quietly patch in the appendix.</p>
+        <p>If an architecture only looks good after you drown it in tokens, you did not find a better model. You found a more expensive average 😭</p>
+        <p>GatedDeltaNet at this scale is saying something louder:</p>
+        <p><strong>less data + strong structure --> real reasoning signal</strong></p>
+        <p>That arrow is the whole lab thesis. ✨</p>
+        <h2>The real benchmark numbers</h2>
+        <p>Here you can see the model performing. Remember: ~300M tokens. Not some billions of it!</p>
+        <div class="table-wrap">
+            <table>
+                <thead>
+                    <tr><th>Model</th><th>Train tokens</th><th>ARC-Easy</th><th>ARC-Challenge</th><th>HellaSwag</th><th>PIQA</th></tr>
+                </thead>
+                <tbody>
+                    <tr><td><strong>Supra-5M-GatedDeltaNet</strong></td><td><strong>~300</strong> 😏</td><td><strong>33.29%</strong></td><td><strong>17.83%</strong></td><td><strong>26.10%</strong></td><td><strong>54.19%</strong></td></tr>
+                    <tr><td>fromziro/Qana-mini-5M</td><td>~21B 😭</td><td>34.97%</td><td>23.21%</td><td>27.60%</td><td>57.18%</td></tr>
+                    <tr><td>AxiomicLabs/GPT-S2-5M</td><td>~75B 😭</td><td>33.92%</td><td>22.87%</td><td>27.87%</td><td>57.56%</td></tr>
+                    <tr><td>User01110/CMA-8M</td><td>~21B 😭</td><td>35.35%</td><td>23.29%</td><td>28.19%</td><td>58.22%</td></tr>
+                </tbody>
+            </table>
+        </div>
+        <p>This is the whole thing. All values are acc_norm. And we're not lying, faking or anything else the values. All is 1:1 spit out from <code>lm_eval</code>. 1:1. For you. Read it again and again and enjoy how good GatedDeltaNet is :D</p>
+        <h2>What we are not saying</h2>
+        <p>We are not pretending 5M params solved intelligence. Surely not 😂</p>
+        <p>We are not shipping a silent "trust us" leaderboard with mystery evals.</p>
+        <p>We are not done. And something improved will come VERY soon! (<em>stay tuned</em>)...</p>
+        <p>The model is two days old. The architecture is clearly not. That combination is what made this model really great.</p>
+        <h2>What is next</h2>
+        <p>More tokens, carefully. Not a mindless 10x dump.</p>
+        <p>Better data mix. Tighter evals. The same spine.</p>
+        <p>If GatedDeltaNet can almost hang with the community SOTA club on a small fraction of the usual diet, we want to know where the ceiling actually is - not where the default recipe says it should be. And we'll find it out - for you 🤗</p>
+        <h2>One last thing</h2>
+        <p>SupraLabs exists to find the models that punch up.</p>
+        <p>This one punches. Defintely.</p>
+        <p>GatedDeltaNet. 5M. ~300M tokens. Two days old. Already bothering the names people quote.</p>
+        <div class="tags">
+              <span class="tag">#research</span>
+              <span class="tag">#small-model</span>
+              <span class="tag">#gated-delta-net</span>
+              <span class="tag">#GDN</span>
+              <span class="tag">#edge-ai</span>
+              <span class="tag">#tinyml</span>
+        </div>
+        </div>
+    </article>
+    <footer>
+        <p class="mono">&copy; 2026 SupraLabs // Built for the community.</p>
+    </footer>
+</div>
+</body>
+</html>