86% of What Coding Agents Do Is Just Reading — Not Solving | Alexander Whedon of Subquadratic

[00:00:00] [SPEAKER_01] In Transformer models, the cost of attention grows with the square of the context length. So doubling context makes the compute roughly four times more expensive. And Alex and his team have solved this issue.

[00:00:18] [SPEAKER_00] When we've done benchmarking the Frontier models with Subbench Pro, we found that 86% of the steps were actually read steps, just trying to do that context engineering before execution review, which was only the last 14%.

[00:00:29] [SPEAKER_01] Is this the end of RAG? If you could put everything that's in your RAG knowledge base into the context window, you wouldn't need that more complicated architecture. Is that right?

[00:00:41] [SPEAKER_00] Yeah, it is. Although I wouldn't say it's the end of RAG, just because it's too extreme. With all these enterprises that are sitting on massive amounts of data, and a lot of that data has yet to be put to use in an AI product, they're getting pushed to do like a $10 million data transformation project before they can even start to build a product on top. And the value prop that we bring to the table is, that is no longer necessary.

[00:01:05] [SPEAKER_01] Hi, this is Eye On A.I.. And today we're talking to Alex Whedon about his company SubQ, which solves a problem that plagues transformer models. In transformer models, the cost of attention grows with the square of a context length.

[00:01:30] [SPEAKER_01] So doubling context makes a compute roughly four times more expensive. That's what we mean by quadratic compute complexity. And Alex and his team have solved this issue. So Alex, can you start by introducing yourself to listeners? And we'll go from there.

[00:01:58] [SPEAKER_00] Alex Whedon Sure. I'm Alex, co-founder and CTO here at SubQquadratic. Alex Whedon I've been in the space for about 10 years. I started working with language models since before transformers existed when LSTMs were still the hot thing. And our mission here is to always be the first to make the most computationally memory and sample efficient algorithms and build foundation models on top of those.

[00:02:27] [SPEAKER_01] Alex Whedon Yeah. And can you explain, Alex Whedon You've tackled this quadratic complexity with sparse attention, am I right? And can you explain what sparse attention is in simple terms and why it matters compared to standard transformer attention?

[00:02:49] [SPEAKER_00] Alex Whedon Sure. So the quadratic bottleneck of attention is introduced because attention compares two token relationships. So you have 1000 tokens, then there are 1000 squared, two token relationships or a million squared, two token relationships for a million tokens. You have this quadratically growing set of math that you're going to perform with an attention. Sparse attention says we don't actually

[00:03:16] [SPEAKER_00] need all of those token relationships. And so sparse attention attempts to find which token relationships are important before running attention on only a subset of the token relationships. And one of the main challenges with sparse attention is that there's always been a quality trade-off, as you can imagine. Alex Whedon In theory, if you're not looking at every token relationship, then you can potentially miss some important ones. Alex Whedon In theory, if you're not looking at the target token.

[00:03:46] [SPEAKER_00] Alex Whedon In theory, if you're not looking at the target token, you can see the target token and all of the other potential tokens. There's some fixed mathematical formula that determines which tokens should be passed through attention. DeepSeek Sparse attention upended this by saying we're actually going to use a separate prediction model

[00:04:16] [SPEAKER_00] to determine which tokens are important. And that worked. And they showed that if you dynamically select the token relationships before running attention over a subset, then you can do so without a quality trade-off. The only problem is that prediction model that selects the token relationships uses full attention.

[00:04:34] [SPEAKER_00] Alex Whedon In theory, if you're not looking at the token and so you still have the quadratic problem. In fact, that separate selection model uses more compute than the sparse attention layers in the full model at just 52.000 tokens, and then continuously grows to an even larger differential. And so what we've done that's unique is we have a sparse attention mechanism that, like DeepSeek Sparse Attention,

[00:05:03] [SPEAKER_00] dynamically selects the token relationships, but does so without the massive cost of full attention and does so without a separate selection mechanism. So the single model does both

[00:05:14] [SPEAKER_01] the selection generation at the same time. And can you give us a practical use case where this quadratic compute complexity makes it too expensive to run the model on something? I mean, code bases seems like an obvious, but are there other situations where you want

[00:05:41] [SPEAKER_01] a very large context window and that it would get unreasonably expensive to do that with regular attention? I mean, regular transformer algorithms. Yeah, absolutely. I mean, I think

[00:05:57] [SPEAKER_00] a very significant portion of enterprise workloads fall into this category. And by the way, I will just clarify now that we're not just after the absolute context window size, we're after higher intelligence within that context window, lower cost of context, lower latency for context. And so these are actually four different axes of context we care about, size, intelligence,

[00:06:21] [SPEAKER_00] cost, and latency. And enterprise workloads are dramatically impacted by these. Everybody's making trade-offs. Everybody's trying to reduce the number of tokens that they're using as much as possible. Everybody is doing context engineering that requires a lot of effort and also narrows the scope of what that system can actually handle. But some of the top use cases that we see are, yes,

[00:06:46] [SPEAKER_00] code is obviously one. Thinking about code review, planning, and long horizon memory, those are all long context problems. Thinking about the fact that when we've done benchmarking of frontier models with SweetBench Pro, we found that 86% of the steps were actually read steps, just trying to do that context engineering before the execution review, which was only the last 14%.

[00:07:15] [SPEAKER_00] And then financial document analysis is another really big one. Financial documents can be a couple hundred pages. And if you really want to analyze trends, then it's helpful to have many of those documents over a period of time or several different companies. In enterprise, you have hundreds of billions of tokens of textual data. And anytime you want to tackle a problem, it's helpful to have

[00:07:42] [SPEAKER_00] like a broad range of documents. The key point that we're conveying and working on with enterprises is how do we do coarser grain retrieval, more retrieval steps at once? How do we avoid chucking documents as much as possible or make larger, more comprehensive chunks, et cetera, to offset the quality loss that we

[00:08:11] [SPEAKER_00] often see with a lot of the context engineering that's done today?

[00:08:15] [SPEAKER_01] Yeah. And that quality offset, or you said earlier a moment ago that it's not only the cost, but it's the intelligence within the context. Is that how you phrased it? Because large context windows,

[00:08:38] [SPEAKER_01] many LLMs don't operate optimally within a very large context window. They don't see things or they get lost in the context. Can you talk about that? How do you solve that? I mean, even though you can now do

[00:09:01] [SPEAKER_00] very large context efficiently. It's a great question. So yeah, the intelligence within the context window is a very important topic. If we just take a step back and think about what frontier context has looked like historically, we didn't get frontier models with more than a 256,000 token context window until March,

[00:09:29] [SPEAKER_00] when the Opus 4.6 general access, 1 million token general access came available. And so until then, most enterprises were building with a less than 256k usable context window. You see that so a lot of people that we talked to, they're using 100,000 tokens or less, and they haven't yet

[00:09:52] [SPEAKER_00] adapted their workflows or rag setups to use more than that. Sometimes they feel a quality trade-off above that with particularly hard problems. Actually, 100,000 is a fairly common cutoff, surprisingly. In the coding space, a lot of people set the compaction limit to 400,000 tokens, because that's where they feel there is a qualitative trade-off within a 1 million token context window. Because of this, in addition to intelligence, I would say there's also an alignment problem

[00:10:22] [SPEAKER_00] within the context window that user preferences with how a model should handle, should leverage 1 million tokens are not super robustly understood today. There are just not enough enterprise applications built to actually leverage a full 1 million token context window to really understand what that should look like. And so as we push into the multi-millions of tokens, this is very much white space, not even

[00:10:50] [SPEAKER_00] just from a technical perspective, but from our product perspective, what the users want. There are all sorts of benchmarks that kind of measure human preferences at short context, but a long context of those don't exist. And then finally, I would just add that one of the main, one of the challenges with large context intelligence and user preference matching is the amount of training that you can do at large context.

[00:11:15] [SPEAKER_00] And so that's one of the main reasons that we set out to create SSA in the first place was that it is cost prohibitive to do large scale training on large context. As far as we know, we are the only people to do robust multi-million token pre-training, which we think differentiates our models. So like the number of times that a model can see how to handle a certain situation at a reason over

[00:11:42] [SPEAKER_00] some context at large scale, as that number goes up, obviously the model gets a lot better and that's really hard to do with quadratically scaling compute costs. So go a little deeper on that. I mean, you've

[00:11:54] [SPEAKER_01] trained and evaluated up to 12 million token context windows. So explain how the, how you avoid the problems that large context windows create. I mean, you, you go a little deeper on how, how the model approaches that.

[00:12:21] [SPEAKER_00] Yeah. So, I mean, I feel like I, I kind of buried the lead on that with one of my earlier answers, but I mean, the problem with sparse attention historically has been that they're fixed pattern mechanisms like sliding window attention. Look at the last 1000 tokens that are closest and

[00:12:45] [SPEAKER_00] and, and, or before and after, but, but these fixed patterns don't robustly model language. So like, there's no guarantee that the first and seventh token are always going to be relevant to each other. Um, it's highly context or content dependent. And so you need a mechanism that accurately models context, um, accurately determines which, uh, relationships are going to be important no matter

[00:13:15] [SPEAKER_00] what the input looks like. And, um, and so while deep sea sparse attention uses a separate model to do that, um, reintroducing the quadratic problem, uh, we do not, we have, um, a mechanism that is much more lightweight and cheaper and leverages the existing model that does the generation to do the selection as

[00:13:39] [SPEAKER_00] well. Um, so it's super cheap, super fast, um, but also dynamic. So every input gets a different, um, attention map and, um, and then we train our, we, we leverage that the fact that we can do this dynamic selection, um, to achieve high quality results while also having efficiency differentials. Um, we, we are, um,

[00:14:06] [SPEAKER_00] uh, around 40 times faster than flash attention before on, uh, B 200s and B 300s. And, um, um, and then we're using 64 times less compute, uh, both of these figures at a million tokens. Um, so this means that we can do a lot more 1 million token training, um, with the same compute or, or 2 million token training or whatever. And, um, so this is, this enables us to introduce a lot

[00:14:36] [SPEAKER_00] of problems that models have yet to be trained on robustly. Um, the reality is, is we're training on a lot of data that hasn't been including in training to dates and introducing capabilities that have yet to be introduced into the models to dates. Um, because we have this mathematical efficiency differential that, uh, just makes the math of doing this training so much different.

[00:14:56] [SPEAKER_01] Just, uh, because my audience isn't all, um, ML engineers. Um, uh, so, so the model, uh, you use sparse attention, um, and you, how does the model decide what tokens to pay attention to? I mean, you were saying that DeepSeq has a separate model that does

[00:15:24] [SPEAKER_01] that prediction, but can you go into how your models with these large context windows then can decide

[00:15:35] [SPEAKER_00] which, which tokens to focus on? Yeah. Um, I mean, that's the highly proprietary part that we, we have to do, but, um, but I would say, uh, I guess I could say we creatively use some early information that's the, um, uh, uh, on the inputs in a way that doesn't create all to all comparisons.

[00:16:03] [SPEAKER_00] Um, which is very high level, but I, I feel that going deeper than that, uh, leaves a lot of breadcrumbs.

[00:16:09] [SPEAKER_01] Uh, well, one thing I, I wanted to ask you when we've spoken a couple of times, but, uh, does this, is this the end of RAG then of having a vector database that the model searches? I mean, could you, if you could put everything that's in your, your RAG knowledge base into the context window, you

[00:16:37] [SPEAKER_01] wouldn't need that more complicated architecture. Is that right?

[00:16:43] [SPEAKER_00] Yeah, it is. Although I wouldn't say it's the end of RAG just because it's too extreme. So, um, the way it changes RAG, uh, what we're looking to do is simplify the way that people build products with RAG or, you know, agentic workflows. Um, examples are today, a lot of documents are chunks

[00:17:09] [SPEAKER_00] that don't necessarily need to be chunked. Um, the chunk sizes are very small, uh, 400, every 400 tokens is like super common. And, um, but like, what if you could have a multi-page multi, multi-section chunk instead, there would be a lot more contiguous context that would prevent, um, misunderstandings. Um, top K, a top K of 10 for the top 10 search results is very common. Well, what if you could

[00:17:36] [SPEAKER_00] pull the top 50? Um, what if instead of doing one type of search, you actually did like 20 parallel searches and dumped all of those search results into the context window? These are ways in which I think we make RAG a lot more, we can enable people to make RAG a lot more robust. If you have 400 million tokens, like there still needs to be some way or the model to process, uh, that data.

[00:18:03] [SPEAKER_00] But we're saying like, don't compress everything into a hundred thousand tokens or less. Um, you can now afford to because of costs and latency. Uh, and then also the intelligence and absolute budget of context. Um, you can now pull a lot more context and do a lot more work with every single step in your agent workflow. Um, like I mentioned before, if we were talking about coding problems, um,

[00:18:29] [SPEAKER_00] Opus 4.6 required an average of 66 steps to tackle a few bunch of pro problems. And, um, and if we assume that we only needed like the last 10%, then we could have gotten away with like six or seven steps. If we had been able to, uh, do, if we'd been able to just put all the code in the context window, or even if we took like three or four steps to, to figure out which parts of the code base are more

[00:18:53] [SPEAKER_00] important, being more aggressive with the context in each step. Um, I mean, that is a much faster, much more efficient process that actually uses more token, sorry, fewer tokens in aggregate. So it's, it's more token efficient and, um, and then it's not going to miss the details. Um, we do hear complaints about, uh, coding agents, missing details and, um, and definitely on the

[00:19:19] [SPEAKER_00] longer horizon. So, so not the end of RAG, but the transformation of RAG as we know it. So.

[00:19:29] [SPEAKER_01] And, um, and you talked about doing multimillion token pre-training rather than just post-training. Uh, can you walk us through how that, how that works and how that changes the model's behavior and

[00:19:44] [SPEAKER_00] practice? Um, yeah. So pre-training is the stage of training where, um, you give the model its core knowledge and to some extent, some core capabilities. Um, but it's very knowledge focused. And so the pre-training simply means you predict the next token over and over again. And, um, and then the

[00:20:10] [SPEAKER_00] model gets penalized based off of how often that next token matches the actual next token in a corpus. So it's crude and rudimentary, but it's also highly scalable. And, uh, it's largely seen as a requirement to extract, um, additional capabilities, um, in the, uh, in the post-training phase.

[00:20:31] [SPEAKER_00] And so, um, research has largely suggested that your ability to create value in the post-training phase is limited by the pre-training that you've done. So if you haven't, for example, done a lot of pre-training on code, then you're going to struggle to post-train a model to be really good at code. And so that's why it's important that we've done this multimillion token pre-training,

[00:20:57] [SPEAKER_00] um, because it means that when we get to post-training, there's already a certain amount of that knowledge there. There's a good foundation to build on and in post-training, we can extrapolate even further. And in fact, this is something we showed in our technical report or a sub-Q 1.1 small model. Um, we actually have a model variant where we trained the model. We pre-trained the model with 1 million token inputs up to 1 million. Um, but a lot of 1 million token pre-training. And we show that

[00:21:23] [SPEAKER_00] that 1 million token pre-training made it so that, um, when we did post-training, uh, uh, largely also only up to 1 million tokens, um, that the model is able to extrapolate how to do retrieval style problems up to 12 million tokens without ever seeing anything in post-training over a million tokens. Um, so that's the power of pre-training. Um, it really, it enables the model to extrapolate much

[00:21:49] [SPEAKER_00] further than if it's seeing stuff for the first time in post-training. Yeah. Um,

[00:21:55] [SPEAKER_01] a lot of people right now, particularly with mythos and, and fable out now from anthropic and glm 5.2 from yeah, your pool or Z AI. Yeah. Um, many people worry about large context models, making it easier to

[00:22:16] [SPEAKER_01] find vulnerabilities in code bases. Uh, it is how, how do you view that? And, and, uh, in, in both offensive and defensive cybersecurity use cases? Yeah. I do think that long context windows, long

[00:22:38] [SPEAKER_00] context reasoning capability does help with finding vulnerabilities in code bases, but that's different than finding vulnerabilities in products. So, um, finding vulnerabilities in code bases enables you to be more defensive. Finding vulnerabilities in products puts your product at risk. Uh, so two very different things. Um, in our case, if you already have all the code in the code base, being able to

[00:23:05] [SPEAKER_00] reason over a lot more of it at once enables you to find vulnerabilities much faster and, and also find vulnerabilities you wouldn't invade would not have been able to find otherwise. Um, and so we see that as something that's really valuable for people being able to make their products safer or more robust. That is, um, some of our design partnerships, um, have, uh, have leaned in that direction. Um,

[00:23:32] [SPEAKER_00] I don't think that long context reasoning specifically provides nearly as much an advantage, um, for being able to find the holes in a product from the outside. Um, because they're more about iterative tests. Um, the context requirement is not necessarily significant. Um, but the way this works is, um,

[00:23:58] [SPEAKER_00] software is built layer upon layer, um, of extraction of abstraction. And so, um, um, you know, they're, they're, they're probably like 20 layers of, uh, of different code components underneath the code that you're writing in any, at any given time. And so, uh, if I'm trying to hack your product, I'm trying to guess which open, I mean, there, there are other things too, but this

[00:24:25] [SPEAKER_00] is a big part of it. I'm trying to guess which open source software you've built on top of, um, um, that, and these open source softwares have known vulnerabilities. And so I'm trying to, uh, I'm trying to, I'm testing to see if any of those known vulnerabilities exist in your software. So it's not really a long context reasoning problem. To be honest, I think more than anything,

[00:24:50] [SPEAKER_00] it's actually a knowledge problem. It's, it's an existing knowledge of all the existing vulnerabilities. And that's why you've actually seen that people can, can get open weight models to, to have mythos level capabilities for, for hacking because, uh, the model needs to specifically know, to be honest, sometimes, uh, like the number of vulnerabilities that, that lead to most of the

[00:25:14] [SPEAKER_00] threat, it's lesser than some people realize, um, because everybody builds on a lot of the same

[00:25:20] [SPEAKER_01] software components. So. Yeah. But couldn't this, uh, I mean, to find those well, you, you said there are known vulnerabilities in open source, uh, code bases, but with mythos or, or, uh, you know, or I mean, with your model that, that can handle massive, uh, contexts,

[00:25:46] [SPEAKER_01] couldn't you then just all the various foundations that are managing open source code bases, they could just, uh, dump it all into your model and, and, uh, discover vulnerabilities that maybe they weren't aware of, patch them. I mean, is, isn't it, uh, advantageous, uh, advantageous for,

[00:26:09] [SPEAKER_01] for the open source community to have solved the context, compute complexity problem so they can scan large code bases, uh, and, and fix vulnerabilities?

[00:26:54] [SPEAKER_01] That's a lot of the most compelling, uh, early design partner use cases you're seeing for a large context.

[00:27:00] [SPEAKER_00] is it a finance code internal documents so um unstructured document um processing and analysis is like the number one right now and the number two is knowledge-based products um within those two categories finance has emerged as a really important one just because finance companies

[00:27:27] [SPEAKER_00] working in the finance space are building such data intensive applications they're processing so much data that's kind of like the core criteria for whether a design partner early access user is a good fit for us right now is how data intensive is their application so finance is a a really big one for us um we have had a lot of interest on the coding side um we are intentionally moving a little

[00:27:51] [SPEAKER_00] bit more slowly on the coding side because um we've kind of realized like how much coding is like um a budget game like how much data budget do you have um how much label data can you train on and so uh that's an area that we're really interested in and are working on but with a bit

[00:28:13] [SPEAKER_00] of a longer time horizon and then um and then it's beyond that it's it's like general enterprise document uh processing for knowledge bases and unstructured inside extraction um those are the main use cases that we're seeing um one thing that's great for us is that i think unstructured document processing and knowledge-based products are in pretty much every enterprise out there

[00:28:41] [SPEAKER_00] um and we're also saying that because we can work with data that is less structured and we can process more data at once it becomes a lot easier for an enterprise to get their first value from data so you have all these enterprises that are sitting on massive amounts of data and a lot of that data has yet to be put to use in an ai product um or maybe it has but in in a very limited fashion and

[00:29:08] [SPEAKER_00] they're getting pushed to do like a 10 10 million dollar data transformation project before they can even start to build a product on top and a value prop that we bring to the table is that is no longer necessary at least not at that level like we can help you um either work with the data with with a lot less structure or create the structure a lot faster um because of the our ability

[00:29:36] [SPEAKER_00] to process a large amount of data at once with a lot less curation that's interesting and as we move

[00:29:45] [SPEAKER_01] into uh agents and particularly coding agents increasingly which is happening uh most coding agents use very surgical context management i mean they they hop from file to file or uh section to section uh

[00:30:07] [SPEAKER_01] uh what do you think uh breaks first when we move to agents that can read everything at once yeah i think

[00:30:19] [SPEAKER_00] that will a couple things will happen um one is we will have systems that can generalize better because every time you add a search engine or you know vector database which is a search engine um or some conditional logic to route between the steps um this human curation really limits the ability of that system to do a lot of different things it makes it so that it works within exactly the scope that you've

[00:30:47] [SPEAKER_00] set it out to to be able to to to achieve and um and uh and the more complicated you make it the less the more narrow that's going to be so i think we'll see agents that can do a lot more um we will also see agents that are a lot faster because you know if you're doing 10 to 60 steps um it's pretty slow so

[00:31:16] [SPEAKER_00] uh i would say agents that are cheaper um we want you know a million tokens to feel like 50 000 tokens in terms of intelligence cost latency etc over time and um and so i think we'll see people build a lot more too to be honest like context engineering step is where the vast majority of

[00:31:39] [SPEAKER_00] human and capital investment is today and so if we can get rid of that then we'll see people like the barrier to entry goes down people build a lot more things and um and when like a single step command can do like 10 steps worth of work what are those 10 to 60 step workflows going to be able to achieve now

[00:32:01] [SPEAKER_00] i mean um it'll be a lot more powerful a lot longer horizon um i guess that's that's another point um we really care about long horizon agents and um the ability to manage context not just through a two-hour

[00:32:24] [SPEAKER_00] session but across you know weeks or months of work is something that i think will be critical and will also require much longer context reasoning than we have today yeah um can we we

[00:32:42] [SPEAKER_01] didn't go into your background or the founding of the company and it's interesting can you give us that

[00:32:49] [SPEAKER_00] uh that story sure yeah so i'm i've been in the ai space for the last 10 years um most recently i was head of generative ai a mid-market and enterprise ai consulting firm called tribe ai um i did over 40 projects with a series of mid-market enterprise companies um with a medium-sized group of folks

[00:33:13] [SPEAKER_00] underneath me and um and uh that's where i got a really good horizontal view of what generative ai implementation was looking like in 2023 and 2024 and that's kind of what convinced me that like it was overdue to move to another model paradigm that there were um bottlenecks in the enterprise implementation layer that were going to be painful for a very long time unless we

[00:33:41] [SPEAKER_00] change the actual mathematics behind the the foundation models themselves um before that i was at uh instagram um working on creator monetization products before that i was at stitch fix where i um built the first text-based recommendation system before that i was at a research lab where i published some research and uh built our first production products for our sister companies glass door and indeed uh before

[00:34:09] [SPEAKER_00] that i was at academia.edu where i built um some of our first ai products for our 200 million user base before that i did research with the world bank of blue cross blue shield and um so i've done a lot of different things i'm not a domain expert um in fact that's one of the things i pride myself in is i'm i like to look at things from a very horizontal perspective what's the key value point that's missing across everything everybody's doing and that's really what we're trying to tackle as a company is like how do

[00:34:38] [SPEAKER_00] we take computational memory and sample efficiency and turn that into downstream business value propositions um context is the first one but there will be many more yeah well and you mentioned memory are you working on memory absolutely so i mean we're working on memory within the context of ssa subquadratic sparse attention the thing we talked about publicly we're also working on non-attention algorithms have

[00:35:04] [SPEAKER_00] been for a while um that uh make it totally necessary to have any form of kvcash um kvcash is this massive thing right now where at the multi-million token range uh can easily use a lot more memory than the model it's the model weights themselves um becomes very hard to um host the context without distributing

[00:35:28] [SPEAKER_00] across multiple um cloud data center grade gpus um makes it impossible in my view to do long-range modeling for robotics on jets and norans which have about as much memory as your phone and um so we see this as a critical problem to solve um for long context modeling for general robotics intelligence um and um just the ability to create much better frontier models and exist today

[00:35:58] [SPEAKER_01] the um this long context uh you mentioned that uh most enterprises don't even fully use 256k uh context uh as you introduce them to multi-million token windows what what are you learning about

[00:36:22] [SPEAKER_00] user preferences and behaviors yeah that i mean that they're not well mapped out and they can vary um so we found for example people will tell us um that uh on a news article they'll ask what are the top three insights from here and the model will give three top insights that are technically accurate but

[00:36:50] [SPEAKER_00] they're not the three they cared about um so like this is the level of alignment that becomes necessary uh with long context reasoning gets very complicated so um one of the things we learned is that we are so far from from this area of modeling being mature i think people say well the frontier models have a million token conics window but like their ability to deliver what people want at that size

[00:37:16] [SPEAKER_00] is just completely we are far removed from that today um and uh and promptability is still a bit lacking um and i think uh like this problem can't be solved at the the prompt level um because it's just so complex and nuanced like we do we do actually need to solve this through training first um where we see a

[00:37:44] [SPEAKER_00] very wide number of problems that um a very wide number of inputs and outputs or problems url um that is um so what we can do is uh we can do a lot of things um and uh we can do a lot of things um

[00:38:01] [SPEAKER_00] um so that's a super high level insight um beyond that um i think reasoning is pretty unsolved oh here's here's an interesting one actually um we did some we created some evals for code-based question answering um and financial document question answering um and we ran we did them at 500 000 tokens um

[00:38:31] [SPEAKER_00] a million tokens per opus 4.7 which is like 800 000 tokens for us um and uh and then as close as possible to 2 million tokens which obviously only we could do and so we found on code-based question answering um the uh there that there were like different levels of difficulties like we could

[00:38:55] [SPEAKER_00] say easy medium and hard and um on the easy medium problems like the models most frontier models and our model could do pretty well pretty close to 100 percent and uh and there was really not much of a concern there on performance but it is quite easy to create an eval set that on financial document analysis

[00:39:18] [SPEAKER_00] that is well below 50 percent um at 500 000 or 800 000 tokens and um so it does seem like obviously there's a bias towards code i think that um the model companies have spent like more time in some areas than others and then also i think there are certain types of contexts that are harder than others like i i've been working with financial documents uh for a long time i've spent a lot of time working

[00:39:47] [SPEAKER_00] with edgar data um and there are a lot of things that make it hard um some of them are intentional but um but the structure of the pdfs can be very challenging for llms to handle um data can be really distributed across the several hundreds of pages up to several hundreds of pages um you've got

[00:40:12] [SPEAKER_00] tab viewer and non-tab viewer data you've got tables that are really hard to parse correctly um all these challenges that i think make these problems harder than code in some ways so uh i guess the tldr here is that long reasoning capability today is very asymmetric across verticals

[00:40:33] [SPEAKER_01] yeah uh on on the uh that that you you mentioned that that you you you give these models to standard transformer models uh a large uh context uh a large chunk of text in the context window and ask it to surface the most

[00:40:59] [SPEAKER_01] significant things and it doesn't come up with what you think is most significant how does this is is is is the reasoning stronger in with your architecture or is it that uh it it sees more of

[00:41:21] [SPEAKER_01] the context i mean why is it better uh with with sub q's model um so the biggest part of it is the training

[00:41:30] [SPEAKER_00] differential um that you know we can um end to end our model training speed is still like uh somewhere between five to ten x faster at a million tokens and then um as you move to two million tokens and beyond that's higher the differential is even bigger than that and so it's easy for us to do a lot more long context training than other people are doing and we're also very focused on generating a lot more

[00:41:58] [SPEAKER_00] of that data through both synthetic generation and human labeling and so you know focus and scale is a big part of that this is phenomenon that we call the bitter lesson in the ai space uh yeah um and so we um so so that's a big part of it i think there's a theoretical benefit that i don't feel like

[00:42:24] [SPEAKER_00] we've fully proven one way or another um but that is that uh we are reducing the noise for the model um so instead of looking at a million squared relationships it'll look at you know scalar value times a million token relationships so has less to consider less to model on top of and um and so uh uh the same way that like rag reduces the noise and could sometimes lead to better results i feel like

[00:42:53] [SPEAKER_00] sparse attention could also reduce the noise and lead to better results but again i don't feel like we fully proven that one way or another it is uh not going to be easy to prove uh because a how do you separate the architecture from the training and b what evals are we going to use that are actually helpful today i think we have a massive dearth of uh high quality long conduct reasoning evals so yeah uh and

[00:43:20] [SPEAKER_01] if someone wants to play around with this um how do they do that so today i mean you reach out to us

[00:43:28] [SPEAKER_00] uh we'd evaluate you for a design partnership um the number of uh we're not giving broad access to the model yet just because uh we have limited compute bandwidth ourselves and are trying to prioritize access to people that we think could help us make a better version of the product and um and then uh and we are continuously trying to increase the amount of compute bandwidth we have um these additional

[00:43:55] [SPEAKER_00] these early design partnerships are also helpful for that too the market validation enables us to increase our capital budgets and um and thus compute budgets um so yeah i mean that's kind of the deal dr is like reach out we'll connect if it seems like an interesting fit we usually go through an evaluation process scope a design partnership and then get started um that'll change over the next couple months in the next couple months we do plan to have complete general access and by then we will have a very

[00:44:24] [SPEAKER_00] different version of the product too it will be uh this product our models are continuously undergoing training to respond to feedback that we're getting from partners so um so if anybody thinks the model is

[00:44:35] [SPEAKER_01] good now it's just going to get a lot better yeah and when you do have general availability is is there going to be a free uh layer or a free trial or or how how are you going to get people in the door

[00:44:54] [SPEAKER_00] so um yeah i think we'll have some level of free access for sure um and we'll have tier pricing and tier rate limits and all that so it'll be pretty standard um there are some non-standard things that i that that we are considering doing one of them is um uh like a higher output to input ratio um today

[00:45:21] [SPEAKER_00] the output token cost is usually three to five x the input token cost we'd like the input token cost to be a lot more cheaper oh sorry a lot cheaper still um to make it we really want people to feel like the input tokens are free like just consider the context that you need to for your problem um so i think we'll see higher ratios there um and then secondly

[00:45:45] [SPEAKER_00] we've hesitated saying this a bit um but we've considered um i think our our pricing curve as your inputs grow could look different than than others because our internal price economics look different for it the external pricing for that could also look different like yes that's all i'll say for now

[00:46:03] [SPEAKER_01] so yeah and and i have to ask um you know there's a lot of talk about using ai for for uh scientific research did you guys use any models to design this i mean uh did you talk to

[00:46:25] [SPEAKER_01] you know claude 4.8 or now i guess fable 5 and and you know how do we solve this problem and and go back and forth with a model i'm curious how how companies like yours are are doing that architecture design whether they're tapping the the knowledge base of the model or or whether it's all

[00:46:55] [SPEAKER_01] uh you know iterative trial and error among humans totally so i mean

[00:47:06] [SPEAKER_00] we're definitely using llms in the process and always trying to as much as possible i don't think you can get there with just an llm uh they're not enough today i don't think they're creative enough either um but they're definitely super helpful and i will say sometimes um i'll just open-endedly ask an llm to come up with an idea and it comes up with something i hadn't thought of so uh they're super helpful for that um but um

[00:47:36] [SPEAKER_00] but we've been talking from probably like somewhere between a year and a year and a half um about how to get to auto research for algorithms before you know way before auto research was termed way before rsi was termed recursive self-improvement um we've been trying to build that product um and so i want to be the first company that creates this high quality flywheel of a model proposing

[00:48:05] [SPEAKER_00] architectural variations creating the experiments um to validate or invalidate those variations um running those experiments looking at the outputs coming up with what the idea is for the next uh set of experiments and architectural variations until it comes up with an architecture that looks vastly different and is vastly better on the axes that we care about um we we will be investing into

[00:48:31] [SPEAKER_01] that very seriously so yeah that's interesting and looking ahead what's the most exciting frontier for you i mean at uh at subq is it bigger contexts and not only at your company but in across ai research in general uh new architectures beyond attention or something else i mean yeah a lot of different

[00:48:57] [SPEAKER_00] things um obviously that i think the auto research thing is really exciting yep um i think that uh but again it's it's not about doing the pre or post training better which is where most people are focused it is about the algorithm itself um we want to move from like shifting the algorithmic paradigm every nine years to doing it every 12 months um so we want to replace transformers or attention entirely

[00:49:28] [SPEAKER_00] in the not create not too distant future and then whatever we replace that with i mean we know what we want to replace it with but um when that happens we want to replace that again within 12 months and just shorten this time as much as possible so that's that's interesting um i think that um from a go-to-market perspective we're really excited about working with enterprises on long context reasoning tasks

[00:49:56] [SPEAKER_00] um long context long horizon agentic tasks as a secondary priority um and then in the somewhat longer term though there is active work being done on it uh we do care about robotics and see that it's highly intertwined with our current work actually um like the reason why uh the figure ai demo was on the conveyor belt flipping boxes is because it requires a 15 second memory span

[00:50:25] [SPEAKER_00] um if you think about asking um if you think about asking a robot to do the laundry the robot has to remember the roadmap of the house to be able to navigate um it needs to remember how to open the laundry machine door it needs to understand it needs to know what cycles are appropriate for which um for which clothing items it needs to remember what these are preferences are for how to fold each of the

[00:50:51] [SPEAKER_00] items it needs to remember where those the clothing goes and then um and so the amount of context that's required there is significant and then additionally the um the uh we are we are in the same place for for um robotics intelligence as you were for language models in 2019 where we train models for specific tasks with thousands of task specific samples that were hard to hard and costly and timely

[00:51:21] [SPEAKER_00] uh time expensive to curate and um and so i think we need a 2020 lmr few shot learners gpt3 moments um for the robotic space uh but if you think about providing a 10 minute video sample for so that the model can learn in context um how to achieve a task without being trained on that task specifically

[00:51:44] [SPEAKER_00] um that is that's like a four million token input and people are struggling to do that in the cloud how do you do that on a phone worth of vram um that is that requires like 100x plus a 100x plus reduction in memory usage

[00:52:06] [SPEAKER_00] and that's a problem we're thinking about a lot um so that's a pretty exciting frontier for us um there are a lot of exciting things that we're doing here um and uh some of the stuff is you know stuff we're trying to to put out in the next six months some of some of it is things that we're trying to put out in the next two years um so we've got a longer term road map we're not just thinking about incremental

[00:52:33] [SPEAKER_00] improvements in the industry we want to figure out what problems are going to matter two to five years from now how do we like take a shortcut to get there so and um how big is the team so we have just under 50 people today uh we've grown a lot in the last couple months um i mean the fundamental research that we did that we've announced was done by just a few people but we've

[00:52:59] [SPEAKER_00] been sitting on a lot of research for a while uh that research was um done in 2025 um and then production our product ties this year um but it was done with a lot fewer resources and people than we have today and so and we've done a lot since then and um yeah i mean it's a little under 50 people today

[00:53:23] [SPEAKER_01] okay um well this is all fascinating and i'm glad that we've met i'm certainly going to be following you guys if uh listeners want to follow what you're doing um is it sub q.ai do you have a blog or

[00:53:44] [SPEAKER_00] something there that people can follow yeah uh sub q.ai is good um we've released a couple blog posts posts i wouldn't say that we're prolific bloggers yet uh we do want to try to push out more but it does take some time um yeah i am trying to post at least once if not a couple times a week on twitter and i do want to start going a little deeper on um some of our uh product and technical and research

[00:54:13] [SPEAKER_00] vision here um in some of those posts and i'll call out different things every once in a while so that could be an interesting place to follow us as well we have our official twitter page and um you know may see some of our other employees being more public as well about some of their work so um social is going to be a pretty good place to stay on top of what we're doing as well

[00:54:32] [SPEAKER_01] yeah okay alex well this is really interesting