Rendered at 02:17:22 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
walrus01 3 hours ago [-]
It would be nice if unsloth published GGUFs would use a version number or something, because now I have multiple different files on local storage that otherwise have exactly the same name.
"Qwen3.8-27B-UD-Q8_K_XL.gguf" for instance.
The one downloaded at least 4 days ago is a different thing and is NOT the "Dynamic 3.0" GGUF which I am now downloading, which I presume will have a different sha256 checksum?
The unsloth page says dynamic 3.0 is released "today", but I have an older copy of qwen3.8 27B Q8 which I downloaded, if I remember right, at least 4-5 days ago...
That's terrible. My computer shipped with a filesystem and curl.
mycall 11 minutes ago [-]
There is always SHA256 as the alternative.
m463 2 hours ago [-]
I wonder if there should be some kind of header sort of like a README inside mixed with something like EXIF information that camera images embed.
walrus01 2 hours ago [-]
there's metadata in a GGUF (as can be seen when loading one into llama-server with verbose level 4) but it doesn't appear to show what the expected sha256 of the file is.
gunalx 2 hours ago [-]
Well that would not work as the metadata is also part of the sha, and if you have a sha predictably contain or hash to itself you have a bad hashing algorithm. (Unless of course you mean sha of the weigths and not the gguf itself)
nagaiaida 6 minutes ago [-]
you can just hash the metadata masking out the hash as all zeroes or anything similar to get around that, like header checksums that ostensibly include themselves
jlbprof 3 hours ago [-]
How can I tell which one I have? I dl'd mine a few days ago.
walrus01 2 hours ago [-]
I have just run a sha256 checksum on both copies now, the one I downloaded 4-5 days ago, and the one that's on the unsloth huggingface page released today, and will see if they're the same or different.
(downloaded in the last few hours after the announcement of dynamic 3.0)
So they're actually the same thing, but the announcement says released today... Please let's not confuse the end users any more than they already are.
CMay 2 hours ago [-]
I think they said they were keeping the old UD 2.0 quant for the larger sizes? So maybe they kept those the same and simply reuploaded them. They said the newer UD 3.0 quant performed worse on some things for the higher quants. So now it's a mix of UD 3.0 and UD 2.0.
Alephinitesimal 4 hours ago [-]
I mostly use local models when the data has personal information. Earlier this year, I felt the coding quality was still not as good as Claude Code.
One thing that works for me is to ask the local model to make some fake data with the same format, let Claude Code work on the fake data, and then bring the code back and run it locally on the real data.
This way the real data never leaves my machine, but I can still use a stronger model for most of the coding.
latentsea 36 minutes ago [-]
Qwen3.8-27B has been the turning point for me. It's not as strong as the absolute frontier, but it's the first time I feel local coding models are actually functionally useable as daily drivers.
xlayn 7 hours ago [-]
Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model.
Today I was trying in fact to see, what's the smallest Qwen3.8-27B that I could run and get good results, say restricting it to 16GB of ram.. so I went, pick up the Qwen3.8-27B-UD-IQ2_XXS.gguf and them BAM, error on MTP... now I understand why after reading your announcement.
Beyond the space saving, why removing the MTP? improves speed exactly for the group that could benefit from it.
gruturo 5 hours ago [-]
The reason for running those insanely low quants is to fit in extremely limited memory budgets. The first thing you sacrifice is speed, then context and accuracy (up to you in which order). IQ2_XXS and below is desperate/proof of concept territory. If you have a spare half gig for the MTP drafter, run a larger quant instead, it will be less incoherent, and damn the speed, it won't be garbage at least. Only around Q4 I'd allocate the comparative luxury of more memory for a speed increase. At least on a dense model. MTP makes a lot more sense (but helps statistically a bit less) on an MoE.
Qwaiting for that 3.8-35B-A3B
danielhanchen 4 hours ago [-]
Hey we did not remove the MTP for sizes above 8GiB - but yes for small GGUFs under 8 ish GiB, we removed the MTP module (IQ2_XXS and lower), because it's 500MiB to 750MiB in size, and on small 8 GiB machines, even 500MiB is needed.
As someone in the comments said we made a separate Q4_0 MTP if that's helpful so you can use that.
But I would suggest using UD-IQ3_XXS for 10.9GB for 16GB machines or Q2_K_XL
xlayn 4 hours ago [-]
Daniel, question I got the Qwen3.8-27B-UD-Q2_K_XL.gguf from https://huggingface.co/unsloth/Qwen3.8-27B-GGUF?show_file_in... and continue with my testing, but the model quickly felt into a loop of asking the same thing over and over again, I have seen the MOE do that but not the dense ones.
And I had similar experiences when Qwen3.8-27B unsloth images just came out with the full Q8_K_XL, I'm using an AMD setup which has modifications to save to disk the kv, but your (assuming you are part of the unsloth team) for some reason have been giving me similar issues.
It can be something in my setup, there is a very high chance of that, but the previous 3.6 images from qwen, the 27B, the 31A3 and 122 they are all unsloth and did work on my setup without issues...
Again could be my setup... let me know if there is any data I can supply to you to debug if needed.
mike-the-brain 7 hours ago [-]
you can still have it, no?
> We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed
xlayn 6 hours ago [-]
my bad, you are totally right, thanks!
walrus01 3 hours ago [-]
Q2 quantization is basically giving a capable model a lobotomy. It will not accurately represent how smart or capable something like qwen 3.8 27B in Q8 will be.
zenoprax 43 minutes ago [-]
Sure, but this is true for all lossy compression (audio, images, etc.)
Given 16GB of VRAM, what will give me the best experience in OpenCode? Currently using Qwen3.8_Q_3
johndough 5 hours ago [-]
Are there benchmarks for the various Qwen3.8-27B quants that actually measure writing code, maybe even with multiple steps? Low KL divergence does not mean much when the model gets stuck in doom loops all the time.
I could of course download and test myself, but that would take days with my internet connection.
InvertedRhodium 5 hours ago [-]
I tested Qwen 3.8 on the Blade CTF last night, it took 3 hours but got the correct answer.
I know that didn’t answer your question but I was looking for a test suite and couldn’t find anything.
After reading the logs, there is far less doom looping than with 3.6, but whether that’s a one off or not is up for debate.
Q4_K_P
Balinares 4 hours ago [-]
I anecdotally observed the same.
Interestingly, it also seems to tend toward self-correcting, which makes lower quantizations borderline usable. There'll be more faffing around, but still converging toward a solution. I wonder if that's a deliberate product of its RL.
danielhanchen 4 hours ago [-]
We made something called Divergence-300 @32 (and later @512) which tests actual inference across 32 tokens on a held out test (Terminal Bench, DeepSWE, Math etc)
We do plan to do larger benchmark suites though!
zdragnar 3 hours ago [-]
Purely an anecdote, but I've found Qwen3.8-27b doesn't doom loop like previous Qwen models would. With that said, it absolutely thinks in circles- it'll prepare to do something, say it is now ready to do it, then follow that with three paragraphs that all start with Acutally... Oh wait, I should check first... Hmm, hmm... I should stop guessing and just do it. Okay, I'm ready to do the thing now... Actually, wait...
It takes forever, but it does actually get around to making things work, and it is more thorough and produces better code than previous qwen models. You just need to let it run quite awhile.
zenoprax 39 minutes ago [-]
I've seen the same thing. I tried the "superpowers" meta-harness and gave it a simple web app task and it spent 4 hours to make a basic timer app. I might try restricting the amount of thinking it is allowed to do to 500-1000 tokens.
Systemerror7A69 3 hours ago [-]
Since it seems like this not only improved sizes but also performance I can't wait for some benchmarks and comparisons. If you don't have a separate GPU for inference, every single GB matters so a comparison between specific Q4 Quants is really interesting to me.
Currently I very much can't decide between going for a bit of a lower Q4 Quant to squeeze out a bit of buffer and ctx or wondering if a slightly higher (IQ4_XS vs Q4_K_M/XL) is worth it
throwa356262 7 hours ago [-]
"We also made some smaller UD-1bit quants with UD-IQ1_S being 6.2GB (without MTP) which retain around 72% top-1% accuracy yet being 89% smaller"
This is crazy! But has anyone tried these lower quants on real projects?
Aurornis 3 hours ago [-]
I tried some 1-bit, 2-bit, and bonsai quants against closed eval sets. They were essentially useless for my case. The little errors accumulate and send the whole output off track quickly.
If you had some use case with very small output sequences they could be interesting to try. I think dropping down to a 9B-class model would produce better results for most cases.
Havoc 3 hours ago [-]
What setup are you using to do said private evaluation? Software wise I mean
kennywinker 7 hours ago [-]
Not 1-bit, but I’m getting pretty good results with some light coding using unsloth’s previous 2-bit quant of qwen3.8-27b. With these new quants i may be able to bump up to 3bit, tho it’s already running so slow (15tok/s average for the first 32k of context) that the speed hit might make it not worth the extra smarts
jjcm 3 hours ago [-]
No Dynamic 3.0 NVFP4 quants just yet from the look of it, as a heads up. Would love to see how those perform relative to others on the curve.
mike-the-brain 7 hours ago [-]
Might be off-topic but: is it possible to perform such a quantization on Apple devices? Something like Mac Studio Ultra M1 (even if it would take weeks/months)?
smcleod 6 hours ago [-]
Unsloth use a property dataset they don't release, however you can indeed create quantisation locally on your machine and it's pretty easy, llama.cpp comes with everything you need.
kristjansson 4 hours ago [-]
Just quantizing takes seconds-to-minutes, llama.cpp provides a nice tool[0]. Improving quality is then a matter of picking specific tensors to maintain at higher accuracy, checking on representative data, and repeating.
The new IQ4XS has been working pretty well so far on 4090 16gb.
kamranjon 6 hours ago [-]
What size context are you able to squeeze in with less than 2gb of headroom? I have had some luck using a quantized kv cache but i fear that also decreases overall quality.
QuantumNomad_ 6 hours ago [-]
Is it possible to use a model that needs around 64 GB VRAM if you have four GPUs with 16 GB VRAM each?
gruturo 5 hours ago [-]
Yes, and if you have the PCIe lanes (say, an x16 lane - actually delivering 16 lanes! - to each GPU) it's also quite performant - it's called a tensor split in llama-server.
If your motherboard/cpu doesn't actually have those (few do outside some xeons, epycs and threadrippers) you can still do it - it's called a layer split and will work even with 1 lane per GPU. Each GPU will work at its maximum speed, but only 1 will be active at any given instant - imagine a relay race.
(Didn't mention which PCIe generation - obviously the higher the better. At v4 and up, even 8 lanes per GPU would be enough for a performant tensor 4-way split)
Edit:
If you have more than 1 user at a time, the GPU can actually all be working all the time, if there are enough parallel requests to serve. But you need enough KV cache for all the sessions you're running in parallel.
xlayn 6 hours ago [-]
I do use 2 amd gpus and I get high 40 for generation, 500 for pp and low 20/100 by the end of the context of 256k.
Yes, you can. Ideally though, you want to minimize the number of cards and maximize the amount of memory in each card.
Using multiple cards is one of the things that the models and software that Unsloth releases does really well in terms of ease of use and relatively good performance.
jambalaya8 4 hours ago [-]
How many do you recommend?
strictnein 3 hours ago [-]
I've run 4, 6, and 8. Adding more video cards increases total available memory, so you can load larger models, but there are definitely some drawbacks.
More cards = more communication over PCIe. The prompts don't come in and just get magically split between each card, they move sequentially through them.
Also, with 2x24GB cards you don't really have 48GB of usable memory to load a model, closer to ~42GB + context.
And then there are power concerns, motherboard limitations (PCIe slots and lanes - a lot of motherboards with multiple 16x PCIe slots don't actually have 16x lanes to each of those slots), and more. 8x GPUs are going to easily draw 2000W on their own, if not substantially more. You'll need wiring and a circuit that can support 3000W without a risk of starting a fire in your wall.
For $5k, a single 32GB 5090 might be a better choice for a lot of people versus 4x3090s with 24GB each. It will definitely perform substantially better on smaller 27B models.
For hardware:
A good motherboard with lots of PCIe lanes (7x full 16x PCIe 4.0), DDR4 support, etc:
Add in a 3xxx series Threadripper PRO, 128 or 256GB of DDR4 (going higher becomes really expensive), and a ~1400 watt power supply. You can underpower/undervolt Nvidia cards really easily, and capping them at 250W loses you minimal performance.
For some reason the unsloth models leave hardly any room for context. I've switched to the regular (non-unsloth) and get about 25 t/s and get about 80,000 more context tokens for the same quant.
Balinares 4 hours ago [-]
Wow, interesting. What KV cache quantization do you use?
walrus01 3 hours ago [-]
Yes, google "llama-server split model multiple GPU" for some concrete examples.
charcircuit 6 hours ago [-]
Of course. Models don't actually require VRAM. Nor do they require regular RAM. You could have 1 GB of RAM and swap the model to disk as you need different parts of it. And if you didn't have enough disks you could access weights via a network connection.
leoooodias 5 hours ago [-]
You don’t even need electricity. You could print the model weights onto millions of sheets of paper, and hire a team of carrier pigeons to fly them into your office one by one. No VRAM!
Coincidentally that’s also Google’s new method for distributing Android sources.
revolvingthrow 5 hours ago [-]
The bitrot would be excessive
segbrk 6 hours ago [-]
Yes, but unless they support NVLink (they don't), it's quite slow.
tetsuo420 6 hours ago [-]
It seems the NVFP4 quants have a preview version of this Unsloth Dynamic 3.0. Is this close to the finished version, or would it be better to switch to one of the newer quants?
josh-wrale 5 hours ago [-]
Sidebar: single threaded inference isn’t good enough anymore
sosodev 5 hours ago [-]
What about do you mean by single threaded? Each token is predicted by using parallel computation on the GPU.
josh-wrale 5 hours ago [-]
Multiple agents need tokens. Should optimize for that instead of one agent blocking the others.
sosodev 4 hours ago [-]
One agent typically blocks the others on a local device because the GPU is already completely utilized either in terms of memory or compute. You can have true parallelism at home, but you need an absurd amount of resources. It's not a simple threading problem.
redox99 60 minutes ago [-]
I have no problem running two or three sequences of qwen 27B with a 3090. It's basically the recommended way, LLM inference without batching is super inefficient.
zozbot234 1 hours ago [-]
The typical bottleneck to wider batching on consumer hardware is memory capacity for the KV-cache, not compute (even unified memory/iGPU-based platforms have enough compute to allow for some batching, and SSD offloading changes the scenario entirely). Qwen models tend to have bulky KV-caches for any given token count. But agentic swarms might end up sharing a large cache prefix, so there's scope for potential gains there.
acuozzo 5 hours ago [-]
Can this help tiny models like Qwen3.5-0.8B?
skyde 3 hours ago [-]
Would converting those quant to MLX preserve the accuracy/size ?
Or this only work with GGUF?
spwa4 6 hours ago [-]
No MLX versions for 3.8 though.
lostmsu 6 hours ago [-]
Cool. Now run TerminalHard and compare to unquantized 27B.
KLD of 1%, or similar error metric that multiplies, on 10000 tokens would give accumulated error of 2,000,000%
sosodev 5 hours ago [-]
That's not how that works. Selecting a different token is not inherently erroneous. A correct solution can still be found despite divergence.
lostmsu 4 hours ago [-]
KLD isn't how that works either. The truth is in the middle and they aren't showing it.
maxbond 5 hours ago [-]
I don't think you can extrapolate that measurement across multiple sequential draws like that. We presumably are comparing against a single trajectory rather than a tree of trajectories. So once we make the wrong choice and step off of the blessed path, we have no way to assign a ranking to the next token; it's error is undefined.
I've seen LLMs self correct in chains of thought ("because of foo and bar, I need to... Wait, bar is not true, so that won't work") so I have to imagine this is a massive overestimate, errors do not necessarily compound.
lostmsu 4 hours ago [-]
I would say they do compound until proven otherwise.
Having "Wait, bar is not true, so that won't work" is not necessarily a correction. In fact, the problem is: across a long text it is a correction of a single mistake, but we are talking about thousands here.
But yes, of course that was a rough estimate. But the problem is - we don't really know what we are measuring here. Maybe there's a 2,000,000x difference of intelligence between coding indexes 52 and 50. By some measure that just feels small because that's how we process it akin to audio db.
Regardless the point is KLD and whatever they came up with is not meaningful. And they did not publish comparisons on real benchmarks.
maxbond 3 hours ago [-]
Just at a sniff test level, don't you think that if the quantization resulted in anything like 2M% error in a pretty typical context length, it would be plain as day? You'd do an A/B test and one of them would look like standard generated text and one of them would veer into incoherence? If not - what on would 2M% error even mean then?
> Regardless the point is KLD and whatever they came up with is not meaningful.
I'm not saying you're wrong, I'm just saying this isn't a meaningful metric either, mostly because it is using a different type of error (divergence along a trajectory) than what was actually measured (divergence at a fixed point) and so can't be used for this purpose. It could establish an upper bound but going by your work that upper bound is so high it may as well be infinite. That's somewhat concerning but doesn't necessarily suggest it performs badly in a typical case (which is how I'd interpret an expectation of 2M% error).
verdverm 4 hours ago [-]
actually, bar is true
but wait, the models constantly go back and forth on these things in their thinking traces, so it is unclear which self correcting is actually correct
maxbond 3 hours ago [-]
It doesn't really matter to this argument, if they are self correcting at all, then we can't assume all errors will permanently injure the trajectory. It's not like dead reckoning or a similar process where there is never an opportunity to reassess. It's more like a long division problem; it is possible to correct errors using in band information, without the external reference dead reckoning would require. (The incidence of false positive self correction does matter to the question of whether the model is actually of comparable quality after the quantization, of course.)
"Qwen3.8-27B-UD-Q8_K_XL.gguf" for instance.
The one downloaded at least 4 days ago is a different thing and is NOT the "Dynamic 3.0" GGUF which I am now downloading, which I presume will have a different sha256 checksum?
The unsloth page says dynamic 3.0 is released "today", but I have an older copy of qwen3.8 27B Q8 which I downloaded, if I remember right, at least 4-5 days ago...
https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
hf download hf://unsloth/Qwen3.8-27B-GGUF \ Qwen3.8-27B-UD-Q4_K_XL.gguf
and then see them with `hf cache ls`.
Prune old versions with `hf cache prune`.
(downloaded in the last few hours after the announcement of dynamic 3.0)
Qwen3.8-27B-UD-Q8_K_XL-unsloth-dynamic3.0$ openssl dgst -sha256 *.gguf
SHA2-256(Qwen3.8-27B-UD-Q8_K_XL.gguf)= af36ecb6b5db1407953345b746c14ac93f0657dda413910b4348683a2d990377
=====separator=========
downloaded at least 4 days ago:
Qwen3.8-27B-UD-Q8_K_XL-unsloth-original$ openssl dgst -sha256 Qwen3.8-27B-UD-Q8_K_XL.gguf
SHA2-256(Qwen3.8-27B-UD-Q8_K_XL.gguf)= af36ecb6b5db1407953345b746c14ac93f0657dda413910b4348683a2d990377
So they're actually the same thing, but the announcement says released today... Please let's not confuse the end users any more than they already are.
One thing that works for me is to ask the local model to make some fake data with the same format, let Claude Code work on the fake data, and then bring the code back and run it locally on the real data.
This way the real data never leaves my machine, but I can still use a stronger model for most of the coding.
Qwaiting for that 3.8-35B-A3B
As someone in the comments said we made a separate Q4_0 MTP if that's helpful so you can use that.
But I would suggest using UD-IQ3_XXS for 10.9GB for 16GB machines or Q2_K_XL
And I had similar experiences when Qwen3.8-27B unsloth images just came out with the full Q8_K_XL, I'm using an AMD setup which has modifications to save to disk the kv, but your (assuming you are part of the unsloth team) for some reason have been giving me similar issues.
I tried https://huggingface.co/mradermacher/Qwen3.8-27B-Uncensored-G... the 8 bit, 6 and 2 bit... the 2 bit almost use the complete KV doing it's thing and didn't loop itself.
It can be something in my setup, there is a very high chance of that, but the previous 3.6 images from qwen, the 27B, the 31A3 and 122 they are all unsloth and did work on my setup without issues...
Again could be my setup... let me know if there is any data I can supply to you to debug if needed.
> We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed
Given 16GB of VRAM, what will give me the best experience in OpenCode? Currently using Qwen3.8_Q_3
I could of course download and test myself, but that would take days with my internet connection.
I know that didn’t answer your question but I was looking for a test suite and couldn’t find anything.
After reading the logs, there is far less doom looping than with 3.6, but whether that’s a one off or not is up for debate.
Q4_K_P
Interestingly, it also seems to tend toward self-correcting, which makes lower quantizations borderline usable. There'll be more faffing around, but still converging toward a solution. I wonder if that's a deliberate product of its RL.
We do plan to do larger benchmark suites though!
It takes forever, but it does actually get around to making things work, and it is more thorough and produces better code than previous qwen models. You just need to let it run quite awhile.
Currently I very much can't decide between going for a bit of a lower Q4 Quant to squeeze out a bit of buffer and ctx or wondering if a slightly higher (IQ4_XS vs Q4_K_M/XL) is worth it
If you had some use case with very small output sequences they could be interesting to try. I think dropping down to a 9B-class model would produce better results for most cases.
[0]: https://github.com/ggml-org/llama.cpp/blob/master/tools/quan...
I use this project: https://github.com/vllm-project/llm-compressor
If your motherboard/cpu doesn't actually have those (few do outside some xeons, epycs and threadrippers) you can still do it - it's called a layer split and will work even with 1 lane per GPU. Each GPU will work at its maximum speed, but only 1 will be active at any given instant - imagine a relay race.
(Didn't mention which PCIe generation - obviously the higher the better. At v4 and up, even 8 lanes per GPU would be enough for a performant tensor 4-way split)
Edit: If you have more than 1 user at a time, the GPU can actually all be working all the time, if there are enough parallel requests to serve. But you need enough KV cache for all the sessions you're running in parallel.
llama-server --host 0.0.0.0 --port 8089 -m Qwen3.8-27B-UD-Q8_u.gguf --spec-type draft-mtp,ngram-mod --spec-draft-n-max 3 --spec-draft-n-min 1
if you have an igpu and want to exclude or just use some gpus you can use
--device Vulkan3,Vulkan2,Vulkan1
in my case vulkan because of amd, you can see your devices with
llama-server2 --list-devices
Available devices: Vulkan0: AMD Radeon Graphics (RADV RAPHAEL_MENDOCINO) (33515 MiB, 29349 MiB free) Vulkan1: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 4911 MiB free) Vulkan2: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 7681 MiB free)
Using multiple cards is one of the things that the models and software that Unsloth releases does really well in terms of ease of use and relatively good performance.
More cards = more communication over PCIe. The prompts don't come in and just get magically split between each card, they move sequentially through them.
Also, with 2x24GB cards you don't really have 48GB of usable memory to load a model, closer to ~42GB + context.
And then there are power concerns, motherboard limitations (PCIe slots and lanes - a lot of motherboards with multiple 16x PCIe slots don't actually have 16x lanes to each of those slots), and more. 8x GPUs are going to easily draw 2000W on their own, if not substantially more. You'll need wiring and a circuit that can support 3000W without a risk of starting a fire in your wall.
For $5k, a single 32GB 5090 might be a better choice for a lot of people versus 4x3090s with 24GB each. It will definitely perform substantially better on smaller 27B models.
For hardware:
A good motherboard with lots of PCIe lanes (7x full 16x PCIe 4.0), DDR4 support, etc:
https://www.asus.com/us/motherboards-components/motherboards...
Add in a 3xxx series Threadripper PRO, 128 or 256GB of DDR4 (going higher becomes really expensive), and a ~1400 watt power supply. You can underpower/undervolt Nvidia cards really easily, and capping them at 250W loses you minimal performance.
KLD of 1%, or similar error metric that multiplies, on 10000 tokens would give accumulated error of 2,000,000%
I've seen LLMs self correct in chains of thought ("because of foo and bar, I need to... Wait, bar is not true, so that won't work") so I have to imagine this is a massive overestimate, errors do not necessarily compound.
Having "Wait, bar is not true, so that won't work" is not necessarily a correction. In fact, the problem is: across a long text it is a correction of a single mistake, but we are talking about thousands here.
But yes, of course that was a rough estimate. But the problem is - we don't really know what we are measuring here. Maybe there's a 2,000,000x difference of intelligence between coding indexes 52 and 50. By some measure that just feels small because that's how we process it akin to audio db.
Regardless the point is KLD and whatever they came up with is not meaningful. And they did not publish comparisons on real benchmarks.
> Regardless the point is KLD and whatever they came up with is not meaningful.
I'm not saying you're wrong, I'm just saying this isn't a meaningful metric either, mostly because it is using a different type of error (divergence along a trajectory) than what was actually measured (divergence at a fixed point) and so can't be used for this purpose. It could establish an upper bound but going by your work that upper bound is so high it may as well be infinite. That's somewhat concerning but doesn't necessarily suggest it performs badly in a typical case (which is how I'd interpret an expectation of 2M% error).
but wait, the models constantly go back and forth on these things in their thinking traces, so it is unclear which self correcting is actually correct