Nope! You don’t know what you’re talking about. At all. But you can have fun running a 1.6 trillion parameter model on CPU at basically 0 tokens per second at scale, MoE or not.
You’ve proved my point that you don’t know what you’re talking about by blindly linking to the git repo. Couldn’t find any source that supports your claim? I wonder why.
Sure you can serve one request at a time to one patient user at a slow token per second rate, which makes running locally viable, but there is no RAM that has the bandwidth to run this model at scale. Even flash would be incredibly slow on CPU with multiple requests. You’d need the high bandwidth of VRAM and to run across multiple GPUs in a scalable way, it requires extremely high bandwidth interconnects between GPUs.
You can run at scale, on huawei. You can also run it on a cpu
Yeah, that is absolutely not what you argued.
Anyway, you’ve conceded that I’m correct that you cannot run it at scale on a CPU, because running on CPU is too slow and inefficient, and that they instead use GPU hardware like Huawei GPUs to run the model at scale. That’s good enough for me!
Not at scale. Even on the new architecture, one really needs some kind of accelerator to make it economical for servers.
Bitnet-like models might change the calculus, but no major trainer had tried that yet.
Yes, you can run it at scale. Which is why it uses Huawei hardware.
You can run it on anything, scaled or not
Nope! You don’t know what you’re talking about. At all. But you can have fun running a 1.6 trillion parameter model on CPU at basically 0 tokens per second at scale, MoE or not.
https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro
You’ve proved my point that you don’t know what you’re talking about by blindly linking to the git repo. Couldn’t find any source that supports your claim? I wonder why.
Sure you can serve one request at a time to one patient user at a slow token per second rate, which makes running locally viable, but there is no RAM that has the bandwidth to run this model at scale. Even flash would be incredibly slow on CPU with multiple requests. You’d need the high bandwidth of VRAM and to run across multiple GPUs in a scalable way, it requires extremely high bandwidth interconnects between GPUs.
Thank you for proving my point. It can be run on a cpu
“It’s slow, it’s inefficient” it still runs
It’s a foundational model just like R1 was.
Shift those goalposts! We went from “at scale” to “it still runs”
Quote me in full.
You can run it at scale, on huawei. You can also run it on a cpu
Okay!
Yeah, that is absolutely not what you argued.
Anyway, you’ve conceded that I’m correct that you cannot run it at scale on a CPU, because running on CPU is too slow and inefficient, and that they instead use GPU hardware like Huawei GPUs to run the model at scale. That’s good enough for me!