Yeah, the connector spec is definitely bad given the incidents we’ve seen. I was just pointing out it’s not actually the mating cycle limit that’s the issue here (especially given older connectors like the 8 pin evidently don’t have this problem, despite equivalent mating cycle ratings), but rather the rest of the spec.

FWIW, lots of power connectors are indeed only rated for a low number of mating cycles (only 30), not just 12VHPWR. This includes the old PCIe 8 pin (same 30 limit). Realistically you will probably not reach the rated limit.

The actual problem, I think, is that the connector could operate without being fully plugged in, especially without current balancing on the side of the card. This is an actual fire hazard, as you now get at least some connections with very high resistance, forcing extreme current on the remaining wires with low resistance connections. This is combined with the fact that the safety margin on 12VHPWR is a lot lower than that of the PCIe 8 pin (which I believe can safely carry a lot more than its 150W rating).

It still won’t help for a couple of reasons. For one, VRAM is fairly abundant on commercial deployments. Even for big models, a company is probably deploying on 1-2 nodes of 8x H200 or newer (hence several TB of VRAM).

But more importantly, commercial inference relies on heavy concurrency. So even if some experts are uncommon, with a lot of concurrent users, they will still fire frequently enough for the performance difference to be felt. And in my own experience, expert use isn’t uniform but it isn’t particularly biased either. This is especially tough since high-concurrency inference can actually be fairly compute bound, but this expert caching system either starves the system of bandwidth (if you require compute on the GPU, then you’re stuck with PCIe speeds which are tiny compared to HBM and even DRAM), or you’re starved of compute (if you do compute on the CPU).

It’s nice for local inference, but yeah, not representative of commercial inference.

Keep in mind many of the optimizations you’re talking about (LRU caching of experts, for example) are only really relevant at the single user local inference scale. As in, an individual wants to run a big model on their machine, but they don’t have enough VRAM to fit the model and KV cache. Accordingly, you’re basically only describing hobbyist and research projects, which aren’t really representative of the AI inference industry as a whole.

Commercial inference keeps everything resident in VRAM, so expert caching isn’t necessary. So these things won’t help costs. A lot of other low hanging fruit (like hierarchical KV cache) has also existed for a long time for production-ready inference engines.