π“Ÿ host1.bitchan.one

𓀐 @v9ug?~+p

⏱ 2d seenβ›“ 0πŸ‘ 0β§– 1

filter [-]

languages:onother

Why were people harping on about llama.cpp's --moe-cache-mib? It seems to be useless. The decode speed is slightly worse than if I just manually tuned --n-cpu-moe and the prefill is much worse. I did pull and rebuild the commit from about an hour ago where they changed the algorithm, doesn't seem to have gotten much better. --n-cpu-moe is still superior.⏱ 2d↩ 1
first is expert cache, second is pre-allocation. Effectiveness probably depends on the hardware and model architecture.⏱ 2d↩