𓁟 host1.bitchan.one

+⧖/lmg/ - a general dedicated to the discussion and development of local language models. News >Claude BANS recomp! >https://www.youtube.com/watch?v=GlRVbIdc_Mk News Archive: https://rentry.org/lmg-news-archive Links: https://rentry.org/LocalModelsLinks Official /lmg/ card: https://files.catbox.moe/cbclyf.png Getting Started https://rentry.org/lmg-lazy-getting-started-guide https://rentry.org/lmg-build-guides https://rentry.org/IsolatedLinuxWebService https://rentry.org/recommended-models https://rentry.org/samplers https://rentry.org/MikupadIntroGuide Further Learning https://rentry.org/machine-learning-roadmap https://rentry.org/llm-training https://rentry.org/LocalModelsPapers Benchmarks LiveBench: https://livebench.ai Programming: https://swe-rebench.com Agentic Coding: https://deepswe.datacurve.ai Context Length: https://github.com/RecapAnon/NoLiMa GPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference Tools GGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator Sampler Visualizer: https://artefact2.github.io/llm-sampling Token Speed Visualizer: https://shir-man.com/tokens-per-second Text Gen. UI, Inference Engines https://github.com/lmg-anon/mikupad https://github.com/oobabooga/text-generation-webui https://github.com/LostRuins/koboldcpp https://github.com/ggerganov/llama.cpp https://github.com/theroyallab/tabbyAPI https://github.com/vllm-project/vllm https://rentry.org/custom-uis /lmg/ - a general dedicated to the discussion and development of local language models. News >Claude BANS recomp! >https://www.youtube.com/watch?v=GlRVbIdc_Mk News Archive: https://rentry.org/lmg-news-archive Links: https://rentry.org/LocalModelsLinks Official /lmg/ card: https://files.catbox.moe/cbclyf.png Getting Started https://rentry.org/lmg-lazy-getting-started-guide https://rentry.org/lmg-build-guides https://rentry.org/IsolatedLinuxWebService https://rentry.org/recommended-models https://rentry.org/samplers https://rentry.org/MikupadIntroGuide Further Learning https://rentry.org/machine-learning-roadmap https://rentry.org/llm-training https://rentry.org/LocalModelsPapers Benchmarks LiveBench: https://livebench.ai Programming: https://swe-rebench.com Agentic Coding: https://deepswe.datacurve.ai Context Length: https://github.com/RecapAnon/NoLiMa GPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference Tools GGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator Sampler Visualizer: https://artefact2.github.io/llm-sampling Token Speed Visualizer: https://shir-man.com/tokens-per-second Text Gen. UI, Inference Engines https://github.com/lmg-anon/mikupad https://github.com/oobabooga/text-generation-webui https://github.com/LostRuins/koboldcpp https://github.com/ggerganov/llama.cpp https://github.com/theroyallab/tabbyAPI https://github.com/vllm-project/vllm https://rentry.org/custom-uis
𓀐 @RNrmh~8d: /lmg/ - a general dedicated to the discussion and development of local language models. News >(10/29) Mistral Large 4 (Le Chonk) 1T-A52B released: https://mistral.ai/news/mistral-large-4-oss >(10/27) NVidia releases Nemotron 3.5 Transcribe 30B-A4B: https://hf.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B >(10/26) XHToken releases Spark-X2.6-1.7B: https://hf.co/XHToken/Spark-X2.6-1.7B >(10/26) LiquidAI releases LFM3 1.7B and 2B: https://hf.co/collections/LiquidAI/lfm3 News Archive: https://rentry.org/lmg-news-archive Links: https://rentry.org/LocalModelsLinks Official /lmg/ card: https://files.catbox.moe/cbclyf.png Getting Started https://rentry.org/lmg-lazy-getting-started-guide https://rentry.org/lmg-build-guides https://rentry.org/IsolatedLinuxWebService https://rentry.org/recommended-models https://rentry.org/samplers https://rentry.org/MikupadIntroGuide Further Learning https://rentry.org/machine-learning-roadmap https://rentry.org/llm-training https://rentry.org/LocalModelsPapers Benchmarks LiveBench: https://livebench.ai Programming: https://swe-rebench.com Agentic Coding: https://deepswe.datacurve.ai Context Length: https://github.com/RecapAnon/NoLiMa GPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference Tools GGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator Sampler Visualizer: https://artefact2.github.io/llm-sampling Token Speed Visualizer: https://shir-man.com/tokens-per-second Text Gen. UI, Inference Engines https://github.com/lmg-anon/mikupad https://github.com/oobabooga/text-generation-webui https://github.com/LostRuins/koboldcpp https://github.com/ggerganov/llama.cpp https://github.com/theroyallab/tabbyAPI https://github.com/vllm-project/vllm https://rentry.org/custom-uis

I saw an anon mention a QAT quant of gemma4-31b that was from google themselves, yet I dont see any quants on their HF. Any anons have suggestions for cope quants of 31b? want to try fitting one entirely into 16gb VRAM

⧖expiring ~8d0/3 kept1w ago

𓀐 @f?7+G7*y

⏱ 1w 👍 0

⬆ upvote 0

[⇥] sort oldest ⏱

⬆[⏱]₿⚠

👍👎✔

𓀐↩⛓⚠

[⇥] flip, oldest first

filter [-]
𓀐 @4hct?gtn: unsloth's goofs are genuinely better in this case, the google goofs are not proper Q4_0 across the board, the embeddings are in Q6_K in theirs for e.g and various testing showed that the qat does behave slightly better if it's actually being run at q4 beside being faster and a bit smaller to boot https://huggingface.co/unsloth/gemma-4-31B-it-qat-GGUF it's not the first time google is being retarded while publishing goofs either, these were the only good goofs for gemma 3: https://huggingface.co/stduhpf/google-gemma-3-27b-it-qat-q4_0-gguf-small+

⬆ upvote 0↩ reply 0

𓀐 @AuYaG4E~: https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-gguf+

⬆ upvote 0↩ reply 0