𓁟 host1.bitchan.one

𓀐 @4t-8BqSQ: /lmg/ - a general dedicated to the discussion and development of local language models. News >Claude BANS recomp! >https://www.youtube.com/watch?v=GlRVbIdc_Mk News Archive: https://rentry.org/lmg-news-archive Links: https://rentry.org/LocalModelsLinks Official /lmg/ card: https://files.catbox.moe/cbclyf.png Getting Started https://rentry.org/lmg-lazy-getting-started-guide https://rentry.org/lmg-build-guides https://rentry.org/IsolatedLinuxWebService https://rentry.org/recommended-models https://rentry.org/samplers https://rentry.org/MikupadIntroGuide Further Learning https://rentry.org/machine-learning-roadmap https://rentry.org/llm-training https://rentry.org/LocalModelsPapers Benchmarks LiveBench: https://livebench.ai Programming: https://swe-rebench.com Agentic Coding: https://deepswe.datacurve.ai Context Length: https://github.com/RecapAnon/NoLiMa GPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference Tools GGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator Sampler Visualizer: https://artefact2.github.io/llm-sampling Token Speed Visualizer: https://shir-man.com/tokens-per-second Text Gen. UI, Inference Engines https://github.com/lmg-anon/mikupad https://github.com/oobabooga/text-generation-webui https://github.com/LostRuins/koboldcpp https://github.com/ggerganov/llama.cpp https://github.com/theroyallab/tabbyAPI https://github.com/vllm-project/vllm https://rentry.org/custom-uis

I didn't set any options for kv quant. I assume it's f16. I can fit iq3_s, ~150k context in my 22GB vram. I didn't load MTP because that doesn't do anything useful and I put mmproj on the CPU since I don't use images that much

⧖expiring ~22d0/1 kept1w ago

𓀐 @HzFkFVmn

⏱ 1w 👍 0

⬆ upvote 0