8bit kv? iirc by default it rotates kv at 4bit and does not at 8bit, you have to set an environment variable for 8bit kv to be rotated and honestly just use f16 kv its kv cache is close to nothing that quantization buys you no practical memory saving

⬆ upvote 0