SEQUOIA: Exact Llama2-70B on an RTX4090 with half-second per-token latency
infini-ai-lab.github.ioLeveraging a large speculation budget, everyone can use RTX 4090 or other consumer (low-cost) GPU, e.g., AMD RX7900 with Sequoia to host very strong LLMs like 70B model without approximation, boosting the applications of AI generated content. In addition, we … Read more