Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

 Originally published in together.ai, Sept 11, 2023. Large Language Models (LLMs) have changed the world. However, generating text with them can be slow and expensive. While methods like speculative decoding have been proposed to accelerate the generation speed, their intricate nature has left many in the open-source community hesitant to embrace them. That’s why we’re