Efficiently serve dozens of fine-tuned models with vLLM on Amazon SageMaker AI and Amazon Bedrock

In this post, we explain how we implemented multi-LoRA inference for Mixture of Experts (MoE) models in vLLM, describe the kernel-level optimizations we performed, and show you how you can benefit from this work. We use GPT-OSS 20B as our primary example throughout this post.

延伸阅读

1. Anthropic无法可靠控制AI智能体，切断内部评估的实时互联网访问
2. Prime Agent 用 Rust 重写：2000 个 AI 智能体协作完成自我重写
3. 乌克兰无人机袭击俄罗斯AI数据中心，Yandex超级计算机受损