FastChat
https://github.com/lm-sys/FastChat
为服务基于大模型的chat应用，
FastChat提供框架。
提供三个功能
training, serving, and evaluating
有评价功能、训练功能（fine tuning）
主要的还是 serving能力， 可以支持大模型的负载均衡
FastChat is an open platform for training, serving, and evaluating large language model based chatbots.
- FastChat powers Chatbot Arena (https://chat.lmsys.org/), serving over 10 million chat requests for 70+ LLMs.
- Chatbot Arena has collected over 500K human votes from side-by-side LLM battles to compile an online LLM Elo leaderboard.
FastChat's core features include:
- The training and evaluation code for state-of-the-art models (e.g., Vicuna, MT-Bench).
- A distributed multi-model serving system with web UI and OpenAI-compatible RESTful APIs.
架构解释
https://zhuanlan.zhihu.com/p/694856151
搭建实践
https://rudeigerc.dev/posts/llm-inference-with-fastchat/
