You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 86bd8db
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: README.md
+1-13Lines changed: 1 addition & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# Doubleword Inference Stack
2
2
3
-
A comprehensive Helm chart for deploying production grade LLMs. This stack is a lightweight, transparent framework to allow you to deploy any model on any inference engine with minimal configuration. It is designed to be flexible, allowing you to easily switch between different models and inference engines without changing your client code.
3
+
A Helm chart for deploying LLMs. This stack is a lightweight, transparent framework to allow you to deploy any model on any inference engine with minimal configuration. It is designed to be flexible, allowing you to easily switch between different models and inference engines without changing your client code.
4
4
5
5
The spirit of this project is to produce a framework that solves distributed serving of LLMs in a non-specific way to the inference engine running the weights.
6
6
@@ -20,15 +20,6 @@ The stack consists of:
20
20
-**Model Groups**: A grouping of kubernetes resources that represent a deployment of an inference engine. Each model group can only have one active model at a time, but can have custom numbers of replicas.
21
21
-**Inference Engines**: Backend services like vLLM, SGLang, TensorRT-LLM that handle the actual inference requests.
22
22
23
-
## Roadmap
24
-
25
-
We are actively developing this stack to support sophisticated LLM deployments. See our [issues](https://github.com/doublewordai/inference-stack/issues) for the latest features and improvements. Some key features we are working on include:
26
-
27
-
-**Model Group Prefix Aware Routing**: <https://github.com/doublewordai/inference-stack/issues/4>
28
-
-**More complicated model group configurations**, such as dynamo: <https://github.com/doublewordai/inference-stack/issues/5>
29
-
-**AutoScaling**: dynamic: <https://github.com/doublewordai/inference-stack/issues/6> and scale to zero: <https://github.com/doublewordai/inference-stack/issues/7>
30
-
-**Inference Engine Support**: Adding example and support for more inference engines, please [open an issue](https://github.com/doublewordai/inference-stack/issues/new) to request specific engines.
31
-
32
23
## Getting Started
33
24
34
25
### Install from OCI Registry
@@ -307,7 +298,4 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
0 commit comments