Version v0.7 of the documentation is no longer actively maintained. The site that you are currently viewing is an archived snapshot. For up-to-date documentation, see the latest version.
Overview
Kubeflow supports two model serving systems that allow multi-framework model serving: KFServing and Seldon Core. Alternatively, you can use a standalone model serving system. This page gives an overview of the options, so that you can choose the framework that best supports your model serving requirements.
Multi-framework serving with KFServing or Seldon Core
KFServing and Seldon Core are both open source systems that allow multi-framework model serving. The following table compares KFServing and Seldon Core. A check mark (✓) indicates that the system (KFServing or Seldon Core) supports the feature specified in that row.
| Feature | Sub-feature | KFServing | Seldon Core |
|---|---|---|---|
| Framework | TensorFlow | ✓ sample | ✓ docs |
| XGBoost | ✓ sample | ✓ docs | |
| scikit-learn | ✓ sample | ✓ docs | |
| NVIDIA TensorRT Inference Server | ✓ sample | ✓ docs | |
| ONNX | ✓ sample | ✓ docs | |
| PyTorch | ✓ sample | ✓ docs | |
| Graph | Transformers | ✓ sample | ✓ docs |
| Combiners | Roadmap | ✓ sample | |
| Routers including MAB | Roadmap | ✓ docs | |
| Analytics | Explanations | ✓ sample | ✓ docs |
| Scaling | Knative | ✓ sample | |
| GPU AutoScaling | ✓ sample | ||
| HPA | ✓ | ✓ docs | |
| Custom | Container | ✓ sample | ✓ docs |
| Language Wrappers | ✓ Python, Java, R | ||
| Multi-Container | ✓ docs | ||
| Rollout | Canary | ✓ sample | ✓ docs |
| Shadow | ✓ | ||
| Istio | ✓ | ✓ |
Notes:
- KFServing and Seldon Core share some technical features, including explainability (using Seldon Alibi Explain) and payload logging, as well as other areas.
- A commercial product, Seldon Deploy, supports both KFServing and Seldon in production.
- KFServing is part of the Kubeflow project ecosystem. Seldon Core is an external project supported within Kubeflow.
Further information:
- KFServing:
- Seldon Core
TensorFlow Serving
For TensorFlow models you can use TensorFlow Serving for real-time prediction. However, if you plan to use multiple frameworks, you should consider KFServing or Seldon Core as described above.
NVIDIA TensorRT Inference Server
NVIDIA TensorRT Inference Server is a REST and GRPC service for deep-learning inferencing of TensorRT, TensorFlow and Caffe2 models. The server is optimized to deploy machine learning algorithms on both GPUs and CPUs at scale.
You can use NVIDIA TensorRT Inference Server as a standalone system, but you should consider KFServing as described above. KFServing includes support for NVIDIA TensorRT Inference Server.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.