Feature Request: Allow EPP to dynamically discover ports based on matching Pods

[delavet]

Background

Currently, EPP uses pods with fixed ports as the basic unit for request scheduling. However, this approach does not adapt well to scenarios where multiple model servers may start dynamicly on different ports. It is hoped that the capability of EPP can be generalized so that EPP can dynamically discover the inference server port from the matched pod, instead of discovering it from the definition of InferencePool.

One potential use case for this scenario is as follows:

https://github.com/llm-d-incubation/llm-d-fast-model-actuation) is a project aiming to speed up scale out, at least in simple model server deployment patterns. The key technologies are using vLLM sleep/wake to maintain some low-GPU-resource idle instances, and a launcher process to spring-load startup of child processes. The launcher will have only one awake child at a time, but also some sleeping ones; switching which is awake is fast. Thus, the launcher pod serves different models at different times. There is a controller that manages this, including adjusting the launcher pod's labels to match the right InferencePool. Remember that even a vLLM instance in sleep mode is handling HTTP requests on its inference port. So the children of one launcher have to have different inference ports. That means that the InferencePools have to have different inference ports. I do not like that, it adds constraints between InferencePool objects. So it would be better if the inference port number(s) could come from a label or annotation on the matching pods.

Proposal

Currently, in the InferencePool API, multiple ports of the backend Model Server are identified through a required targetPorts field.

https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/api/v1/inferencepool_types.go#L81C2-L81C13

However, with this approach, it is not possible to identify inference servers dynamically started on different ports within a pod.

We suggest making the targetPorts field optional and proposing a new annotation-based method in the EPP Model Server Protocol to help EPP dynamically discover model servers running on multiple dynamic ports, thereby serving the aforementioned use cases. Specifically, an example of the annotations would look like this:

annotations:
 inference.networking.x-k8s.io/port-discovery: [{"inferencePool":"qwen-pool","number":8007}]

This can be parsed into the following data structure:

type InferencePort struct {
   InferencePool string            `json:"inferencePool"`
   Number        int32             `json:"number"`
   Attributes    map[string]string `json:"attributes,omitempty"`
}

type InferencePorts []InferencePort

• Each InferencePort declares a model server instance running on that port.
• The inferencePool field specifies which inference pool the instance belongs to (since services running on different ports may belong to different inference pools).
• For any additional requirements, an optional Attributes field allows users to attach metadata to the model server running on that port. This is particularly useful when the inference servers, launched on different ports, are responsible for distinct roles, such as prefill and decode.

At the implementation level, EPP can listen to pods and, based on predefined annotations, transform a pod into one or more "virtual pods", which are then recorded in the local datastore. This approach ensures full compatibility with other EPP scheduling logic based on pods. This implementation has already been proposed in #1663 to support DP. We only need to modify the creation logic of the "virtual pods" so that EPP can create "virtual pods" according to the annotations.

@kfswain
cc @MikeSpreitzer @osswangxining @shmuelk

[shmuelk]

We may get significant push back from the Gateway's to add this very application specific logic into the code that builds the "cluster" for Envoy.

If the EPP were a standalone proxy that did it's own routing, this would be in the direction of a great solution.

I don't see a need to add to the annotations the name of the InferencePool. The InferencePool can be left with the Pod Selector which will select the pods that are part of the pool. The annotations would be used to discover the ports being used. I don't see a real need for different ports on the same pod to be part of different InferencePools. In fact it would be much better if we supported multiple base models in the same InferencePool, thus eliminating many hoops for running a large cluster with many base models on different servers.

[MikeSpreitzer]

I think that the minimal need for the use case here is an annotation carrying an array of port numbers --- simply a pod-local alternative to

gateway-api-inference-extension/api/v1/inferencepool_types.go

Line 81 in a1a8cc8

TargetPorts []Port `json:"targetPorts,omitempty"`

. One of these Pods has only one awake instance of vLLM (or perhaps, in a data-parallel case, one array of awake instances) at a time and matches only one InferencePool at a time, so it is unnecessary for the annotation to name the InferencePool.

[delavet]

Thank you for the feedback. After synthesizing everyone's input, I have reorganized the solution to address the scenarios raised in the issue:

Make the targetPorts field in the InferencePool optional and provide an alternative array in the pod annotation to discover the ports on which the inference engine serves within the pod. For example:

annotations:
inference.networking.x-k8s.io/port-discovery: '8007'

Pros:

• Able to resolve the scenario described in the issue with minimal changes.

Cons:

• May pose a challenge for the gateway implementation because it would be difficult for the gateway to construct the upstream Envoy cluster.