XLA Serving Engine#
A C++ inference runtime and model packaging toolkit for the OpenXLA ecosystem.
Export once from JAX or PyTorch (torchax); serve on CPU, NVIDIA GPU, or Cloud
TPU from a single bundle, with no Python in the request path.
XLA Serving Engine (XSE) takes models trained in JAX—or lowered to JAX from PyTorch via torchax—into production without rewriting them or converting them to another graph format.
On the export side, xla_saved_model packages your model’s StableHLO
computation, its Orbax weight checkpoint,
and any host-side pre- and post-processing into a self-contained directory
called an XLA SavedModel. On the serving side, a standalone C++ runtime
loads that directory, compiles it for the attached hardware via PJRT (or loads
a precompiled executable), dynamically batches incoming requests, and serves
predictions over gRPC or in-process.
XSE is designed for recommenders and rankers, vision and multimodal encoders, encoder-style language models, and short fixed-horizon generation where tail latency and hardware efficiency per request matter. It is not a continuous-batching chat engine; for open-ended autoregressive LLM serving with a paged KV cache, use vLLM (with vllm-jax or vllm-torchtpu on TPUs) or SGLang (with sglang-jax or sglang-torchtpu on TPUs).
Package a JAX function, Flax module, Keras model, or PyTorch module into a portable bundle with symbolic batch dimensions.
Run a prebuilt container on CPU, CUDA, or Cloud TPU, validate in-process from
Python with ModelRunner, or embed the C++ Interpreter.
Configure dynamic batching, host-side preprocessing, startup warmup, and ahead-of-time (AOT) compilation directly in the bundle.
Thirty-second tour#
Export a model in Python:
import xla_saved_model
xla_saved_model.save(
xla_saved_model.JaxModel(
apply_fn_bundle=xla_saved_model.ApplyFnBundle(
apply_fn=apply_fn, # (params, inputs) -> outputs
input_polymorphic_shapes={'features': 'b, _'}, # batch dim stays symbolic
jax_export_kwargs={'platforms': ['cpu', 'cuda', 'tpu']},
),
params=params,
),
export_path='/tmp/classifier',
serving_configs=[xla_saved_model.ServingConfig(input_signature=input_signature)],
)
Start the gRPC server container:
docker run \
--net=host \
--mount type=bind,src=/tmp/classifier,dst=/models/classifier,readonly \
us-docker.pkg.dev/xla-serving-engine/xla-serving/cpu-jax-only:latest \
--model_name=classifier \
--model_path=/models/classifier \
--orbax_grpc_server_platform=cpu \
--use_insecure_grpc_credentials=true
The server answers PredictionService/Predict and
PredictionService/GetModelMetadata on port 8500. The
quickstart runs this end to end with a working model, an
in-process check, and a Python gRPC client.
Where to go next#
Exploring XSE? Start with What XLA Serving Engine is for why XSE exists, which workloads it fits, and how the runtime is structured, then read Concepts for the five core terms.
Ready to try it? Follow Installation and the ten-minute Quickstart.
Building a pipeline? Read Exporting models, Serving and runtime, and Pre- and post-processing.
Going to production? Configure Batching, Warmup, and Ahead-of-time compilation, and inspect or update bundles with
xsm-cli.Evaluating performance? Read the DLRM serving benchmark comparing XSE against single- and multi-process Python JAX servers on Cloud TPU, with end-to-end reproduction instructions and full source in
examples/dlrm_benchmark/.