Amazon announced that SGLang-Neuron is publicly available through an AWS Deep Learning Container. RadixArk released it in collaboration with Annapurna Labs. It connects the SGLang serving framework to AWS Trainium, the company's family of AI chips.
What it does
With the plugin, teams can serve models on Trainium and generate reinforcement learning rollouts. They launch a model through SGLang's server interface and call the endpoint with the same generation APIs as before. Under that interface the code runs on native PyTorch.
How it is built
SGLang-Neuron is an out-of-tree plugin. Once installed, it registers a neuron platform, and setting the environment variable SGLANG_PLATFORM=neuron selects it at server start. Because it sits outside the main project, Trainium support and SGLang core can be developed separately, which AWS says lets both iterate faster.
The division of work is as follows:
- SGLang core keeps the API server, the request lifecycle, RadixAttention and prefix-cache bookkeeping.
- The plugin supplies the Trainium-specific models, attention, mixture-of-experts support and the graph runner.
- TorchNeuron makes Trainium appear as a native PyTorch device. Model execution uses standard PyTorch operations plus kernels written in the Neuron Kernel Interface. Everything is compiled with torch.compile.
The plugin also supports prefill/decode disaggregation. One server computes the prompt's KV cache and hands it to a second server that generates the remaining tokens. The transfer uses NIXL over Elastic Fabric Adapter.
What the walkthrough shows
AWS demonstrates the setup on a Trainium3 Amazon EC2 instance. You start the container, launch the SGLang server with the OpenAI gpt-oss-120b model, and send a chat completion request to a local endpoint. The post points to a getting started guide and the repository documentation for supported models, features and validated configurations.
What the source does not say
The announcement gives no pricing, no performance or cost comparison with other hardware, and no list of supported models beyond the example. Whether Trainium is cheaper or faster for a given workload is unconfirmed and has to be tested.