Inside a GPU server: parts and their jobs

The rack and the server are different

A rack is the cabinet that holds equipment. A server is a computer mounted within it, alongside switches and power-distribution equipment. One rack can hold multiple servers. Their arrangement depends on power capacity, cooling, service access and workload. The explorer separates the facility, rack and server views so small parts can be inspected without pretending they are visible from outside the building.

GPUs perform parallel numerical work

AI models involve repeated operations on arrays of numbers. GPUs provide many parallel execution resources for this work. Performance also depends on how quickly data reaches those resources: a powerful accelerator can spend time waiting for memory or communication. The number of GPU packages shown in the model is representative and should not be read as a standard server specification.

The CPU still has an important role

The CPU runs the operating system and coordinates application tasks, networking, input preparation and accelerator work. It can perform computation that does not suit the GPU, and it helps manage the server as a whole. Calling a system a GPU server does not mean its CPU has disappeared or that all instructions execute on an accelerator.

System RAM and GPU memory serve different needs

System RAM supports the CPU and host software. GPU memory holds data close to the accelerator, including model parameters and working state. These are distinct memory resources, although some architectures provide more integrated access than others. Memory capacity affects what can fit; memory bandwidth affects how quickly data can be supplied. More capacity alone does not guarantee faster generation.

Storage keeps data beyond a running job

SSDs or other storage hold operating-system files, model files, datasets and checkpoints. Storage retains data without the working-memory role of RAM. A service may load model parameters into accelerator memory and keep them there to answer many requests. It does not necessarily read a complete model from disk again for every prompt.

Links connect devices and servers

Inside a server, PCIe and, where provided, dedicated accelerator links connect processors and GPUs. Network adapters connect the server to switches and other servers. A model may be divided across devices, requiring intermediate results to move between them. Network layout, bandwidth and software coordination therefore matter alongside raw compute capability. Not every workload needs multiple servers.

Cooling and power make computation possible

Power supplies convert the incoming electrical supply to the voltages the server needs. Fans move air over components, while cold plates can carry heat away from high-power chips using circulating coolant. A liquid-cooled server may still need fans for other parts. Open the representative server in the explorer and use the exploded view to connect each component’s position with its role.

Connect the steps

Loading a model is different from using it

Think of opening a large project before editing it. Stored files must become accessible working data before computation can use them. In a server, storage, system RAM and GPU memory have different jobs. A model that exceeds one GPU’s available memory may require partitioning or another deployment strategy; faster arithmetic alone does not solve a capacity limit.