CUDA + CUDA Llm Inference using RPC

Introduction:

Previously, we tested combining two GPUs running CUDA and SYCL over RPC. After that experiment, I wondered what kind of results I could get if I went with CUDA + CUDA instead. For that reason, I decided to throw the 4080 Mobile GPU on my laptop into the equation. So, out with the Intel A770, in with the 4080 Mobile. On the 4080 side, instead of VM passthrough, we ran it directly via Docker. On the 1080ti side, we stripped away the Kubernetes in between and set it up so that the 1080ti is passed through to one VM, while the main RPC server runs on another VM.

Compared to our previous test, the 4080 is already under active use on the laptop—the display is relying on it. Because of this, we can't fully utilize its VRAM, so it's best to take GPU utilization on this side with a grain of salt. In total, there is about 23GB of VRAM available, but for effective usage, we can treat it as roughly 18GB. Naturally, the model we chose will also change.

Let's recap the infrastructure and take a look at our link speeds.

root@virtbase:~# lspci | grep -i vga
0e:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)

root@virtbase:~# lspci -vv -s 0e:00.0 | grep -E "LnkCap:|LnkSta:"
        LnkCap: Port #2, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
        LnkSta: Speed 2.5GT/s (downgraded), Width x4 (downgraded)

01:00.0 VGA compatible controller: NVIDIA Corporation AD104M [GeForce RTX 4080 Max-Q / Mobile] (rev a1)
root@EWUL13:~# lspci -vv -s 01:00.0 | grep -E "LnkCap:|LnkSta:"
        LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L1, Exit Latency L1 <16us
        LnkSta: Speed 2.5GT/s (downgraded), Width x8 (downgraded)

Development:

We didn't make any changes to our container image for the 1080ti; it can keep using the same container. Back in the day, since we hadn't pinned it to a specific commit inside the Dockerfile, it was building with the latest commit available. To make sure all containers wouldn't end up built with different versions of llama.cpp, we rebuilt them all simultaneously once again.

On the 4080 side, compared to the 1080ti, it's supported by a newer CUDA version, meaning the driver (610) and CUDA (13.3.1) versions are different. For that reason, we created a container for it in a new Dockerfile using CUDA version 13.3.1 and the same 13.3.1 runtime.

As for the main RPC server, it didn't undergo any changes either—we just rebuilt its container image and spun it up.

Here is a breakdown of what changed across the entire setup:

Main RPC server:

• The IP addresses it uses for RPC have changed.
• The LLM model has changed.

1080ti:

• Moved out of OpenStack Magnum and directly onto a VM on the host. Nested virtualization was reduced.

4080 Mobile:

• Runs directly inside a container on the host OS, running the LLM alongside regular OS usage (sharing the card with Xorg, Firefox, etc.).

Conclusion:

So, what happened this time? Our model choice was gpt-oss-20b-Q4_K_M.gguf by unsloth. After loading, the VRAM usage was 7208MiB on the 4080 side and 7450MiB on the 1080ti side. So in total, we loaded a ~14.3GiB model distributed across two GPUs.

There is an interesting situation worth pointing out. When we compile llama.cpp with CUDA, we understand that it creates a graph to speed up token generation. However, due to our GPU generations and the features supported/provided on the CUDA side, one GPU supports this graph feature while the other does not. Because of this, we see logs like that in the containers. To avoid dealing with this, it might make sense to disable the graph feature on both when compiling RPC. That would eliminate one more variable from the equation, but if it offers even a tiny benefit on the 4080 side, we'd be missing out on that. Oh, and on top of that, there is also an open issue regarding a Memory Leak in this area.

4080mobile:
Accepted client connection
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
ggml_backend_cuda_graph_compute: CUDA graph warmup reset
ggml_backend_cuda_graph_compute: CUDA graph warmup complete

1080ti:
Accepted client connection
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture

When we used CUDA + SYCL, we landed around 7 tokens per second, and our GPUs ran at very low utilization. Now let's take a look at how things stand for CUDA + CUDA.

llama-server  | 26.07.321.601 I slot print_timing: id  3 | task 1203 | n_decoded =   2268, tg =  50.24 t/s, tg_3s =  47.92 t/s
llama-server  | 26.10.337.029 I slot print_timing: id  3 | task 1203 | n_decoded =   2418, tg =  50.21 t/s, tg_3s =  49.74 t/s
llama-server  | 26.13.341.406 I slot print_timing: id  3 | task 1203 | n_decoded =   2562, tg =  50.07 t/s, tg_3s =  47.93 t/s
llama-server  | 26.16.356.827 I slot print_timing: id  3 | task 1203 | n_decoded =   2707, tg =  49.96 t/s, tg_3s =  48.09 t/s
llama-server  | 26.17.132.543 I slot print_timing: id  3 | task 1203 | prompt eval time =    2047.80 ms /   972 tokens (    2.11 ms per token,   474.66 tokens per second)
llama-server  | 26.17.132.549 I slot print_timing: id  3 | task 1203 |        eval time =   54956.09 ms /  2745 tokens (   20.02 ms per token,    49.95 tokens per second)
llama-server  | 26.17.132.550 I slot print_timing: id  3 | task 1203 |       total time =   57003.89 ms /  3717 tokens
llama-server  | 26.17.132.550 I slot print_timing: id  3 | task 1203 |    graphs reused =       3917

Since the underlying technology didn't change this time around, there was no need for translation overhead during the back-and-forth communication over RPC. In terms of theoretical performance, the A770's specs are actually higher than those of the 4080 Mobile specs, but naturally, the ecosystem's heavy focus on CUDA and NVIDIA's years of dominance in this field make a massive difference. In our previous attempt, we were hovering around 7 tokens per second, but here we are seeing 48–50 tokens per second. While all this is happening, GPU utilization rates are also showing a distinct difference compared to our last test.

For both GPUs, minimum utilization stays around 25%, with average usage consistently above that mark. Eyeballing it, the 4080 settles around 30%, whereas the 1080ti hits 30% and frequently goes above it—in fact, the 1080ti even sees momentary spikes up to 70%.

If we run the same model with the exact same context size (131072) solely on the laptop, configuring it so that 10 layers reside on the GPU, we see a value around 30–32 tokens per second while consuming 7800MiB of vram. Here, GPU utilization sits around 25%, and on the generation side, we get roughly 30 tokens per second. Of course, in this observation, I am completely ignoring system CPU and RAM loads—we're talking about a setup with a 13980HX and 64GB of 5200MHz RAM.

To sum things up, I still don't think this is at a point where it delivers real value to the end user. The main RPC server crashes quite frequently—in fact, I can practically pinpoint these crashes to specific words in the prompt. Yes, every single prompt I send containing that word causes the main RPC server to crash :) But setting that aside and looking purely at the jump from 30 tokens per second to 48 tokens per second:

• It's usable in environments meant for experimentation.
• It can be leveraged when we genuinely need the extra VRAM.
• It can be used if this happens to be our only viable path forward.

Outside of these scenarios, I don't think it has reached its time or place just yet. Naturally, as I say all this, it's worth noting that the network connection between them is just 1G.

You can access the previous CUDA + SYCL article down below:

• CUDA + SYCL Llm Inference using RPC

References:

• GGML llama.cpp

Thanks:

Main Photo: Michael Jasmund on Unsplash