Introduction:
I use my old desktop PC for experiments like virtualization and such. It’s kind of a sandbox pit where I can just mess around with whatever comes to mind; it accepts whatever workloads you throw at it and runs them, for better or worse. Thanks to a friend handing over a CPU they had lying around in a corner, this machine jumped from 8 cores / 16 threads to 16 cores / 32 threads. It’s got 64GB of DDR4 RAM, a few NVMes plugged in, various Bluetooth adapters and what-not... Plus, on top of all that, there’s an Nvidia 1080 Ti and an Intel A770 in there—a total Frankenstein build. Because of the heavy PCIe lane population, we can’t really say these GPUs run super efficiently, but they work well enough to do what we want.
I’ve thrown a bunch of VMs onto this machine, 3 of which are used for OpenStack—1 controller and 2 compute nodes. On the storage side, there’s an NFS VM, which I hooked up to OpenStack one way or another. When it comes to passing through those GPUs to OpenStack, everything is deeply nested. So the A770 gets passed to a VM, and from there, OpenStack passes it down to its own spawned VM... A bit of an IOMMU fight on the host server, but it works when you manually edit the virsh XML. Or, let's say, it works once you beat it into submission enough. So yeah, that’s the story.
As you know, the 1080 Ti comes with 11GB of VRAM and the A770 comes with 16GB of VRAM (I think 8GB versions exist too). While one side runs CUDA, the other can run oneAPI, SYCL, etc. Technically, there’s 27GB of VRAM in total, but I wanted to see if it’s actually possible to use them as a "total" pool—not for performance, just for the sake of tinkering.
First, let me give you the PCIe link speeds for the GPUs, just to set expectations low.
root@virtbase:~# lspci | grep -i vga
0c:00.0 VGA compatible controller: Intel Corporation DG2 [Arc A770] (rev 08)
0e:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)
root@virtbase:~# lspci -vv -s 0c:00.0 | grep -E "LnkCap:|LnkSta:"
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
LnkSta: Speed 2.5GT/s, Width x1
root@virtbase:~# lspci -vv -s 0e:00.0 | grep -E "LnkCap:|LnkSta:"
LnkCap: Port #2, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
LnkSta: Speed 2.5GT/s (downgraded), Width x4 (downgraded)My goal is to spin up an RPC server with llama.cpp and have it distribute a model across the 1080 Ti and A770 (See: Tensor parallelism, Pipeline parallelism). For this, alongside the main server, we’ll spin up an RPC server running CUDA with the 1080 Ti and another RPC server running SYCL with the A770—both through llama.cpp. The main server will then handle the necessary workload distribution between them.
The idea is nice—will it work? Maybe. Buzzwords and jargon flying all over the place: LLM-Hetero, RPC-Hybrid LLM Inference... Saying "what if it actually works?", we dive right in.
Development:
My infrastructure is OpenStack Gazpacho, and on top of it, I’m running OpenStack Magnum with Vexxhost's Magnum Cluster API driver—plus, of course, Octavia and stuff are in the mix too. With this Magnum setup, I spin up Kubernetes clusters off of custom images I created myself. Kubernetes version 1.36.2.
Here's how things look in our Kubernetes cluster: 1 master, 1 1080 Ti worker, and 1 A770 worker.
erdem@EWUL13:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-ngar4-jn72v-c4c7h Ready control-plane,master 20h v1.36.2
kube-ngar4-worker-gpu-1080ti-mnvpd-hhh6v-6klk6 Ready worker 20h v1.36.2
kube-ngar4-worker-gpu-a770-mrzqm-6r6xc-h5wl9 Ready worker 20h v1.36.2On our worker nodes, I installed the necessary drivers at the host level. Since the 1080 Ti is basically an abandoned card, the Nvidia Driver version is 580, and the CUDA version is 12.9.2. On the A770, things are a bit different—the necessary driver is actually built right into the kernel (I guess 6+ onwards, i915 and Xe come baked in, though I might be mixing it up with the HWE kernel). But if you build your own node images like me and strip everything down to save on size, watch out: even though the A770 drivers are installed, you might need to manually copy over the required firmware files, then do an initramfs update and all that...
sudo wget -P /lib/firmware/i915 "https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/i915/dg2_guc_70.bin"
sudo wget -P /lib/firmware/i915 "https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/i915/dg2_dmc_ver2_08.bin"Anyway, without dragging it out, let's share the Dockerfile and the RPC servers.
Dockerfile for CUDA:
# ---- build
FROM nvidia/cuda:12.9.2-devel-ubuntu24.04 AS build
RUN apt-get update && apt-get install -y --no-install-recommends git build-essential cmake ninja-build && rm -rf /var/lib/apt/lists/*
WORKDIR /build
RUN git clone --depth 1 https://github.com/ggml-org/llama.cpp.git .
RUN cmake -B build -G Ninja -DGGML_CUDA=ON -DGGML_RPC=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXE_LINKER_FLAGS=-Wl,--allow-shlib-undefined && cmake --build build --target ggml-rpc-server -j"$(nproc)"
RUN ldd build/bin/ggml-rpc-server
# ---- runtime
FROM nvidia/cuda:12.9.2-runtime-ubuntu24.04
RUN apt-get update && apt-get install -y --no-install-recommends libgomp1 && rm -rf /var/lib/apt/lists/*
COPY --from=build /build/build/bin/ggml-rpc-server /usr/local/bin/ggml-rpc-server
COPY --from=build /build/build/bin/*.so* /usr/local/lib/
RUN ldconfig
ENTRYPOINT ["/usr/local/bin/ggml-rpc-server"]
CMD ["-H", "0.0.0.0", "-p", "50052"]Dockerfile for SYCL (I couldn't get it running on the runtime image, didn't bother troubleshooting too much, so out came a 4.4GB beast):
# ---- build
FROM intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04 AS build
RUN apt-get update && apt-get install -y --no-install-recommends git cmake ninja-build && rm -rf /var/lib/apt/lists/*
WORKDIR /build
RUN git clone --depth 1 https://github.com/ggml-org/llama.cpp.git .
#SHELL ["/bin/bash", "-c"]
RUN source /opt/intel/oneapi/setvars.sh --force && cmake -B build -G Ninja -DGGML_SYCL=ON -DGGML_RPC=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx -DCMAKE_BUILD_TYPE=Release && cmake --build build --target ggml-rpc-server -j"$(nproc)"
RUN ldd build/bin/ggml-rpc-server
# ---- runtime
FROM intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04
RUN apt-get update && apt-get install -y --no-install-recommends libgomp1 && rm -rf /var/lib/apt/lists/*
COPY --from=build /build/build/bin/ggml-rpc-server /usr/local/bin/ggml-rpc-server
COPY --from=build /build/build/bin/*.so* /usr/local/lib/
RUN ldconfig
COPY entrypoint-sycl.sh /usr/local/bin/entrypoint-sycl.sh
RUN chmod +x /usr/local/bin/entrypoint-sycl.sh
ENTRYPOINT ["/usr/local/bin/entrypoint-sycl.sh"]
CMD ["-H", "0.0.0.0", "-p", "50052"]Entrypoint for the A770 (No matter where I set ZES_ENABLE_SYSMAN, I couldn't get around the warning it threw in the logs. Couldn't get a clean, proper GPU Util reading with xpu-smi either):
export ZES_ENABLE_SYSMAN=1
source /opt/intel/oneapi/setvars.sh --force
exec /usr/local/bin/ggml-rpc-server "$@"RPC Dockerfile for the main server:
FROM ubuntu:24.04 AS build
RUN apt-get update && apt install -y --reinstall ca-certificates && update-ca-certificates && apt-get install -y --no-install-recommends git build-essential cmake ninja-build && rm -rf /var/lib/apt/lists/*
WORKDIR /build
RUN git clone --depth 1 https://github.com/ggml-org/llama.cpp.git .
RUN cmake -B build -G Ninja -DGGML_RPC=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build --target llama-server -j"$(nproc)"
# ---- runtime image ----
FROM ubuntu:24.04
RUN apt-get update && apt-get install -y --no-install-recommends libgomp1 ca-certificates curl && rm -rf /var/lib/apt/lists/*
COPY --from=build /build/build/bin/llama-server /usr/local/bin/llama-server
COPY --from=build /build/build/bin/*.so* /usr/local/lib/
RUN ldconfig
ENTRYPOINT ["/usr/local/bin/llama-server"]Once you package all this up, all that's left is spinning them up. Let me share these trimmed down a bit so the post doesn't drag on too long.
CUDA:
...
containers:
- name: rpc-server
image: your-repo-address/rpc:cuda
args: ["-H", "0.0.0.0", "-p", "50052"]
ports:
- containerPort: 50052
name: rpc
resources:
limits:
nvidia.com/gpu: 1
...
requests:
nvidia.com/gpu: 1
...
---
apiVersion: v1
kind: Service
metadata:
name: rpc-cuda
namespace: llm-hetero
labels:
app: rpc-cuda
spec:
clusterIP: None # headless — single pod, stable DNS: rpc-cuda.llm-hetero.svc.cluster.local
selector:
app: rpc-cuda
ports:
- port: 50052
targetPort: 50052
name: rpcSYCL: (Definitely check out this link for GGML_SYCL_ENABLE_OPT)
...
containers:
- name: rpc-server
image: your-repo-address/rpc:sycl
args: ["-H", "0.0.0.0", "-p", "50052"]
env:
- name: GGML_SYCL_ENABLE_OPT
value: "0"
...
ports:
- containerPort: 50052
name: rpc
resources:
limits:
gpu.intel.com/i915: 1
...
requests:
gpu.intel.com/i915: 1
...
---
apiVersion: v1
kind: Service
metadata:
name: rpc-sycl
namespace: llm-hetero
labels:
app: rpc-sycl
spec:
clusterIP: None # headless — single pod, stable DNS: rpc-sycl.llm-hetero.svc.cluster.local
selector:
app: rpc-sycl
ports:
- port: 50052
targetPort: 50052
name: rpcMAIN: (On the main server, I'm downloading the model over my local network, and it gets written to a persistent volume) Additionally, we have a load-balancer section so a load balancer gets created on the OpenStack side and we can hit the interface.
...
initContainers:
- name: fetch-model
image: curlimages/curl:8.9.1
envFrom:
- configMapRef:
name: model-source
command:
- sh
- -c
- |
if [ ! -f /models/model2.gguf ]; then
echo "Downloading $MODEL_URL"
curl -fL --retry 3 --retry-delay 5 "$MODEL_URL" -o /models/model2.gguf.tmp
mv /models/model2.gguf.tmp /models/model2.gguf
else
echo "Model already cached"
fi
volumeMounts:
- name: model-storage
mountPath: /models
containers:
- name: llama-server
image: your-repo-address/rpc:main
args:
- "-m"
- "/models/model2.gguf"
- "--host"
- "0.0.0.0"
- "--port"
- "8080"
- "-ngl"
- "99"
- "--rpc"
- "rpc-sycl.llm-hetero.svc.cluster.local:50052,rpc-cuda.llm-hetero.svc.cluster.local:50052"
- "-c"
- "131072"
ports:
- containerPort: 8080
name: http
---
apiVersion: v1
kind: Service
metadata:
name: main-server
namespace: llm-hetero
labels:
app: main-server
spec:
type: LoadBalancer
selector:
app: main-server
ports:
- port: 8080
targetPort: 8080
name: httpWith all of this and after waiting long enough, the pods settle into place.
Conclusion:
Now for the result: for the Qwen3.6-35B-A3B-Q4_K_M.gguf model, with a context size of 131072, we can offload all layers to the GPUs. Right off the bat, VRAM usage sits at 9700 MB on the 1080 Ti side and 13600 MB on the A770 side. Out of a total of 27 GB VRAM, you could say we hit around 22.75 GB just starting out.
1080 Ti Nvidia-Smi: 1080TI Nvidia-Smi:
...
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
| 0 NVIDIA GeForce GTX 1080 Ti On | 00000000:00:05.0 Off | N/A |
| 0% 38C P8 14W / 320W | 9751MiB / 11264MiB | 0% Default |
| | | N/A |
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
| 0 N/A N/A 9813 C /usr/local/bin/ggml-rpc-server 9746MiB |A770 xpu-smi:
root@kube-ngar4-worker-gpu-a770-mrzqm-6r6xc-h5wl9:~# xpu-smi dump --metrics 1,18 --interval 2
DeviceId power.draw (W) memory.used (MiB)
0 36.18 13662.97265625So, after all that effort, did we manage to get decent tokens per second? Of course not. If the GPU utilization numbers reported by the A770 are accurate, it never passed 40%, and the 1080 Ti rarely even spiked up to 12% utilization. So basically, all we managed to do was load the layers onto the GPUs; we didn't even get to reap the actual benefit of the GPUs themselves. Take a look, I'm sharing the results below.
7.49.302.746 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 131072, kv_unified = 'true'
7.49.352.225 I srv init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
7.49.352.287 I srv llama_server: model loaded
7.49.352.292 I srv llama_server: listening on http://0.0.0.0:8080
87.11.318.373 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
87.11.318.534 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
87.26.895.011 I slot print_timing: id 3 | task 0 | n_decoded = 100, tg = 8.41 t/s, tg_3s = 8.41 t/s
87.29.986.842 I slot print_timing: id 3 | task 0 | n_decoded = 124, tg = 8.28 t/s, tg_3s = 7.76 t/s
87.33.035.898 I slot print_timing: id 3 | task 0 | n_decoded = 150, tg = 8.32 t/s, tg_3s = 8.53 t/s
87.36.063.507 I slot print_timing: id 3 | task 0 | n_decoded = 174, tg = 8.26 t/s, tg_3s = 7.93 t/s
87.39.228.127 I slot print_timing: id 3 | task 0 | n_decoded = 191, tg = 7.89 t/s, tg_3s = 5.37 t/s
87.42.282.798 I slot print_timing: id 3 | task 0 | n_decoded = 207, tg = 7.59 t/s, tg_3s = 5.24 t/s
87.45.392.313 I slot print_timing: id 3 | task 0 | n_decoded = 225, tg = 7.40 t/s, tg_3s = 5.79 t/s
87.48.422.879 I slot print_timing: id 3 | task 0 | n_decoded = 246, tg = 7.36 t/s, tg_3s = 6.93 t/sWhen I tested the same model on a laptop with a mobile 4080—with only 10 layers on the GPU — I was able to see values around 18 tokens per second. Now, if you ask, "Why did you even try this then?", my goal was first to see if it would work at all, and second, to figure out whether RPC is actually useful if the only way to get a model to run in the first place is by distributing it.
It turns out this can be done via RPC ... we can hook up multiple GPU-equipped servers over TCP and get these models running one way or another. Did I reach my goal? Yes. But would we sit around waiting for a reasoning model's output at this token speed? I don't think so.
References:
Thanks:
Main Photo: Frames For Your Heart on Unsplash