Running llama-cpp-python in Ubuntu 24.04 with AMD Radeon GPU #

The llama-cpp-python package provides Python bindings for llama.cpp that gives low-level access to C API and high-level APIs (OpenAI-like API) in Python scripts. It also implements OpenAI-compatible API web server [1].

The installation instructions are included in their README.md.

Before installing llama.cpp, we need to make sure the relevant GPU drivers are installed (for AMD GPU see [2], for NVidia see [3]).

Refs #

• [1] https://github.com/abetlen/llama-cpp-python
• [2] https://rocm.docs.amd.com/projects/radeon-ryzen/en/latest/
• [3] https://www.nvidia.com/en-in/drivers/