|
| 1 | +# Deploying Ollama and Open WebUI on OKE |
| 2 | + |
| 3 | +In this tutorial, we will explain how to use a Mistral AI large language model (LLM) in a browser using the Open WebUI graphical interface. The LLM will be served using the Ollama framework and the overall infrastructure will rely on an Oracle Kubernetes Engine cluster with a NVIDIA A10 GPU based node pool. |
| 4 | + |
| 5 | +## Prerequisites |
| 6 | + |
| 7 | +To run this tutorial, you will need: |
| 8 | +* An OCI tenancy with limits set for A10 based instances |
| 9 | + |
| 10 | +## Deploying the infrastructure |
| 11 | + |
| 12 | +### Creating the OKE cluster with a CPU node pool |
| 13 | + |
| 14 | +The first step consists in creating a Kubernetes cluster. Initially, the cluster will be configured with a CPU node pool only. GPU node pool will be added afterwards. |
| 15 | + |
| 16 | +<p align="center"> |
| 17 | + <img width="500" height="405" src="https://github.com/oracle-devrel/technology-engineering/blob/ollama-openwebui-mistral/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/ollama-openwebui-mistral/assets/images/oke-quick-create.png"> |
| 18 | +</p> |
| 19 | + |
| 20 | +The easiest way is to use the Quick Create cluster assistant with the following options: |
| 21 | +* Public Endpoint, |
| 22 | +* Self-Managed nodes, |
| 23 | +* Private workers, |
| 24 | +* VM.Standard.E5.Flex compute shapes, |
| 25 | +* Oracle Linux OKE specific image. |
| 26 | + |
| 27 | +### Accessing the cluster |
| 28 | + |
| 29 | +Click Access Cluster, choose Cloud Shell Access or Local Access and follow the instructions. If you select Local Access, you must first install and configure the [OCI CLI package](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/cliconcepts.htm). We can now check that the nodes are there: |
| 30 | +``` |
| 31 | +kubectl get nodes |
| 32 | +``` |
| 33 | + |
| 34 | +### Adding a GPU node pool |
| 35 | + |
| 36 | +Once the cluster is available, we can add a GPU node pool. Go to `Node pools` on the left panel and click on `Add node pool` and use the following options: |
| 37 | +* Public endpoint, |
| 38 | +* Self-Managed Nodes, |
| 39 | +* VM.GPU.A10.1 nodes, |
| 40 | +* Oracle Linux GPU OKE image. |
| 41 | +* Specify a custom boot volume size of 250 GB and add an Initialization script (Advanced options) to apply the changes. |
| 42 | +``` |
| 43 | +#!/bin/bash |
| 44 | +curl --fail -H "Authorization: Bearer Oracle" -L0 http://169.254.169.254/opc/v2/instance/metadata/oke_init_script | base64 --decode >/var/run/oke-init.sh |
| 45 | +bash /usr/libexec/oci-growfs -y |
| 46 | +bash /var/run/oke-init.sh |
| 47 | +``` |
| 48 | +Click on Create to add the GPU instances and wait for the node pool to be `Active` and the nodes to be in the `Ready` state. Check again the nodes that are available: |
| 49 | +``` |
| 50 | +kubectl get nodes |
| 51 | +``` |
| 52 | +Check device visibility on the GPU node whose name is `xxx.xxx.xxx.xxx`: |
| 53 | +``` |
| 54 | +kubectl describe nodes xxx.xxx.xxx.xxx | grep gpu |
| 55 | +``` |
| 56 | +You will get the following output: |
| 57 | +``` |
| 58 | + nvidia.com/gpu=true |
| 59 | +Taints: nvidia.com/gpu=present:NoSchedule |
| 60 | + nvidia.com/gpu: 1 |
| 61 | + nvidia.com/gpu: 1 |
| 62 | + kube-system nvidia-gpu-device-plugin-8ktcj 50m (0%) 50m (0%) 200Mi (0%) 200Mi (0%) 4m48s |
| 63 | + nvidia.com/gpu 0 0 |
| 64 | +``` |
| 65 | + |
| 66 | +### Installing the NVIDIA GPU Operator |
| 67 | + |
| 68 | +You can access the cluster either using Cloud Shell or using a standalone instance. The NVIDIA GPU Operator enhances the GPU features visibility in Kubernetes. The easiest way to install it is to use `Helm` ([Installing Helm](https://helm.sh/docs/intro/install/)). |
| 69 | +``` |
| 70 | +helm repo add nvidia https://helm.ngc.nvidia.com/nvidia |
| 71 | +helm repo update |
| 72 | +helm install gpu-operator nvidia/gpu-operator --namespace gpu-operator --create-namespace |
| 73 | +``` |
| 74 | +Check again the device visibility on the GPU node: |
| 75 | +``` |
| 76 | +kubectl describe nodes xxx.xxx.xxx.xxx | grep gpu |
| 77 | +``` |
| 78 | +You will get the following output: |
| 79 | +``` |
| 80 | + nvidia.com/gpu=true |
| 81 | + nvidia.com/gpu-driver-upgrade-state=upgrade-done |
| 82 | + nvidia.com/gpu.compute.major=8 |
| 83 | + nvidia.com/gpu.compute.minor=6 |
| 84 | + nvidia.com/gpu.count=1 |
| 85 | + nvidia.com/gpu.deploy.container-toolkit=true |
| 86 | + nvidia.com/gpu.deploy.dcgm=true |
| 87 | + nvidia.com/gpu.deploy.dcgm-exporter=true |
| 88 | + nvidia.com/gpu.deploy.device-plugin=true |
| 89 | + nvidia.com/gpu.deploy.driver=pre-installed |
| 90 | + nvidia.com/gpu.deploy.gpu-feature-discovery=true |
| 91 | + nvidia.com/gpu.deploy.node-status-exporter=true |
| 92 | + nvidia.com/gpu.deploy.operator-validator=true |
| 93 | + nvidia.com/gpu.family=ampere |
| 94 | + nvidia.com/gpu.machine=Standard-PC-i440FX-PIIX-1996 |
| 95 | + nvidia.com/gpu.memory=23028 |
| 96 | + nvidia.com/gpu.mode=compute |
| 97 | + nvidia.com/gpu.present=true |
| 98 | + nvidia.com/gpu.product=NVIDIA-A10 |
| 99 | + nvidia.com/gpu.replicas=1 |
| 100 | + nvidia.com/gpu.sharing-strategy=none |
| 101 | + nvidia.com/vgpu.present=false |
| 102 | + nvidia.com/gpu-driver-upgrade-enabled: true |
| 103 | +Taints: nvidia.com/gpu=present:NoSchedule |
| 104 | + nvidia.com/gpu: 1 |
| 105 | + nvidia.com/gpu: 1 |
| 106 | + gpu-operator gpu-feature-discovery-9jmph 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m1s |
| 107 | + gpu-operator gpu-operator-node-feature-discovery-worker-t6b75 5m (0%) 0 (0%) 64Mi (0%) 512Mi (0%) 3m16s |
| 108 | + gpu-operator nvidia-container-toolkit-daemonset-t5tpc 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m3s |
| 109 | + gpu-operator nvidia-dcgm-exporter-2jvhz 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m2s |
| 110 | + gpu-operator nvidia-device-plugin-daemonset-zbk2b 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m2s |
| 111 | + gpu-operator nvidia-operator-validator-wpkxt 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3m3s |
| 112 | + kube-system nvidia-gpu-device-plugin-8ktcj 50m (0%) 50m (0%) 200Mi (0%) 200Mi (0%) 12m |
| 113 | + nvidia.com/gpu 0 0 |
| 114 | + Normal GPUDriverUpgrade 2m52s nvidia-gpu-operator Successfully updated node state label to upgrade-done |
| 115 | +``` |
| 116 | + |
| 117 | +## Deploying Ollama |
| 118 | + |
| 119 | +### Creating Ollama deployment |
| 120 | + |
| 121 | +[Ollama](https://ollama.com/) is an open source framework for deploying and training language models on a local machine such as a cloud instance. To deploy Ollama, simply use the `ollama-deployment.yml` manifest. |
| 122 | +``` |
| 123 | +kubectl apply -f ollama-deployment.yaml |
| 124 | +``` |
| 125 | +Check that the deployment is ready: |
| 126 | +``` |
| 127 | +kubectl get all |
| 128 | +``` |
| 129 | + |
| 130 | +### Pulling the model from pod |
| 131 | + |
| 132 | +The `ollama` image does not come with any models. Therefore, it is necessary to download it manually. Enter the container: |
| 133 | +``` |
| 134 | +kubectl exec -ti ollama-deployment-pod -- /bin/bash |
| 135 | +``` |
| 136 | +where `ollama-deployment-pod` is the name of the pod displayed by the `kubectl get pods` command. |
| 137 | +Pull the desired model(s), here Mistral 7B version 0.3, simply referred to as `mistral`: |
| 138 | +``` |
| 139 | +ollama pull mistral |
| 140 | +``` |
| 141 | +For more model options, the list of all supported models can be found in [here](https://ollama.com/search). |
| 142 | + |
| 143 | +Optionnally, the model can be tested from the container: |
| 144 | +``` |
| 145 | +ollama run mistral |
| 146 | +>>> Tell me about Mistral AI. |
| 147 | + Mistral AI is a cutting-edge company based in Paris, France, developing large language models. Founded by CTO Edouard Dumoulin and CEO Thibault Favodi in 2021, Mistral AI aims to create advanced artificial intelligence technologies that can understand, learn, and generate human-like text with a focus on French and European languages. |
| 148 | +
|
| 149 | +Mistral AI is backed by prominent European investors, including Daphni, Founders Future, and Iris Capital, among others, and has received significant financial support from the French government to further its research and development in large language models. The company's ultimate goal is to contribute to France's technological sovereignty and help shape the future of artificial |
| 150 | +intelligence on the European continent. |
| 151 | +
|
| 152 | +One of Mistral AI's most notable projects is "La Mesure," a large-scale French language model that has achieved impressive results in various natural language processing tasks, such as text generation and understanding. The company is dedicated to pushing the boundaries of what AI can do and applying its technology to real-world applications like education, entertainment, and more. |
| 153 | +
|
| 154 | +>>> /bye |
| 155 | +``` |
| 156 | +Exit the container by simply typing `exit`. |
| 157 | + |
| 158 | + |
| 159 | +### Creating an Ollama service |
| 160 | + |
| 161 | +A Service is necessary to make the model accessible from outside of the node. The Ollama (load balancer with a public IP address) service can be created using the `ollama-service.yaml` manifest: |
| 162 | +``` |
| 163 | +kubectl apply -f ollama-service.yaml |
| 164 | +``` |
| 165 | + |
| 166 | +## Deploying Open WebUI |
| 167 | + |
| 168 | +### Creating Open WebUI deployment |
| 169 | + |
| 170 | +Open WebUI is a user-friendly self-hosted AI platform that supports multiple LLM runners including Ollama. It can be deployed using the `openwebui-deployment.yaml` manifest. First set the `OLLAMA_BASE_URL` value in the manifest and apply it: |
| 171 | +``` |
| 172 | +kubectl apply -f openwebui-deployment.yaml |
| 173 | +``` |
| 174 | + |
| 175 | +### Creating Open WebUI service |
| 176 | + |
| 177 | +Like Ollama, OpenWebUI requires a Service (load balancer with a public IP address) to be reached. The Open WebUI service can be created using the `openwebui-service.yaml` manifest: |
| 178 | +``` |
| 179 | +kubectl apply -f openwebui-service.yaml |
| 180 | +``` |
| 181 | + |
| 182 | +## Testing the platform |
| 183 | + |
| 184 | +An easy way to check that everything is running is to run the following command: |
| 185 | +``` |
| 186 | +kubectl get all |
| 187 | +``` |
| 188 | +Go to `http://XXX.XXX.XXX.XXX:81` where XXX.XXX.XXX.XXX is the external IP address of the Open WebUI load balancer and click on `Get started` and create admin account (local). |
| 189 | + |
| 190 | +If no model can be found, go to `Profile > Settings > Admin Settings > Connections > Manage Ollama API Connections` and verify that the Ollama address matches the Ollama service load balancer external IP address and check the connection by clicking on the `Configure icon > Verify Connection`. |
| 191 | + |
| 192 | +You can now start chatting with the model. |
| 193 | + |
| 194 | + |
| 195 | + |
| 196 | +## Deleting the platform |
| 197 | + |
| 198 | +If you want to delete all the platform, first delete all the resources deployed in the OKE cluster: |
| 199 | +``` |
| 200 | +kubectl delete all --all |
| 201 | +``` |
| 202 | +Then, the OKE cluster can be deleted from the OCI console. |
| 203 | + |
| 204 | + |
| 205 | +## External links |
| 206 | + |
| 207 | +* [Mistral AI official website](https://mistral.ai/) |
| 208 | +* [Ollama official website](https://ollama.com/) |
| 209 | +* [Open WebUI official website](https://openwebui.com/) |
| 210 | + |
| 211 | +## License |
| 212 | + |
| 213 | +Copyright (c) 2025 Oracle and/or its affiliates. |
| 214 | + |
| 215 | +Licensed under the Universal Permissive License (UPL), Version 1.0. |
| 216 | + |
| 217 | +See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE) for more details. |
0 commit comments