Skip to content

Commit 7e62175

Browse files
chyundunovDatamonsterschyundunovDatamonsters
authored
Adding files to deploy CodeTrans application on AMD GPU (#1138)
Signed-off-by: Chingis Yundunov <[email protected]>
1 parent 152adf8 commit 7e62175

File tree

4 files changed

+447
-0
lines changed

4 files changed

+447
-0
lines changed

‎CodeTrans/docker_compose/amd/gpu/rocm/README.md‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
# Build and deploy CodeTrans Application on AMD GPU (ROCm)
2+
3+
## Build images
4+
5+
### Build the LLM Docker Image
6+
7+
```bash
8+
### Cloning repo
9+
git clone https://github.com/opea-project/GenAIComps.git
10+
cd GenAIComps
11+
12+
### Build Docker image
13+
docker build -t opea/llm-tgi:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/llms/text-generation/tgi/Dockerfile .
14+
```
15+
16+
### Build the MegaService Docker Image
17+
18+
```bash
19+
### Cloning repo
20+
git clone https://github.com/opea-project/GenAIExamples
21+
cd GenAIExamples/CodeTrans
22+
23+
### Build Docker image
24+
docker build -t opea/codetrans:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f Dockerfile .
25+
```
26+
27+
### Build the UI Docker Image
28+
29+
```bash
30+
cd GenAIExamples/CodeTrans/ui
31+
### Build UI Docker image
32+
docker build -t opea/codetrans-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile .
33+
```
34+
35+
## Deploy CodeTrans Application
36+
37+
### Features of Docker compose for AMD GPUs
38+
39+
1. Added forwarding of GPU devices to the container TGI service with instructions:
40+
41+
```yaml
42+
shm_size: 1g
43+
devices:
44+
- /dev/kfd:/dev/kfd
45+
- /dev/dri/:/dev/dri/
46+
cap_add:
47+
- SYS_PTRACE
48+
group_add:
49+
- video
50+
security_opt:
51+
- seccomp:unconfined
52+
```
53+
54+
In this case, all GPUs are thrown. To reset a specific GPU, you need to use specific device names cardN and renderN.
55+
56+
For example:
57+
58+
```yaml
59+
shm_size: 1g
60+
devices:
61+
- /dev/kfd:/dev/kfd
62+
- /dev/dri/card0:/dev/dri/card0
63+
- /dev/dri/render128:/dev/dri/render128
64+
cap_add:
65+
- SYS_PTRACE
66+
group_add:
67+
- video
68+
security_opt:
69+
- seccomp:unconfined
70+
```
71+
72+
To find out which GPU device IDs cardN and renderN correspond to the same GPU, use the GPU driver utility
73+
74+
### Go to the directory with the Docker compose file
75+
76+
```bash
77+
cd GenAIExamples/CodeTrans/docker_compose/amd/gpu/rocm
78+
```
79+
80+
### Set environments
81+
82+
In the file "GenAIExamples/CodeTrans/docker_compose/amd/gpu/rocm/set_env.sh " it is necessary to set the required values. Parameter assignments are specified in the comments for each variable setting command
83+
84+
```bash
85+
chmod +x set_env.sh
86+
. set_env.sh
87+
```
88+
89+
### Run services
90+
91+
```
92+
docker compose up -d
93+
```
94+
95+
# Validate the MicroServices and MegaService
96+
97+
## Validate TGI service
98+
99+
```bash
100+
curl http://${HOST_IP}:${CODETRANS_TGI_SERVICE_PORT}/generate \
101+
-X POST \
102+
-d '{"inputs":" ### System: Please translate the following Golang codes into Python codes. ### Original codes: '\'''\'''\''Golang \npackage main\n\nimport \"fmt\"\nfunc main() {\n fmt.Println(\"Hello, World!\");\n '\'''\'''\'' ### Translated codes:","parameters":{"max_new_tokens":17, "do_sample": true}}' \
103+
-H 'Content-Type: application/json'
104+
```
105+
106+
## Validate LLM service
107+
108+
```bash
109+
curl http://${HOST_IP}:${CODETRANS_LLM_SERVICE_PORT}/v1/chat/completions \
110+
-X POST \
111+
-d '{"query":" ### System: Please translate the following Golang codes into Python codes. ### Original codes: '\'''\'''\''Golang \npackage main\n\nimport \"fmt\"\nfunc main() {\n fmt.Println(\"Hello, World!\");\n '\'''\'''\'' ### Translated codes:"}' \
112+
-H 'Content-Type: application/json'
113+
```
114+
115+
## Validate MegaService
116+
117+
```bash
118+
curl http://${HOST_IP}:${CODEGEN_BACKEND_SERVICE_PORT}/v1/codetrans \
119+
-H "Content-Type: application/json" \
120+
-d '{"language_from": "Golang","language_to": "Python","source_code": "package main\n\nimport \"fmt\"\nfunc main() {\n fmt.Println(\"Hello, World!\");\n}"}'
121+
```

‎CodeTrans/docker_compose/amd/gpu/rocm/compose.yaml‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
services:
5+
codetrans-tgi-service:
6+
image: ghcr.io/huggingface/text-generation-inference:2.3.1-rocm
7+
container_name: codetrans-tgi-service
8+
ports:
9+
- "${CODETRANS_TGI_SERVICE_PORT:-8008}:80"
10+
volumes:
11+
- "/var/lib/GenAI/codetrans/data:/data"
12+
shm_size: 1g
13+
environment:
14+
no_proxy: ${no_proxy}
15+
http_proxy: ${http_proxy}
16+
https_proxy: ${https_proxy}
17+
TGI_LLM_ENDPOINT: ${CODETRANS_TGI_LLM_ENDPOINT}
18+
HUGGING_FACE_HUB_TOKEN: ${CODEGEN_HUGGINGFACEHUB_API_TOKEN}
19+
HUGGINGFACEHUB_API_TOKEN: ${CODEGEN_HUGGINGFACEHUB_API_TOKEN}
20+
devices:
21+
- /dev/kfd:/dev/kfd
22+
- /dev/dri/:/dev/dri/
23+
cap_add:
24+
- SYS_PTRACE
25+
group_add:
26+
- video
27+
security_opt:
28+
- seccomp:unconfined
29+
ipc: host
30+
command: --model-id ${CODETRANS_LLM_MODEL_ID}
31+
codetrans-llm-server:
32+
image: ${REGISTRY:-opea}/llm-tgi:${TAG:-latest}
33+
container_name: codetrans-llm-server
34+
ports:
35+
- "${CODETRANS_LLM_SERVICE_PORT:-9000}:9000"
36+
ipc: host
37+
environment:
38+
no_proxy: ${no_proxy}
39+
http_proxy: ${http_proxy}
40+
https_proxy: ${https_proxy}
41+
TGI_LLM_ENDPOINT: "http://codetrans-tgi-service"
42+
HUGGINGFACEHUB_API_TOKEN: ${CODETRANS_HUGGINGFACEHUB_API_TOKEN}
43+
restart: unless-stopped
44+
codetrans-backend-server:
45+
image: ${REGISTRY:-opea}/codetrans:${TAG:-latest}
46+
container_name: codetrans-backend-server
47+
depends_on:
48+
- codetrans-tgi-service
49+
- codetrans-llm-server
50+
ports:
51+
- "${CODETRANS_BACKEND_SERVICE_PORT:-7777}:7777"
52+
environment:
53+
no_proxy: ${no_proxy}
54+
https_proxy: ${https_proxy}
55+
http_proxy: ${http_proxy}
56+
MEGA_SERVICE_HOST_IP: ${HOST_IP}
57+
LLM_SERVICE_HOST_IP: "codetrans-llm-server"
58+
ipc: host
59+
restart: always
60+
codetrans-ui-server:
61+
image: ${REGISTRY:-opea}/codetrans-ui:${TAG:-latest}
62+
container_name: codetrans-ui-server
63+
depends_on:
64+
- codetrans-backend-server
65+
ports:
66+
- "${CODETRANS_FRONTEND_SERVICE_PORT:-5173}:5173"
67+
environment:
68+
no_proxy: ${no_proxy}
69+
https_proxy: ${https_proxy}
70+
http_proxy: ${http_proxy}
71+
BASE_URL: ${CODETRANS_BACKEND_SERVICE_URL}
72+
BASIC_URL: ${CODETRANS_BACKEND_SERVICE_URL}
73+
ipc: host
74+
restart: always
75+
codetrans-nginx-server:
76+
image: ${REGISTRY:-opea}/nginx:${TAG:-latest}
77+
container_name: codetrans-nginx-server
78+
depends_on:
79+
- codetrans-backend-server
80+
- codetrans-ui-server
81+
ports:
82+
- "${CODETRANS_NGINX_PORT:-80}:80"
83+
environment:
84+
- no_proxy=${no_proxy}
85+
- https_proxy=${https_proxy}
86+
- http_proxy=${http_proxy}
87+
- FRONTEND_SERVICE_IP=${CODETRANS_FRONTEND_SERVICE_IP}
88+
- FRONTEND_SERVICE_PORT=${CODETRANS_FRONTEND_SERVICE_PORT}
89+
- BACKEND_SERVICE_NAME=${CODETRANS_BACKEND_SERVICE_NAME}
90+
- BACKEND_SERVICE_IP=${CODETRANS_BACKEND_SERVICE_IP}
91+
- BACKEND_SERVICE_PORT=${CODETRANS_BACKEND_SERVICE_PORT}
92+
ipc: host
93+
restart: always
94+
95+
networks:
96+
default:
97+
driver: bridge

‎CodeTrans/docker_compose/amd/gpu/rocm/set_env.sh‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#!/usr/bin/env bash
2+
3+
# Copyright (C) 2024 Intel Corporation
4+
# SPDX-License-Identifier: Apache-2.0
5+
6+
### The IP address or domain name of the server on which the application is running
7+
export HOST_IP=direct-supercomputer1.powerml.co
8+
9+
### Model ID
10+
export CODETRANS_LLM_MODEL_ID="Qwen/Qwen2.5-Coder-7B-Instruct"
11+
12+
### The port of the TGI service. On this port, the TGI service will accept connections
13+
export CODETRANS_TGI_SERVICE_PORT=18156
14+
15+
### The endpoint of the TGI service to which requests to this service will be sent (formed from previously set variables)
16+
export CODETRANS_TGI_LLM_ENDPOINT="http://${HOST_IP}:${CODETRANS_TGI_SERVICE_PORT}"
17+
18+
### A token for accessing repositories with models
19+
export CODETRANS_HUGGINGFACEHUB_API_TOKEN=''
20+
21+
### The port of the LLM service. On this port, the LLM service will accept connections
22+
export CODETRANS_LLM_SERVICE_PORT=18157
23+
24+
### The IP address or domain name of the server for CodeTrans MegaService
25+
export CODETRANS_MEGA_SERVICE_HOST_IP=${HOST_IP}
26+
27+
### The endpoint of the LLM service to which requests to this service will be sent
28+
export CODETRANS_LLM_SERVICE_HOST_IP=${HOST_IP}
29+
30+
### The ip address of the host on which the container with the frontend service is running
31+
export CODETRANS_FRONTEND_SERVICE_IP=192.165.1.21
32+
33+
### The port of the frontend service
34+
export CODETRANS_FRONTEND_SERVICE_PORT=18155
35+
36+
### Name of GenAI service for route requests to application
37+
export CODETRANS_BACKEND_SERVICE_NAME=codetrans
38+
39+
### The ip address of the host on which the container with the backend service is running
40+
export CODETRANS_BACKEND_SERVICE_IP=192.165.1.21
41+
42+
### The port of the backend service
43+
export CODETRANS_BACKEND_SERVICE_PORT=18154
44+
45+
### The port of the Nginx reverse proxy for application
46+
export CODETRANS_NGINX_PORT=18153
47+
48+
### Endpoint of the backend service
49+
export CODETRANS_BACKEND_SERVICE_URL="http://${HOST_IP}:${CODETRANS_BACKEND_SERVICE_PORT}/v1/codetrans"

0 commit comments

Comments
 (0)