Skip to content

Commit 169f44b

Browse files
alexsin368pre-commit-ci[bot]
alexsin368
and
pre-commit-ci[bot]
authored
Codetrans: enable remote endpoints (#2195)
Signed-off-by: alexsin368 <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent 9417e3b commit 169f44b

File tree

3 files changed

+102
-4
lines changed

3 files changed

+102
-4
lines changed

‎CodeTrans/code_translation.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
MEGA_SERVICE_PORT = int(os.getenv("MEGA_SERVICE_PORT", 7777))
1818
LLM_SERVICE_HOST_IP = os.getenv("LLM_SERVICE_HOST_IP", "0.0.0.0")
1919
LLM_SERVICE_PORT = int(os.getenv("LLM_SERVICE_PORT", 9000))
20+
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY", None)
2021

2122

2223
class CodeTransService:
@@ -31,8 +32,10 @@ def add_remote_service(self):
3132
name="llm",
3233
host=LLM_SERVICE_HOST_IP,
3334
port=LLM_SERVICE_PORT,
35+
api_key=OPENAI_API_KEY,
3436
endpoint="/v1/chat/completions",
3537
use_remote_service=True,
38+
service_type=ServiceType.LLM,
3639
)
3740
self.megaservice.add(llm)
3841

‎CodeTrans/docker_compose/intel/cpu/xeon/README.md‎

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -137,10 +137,33 @@ Key parameters are configured via environment variables set before running `dock
137137

138138
In the context of deploying a CodeTrans pipeline on an Intel® Xeon® platform, we can pick and choose different large language model serving frameworks. The table below outlines the various configurations that are available as part of the application. These configurations can be used as templates and can be extended to different components available in [GenAIComps](https://github.com/opea-project/GenAIComps.git).
139139

140-
| File | Description |
141-
| -------------------------------------- | ----------------------------------------------------------------------------------------- |
142-
| [compose.yaml](./compose.yaml) | Default compose file using vllm as serving framework and redis as vector database |
143-
| [compose_tgi.yaml](./compose_tgi.yaml) | The LLM serving framework is TGI. All other configurations remain the same as the default |
140+
| File | Description |
141+
| -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
142+
| [compose.yaml](./compose.yaml) | Default compose file using vllm as serving framework and redis as vector database. |
143+
| [compose_tgi.yaml](./compose_tgi.yaml) | The LLM serving framework is TGI. All other configurations remain the same as the default. |
144+
| [compose_remote.yaml](./compose_remote.yaml) | The LLM used is hosted on a remote server and an endpoint is used to access this model. vLLM is the serving framework. Additional environment variables need to be set before running. See [instructions](#running-llm-models-with-remote-endpoints) below. |
145+
146+
### Running LLM models with remote endpoints
147+
148+
When models are deployed on a remote server, a base URL and an API key are required to access them. To set up a remote server and acquire the base URL and API key, refer to [Intel® AI for Enterprise Inference](https://www.intel.com/content/www/us/en/developer/topic-technology/artificial-intelligence/enterprise-inference.html) offerings.
149+
150+
Set the following environment variables.
151+
152+
- `REMOTE_ENDPOINT` is the HTTPS endpoint of the remote server with the model of choice (i.e. https://api.example.com). **Note:** If the API for the models does not use LiteLLM, the second part of the model card needs to be appended to the URL. For example, set `REMOTE_ENDPOINT` to https://api.example.com/Llama-3.3-70B-Instruct if the model card is `meta-llama/Llama-3.3-70B-Instruct`.
153+
- `API_KEY` is the access token or key to access the model(s) on the server.
154+
- `LLM_MODEL_ID` is the model card which may need to be overwritten depending on what it is set to in `set_env.sh`.
155+
156+
```bash
157+
export REMOTE_ENDPOINT=<https-endpoint-of-remote-server>
158+
export API_KEY=<your-api-key>
159+
export LLM_MODEL_ID=<model-card>
160+
```
161+
162+
After setting these environment variables, run `docker compose` with `compose_remote.yaml`:
163+
164+
```bash
165+
docker compose -f compose_remote.yaml up -d
166+
```
144167

145168
## Validate Microservices
146169

‎CodeTrans/docker_compose/intel/cpu/xeon/compose_remote.yaml‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Copyright (C) 2024 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
services:
5+
llm:
6+
image: ${REGISTRY:-opea}/llm-textgen:${TAG:-latest}
7+
container_name: codetrans-xeon-llm-server
8+
ports:
9+
- "9000:9000"
10+
ipc: host
11+
environment:
12+
no_proxy: ${no_proxy}
13+
http_proxy: ${http_proxy}
14+
https_proxy: ${https_proxy}
15+
LLM_ENDPOINT: ${REMOTE_ENDPOINT}
16+
LLM_MODEL_ID: ${LLM_MODEL_ID}
17+
OPENAI_API_KEY: ${API_KEY}
18+
LLM_COMPONENT_NAME: ${LLM_COMPONENT_NAME}
19+
HF_TOKEN: ${HF_TOKEN}
20+
restart: unless-stopped
21+
codetrans-xeon-backend-server:
22+
image: ${REGISTRY:-opea}/codetrans:${TAG:-latest}
23+
container_name: codetrans-xeon-backend-server
24+
depends_on:
25+
- llm
26+
ports:
27+
- "${BACKEND_SERVICE_PORT:-7777}:7777"
28+
environment:
29+
- no_proxy=${no_proxy}
30+
- https_proxy=${https_proxy}
31+
- http_proxy=${http_proxy}
32+
- MEGA_SERVICE_HOST_IP=${MEGA_SERVICE_HOST_IP}
33+
- LLM_SERVICE_HOST_IP=${LLM_SERVICE_HOST_IP}
34+
ipc: host
35+
restart: always
36+
codetrans-xeon-ui-server:
37+
image: ${REGISTRY:-opea}/codetrans-ui:${TAG:-latest}
38+
container_name: codetrans-xeon-ui-server
39+
depends_on:
40+
- codetrans-xeon-backend-server
41+
ports:
42+
- "${FRONTEND_SERVICE_PORT:-5173}:5173"
43+
environment:
44+
- no_proxy=${no_proxy}
45+
- https_proxy=${https_proxy}
46+
- http_proxy=${http_proxy}
47+
- BASE_URL=${BACKEND_SERVICE_ENDPOINT}
48+
ipc: host
49+
restart: always
50+
codetrans-xeon-nginx-server:
51+
image: ${REGISTRY:-opea}/nginx:${TAG:-latest}
52+
container_name: codetrans-xeon-nginx-server
53+
depends_on:
54+
- codetrans-xeon-backend-server
55+
- codetrans-xeon-ui-server
56+
ports:
57+
- "${NGINX_PORT:-80}:80"
58+
environment:
59+
- no_proxy=${no_proxy}
60+
- https_proxy=${https_proxy}
61+
- http_proxy=${http_proxy}
62+
- FRONTEND_SERVICE_IP=${FRONTEND_SERVICE_IP}
63+
- FRONTEND_SERVICE_PORT=${FRONTEND_SERVICE_PORT}
64+
- BACKEND_SERVICE_NAME=${BACKEND_SERVICE_NAME}
65+
- BACKEND_SERVICE_IP=${BACKEND_SERVICE_IP}
66+
- BACKEND_SERVICE_PORT=${BACKEND_SERVICE_PORT}
67+
ipc: host
68+
restart: always
69+
70+
networks:
71+
default:
72+
driver: bridge

0 commit comments

Comments
 (0)