+{"codeList":["mq:\n type: woodpecker\n","# Related configuration of woodpecker, used to manage Milvus logs of recent mutation operations, output streaming log, and provide embedded log sequential read and write.\nwoodpecker:\n meta:\n type: etcd # The Type of the metadata provider. currently only support etcd.\n prefix: woodpecker # The Prefix of the metadata provider. default is woodpecker.\n client:\n segmentAppend:\n queueSize: 10000 # The size of the queue for pending messages to be sent of each log.\n maxRetries: 3 # Maximum number of retries for segment append operations.\n segmentRollingPolicy:\n maxSize: 256M # Maximum size of a segment.\n maxInterval: 10m # Maximum interval between two segments, default is 10 minutes.\n maxBlocks: 1000 # Maximum number of blocks in a segment\n auditor:\n maxInterval: 10s # Maximum interval between two auditing operations, default is 10 seconds.\n logstore:\n segmentSyncPolicy:\n maxInterval: 200ms # Maximum interval between two sync operations, default is 200 milliseconds.\n maxIntervalForLocalStorage: 10ms # Maximum interval between two sync operations local storage backend, default is 10 milliseconds.\n maxBytes: 256M # Maximum size of write buffer in bytes.\n maxEntries: 10000 # Maximum entries number of write buffer.\n maxFlushRetries: 5 # Maximum size of write buffer in bytes.\n retryInterval: 1000ms # Maximum interval between two retries. default is 1000 milliseconds.\n maxFlushSize: 2M # Maximum size of a fragment in bytes to flush.\n maxFlushThreads: 32 # Maximum number of threads to flush data\n segmentCompactionPolicy:\n maxSize: 2M # The maximum size of the merged files.\n maxParallelUploads: 4 # The maximum number of parallel upload threads for compaction.\n maxParallelReads: 8 # The maximum number of parallel read threads for compaction.\n segmentReadPolicy:\n maxBatchSize: 16M # Maximum size of a batch in bytes.\n maxFetchThreads: 32 # Maximum number of threads to fetch data.\n storage:\n type: minio # The Type of the storage provider. Valid values: [minio, local]\n rootPath: /var/lib/milvus/woodpecker # The root path of the storage provider.\n","kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/main/config/samples/milvus_cluster_woodpecker.yaml\n\n","kubectl get pods\nkubectl get milvus my-release -o yaml | grep -A2 status\n","NAME READY STATUS RESTARTS AGE\nmy-release-etcd-0 1/1 Running 0 17m\nmy-release-etcd-1 1/1 Running 0 17m\nmy-release-etcd-2 1/1 Running 0 17m\nmy-release-milvus-datanode-7f8f88499d-kc66r 1/1 Running 0 16m\nmy-release-milvus-mixcoord-7cd7998d-x59kg 1/1 Running 0 16m\nmy-release-milvus-proxy-5b56cf8446-pbnjm 1/1 Running 0 16m\nmy-release-milvus-querynode-0-558d9cdd57-sgbfx 1/1 Running 0 16m\nmy-release-milvus-streamingnode-58fbfdfdd8-vtxfd 1/1 Running 0 16m\nmy-release-minio-0 1/1 Running 0 17m\nmy-release-minio-1 1/1 Running 0 17m\nmy-release-minio-2 1/1 Running 0 17m\nmy-release-minio-3 1/1 Running 0 17m\n","kubectl delete milvus my-release\n","helm install my-release zilliztech/milvus \\\n --set image.all.tag=v2.6.0 \\\n --set pulsarv3.enabled=false \\\n --set woodpecker.enabled=true \\\n --set streaming.enabled=true \\\n --set indexNode.enabled=false\n","helm install my-release zilliztech/milvus \\\n --set image.all.tag=v2.6.0 \\\n --set cluster.enabled=false \\\n --set pulsarv3.enabled=false \\\n --set standalone.messageQueue=woodpecker \\\n --set woodpecker.enabled=true \\\n --set streaming.enabled=true\n","mkdir milvus-wp && cd milvus-wp\ncurl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh -o standalone_embed.sh\n\n# Create user.yaml to enable Woodpecker with local filesystem\ncat > user.yaml <<'EOF'\nmq:\n type: woodpecker\nwoodpecker:\n storage:\n type: local\n rootPath: /var/lib/milvus/woodpecker\nEOF\n\nbash standalone_embed.sh start\n","mkdir milvus-wp-compose && cd milvus-wp-compose\nwget https://github.com/milvus-io/milvus/releases/download/v2.6.0/milvus-standalone-docker-compose.yml -O docker-compose.yml\n# By default, the Docker Compose standalone uses Woodpecker\nsudo docker compose up -d\n# If you need to change Woodpecker parameters further, write an override:\ndocker exec -it milvus-standalone bash -lc 'cat > /milvus/configs/user.yaml <<EOF\nmq:\n type: woodpecker\nwoodpecker:\n logstore:\n segmentSyncPolicy: \n maxFlushThreads: 16\n storage:\n type: minio\nEOF'\n\n# Restart the container to apply the changes\ndocker restart milvus-standalone\n","from pymilvus import MilvusClient\nimport random\n\n# 1. Set up a Milvus client\nclient = MilvusClient(\n uri=\"http://<Proxy Pod IP>:27017\",\n)\n\n# 2. Create a collection\nres = client.create_collection(\n collection_name=\"test_milvus_wp\",\n dimension=512,\n metric_type=\"IP\",\n shards_num=2,\n)\nprint(res)\n\n# 3. Insert randomly generated vectors\ncolors = [\"green\", \"blue\", \"yellow\", \"red\", \"black\", \"white\", \"purple\", \"pink\", \"orange\", \"brown\", \"grey\"]\ndata = []\n\nbatch_size = 1000\nbatch_count = 2000\nfor j in range(batch_count):\n start_time = time.time()\n print(f\"Inserting {j}th vectors {j * batch_size} startTime{start_time}\")\n for i in range(batch_size):\n current_color = random.choice(colors)\n data.append({\n \"id\": (j*batch_size + i),\n \"vector\": [ random.uniform(-1, 1) for _ in range(512) ],\n \"color\": current_color,\n \"color_tag\": f\"{current_color}_{str(random.randint(1000, 9999))}\"\n })\n res = client.insert(\n collection_name=\"test_milvus_wp\",\n data=data\n )\n data = []\n print(f\"Inserted {j}th vectors endTime:{time.time()} costTime:{time.time() - start_time}\")\n"],"headingContent":"Use Woodpecker","anchorList":[{"label":"Use Woodpecker","href":"Use-Woodpecker","type":1,"isActive":false},{"label":"Overview","href":"Overview","type":2,"isActive":false},{"label":"Quick start","href":"Quick-start","type":2,"isActive":false},{"label":"Configuration","href":"Configuration","type":2,"isActive":false},{"label":"Deployment modes","href":"Deployment-modes","type":2,"isActive":false},{"label":"Object storage compatibility for storage.type=minio","href":"Object-storage-compatibility-for-storagetypeminio","type":2,"isActive":false},{"label":"Deployment guides","href":"Deployment-guides","type":2,"isActive":false},{"label":"Enable Woodpecker for a Milvus Cluster on Kubernetes (Milvus Operator, storage=minio)","href":"Enable-Woodpecker-for-a-Milvus-Cluster-on-Kubernetes-Milvus-Operator-storageminio","type":3,"isActive":false},{"label":"Enable Woodpecker for a Milvus Cluster on Kubernetes (Helm Chart, storage=minio)","href":"Enable-Woodpecker-for-a-Milvus-Cluster-on-Kubernetes-Helm-Chart-storageminio","type":3,"isActive":false},{"label":"Enable Woodpecker for Milvus Standalone in Docker (storage=local)","href":"Enable-Woodpecker-for-Milvus-Standalone-in-Docker-storagelocal","type":3,"isActive":false},{"label":"Enable Woodpecker for Milvus Standalone with Docker Compose (storage=minio)","href":"Enable-Woodpecker-for-Milvus-Standalone-with-Docker-Compose-storageminio","type":3,"isActive":false},{"label":"Throughput tuning tips","href":"Throughput-tuning-tips","type":2,"isActive":false},{"label":"Latency","href":"Latency","type":2,"isActive":false}]}
0 commit comments