Conanfiles

Summary

Custom Conan 2.x recipes for Milvus C++ dependencies. This repo holds conanfile.py recipes, patches, and configurations for third-party libraries that Milvus's C++ core depends on.

All recipes are organized under the recipes/ directory, with each subdirectory representing a single package (e.g. recipes/zlib/, recipes/arrow/).

Recipes can be built and uploaded to Milvus's private Conan Artifactory either by triggering the GitHub Actions workflows, or by running the local scripts directly (requires Artifactory credentials).

Prepare

Requirements:

• Python 3.8 or later
• Conan 2.25.1

Install Conan:

pip install --user conan==2.25.1 pyyaml

Initialize Conan:

conan profile detect

This detects your default build profile (compiler, arch, OS). The profile is stored at ~/.conan2/profiles/default.

Submit Your Recipe

For a step-by-step walkthrough — including recipe structure, verification, and publishing — see PUBLISH_GUIDELINE.md.

To add a new recipe (or a new version of an existing recipe), submit a pull request to the master branch:

1. Create a new folder under recipes/<package>/ with this layout:

   recipes/<package>/
   ├── config.yml                # maps versions to recipe folders
   └── all/                      # (or version-specific folder, e.g. v2024/, 3.x.x/)
       ├── conanfile.py          # Conan 2.x recipe (required_conan_version = ">=2.0")
       ├── conandata.yml         # source URLs and sha256 hashes
       └── test_package/         # small test consumer
           ├── conanfile.py
           └── CMakeLists.txt
   

2. The recipe must follow the Conan 2.x API (from conan import ConanFile, CMakeToolchain, CMakeDeps, etc.). Conan 1.x recipes (from conans import ConanFile) are not accepted.

3. Verify the recipe builds locally before submitting:

   ./scripts/build-and-push.sh <package> <version> --no-upload

4. Open a PR against master. Once merged, maintainers can trigger the build-and-push.yml workflow to publish it to Artifactory.

Publish Workflows (GitHub Actions)

Triggering these workflows requires write access (or higher) to the conanfiles repo. All the workflows authenticate to Artifactory using the JFROG_USERNAME2 and JFROG_PASSWORD2 GitHub repository secrets (configured in Settings → Secrets and variables → Actions).

sync-to-artifactory.yml

Downloads a package from ConanCenter (upstream) and re-uploads it (with all transitive deps) to JFrog Artifactory. Used when you want a vanilla upstream package mirrored into your private registry.

Inputs:

Input	Required	Description
package	Yes	Package name (e.g. zlib)
version	Yes	Package version (e.g. 1.3.1)
repository	No	Target Artifactory repository: production or testing (default: production)

Job: sync-conan-package

Step	Command	Description
Install Conan	pip install --user conan==2.25.1	Install Conan 2.25.1 and detect default profile
Sync to Artifactory	./scripts/sync-to-artifactory.sh	Delegates to scripts/sync-to-artifactory.sh with workflow inputs passed as arguments (see scripts/sync-to-artifactory.sh for details)

---

build-and-push.yml

Builds a package from a local recipe in this repo (custom/patched), then uploads it to Artifactory. Used for Milvus-specific recipes that differ from upstream.

When triggering the workflow from the GitHub Actions page (Run workflow button), you can configure the following:

• Use workflow from (branch/tag) — the built-in GitHub Actions selector for which branch or tag's workflow and recipes to use
• The input fields listed below

Inputs:

Input	Required	Description
package	Yes	Package name (e.g. folly)
version	Yes	Package version (e.g. 2024.08.12.00)
conanfile_path	Yes	Relative path to conanfile.py within the package dir (default: all/conanfile.py)
repository	No	Target Artifactory repository: production or testing (default: production)
user_channel	No	User/channel for the package (e.g. milvus/dev)
extra_options	No	Extra conan options (e.g. -o pkg:opt=val -c conf=val)

Job: build-and-push

Step	Command	Description
Install Conan and dependencies	pip install --user conan==2.25.1 pyyaml	Install Conan 2.25.1, pyyaml, and detect default profile
Build and push	./scripts/build-and-push.sh	Delegates to scripts/build-and-push.sh with workflow inputs passed as arguments (see scripts/build-and-push.sh for details)

---

try-build.yml

Builds a package from a local recipe without uploading anything to Artifactory. Used to validate recipes on different OS before publishing.

Inputs:

Input	Required	Description
package	Yes	Package name (e.g. milvus-common)
version	Yes	Package version (e.g. 1.0.0-28a2b48)
conanfile_path	Yes	Relative path to conanfile.py within the package dir (default: all/conanfile.py)
runner	Yes	GitHub-hosted runner: ubuntu-24.04-arm or macos-15
settings_mode	Yes	Conan settings mode: runner-default or linux-gcc-11 (default: runner-default)
user_channel	No	User/channel for the package (e.g. milvus/dev)
extra_options	No	Extra conan options (e.g. -s compiler.cppstd=20 -o pkg:opt=val)

Settings modes:

Mode	Description
runner-default	Uses conan profile detect from the selected runner for compiler/compiler version/libcxx, while forcing Release and default C++17. This is the recommended mode for both ubuntu-24.04-arm and macos-15; add -s compiler.cppstd=<std> in extra_options when a recipe needs a different C++ standard.
linux-gcc-11	Uses the repo's existing Linux production settings: GCC 11, libstdc++11, C++17, Release. Use this when you need compatibility with existing Linux package IDs.

Job: try-build

Step	Command	Description
Install Conan and dependencies	python -m pip install conan==2.25.1 pyyaml	Install Conan 2.25.1, pyyaml, and detect the runner profile
Try build	./scripts/build-and-push.sh --repository testing --no-upload	Builds the recipe through the standard script but skips Artifactory upload

Example inputs for milvus-common:

Input	Value
package	milvus-common
version	1.0.0-28a2b48
conanfile_path	all/conanfile.py
runner	ubuntu-24.04-arm
settings_mode	runner-default
user_channel	milvus/dev
extra_options	-s compiler.cppstd=20

Scripts for Manual Publish

scripts/build-milvus-deps.sh

Builds all 56 Milvus C++ dependencies from this repo's recipes in the correct dependency order, using conan create with Milvus-specific settings and options. Packages are built in 8 phases, from foundational libraries (zlib, lz4) up to complex ones (arrow, folly). This was run once during the initial population of the default-conan-local2 Artifactory.

See MILVUS_DEPENS.md for the full Milvus dependency graph (resolved from milvus/internal/core/conanfile.py), including version pins, custom @milvus/dev channel packages, and build order.

Usage:

./scripts/build-milvus-deps.sh                # build only
./scripts/build-milvus-deps.sh --upload       # build and upload recipes to Artifactory

Build settings:

• Compiler: gcc 11, libstdc++11, C++17, Release
• Parallel jobs: 6
• CMake compatibility: CMAKE_POLICY_VERSION_MINIMUM=3.5 (for CMake 4.x)

Upload: When --upload is specified, built recipes are uploaded to the default-conan-local2 Artifactory remote. Requires JFROG_USERNAME2 and JFROG_PASSWORD2 environment variables (or prompts interactively).

scripts/build-and-push.sh

Builds a single Conan package from this repo's recipes and uploads the recipe and its dependencies (recipe-only) to JFrog Artifactory. Uses the conan create --format=json build graph to determine exactly which packages to upload, so it is safe to run locally without uploading unrelated packages from the cache. Supports both production and testing remotes. This script is also used by the build-and-push.yml GitHub Actions workflow.

Usage:

./scripts/build-and-push.sh <package> <version> [options]

Options:

Option	Description
--conanfile <path>	Relative path to conanfile.py within the package dir (default: all/conanfile.py)
--user-channel <u/c>	User/channel for the package (e.g. milvus/dev)
--extra-options <opts>	Extra conan options (e.g. -o pkg:opt=val)
--settings-mode <mode>	Conan settings mode: linux-gcc-11 or runner-default (default: linux-gcc-11)
--repository <repo>	Target Artifactory repository: production or testing (default: production)
--no-upload	Build only, skip uploading to Artifactory

Examples:

./scripts/build-and-push.sh zlib 1.3.1
./scripts/build-and-push.sh protobuf 5.27.0 --user-channel milvus/dev
./scripts/build-and-push.sh protobuf 5.27.0 --user-channel milvus/dev --repository testing
./scripts/build-and-push.sh folly 2024.08.12.00 --conanfile v2024/conanfile.py --user-channel milvus/dev --extra-options "-o folly/*:shared=True"
./scripts/build-and-push.sh milvus-common 1.0.0-28a2b48 --user-channel milvus/dev --settings-mode runner-default --extra-options "-s compiler.cppstd=20" --no-upload
./scripts/build-and-push.sh zlib 1.3.1 --no-upload    # build only

Authentication: Uploading to Artifactory requires JFROG_USERNAME2 and JFROG_PASSWORD2 environment variables (or prompts interactively).

Build metadata: After a successful upload, the script automatically sets the following properties on the uploaded recipe in JFrog Artifactory:

Property	Description
build.branch	Current git branch name
build.commit	HEAD commit SHA of this repo
build.datetime	UTC timestamp of the build
build.author	Git author of the latest commit

scripts/sync-to-artifactory.sh

Downloads a package from ConanCenter (upstream) and uploads it along with all transitive dependencies (recipes and binaries) to JFrog Artifactory. Uses the conan install --format=json install graph to determine exactly which packages to upload, so it is safe to run locally without uploading unrelated packages from the cache. Supports both production and testing remotes. This script is also used by the sync-to-artifactory.yml GitHub Actions workflow.

Usage:

./scripts/sync-to-artifactory.sh <package> <version> [options]

Options:

Option	Description
--repository <repo>	Target Artifactory repository: production or testing (default: production)
--no-upload	Download only, skip uploading to Artifactory

Examples:

./scripts/sync-to-artifactory.sh openssl 3.3.2
./scripts/sync-to-artifactory.sh libcurl 8.10.1 --repository testing
./scripts/sync-to-artifactory.sh prometheus-cpp 1.2.4 --no-upload    # download only

Authentication: Uploading to Artifactory requires JFROG_USERNAME2 and JFROG_PASSWORD2 environment variables (or prompts interactively).

Browsing Uploaded Recipes

After a recipe is uploaded, you can browse it via the JFrog web UI:

• Production: https://milvus01.jfrog.io/ui/repos/tree/Properties/default-conan-local2
• Testing: https://milvus01.jfrog.io/ui/repos/tree/Properties/testing2

Navigate to a recipe's conanfile.py and open the Properties tab to see the build metadata (build.author, build.branch, build.commit, build.datetime).

Example: folly/2024.08.12.00@milvus/dev properties:

https://milvus01.jfrog.io/ui/repos/tree/Properties/default-conan-local2/milvus/folly/2024.08.12.00/dev/f9b2bdf162c0ec47cb4e5404097b340d/export/conanfile.py