[question] Handling failed builds in products pipeline flow

[akielbas]

What is your question?

Our team has been working on implementing the products pipeline flow from the CI Tutorial in the docs.

We do have similar Artifactory repos that correspond to packages, products and develop.

We have the build working pretty well with a re-usable GitHub Actions workflow. We can feed an arbitrary conanfile and a list of target profiles / build systems and have it build everything that is missing in a dynamic way, uploading to products along the way. What's still work in progress is the promotion workflow.

In the documentated example the entire flow can be summarized as:

1. Create lockfile
2. Generate build order for every desired configuration
3. Merge build orders (with --reduce)
4. Iterate through packages that need to be built, for each:
   a) Build missing package
   b) Convert build output to list
   c) Upload list to products repo (in practice will only be the one package that was just built, everything else will generally already be in products repo because it must have been built earlier per build order).
5. Combine uploaded lists, preserve this somewhere
6. Test the built packages in products repo
7. Later promote uploaded list via conan download / upload or conan art:promote using persisted list from step 5.

This works pretty well, but I'm curious what you recommend for the following scenario:

1. Let's assume we have a simple/flat build order of PkgA/1.0, PkgB/1.0 and PkgC/1.0.
2. Run steps 1-4 above and say that PkgC/1.0 fails to build but PkgA/1.0 and PkgB/1.0 have built successfully and have been uploaded to products repo.
3. We don't continue to step 5 because of the failure and go resolve the issue with PkgC/1.0 build (could be bad version or a build system issue, whatever).
4. We try to run steps 1-4 again and this time the reduced build order is only PkgC/1.0 (because PkgA/1.0 and PkgB/1.0 already exist in products repo from earlier attempt), this completes successfully.
5. When we get to step 7 above we only have the list with PkgC/1.0 so PkgA/1.0 and PkgB/1.0 are not promoted.

I see several ways to solve this but was wondering what others think:

1. We could clear the products repo between attempts. Not ideal because we share this repo among multiple products. I've seen mention that some teams dynamically create Artifactory repos on the fly for each run but this is not an option for us unfortunately.
2. We could go to each attempt (multiple workflow runs) and collect all the uploaded lists and combine those. Not ideal trying to find all these individual workflow runs.
3. We could compute a complete build graph for each configuration, convert each to a list, combine and use that as the promotion list instead of basing it on what was built. Might be a little wasteful when using conan download / upload promotion (maybe there is a way to filter out packages that are already in develop?) but perhaps it is not a big deal with conan art:promote.

We're kind of leaning toward option 3 here, is that an OK approach? Are we totally missing something?

Thank you in advance!

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[akielbas]

I've implemented option 3 above and it is working quite well, still interested in hearing what's recommended.

We no longer collect and aggregate the built package graphs / pkglists as shown in the tutorial.

Instead after we complete any succesful build process attempt we do the following:

1. compute a new graph info for each configuration
2. turn each build graph info into a package list
3. merge all package lists into one
4. use art:promote to copy from products remote to develop (need to additionally specify --remote for products because usually some packages are already in develop).

[memsharded]

Hi @akielbas

Thanks for your question

This works pretty well, but I'm curious what you recommend for the following scenario:

So this would be the scenario of an interrupted/broken build

From your comment in

We try to run steps 1-4 again and this time the reduced build order is only PkgC/1.0 (because PkgA/1.0 and PkgB/1.0 already exist in products repo from earlier attempt), this completes successfully.

I understand that the breakage doesn't really need a code change in PkgC, but it was a glitch or something, not an API break that requires a code change.

Indeed, PkgA and PkgB don't need to be built, they were already built and uploaded. This is one of the advantages of the approach, it allows to re-use binaries that result from a partial build in further later builds, avoiding unnecessary costs even if the build as a whole failed.

When we get to step 7 above we only have the list with PkgC/1.0 so PkgA/1.0 and PkgB/1.0 are not promoted.

I see, yes, if this is managed as a clean build from scratch, it might fail to promote PkgA and PkgB.

We could clear the products repo between attempts. Not ideal because we share this repo among multiple products. I've seen mention that some teams dynamically create Artifactory repos on the fly for each run but this is not an option for us unfortunately.

I agree, this looks inefficient at best, and potentially problematic

We could go to each attempt (multiple workflow runs) and collect all the uploaded lists and combine those. Not ideal trying to find all these individual workflow runs.

This doesn't sound that bad, as you had actually preserved the package list in Step 5 above, then, if this is an incremental, combined view of the packages that have been created, it wouldn't be necessary to find all workflows, just the parent/latest one, and pick it from there?

We could compute a complete build graph for each configuration, convert each to a list, combine and use that as the promotion list instead of basing it on what was built. Might be a little wasteful when using conan download / upload promotion (maybe there is a way to filter out packages that are already in develop?) but perhaps it is not a big deal with conan art:promote.

Yes, and even for the download+upload promotion, the truth is that the checks that the Conan client does to avoid uploading/downloading things that are already in the server or in the cache should save most of the potentially unnecessary costs, so this looks like a pragmatic approach. And it shouldn't be an issue with direct API promotion like the conan art:promote one.

[memsharded]

I've implemented option 3 above and it is working quite well, still interested in hearing what's recommended.

Interesting, thanks for the feedback!

In any case, yes, the approach sounds pragmatic and more robust against re-launched partial builds.
The case explained in the tutorial is focused on the "happy path", trying to be as efficient as possible.

compute a new graph info for each configuration
turn each build graph info into a package list

The information should already be there in the conan graph build-order result, but it is true that this doesn't convert that easily to a package list.

Maybe this is something that we want to revisit, maybe a feature request to create package-lists out of build-orders, that way the extra conan grpah info could be saved.

[akielbas]

Thank you, I appreciate your feedback!

[memsharded]

I am trying a new feature in PR #19025, but I am not convinced at all, it seems this approach could have its flaws.

I am thinking that a robust flow for this should probably be designed around a persistent pkglist in the CI that aggregates the packages that have been built and uploaded.

So moving from the proposal in the tutorial:

pkg_lists = []  # to aggregate the uploaded package-lists
for level in to_build:
    for recipe in level:  # This could be executed in parallel
        ref = recipe["ref"]
        for packages_level in recipe["packages"]:
            for package in packages_level:
                ...
                run(f"conan install {build_args} {build_type} --lockfile=conan.lock --format=json", file_stdout="graph.json")
                run("conan list --graph=graph.json --format=json", file_stdout="built.json")
                filename = f"uploaded{len(pkg_lists)}.json"
                run(f"conan upload -l=built.json -r=products -c --format=json", file_stdout=filename)
                pkg_lists.append(filename)

and

$ conan pkglist merge -l uploaded0.json -l uploaded1.json -l uploaded2.json -l uploaded3.json --format=json > uploaded.json

To something like:

# create a persistent "uploaded.json" file in the CI, initially empty ``{}``
for level in to_build:
    for recipe in level:  # This could be executed in parallel
        ref = recipe["ref"]
        for packages_level in recipe["packages"]:
            for package in packages_level:
                ...
                run(f"conan install {build_args} {build_type} --lockfile=conan.lock --format=json", file_stdout="graph.json")
                run("conan list --graph=graph.json --format=json", file_stdout="built.json")
                filename = f"uploaded{len(pkg_lists)}.json"
                run(f"conan upload -l=built.json -r=products -c --format=json", file_stdout=filename)
                # Incremental merge over the persistent uploaded.json
                run(f"conan pkglist merge -l=uploaded.json -l=built.json --format=json", file_stdout=filename)

That way at any time, the uploaded.json contains the latest aggregated list of built+uploaded packages, and if it is persistent in the CI pipeline, it is easy to use it for the promotion, etc.

Wdyt?

[akielbas]

Thanks, I will take a look at the PR for the new feature.

Regarding the doc proposal, I think that is an good suggestion. That said, for our implementation I would hestitate to use it for two reasons:

1. It is some additional complexity to have to store and keep track of these persistent upload package lists. We use GitHub Actions for CI/CD and there isn't common artifact storage that we can use across multiple runs. We would have to use Artifactory (which is fine).
2. Although rare, we may accumulate package builds that don't end up needed in the final build of product.

Expanding on item 2 above:
(Disclaimer: We might be approaching this incorrectly so let me know what you think.)

Since implementing our product workflow we've used it a couple of items and generally it is working really well. There have been failures however not relating purely to build machine issues, stuff not caught in individual packge build:

1. Recipe errors (conflicting requires in two new separate packages that weren't anticipated at individual package build time)
2. Source code errors when building with different Conan profiles (ones we didn't use in packge build)
3. Runtime bugs in libraries when they're combined with other new libraries (vs. package build which generally tests individual library against what is already in develop/production).

As these come up we fix them upstream (fixes sources, add new tests, new package builds, etc) but realistically we'll never be able to catch everything ahead of time so it falls to product build.

During product build we'd either see failure at build time or at runtime when testing product built against products repo. We'd fix these upstream (new recipe revisions, new package versions) and run the product build again.

These additional iterations may change the build graphs and we may not end up needing certain packages which were successfully built/uploaded in earlier iterations.

Again this could be pretty rare, and maybe not worth worrying about, but was trying to be as efficient as possible here.

I still like the idea of just computing a list of packages from build graph that are in the source repo and not the target repo and promoting those, regardless of when they were built.