Kubernetes ingress controller samples need improvement

[Joren-Thijs-KasparSolutions]

Some details

I recently tried the kubernetes ingress samples and although i got it working. The documentation in the docs folder could use some improvement. I ran into two concrete issues.

Firsts it talks about a dockerfile you can build in .\src\Kubernetes.Controller\Dockerfile but this file does not seem to exist, not even in the main branch. Can you give clarification on this?

Second I tried to build the dockerfile from the ingress sample .\samples\KuberenetesIngress.Sample\Combined\Dockerfile. But this kept getting stuck on step 12/16 trying to do a dotnet restore.

reverse-proxy git:(main) docker build . -t acrkasparinternalunivweu.azurecr.io/yarp-controller:1.0.0 -f ./samples/KubernetesIngress.Sample/Combined/Dockerfile                                            
[+] Building 584.0s (20/26)                                                        docker:desktop-linux
 => [internal] load build definition from Dockerfile                                               0.0s
 => => transferring dockerfile: 1.53kB                                                             0.0s
 => [internal] load .dockerignore                                                                  0.0s
 => => transferring context: 461B                                                                  0.0s
 => [internal] load metadata for docker.io/amd64/buildpack-deps:jammy-curl                         1.5s
 => [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:6.0                               0.0s
 => [auth] amd64/buildpack-deps:pull token for registry-1.docker.io                                0.0s
 => [base 1/2] FROM mcr.microsoft.com/dotnet/aspnet:6.0                                            0.0s
 => [publish  1/16] FROM docker.io/amd64/buildpack-deps:jammy-curl@sha256:6c7cd91d715ab799960671f  0.0s
 => [internal] load build context                                                                  0.1s
 => => transferring context: 200.53kB                                                              0.1s
 => CACHED [base 2/2] WORKDIR /app                                                                 0.0s
 => CACHED [final 1/2] WORKDIR /app                                                                0.0s
 => CACHED [publish  2/16] WORKDIR /src                                                            0.0s
 => CACHED [publish  3/16] COPY [global.json, ]                                                    0.0s
 => CACHED [publish  4/16] RUN curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin -  0.0s
 => [publish  5/16] COPY [samples/KubernetesIngress.Sample/Combined/Yarp.Kubernetes.IngressContro  0.0s
 => [publish  6/16] COPY [src/ReverseProxy/Yarp.ReverseProxy.csproj, src/ReverseProxy/]            0.0s
 => [publish  7/16] COPY [src/Kubernetes.Controller/Yarp.Kubernetes.Controller.csproj, src/Kubern  0.0s
 => [publish  8/16] COPY [src/Directory.Build.props, src/]                                         0.0s
 => [publish  9/16] COPY [Directory.Build.*, ./]                                                   0.0s
 => [publish 10/16] COPY [NuGet.config, ]                                                          0.0s
 => [publish 11/16] COPY [eng/Versions.props, eng/]                                                0.0s
 => [publish 12/16] RUN /root/.dotnet/dotnet restore samples/KubernetesIngress.Sample/Combined/  582.3s
 => => # Write your first app: https://aka.ms/dotnet-hello-world                                       
 => => # Find out what's new: https://aka.ms/dotnet-whats-new                                          
 => => # Explore documentation: https://aka.ms/dotnet-docs                                             
 => => # Report issues and find source on GitHub: https://github.com/dotnet/core                       
 => => # Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli             
 => => # --------------------------------------------------------------------------------------        

This can go on exceeding 600 seconds at wich point i just give up.

The way to fix this for me was by introducing container properties in the ./samples/KuberenetesIngress.Sample/Combined/Yarp.Kubernetes.IngressController.csproj file and leveraging Microsoft.NET.Build.Containers (Buillt into net7 sdk) to publish the project as a dockerfile.

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFrameworks>net6.0;net7.0;net8.0</TargetFrameworks>
    <UserSecretsId>78d1f3b4-abce-4c5a-b914-3321fab1f8d0</UserSecretsId>
    <DockerDefaultTargetOS>Linux</DockerDefaultTargetOS>
    <IsPackable>$('System.TeamProject') != 'internal'</IsPackable>

    <!-- I added these -->
    <ContainerRepository>yarp-controller</ContainerRepository>
    <ContainerImageTag>1.0.0</ContainerImageTag>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Serilog.Extensions.Logging" Version="3.0.1" />
    <PackageReference Include="Serilog.Formatting.Compact" Version="1.1.0" />
    <PackageReference Include="Serilog.Sinks.Console" Version="3.1.1" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\..\..\src\Kubernetes.Controller\Yarp.Kubernetes.Controller.csproj" />
  </ItemGroup>

</Project>

dotnet publish --os linux --arch x64 /t:PublishContainer -f net7.0 -c Release

This worked great and built image yarp-controller:1.0.0 locally.

Next I used docker tag yarp-controller:1.0.0 my-container-registry.azurecr.io/yarp-controller:1.0.0 to tag it for the our Azure container registry.

Then i used docker push my-container-registry.azurecr.io/yarp-controller:1.0.0 to push it to our container registry.

So my suggstion is to update the doc to

• leverage Microsoft.NET.Build.Containers so it works better.
• remove the first paragraph referencing .\src\Kubernetes.Controller\Dockerfile which does not exist
• include instructions for applying docker tag after the build so you don't run into docker registry authentication isues while building
• either update the paths to use / instead of \ or include seperate code blocks for MacOS/Linux users.

I am using a M2 Macbook Pro with .NET 7 and dockerdesktop 4.22.1

I would also like to ask if you would except a PR from me to improve this?

How many backends are in your application?

• 1-2
• 3-5
• 6-10
• 10+

How do you host your application?

• Kubernetes
• Azure App Service
• Azure VMs
• Other Cloud Provider (please include details below)
• Other Hosting model (please include details below)

What did you think of YARP?

I am really impressed with the project so far but it lacks documentation. Especially in the route vs cluster concepts department.

[BasLijten]

I build the same code today. Although I agree on better documentation, the combined dockerfile did work and the restore worked succesfully.

[Joren-Thijs-KasparSolutions]

@BasLijten can you share your terminal output from the build?
How long did the restore step take? What machine/architecture are you running on?

[BasLijten]

i would have to rerun the build at a later moment, but having read your comment I was ready for failure. I think it took about (max) 1 minute. I am running a dell Legion 7: Windows 11, i9-13900HX. if i recall correctly, I ran the build on WSL2 (ubuntu 22), that might be a difference.

[Joren-Thijs-KasparSolutions]

@BasLijten I tried building the dockerfile on my windows laptop and indeed it works there without issues.
Both on Windows 11 and WSL2. I wonder if it's an arm64 vs x86 issue with the install.sh script it downloads in the dockerfile? Althoug i inspected it and it looks like it automatically detects wheter it's running on arm64.

Either way I still think it would be beter to just use dotnet publish for building container images since this works consistetly across platfroms.

[adityamandaleeka]

We should fix the broken docs/links. The build issue is interesting; we do build this in our CI successfully so perhaps there's an environmental problem involved here.

[Joren-Thijs-KasparSolutions]

@adityamandaleeka would you except a PR adding the conatainer publishing properties to the .csproj file and adding detailed steps for using them? Maybe under a MacOS specific section?