Adding simple progress status on the buildmessage.

[BielLopes]

What problem will this solve?
When compiling the project using the generated Makefile, the standard output is very basic and provides no indication of the overall compilation process.
The standard output looks like this:

==== Building program1 (debug) ====
Creating obj/Debug
Running prebuild commands
file1.cpp
file2.cpp
file3.cpp

There's no inherent problem with that solution, but I'm thinking of ways to improve this and offer a better development experience for Premake users.
The prototype I'm working on will produce output like this:

==== Building program1 (debug) ====
Creating obj/Debug
Running prebuild commands
[1/3] Compiling file1.cpp (program1)
[2/3] Compiling file2.cpp (program1)
[3/3] Compiling file3.cpp (program1)

I want to contribute to Premake as much as I can, and I have some thoughts about the applicability of my solution. I'd like to know if I should proceed and what else I should consider to enable a more stable and complete solution for this improvement.

What might be a solution?
After a deep investigation, I could not find any "well-established" way to show a nice progress indicator using just Make. I found a beautiful solution on stackoverflow that uses only Make syntax, but it has many problems associated with the overhead of recursive Make calls and multi-threaded execution.
Very determined to find a way to make this possible, I started a brainstorming process to enable this with no external tool dependencies, not even Lua itself.
Even though the final solution should not depend on anything other than Make and native tools on every platform (such as standard command lines, shell scripts/PowerShell), the prototype I created is implemented in Lua, with the assumption that all operations can be translated to platform-specific tools, as I will discuss in this issue.

What other alternatives have you already considered?
The alternatives I considered were switching to CMake, Ninja, or Meson, as they all implement progress indication when compiling code.
My thought is to bring the same capabilities to Premake as these tools.

Anything else we should know?
To explore and test the prototype I am developing, you can access this fork for Premake5 and try to compile Premake5 itself with another Premake5 pre-installed on your environment. However, due to technical limitations, this fork should only work in Linux environments. To work properly, you will need a Lua interpreter (I tested with 'Lua 5.4.7 Copyright (C) 1994-2024 Lua.org, PUC-Rio'), and you will need to install the luafilesystem and luaposix modules (I used luarocks {sudo luarocks install luafilesystem && sudo luarocks install luaposix}). Ensure they are globally available for your Lua interpreter. All changes from your master branch are available in that commit.

The central thing I want to discuss in this ISSUE is the opinion of the Premake5 maintainers about the viability and applicability of such a feature. Specifically, does the complexity that this feature adds to the Premake5 project compensate for the benefits of better visual information?

I will try to argue in favor of an implementation approach that minimizes breaking risks and aligns with the Premake philosophy.
To be direct, the main change I am proposing is the way the compilation message is invoked. Currently, the default message is @echo "$(notdir $<)", and in my prototype, I created the "PrettyMessage" rule to replace that with $(SILENT) lua progress.lua $(notdir $<). However, the final goal is to achieve progress.sh $(notdir $<) for Unix systems or progress.bat $(notdir $<) on Windows. Such a change goes against an 'implicit' principle of Premake, which is to generate Makefiles to be invoked with Make. This happens because everything Premake generates is just Make code, and what I am proposing will generate script language code.
I will argue that even if this solution changes Premake to a point where it generates a Shell Script or a Batch Script, that is not a problem, since the script only uses native platform-specific tools.
To understand what I mean by 'native platform-specific tools', let's explore a bit of the progress.lua script. One of the main parts of the script is the lock system that enables multi-thread consistency for progress calculation:

--#############################################
-- LOCK SYSTEM FOR MULTITHREADED COMPILATION
-----------------------------------------------

    ---Open the lock flag file
    ---@param path string The path to the lock file that is used as a flag
    ---@return integer fd The file identifier
    local function glock(path)
        local fd, err = fcntl.open(path, fcntl.O_RDWR | fcntl.O_CREAT, 0666)
        if not fd then
            print(" Error opening lock file:", err)
            os.exit(1)
        end
        return fd
    end

    ---Create an exclusive lock on the file, blocking any other read or write operation
    ---@param path string The path to the lock file that is used as a flag
    ---@param dpath string The path to the data file
    local function xlock(path, dpath)
        if not io.open(path, "r") or not io.open(dpath, "r") then
            os.exit(0)
        end
        local fd = glock(path)
        local lock = { l_type = fcntl.F_WRLCK, l_whence = fcntl.SEEK_SET, l_start = 0, l_len = 0 }
        local success, err = fcntl.fcntl(fd, fcntl.F_SETLKW, lock) -- Blocking lock
        if not success then
            print(" Error acquiring exclusive lock:", err)
            os.exit(1)
        end
    end

    ---Remove the shared or exclusive lock
    ---@param path string The path to the lock file that is used as a flag
    local function unlock(path)
        local fd = glock(path)
        local lock = { l_type = fcntl.F_UNLCK, l_whence = fcntl.SEEK_SET, l_start = 0, l_len = 0 }
        fcntl.fcntl(fd, fcntl.F_SETLK, lock) -- Non-blocking unlock
    end

In the code above, I am using the fcntl submodule from the luaposix module to be able to create a lock system between multiple threads. The glock function will take a path to a file that will work as a flag and open it. Then, xlock and unlock will work with this flag file to communicate with other threads about whether other files are available for reading/writing. When xlock marks the flag file as locked, no other thread will be able to proceed until an unlock operation is triggered, and threads will await their turn to acquire the lock and manipulate the progress file.
"Okay Gabriel, this has the potential to solve the problem of multiple threads calculating progress, but it doesn't solve the 'native platform-specific' tool issue!" This is true. As implemented, we still have the problem of finding a native tool on each platform to perform the same logic. In Linux, we have flock, a CLI tool present in the util-linux package (like kill, mount, fdisk, etc.), which is virtually omnipresent on all Linux distributions. For Windows, things are a little different. I didn't find a direct alternative for flock in Batch Script that can lock a file from other processes and where the kernel guarantees the lock is released when the process is killed. However, in PowerShell Scripting Language, we have similar behavior when opening a file with the C# API: if a file is opened by one process, other processes will throw an error when trying to open it. So, the solution is to simply loop over a try-catch operation to open the file, and only one thread will have read/write access to the progress data at a time. The idea of using a progress.bat script is for it to be invoked from CMD and PowerShell, but internally call PowerShell to compute and log the message (I haven't been a Windows user for a long time, so if someone knows how to improve this logic for Windows, please share).
Now, for macOS, FreeBSD, Solaris, Haiku, and other Unix-like systems, they might not have a simple native CLI command to achieve this behavior. However, since all of them implement the lockf() syscall, it's possible to find a solution, even if it requires compiling a small C program. In the case of macOS, Ruby is natively installed and can perform syscalls or auto-install modules if not already present.
Another solution is to use the Lua code and download the Lua interpreter and libraries when the gmake subcommand is executed. But again, this adds extra dependencies to the project, though it can reduce the complexity of maintaining multiple script language generations for each operating system.
I know this feature breaks the simplicity of Premake and adds considerable complexity for just a visual effect. In that case, a more realistic approach could be to "fork" the gmake subcommand into something like gmake++. This can keep the original purpose of gmake and give the user the possibility to use a more expressive version of it.

Doubts about how Premake works

1. In my code design process, I faced the problem of calculating the total number of compilable targets of a project. Here is the code I wrote for this in premake5.lua in my premake-core fork:

    ---Collect unique files for every pattern, avoiding duplications
    ---@param filters table Array with the glob pattern (strings).
    ---@return table filesmap keys are filepaths, value = bool
    function collectfiles(filters, excludes)
        local filesmap = {}

        for _, pattern in ipairs(filters) do
            -- Use native premake5 API to access globs
            local matches = os.matchfiles(pattern)
            for _, filepath in ipairs(matches) do
                if filepath:match("%.c$") then
                    filesmap[filepath] = true
                end
            end
        end

        for _, pattern in ipairs(excludes) do
            -- Use native premake5 API to access globs
            local matches = os.matchfiles(pattern)
            for _, filepath in ipairs(matches) do
                if filepath:match("%.c$") then
                    filesmap[filepath] = false
                end
            end
        end

        return filesmap
    end

    ---Join the array as a string formatted as "'a','b',...'n'".
    ---@param files table String array (ex: {"src/file1.cpp", "src/file2.cpp"})
    ---@return string quoted Filenames joined with simple quotes and comma (ex: "'src/file1.cpp','src/file2.cpp'")
    function joinfiles(files)
        local quoted = {}
        for path, include in pairs(files) do
            if include then
                table.insert(quoted, string.format("'%s'", path))
            end
        end
        -- concat using comma
        return table.concat(quoted, ",")
    end

    local cfiles = joinfiles(collectfiles({ "src/**.c", "modules/**.c" }, { "contrib/**.c", "binmodules/**.c" }))

    -- Rule to compile and log progress
    rule "PrettyMessage"
        fileextension ".c"
        buildcommands {
            'lua progress.lua $(notdir $<)',
            '$(CC) $(ALL_CFLAGS) $(FORCE_INCLUDE) -o "$@" -MF "$(@:%.o=%.d)" -c "$<"',
        }
        buildoutputs { '$(OBJDIR)/%{file.basename}.o' }


    project "Premake5"
        targetname  "premake5"
        language    "C"
        kind        "ConsoleApp"
        rules       { "PrettyMessage" }

        files
        {
            "*.txt", "**.lua",
            "src/**.h", "src/**.c",
            "modules/**"
        }

        excludes
        {
            "contrib/**.*",
            "binmodules/**.*"
        }

Why do the files and excludes sections indicate more than just .c files? In my personal project, I didn't see any difference by just including .c files, because, from Make's perspective, all object files (.o or .obj) are only generated from a .c or .cpp file, and never from .h, .txt, or .lua files. What difference do such filters make on the generated Makefile? My approach was to just add the .c files to the files and excludes lists, iterate over them one after another, setting a true on each instance in the first list and false in the second, indicating whether the file will or will not be compiled (collectfiles({ "src/**.c", "modules/**.c" }, { "contrib/**.c", "binmodules/**.c" })).

2. Is the default behavior of Premake to put the generation of .d/.dep files in the %{cfg.objdir} folder with the -MF "$(@:%.o=%.d)" flag? Without a custom rule, can the user disable this? Is the algorithm I developed to calculate/predict the total number of files that Make will compile basically to iterate over each file in the .d file and verify if the timestamp of the last modification is greater than the object file (.o/.obj) itself, correct?

Another Considerations

• I'm a Rust programmer, so please be kind to me as I'm not an expert in C/C++. I started using Premake a month ago on a personal project to create an interpreted language like Lua/Python using C++, and Premake looks like the most friendly and idiomatic build tool I found.
• Should this be something to implement for Premake6, since 5.x is already in beta?
• The integration with tokens and things like the buildmessage parameters should be a separate issue, right?
• On my fork, the progress build message has nice colors, but for accessibility purposes, it's better to have no color as a default behavior and add a configuration option to color the text, enabling the user to choose a custom color setting policy, such as a random/specific color for each project, which will allow differentiating each target visually in a multi-threaded compilation.
• Anything else I should know about how Premake and Make work that I'm not seeing and should consider for this feature?

[Jarod42]

The alternatives I considered were switching to CMake, Ninja, or Meson

There are premake modules for CMake and Ninja (I contribute to both of them).
Ninja has several integrated features (as the "progress bar"), so Ninja is a good alternative to Make for that.
There are "discussions" about having a ninja module as core module (not necessary the existing one though).

Even though the final solution should not depend on anything other than Make and native tools on every platform (such as standard command lines, shell scripts/PowerShell), the prototype I created is implemented in Lua, with the assumption that all operations can be translated to platform-specific tools, as I will discuss in this issue.

I like the fact that generated "solution" are stand alone (doesn't depends of premake).
a priori, having extra script seems strange to me.

In my code design process, I faced the problem of calculating the total number of compilable targets of a project.

There are not only compilation, but

• directory creation
• pre/post build
• custom build
• custom rules
• link
• different build actions as copy (directories)
• incremental build (so number has to be recomputed at each build)

I don't think it is the premake's task to do that.
I even think that collecting those data is done when handling the dependencies, so it would mostly do the job twice (one by premake, one by make).

BTW, I'm a simple external contributor, and I got often idea different than the owners of the projects.

[tritao]

I would consider using https://github.com/jimon/premake-ninja as well, personally that's what I've been using in my projects and it works rather well.

As @Jarod42 said, there have been some discussions about integrating such support as a core module. Of course that does not mean the make module cannot be improved, but personally I'd rather see ninja adopted as the default backend, rather than make as its simpler, faster and more robust than make.

I would even go one step further and integrate https://github.com/michaelforney/samurai inside Premake so it can build projects on its own, but that's another discussion.

[BielLopes]

Thanks so much, learn a lot, really.
I think I will proceed to create a Premake module, looks like a more natural path to achieve my goal.
But I agree with you @tritao, Premake should officially support ninja/samurai natively.