ANSI Colors support

[14NGiestas]

Colors Module

In order to support colors and styling (bold, underline and etc) I propose a small utility to handle colored output, namely stdlib_colors. The use cases can vary, but the main application would be the richer logger and errors which helps a lot to visually identify what kind of message is right away.

Figure 1 - coloredlogs python library showing a use case

Few remarks from the community in #193

@mobius-eng commented on 20 May
(...)
Julia (and Python and R) are meant to be used interactively. Thus, colors play an important role with the output to the terminal. By contrast, Fortran programs are meant to be run in the "batch mode" with log going to the file.

@aradi commented on 24 May
As for colors: I am fine, as long as they are optional. I think most people typically run Fortran programs in batch systems with redirected output. Seeing a lot of color control sequences when opening the output files is rather disturbing...

@jvdp1 commented on 23 May
(...)
Colors are not needed for me.

@wclodius2
A problem with using colors is that output_unit can be directed to a file where the "color codes” will be distracting.

Proof of Concept and fixing the issue with redirection

• Well, gnu utils have support to colored output for eons and when it gets redirected it handles correctly, how is that possible? So I did some research in order to figure out how they are made and found this.
• So basically we test if there is a teletypewriter device (A interactive terminal) connected with the isatty function (from <unistd.h>), if isatty returns true then we write the ANSI colors, otherwise no colors in the output.
• A lot of compiler vendors support this function as a extension, however is not a standard so we can't rely on it. That being said I think it should be included in the stdlib_system module as a new function binding the C one (and alternatives to other platforms).
• In order to show that it works I made a small change in a program to output colors from the fortran-wiki and its here

Some other related projects that could inspire the API

• FACE by @szaghi
• Foul, as pointed by @arjenmarkus]

[certik]

I think something along the FACE and Foul libraries would be good for stdlib. I like using colors a lot in a terminal, it helps.

The ANSI escape sequences for colors work out of the box on Linux / macOS, but must be turned on to work on Windows.

So one needs some basic terminal support. I created this library in C++ to do colors on all platforms as well as keyboard input:

https://github.com/certik/terminal

This allows to create things like interactive prompts, get arrows working for cursor movements, and to draw anywhere on the screen. Similar to the ncurses Linux library, but multi-platform and only using the ANSI sequences that are now supported on all platforms (including Windows), which simplifies things.

[urbanjost]

I like the FACE interface. I have a similar library but it is not as completely or nicely developed. For something fancier than colors I generally use ncurses from Fortran http://www.urbanjost.altervista.org/LIBRARY/libscreen/ncurses/pdsrc/ncurses_from_Fortran.html but I think that is a bit beyond the stdlib but would be appropriate for fpm if it supported C wrappers as well as Fortran. Specifically for xterm(1) I have a library that lets you set fonts, window size and position, window title, etc. The esc(1) program in the GPF collection actually combines the two

when called without CLI options. So I use escape sequences quite extensively in one form or another but find using ncurses preferable for something past basic text formatting; so I personally support adding some basic ANSI color interface but think that getting a mostly-Fortran fpm(1) package manager up and getting the authors of packages like those mentioned to add their libraries to the registry more appropriate myself. There are ISO_C_BINDING interfaces for terminfo/termlib out there too which give you extensive terminal escape sequence support as well; although the ability to support different terminals is not nearly as important now that almost everyone uses a terminal emulator instead of a terminal, and those tend to emulate the ANSI terminals like VT102 or do not support escape sequences at all.

[certik]

@urbanjost yes, I agree that going beyond colors should go into a separate fpm package.

I would recommend in general to not depend on platform specific libraries, such as ncurses (which doesn't work on Windows in the native cmd.exe terminal), and rather always implement things in a platform independent (multiplatform) way. I have done that in C++ in the terminal library, and the same thing can be done in pure Fortran as an fpm package also.

I agree that the reason that can be done is that nowadays everybody supports some subset of the ANSI escape sequences, and so one can just use those.

[urbanjost]

I put M_escape
onto github as it uses a different API model than the others referenced so far, although the parsing method used is just a prototype it is functional for basic string coloring.

I found it interesting to see what language have color built-in and how many have a standard module or library for it in Rosetta Code.

[certik]

That seems similar to the approach here: https://python-prompt-toolkit.readthedocs.io/en/master/pages/printing_text.html#html It works. But it's slower, because you have to parse the string. So it depends on the application.

…

On Sat, Sep 12, 2020, at 8:15 PM, urbanjost wrote: I put M_escape <https://github.com/urbanjost/M_escape> image <https://user-images.githubusercontent.com/29845229/93008620-c2065e80-f544-11ea-84d1-0fbba159c769.png> onto github as it uses a different API model than the others referenced so far, although the parsing method used is just a prototype it is functional for basic string coloring. I found it interesting to see what language have color built-in and how many have a standard module or library for it in Rosetta Code. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#229 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAFAWBXFUPNUIPL4BX5VCLSFQTMBANCNFSM4Q74M77Q>.

[urbanjost]

Speed is not always an issue in that you can process fixed strings once, as in
error=esc("ERROR")
and then just use the error variable; but usually adding color for human consumption at a terminal so I would not imagine using this with large amounts of data; but I agree.

[urbanjost]

The advantage of the approach of replacing in-band escape sequences
with formatting directives contained on each line is that it is easy
to turn off when running batch, but more importantly your program can
be run in "raw" mode and write a file with the directives in it that
can then be read back in by a simple filter program that strips it
back to plain text or displays it to a screen in color or converts it
to HTML or Adobe PDF. By making each line self-contained by default
this can still be done with any selected group of lines from the file.

I need to add some example filters (a loop that reads the "raw" files and displays them as plain text or as color on screen would liternally be a read/write loop that calls the esc(3f) routine, but an HTML and PDF example ala like the M_ncurses does for ncurses dumps (to HTML) or the asa2pdf(1) program does for "ASA carriage control" might make a stronger case); but have not seen a lot of interest in this so far(?) so it is on the "one of these days" lists for now. People can write HTML pretty easily from Fortran, after all. This actually has some advantages because it is simple and assumes fix spaced output and keeps each line independent by default. So that is how I justify the parsing overhead. It compliments traditional batch use but still lets you throw a little sparkle in your output.

[certik]

It's a question what to use for markup to show colors. HTML is certainly one way to do that, used in python-prompt-toolkit. It's a clean way to do that.

To be honest though, I still kind of like the simplicity of just outputting ANSI sequences, as you don't have to worry about any kind of HTML, you just output the sequences directly, here is an example from C++:

        std::string text = "Some text with "
            + color(fg::red) + color(bg::green) + "red on green"
            + color(bg::reset) + color(fg::reset) + " and some "
            + color(style::bold) + "bold text" + color(style::reset) + ".";
        std::cout << text << std::endl;

You can use less -r to see files with ANSI escape sequences. It works great. One can then convert it to HTML, I've used the ansi2html filter in Jinja in the past when converting a terminal output to html:

• https://nbconvert.readthedocs.io/en/4.2.0/architecture.html#templates-and-filters

One way to design color output is to first provide the color function together with the predefined fg, bg and style enumerations (somehow in Fortran) and then build the HTML layer on top. People who prefer HTML can use it, people who prefer a more direct style can use the color function directly.

[urbanjost]

I use the ansi2html(1) bash shell as well. I changed M_escape to have not only the pseudo-XML mode, but a function-based and direct ANSI escape sequence mode and added an fpm(1) config for it and simple filters called plain(1) and light(1) to read the pseudo-XML files and print the output to stdout with and without color as examples for discussion. I was going to do an OOP interface too, but I think the other resources listed here cover that. Comments welcome. I put a list of resources that compliments those mentioned here as well. If anyone tries it I would be interested in a vote as to which interface people prefer. The README files shows an example of each of the three modes.

[certik]

@urbanjost thanks! I like your example as the lowest level API:

   program direct
      use M_escape, only : &
     ! FOREGROUND COLORS
        & fg_red, fg_cyan, fg_magenta, fg_blue, fg_green, fg_yellow, fg_white, fg_ebony, fg_default, &
     ! BACKGROUND COLORS
        & bg_red, bg_cyan, bg_magenta, bg_blue, bg_green, bg_yellow, bg_white, bg_ebony, bg_default, &
     ! ATTRIBUTES
        & bold, italic, inverse, underline,  unbold, unitalic, uninverse, ununderline,  reset, &
     ! DISPLAY
        & clear
      implicit none
	write(*,'(*(g0))')fg_red,bg_green,bold,'Hello!',reset
   end program direct

I think that works great. With this, and your html API, I think that covers most use cases.

[14NGiestas]

I think that works great. With this, and your html API, I think that covers most use cases.

I think this kind of low level API is great, but we should keep in mind the complaint about the redirection issue too. So i think it should have a higher level API with a function that handles all the functionality:

program
    use stdlib_colors only: color & ! other suggestions: colorize, stylize...
                            fg_blue, sgr_bold
    implicit none
    write(*,*) color('Hello... ',      fg='red',  bg='yellow'), &
               color("it's me you're ",fg='blue', bg='yellow'), &
               color('LOOKING FOR?',   fg='red',  bg='yellow', style='bold')
    write(*,*) color('I can see it in your <b>eyes</b>') 
    write(*,*) color('I can see it in your <y>smile</y>') 
    write(*,*) 'Tell me how to win your ', color('<3', 'red')
end program

This would apply the escapes and the reset and handle the file redirection thing.
Using only the XML tags and the low level API would leave behind some kind of procedural colorization use case: imagine parsing a json and colorize it in some way, i think would be easier to just read the color name and style and pass to a function rather than build a XML string to escape or use a select case to apply the right low level escape.

[certik]

complaint about the redirection

Can you explain what you mean by this? Is it to change the output from ANSI sequences to HTML using the same code?

The API you proposed is very good also, and it can be built on top of the low level API we discussed above.

[14NGiestas]

I mean the redirection of such outputs to a file, since they would write the ansi escapes sequences too, polluting it (see "Few remarks from the community" section of this issue).
EDIT: And yes I think the higher API should be built upon the low level one.

[certik]

I see. I feel this must go into the API that is above the low level API.

I see his architecture all over many of the other issues in stdlib. In this case, the API layers as I can see them could be (from lowest to highest):

• lowest level: the goal is to output the correct ANSI string without having to deal with ansi sequences directly, but there should be no over head and it should be as simple as possible. The two proposed solutions would both work:

  • print *, fg_red, bg_green, bold, 'Hello!', reset
  • print *, color('Hello!', fg=fg_red, bg=bg_green, style=style_bold) (notice the fg_red is not a string "red" to avoid string parsing / comparison, and just efficiently output the ANSI sequence)

• higher level: a function color that can work in many ways, output both ANSI or HTML output and allows to turn off colors

• high level: some intermediate representation (XML) that can be parsed into, and then manipulated to any output (HTML, ANSI, latex, ...)