ANSI symbols for manipulating line position are not handled in web ui output

[yarikoptic]

The simplest example to demonstrate is to run e.g.

print("someline\n2ndline\n\x1b[Aembed ")

in both ipython console (where it works correctly) where it would override 2ndline with embed , and then in the web notebook, where it would not.

Why I care... tqdm module is quite handy for providing progress bars, and uses this "line navigation" ansi characters to be able to output multiple progress bars in the terminal at the same time. They do provide progressbars using ipython widgets BUT you need to code for the notebook specifically. Since I was told that there is no way within python process to know either it is currently outputing to web ui or console, we simply can't provide dynamic switching between one implementation and another :-( any advice on how to handle this situation would be very welcomed! Ideally I would like to see web frontend providing support for basic ANSI navigation symbols since it should be relatively easy to implement their support (I think ;) ).

note: originally filed against ipython/ipython: ipython/ipython#10643

[MilesCranmer]

Status of this? I'm looking into this for the issue: cloud-oak/ProgressBars.jl#49 where ANSI navigation is used for status bar outputs.

@Carreau I saw some PRs for ANSI sequences from you. Would you happen to know whether ANSI navigation sequences are possible? Sorry if this tag is misplaced.

Cheers,
Miles

[Carreau]

I'm not sure it is handled, and IIRC one of the problem is that terminals have a predefined "hight", while web browser does not, so it's sometime fairly complicated what to do. Plu sIIRC< there are some worries of having DOS as some of the early attempts at paring ansi sequences were O(n^2).

There is also some concern about intermingling stdout/err I believe, though I guess that is easier to solve.

That was for notebook, maybe jupyterLab has a better solution, in particular lab can simply use components published on NPM, so if one if able to do the right thing...why not?

Unfortunately I'm not involved much in JupyterLab codebase, so I can't tell you.

[yarikoptic]

Seeing how badly tqdm progress bars behave in output of !externally_ran_commands, I spent some time trying to figure out actual underlying reason, and then most appropriate issue, and then ran into this issue I filed 8 years ago! Feeling proud and sad at the same time ;)

Some additional observations:

• jupyter pretends well to support fully blown xterm, sets TERM and tty reports capabilities to navigate lines

but my original example still demonstrates the issue of it not being able to perform a very basic "go 1 line up" for which it does not need to know full size of terminal at all etc:

although in a proper terminal looks like

I think the best here would be to implement handing of the basic set of ANSI control commands correctly, such as line navigation and clearing, and potentially going to the top level corner etc -- I really do not see a technical challenge to have those implemented as long as there is control over the content of the output box (which I guess there is) -- it is really just manipulation of text and lines within and re-rendering.

Otherwise, even basic nesting of a tqdm progressbar, like

#!/usr/bin/env python3
from tqdm import tqdm
from time import sleep

for _ in tqdm(range(2), "All", ncols = 80, position = 0):
    for _ in tqdm(range(10), "Sub", ncols = 80, position = 1, leave = False):
        sleep(0.01)

produces ugly

[krassowski]

FYI there is:

from tqdm.notebook import tqdm

which uses notebook widgets and does not have that issue.

[yarikoptic]

@krassowski this is about output from tools in a "terminal", not running python code natively to render output in the notebook using widgets. Part of the issue here that jupyter does not set TERM env var to some dumb or alike, it claims to provide xterm. so underlying tools which are to use that terminal, have no way to get a clue that it is deficient and lack support of ANSI control codes.

[yarikoptic]

and if the tools do use that `tqdm.notebook` explicitly after figuring out somehow that they are under very special jupyter notebook environment, you get no progress/completion reports, just announcement that it starts that it is to start -- so not a solution really either

But overall issue is not just about tqdm, but any tool which might use ANSI control symbols (top?), hence a proper solution should not be tqdm specific.