How to generate graphs with common-lisp-jupyter?

[ryukinix]

Hi! Sorry for bothering you again. I'm just very happy to see this development. I see you add recently a collection of useful widgets. There is some way to to embedded graphs in this kernel?

It would be so awesome!

https://github.com/martinkersner/cl-plot Something like this or better.

[yitzchak]

No need to apologize.

Unless you want the graphs to have interactive sliders or such, you probably don't need widgets.

Media can be displayed with the functions defined in results.lisp The displayed media can come from a file or an inline value. In the case of something like a gnuplot wrapper you will probably need to write the result to a file then display the result in the notebook with jupyter:file. There are a few examples in the about notebook.

If I understand cl-plot correctly then you would do something like this.

(defparameter *dataframe*
  '((1 1)
    (2 2)
    (3 1.5)
    (4 3)
    (5 2.5)
    (6 4)))

(defparameter *fig* (make-instance 'figure))

(xlabel *fig* "x-label")
(ylabel *fig* "y-label")
(title *fig* "Basic example")
(scatter *fig* *dataframe*)

(save *fig* "fig.png")
(jupyter:file "fig.png")

If you are using JupyterLab there is also an extension for Vega-Lite that allows one to pass the JSON based Vega-Lite description right to the front end. Eventually I may make some wrapper code for this.

(jupyter:inline-result
  (jsown:new-js
    ("$schema" "https://vega.github.io/schema/vega-lite/v3.json")
    ("description" "A simple bar chart with embedded data.")
    ("data" (jsown:new-js
      ("values" (list
        (jsown:new-js ("a" "A") ("b"  28)) (jsown:new-js("a" "B") ("b" 55))  (jsown:new-js("a" "C") ("b" 43)) 
        (jsown:new-js("a" "D") ("b" 91)) (jsown:new-js("a" "E") ("b" 81))  (jsown:new-js("a" "F") ("b" 53)) 
        (jsown:new-js("a" "G") ("b" 19)) (jsown:new-js("a" "H") ("b" 87))  (jsown:new-js("a" "I") ("b" 52))))))
    ("mark" "bar")
    ("encoding" (jsown:new-js
      ("x" (jsown:new-js ("field" "a") ("type" "ordinal")))
      ("y" (jsown:new-js ("field" "b") ("type" "quantitative"))))))
  "application/vnd.vegalite.v2+json")

[ryukinix]

Thank you! I'll try that!

[Symbolics]

If you are using JupyterLab there is also an extension for Vega-Lite that allows one to pass the JSON based Vega-Lite description right to the front end. Eventually I may make some wrapper code for this.

What is the name of the recommended extension for Vega-Lite?

[yitzchak]

It's called @jupyterlab/vega5-extension. You can install from the extension manager in the lab interface or the following on the command line.

jupyter-labextension install @jupyterlab/vega5-extension 

[Symbolics]

Is the extension still required? I was able to generate some Vega-Lite graphs with JupyterLab 3.0.9 by creating a spec file and double clicking on it. See https://jupyterlab.readthedocs.io/en/stable/user/file_formats.html#vega-vega-lite. I admit though, it's a rather awkward way to create a plot.

You mentioned a wrapper. I tried wrapping a Vega Lite spec composed of alists, which seems a natural fit to me, and it worked well until the yason developer(s) reverted the recursive behaviour because it broke some existing functionality slightly, and were unwilling to address it properly because it would mean a breaking change to the API. This is unfortunate because developing a plot spec from lisp is more natural, and you can splice in data with the backquote mechanism. I suppose an alternative JSON library could be tried; I used yason because it's the library used by json-mop and it would be nice to go backwards and forwards between the two.

Did you give any thought to how you might wrap a Vega-Lite JSON spec?

[yitzchak]

The extension is for rendering the graph in a notebook output cell. I think it is still required if you want to do that.

I haven't thought about wrapping the spec much, but I if you are just trying to create inline graphs I would probably just write convenience functions that output the JSON in the above format for each kind of graph you want to do.

Right now common-lisp-jupyter is using jsown internally. That doesn't prevent you from using whatever JSON library you want though. Eventually I want to switch to my own library, shasht, which was designed to pass JSONTextSuite which contains over 300 different parsing tests. You can see differences between the various Common Lisp libraries here. If you are just generating JSON that this probably doesn't matter much, but common-lisp-jupyter is doing a lot of JSON parsing internally.

[Symbolics]

Interesting. I never heard of shasht before. I'll have a look. I might just try to work around any issues in yason with a macro, or maybe work with the yason guys to get those changes moved back in somehow.

[Symbolics]

Can shasht handle encoding of nested alists? For example in the above (though not an alist, imagine it is):

("encoding" (jsown:new-js
      ("x" (jsown:new-js ("field" "a") ("type" "ordinal")))
      ("y" (jsown:new-js ("field" "b") ("type" "quantitative"))))))

?

[yitzchak]

It can. It is in quicklisp, but probably not used by anyone yet. I haven't put much effort in to documentation yet, but you can see some of the configuration stuff in config.lisp

* (ql:quickload :shasht)
To load "shasht":
  Load 1 ASDF system:
    shasht
; Loading "shasht"
[package shasht].........
(:SHASHT)
* (let ((shasht:*write-alist-as-object* t)) 
    (shasht:write-json '(("fu" . 1) ("bar" . (("quux" . 7))))))
{
  "fu": 1,
  "bar": {
    "quux": 7
  }
}
(("fu" . 1) ("bar" ("quux" . 7)))

[Symbolics]

If you are using JupyterLab there is also an extension for Vega-Lite that allows one to pass the JSON based Vega-Lite description right to the front end. Eventually I may make some wrapper code for this.

Do you have an example of passing Vega-Lite JSON to directly to the front end? If, say, we were to encode a spec with an alist like this:

(let ((shasht:*write-alist-as-object* t))
  (shasht:write-json
'(("$schema" . "https://vega.github.io/schema/vega-lite/v5.json")
 ("description" . "A simple bar chart with embedded data.")
 ("data"
  ("values"
   . #((("a" . "A") ("b" . 28)) (("a" . "B") ("b" . 55))
       (("a" . "C") ("b" . 43)) (("a" . "D") ("b" . 91))
       (("a" . "E") ("b" . 81)) (("a" . "F") ("b" . 53))
       (("a" . "G") ("b" . 19)) (("a" . "H") ("b" . 87))
       (("a" . "I") ("b" . 52)))))
 ("mark" . "bar")
 ("encoding"
  ("x" ("field" . "a") ("type" . "nominal") ("axis" ("labelAngle" . 0)))
  ("y" ("field" . "b") ("type" . "quantitative"))))
   ))

Is there a way to have Jupyter Lab render the plot?

[yitzchak]

You don't need to serialize the JSON. You need to pass the JSOWN style json to jupyter:inline-result.

(jupyter:inline-result
  (jsown:new-js
    ("$schema" "https://vega.github.io/schema/vega-lite/v3.json")
    ("description" "A simple bar chart with embedded data.")
    ("data" (jsown:new-js
      ("values" (list
        (jsown:new-js ("a" "A") ("b"  28)) (jsown:new-js("a" "B") ("b" 55))  (jsown:new-js("a" "C") ("b" 43)) 
        (jsown:new-js("a" "D") ("b" 91)) (jsown:new-js("a" "E") ("b" 81))  (jsown:new-js("a" "F") ("b" 53)) 
        (jsown:new-js("a" "G") ("b" 19)) (jsown:new-js("a" "H") ("b" 87))  (jsown:new-js("a" "I") ("b" 52))))))
    ("mark" "bar")
    ("encoding" (jsown:new-js
      ("x" (jsown:new-js ("field" "a") ("type" "ordinal")))
      ("y" (jsown:new-js ("field" "b") ("type" "quantitative"))))))
  "application/vnd.vegalite.v3+json")

Also, it looks like you no longer need to install the separate extension as it now included by default with JupyterLab.

[Symbolics]

I was thinking of a way to save a JSON spec in a variable, or as part of a plot class. The jsown is a bit wordy to work with as a lisp-side spec. Are there any ways an alist can be coerced into the JSOWN (or other format) without inserting a bunch of jsown:new-js function calls into the DSL?

[yitzchak]

The JSOWN objects are just alists with a :obj head.

'(:obj ("a" . "A") ("b" . 28))

[Symbolics]

I see. In fact if you look at the new-js output:

(:OBJ ("$schema" . "https://vega.github.io/schema/vega-lite/v3.json")
 ("description" . "A simple bar chart with embedded data.")
 ("data" :OBJ
  ("values" (:OBJ ("a" . "A") ("b" . 28)) (:OBJ ("a" . "B") ("b" . 55))
   (:OBJ ("a" . "C") ("b" . 43)) (:OBJ ("a" . "D") ("b" . 91))
   (:OBJ ("a" . "E") ("b" . 81)) (:OBJ ("a" . "F") ("b" . 53))
   (:OBJ ("a" . "G") ("b" . 19)) (:OBJ ("a" . "H") ("b" . 87))
   (:OBJ ("a" . "I") ("b" . 52))))
 ("mark" . "bar")
 ("encoding" :OBJ ("x" :OBJ ("field" . "a") ("type" . "ordinal"))
  ("y" :OBJ ("field" . "b") ("type" . "quantitative"))))

I can see the raw JSOWN.

I was thinking that the alist->JSON encoding happened in one of the cells, and then was passed to Jupyter, but it seems instead that Jupyter receives the JSOWN spec to interpret. Are we able to influence this path and somehow send the final JSON spec to the Vega renderer?

[yitzchak]

Its JSOWN style because common-lisp-jupyter is using JSOWN and inline-result just passes the contents directly to the message encoder which handles the serialization to JSON since the Jupyter protocol is JSON layered on top of ZeroMQ.

It might be possible to add some options to inline-result which influence the serialization. I'll add it as an item in the shasht PR.