example for parsing indented languages

[Geal]

indented languages must keep a context indicating the indentation level

[LaylBongers]

Here's a really basic example that parses a language that looks like this:

value

parent
    child
        nestedchild
    otherchild

parent2
    child2

use nom::{IResult, alphanumeric, line_ending};
use nom::types::{CompleteStr};

#[derive(Debug)]
pub enum Value {
    Value(String),
    List(String, Vec<Value>),
}

#[inline(always)]
pub fn document(input: CompleteStr) -> IResult<CompleteStr, Vec<Value>> {
    terminated!(input,
        map!(
            opt!(apply!(lists_values, 0)),
            |v| v.unwrap_or(Vec::new())
        ),
        eof!()
    )
}

#[inline(always)]
fn lists_values(input: CompleteStr, indent_level: usize) -> IResult<CompleteStr, Vec<Value>> {
    many1!(input,
        terminated!(
            alt!(
                complete!(apply!(value_list, indent_level)) |
                apply!(value_value, indent_level)
            ),
            many0!(line_ending)
        )
    )
}

#[inline(always)]
fn value_list(input: CompleteStr, indent_level: usize) -> IResult<CompleteStr, Value> {
    do_parse!(input,
name:   apply!(value, indent_level) >>
list:   apply!(lists_values, indent_level+1) >>
        ( Value::List(name, list) )
    )
}

#[inline(always)]
fn value_value(input: CompleteStr, indent_level: usize) -> IResult<CompleteStr, Value> {
    map!(input,
        apply!(value, indent_level),
        |s| Value::Value(s)
    )
}

#[inline(always)]
fn value(input: CompleteStr, indent_level: usize) -> IResult<CompleteStr, String> {
    delimited!(input,
        count!(indent_any, indent_level),
        map!(alphanumeric, |s| { s.0.to_string() }),
        line_ending
    )
}

named!(indent_any<CompleteStr, CompleteStr>,
    alt_complete!(
        tag!("    ") |
        tag!("\t")
    )
);

[elbaro]

A non-trivial indentation example doing the following would be great:

• Avoid adding indent_level to every function and use INDENT/DEDENT abstraction (as in Python)
• How to generate INDENT/DEDENT tokens, while supporting multiline-string-literals
• Support mixed space/tabs as indentation. Python allow 'tab|4spaces|tab' but the next line cannot be 'tab|tab|tab'. The parser is required to keep track of the indentation history.

[jlkiri]

Hi everyone, I needed to parse indentation in one of my projects and found one approach pretty useful so I created a POC here (with nom 7.0):

https://github.com/jlkiri/nom-indentation-poc

The basic idea is to keep a mutable reference to an indentation counter (struct) and capture the reference in a closure that is a valid nom parser. Then on each invocation of this parser we count the number of tabs in the current line and if it is more than the current amount of indentation we generate INDENT token and if it is less we generate DEDENT token. That's it. The parser assumes that nothing can be multiline though, so I'm not sure how it would be different.
If you think it's OK, I guess it could be used as an example.