Preserving blank space at the end of sentence

[monks1975]

Would like it if you could support preserving blank space at the end of sentence when using doc.sentences().out(). A few other sbd libraries do this and it's useful to have. Thanks.

[sanjayaksaxena]

Hello @monks1975

In the current implementation we are specifically trimming leading/trailing spaces, please refer to as.text. Changing this would lead to a breaking change for the community.

We can possibly add as.untrimmedText helper? Any suggestions/ideas to make it even better from end-user perspective?

Best,
Sanjaya

[monks1975]

I think adding an 'as' helper would be perfect for this functionality. I don't know the ins and outs of the your apis yet as am coming to this lib fairly fresh (although I use the wink bayes classifier in this project for some time). Right now I am using Compromise for sentence tokenisation and this is the api I use currently:

    const doc = nlp(text);
    const tokens = doc
      .sentences()
      .json({ terms: { whitespace: true } })
      .map(c =>
        c.terms
          .map(t => {
            return t.pre + t.text + t.post;
          })
          .join('')
      );

In this case I am mapping over the terms created for each sentence and joining the pre and post whitespace back to the text.

Best,
Jon

[sanjayaksaxena]

You could use the following code meanwhile:

const winkNLP = require( 'wink-nlp' );
const model = require( 'wink-eng-lite-web-model' );
const nlp = winkNLP( model );
const its = nlp.its;

const text = 'U.S.A is my birth place.   I was born on 06.12.1924.I climbed Mt. Everest.    It was fun!';
const doc = nlp.readDoc( text );
doc.sentences().each(s => {
  const sentence = [];
  s.tokens().each( t => sentence.push( t.out(its.precedingSpaces), t.out() ));
  console.log(sentence.join(''))
});

// —> "U.S.A is my birth place."
// —> "   I was born on 06.12.1924."
// —> "I climbed Mt. Everest."
// —> "    It was fun!"

[monks1975]

Thank you, will try this out!