Search broken with umlaut?

[danielnaber]

If you search for gütersloh, you'll get 10.000+ results, many of which are invalid. Searching for gutersloh works better. Similar for würzburg. Does the umlaut mess up the search?

[stefanw]

I don't think it's the umlaut, but we're using word decompounding (see ES Dockerfile and our analyzer config). The decompounding is based on heuristics and will split up "Gütersloh".

curl --request POST \
  --url http://localhost:9200/fragdenstaat_de_foirequest/_analyze \
  --header 'content-type: application/json' \
  --data '{
  "analyzer": "fds_analyzer",
  "text": "Gütersloh"
}'

{
  "tokens": [
    {
      "token": "gutersloh",
      "start_offset": 0,
      "end_offset": 9,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "guter",
      "start_offset": 0,
      "end_offset": 9,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "gut",
      "start_offset": 0,
      "end_offset": 9,
      "type": "<ALPHANUM>",
      "position": 0
    }
  ]
}

The idea is that German words can become very long („Informationsfreiheitsgesetz“) and we want to index the individual words. However, this approach has led to some confusion by users and I'm wondering if there's a better approach.

Here's our help page in German about search which recommends putting exact search terms in quotes to avoid decompounding. But if your search needs a help page, maybe it needs improvement. :)

You have linguistic background iirc, what would you recommend?

[danielnaber]

You have linguistic background iirc, what would you recommend?

You could use a better word splitter like https://danielnaber.de/jwordsplitter/ maybe, but I'm not sure if it's worth the effort. Maybe just be more transparent in the UI about what has been searched internally? Like a search engine that tells you it searched for something and then offers "xyz" (including the quotes) to only include that very word.