Add support for Malayalam

[grhoten]

This is a GSoC Project idea.

Difficulty/Size: Medium

Right now, the Unicode Inflection project supports Arabic, Danish, German, English, Spanish, French, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Malay, Norwegian, Dutch, Portuguese, Russian, Swedish, Thai, Turkish and Chinese. Supporting more languages is desired.

Here is background material for the Unicode Inflection concepts.

• UTW 2023 Automatic Grammar Agreement in Message Formatting
• S12T1 Authoring Grammatically Correct Conversational Templates for Siri
• Let's Come To An Agreement About Our Words :: IMUG 2017.02.16

Expected Outcomes

• Unicode Inflection code will be able to inflect nouns and personal pronouns for the language being supported. Examples include:
  • object + plural → objects
  • city + plural,genitive → cities’
• Optionally inflect articles, prepositions, adjectives and verbs as necessary for a given language.
• All tests of supported functionality should pass.
• Support a language that isn’t already supported that has sufficient Wikidata, examples include:
  • Estonian, Malayalam, Greek, Czech, Norwegian (Nynorsk), Slovak, Ukrainian, Bangla, Punjabi, Polish, Urdu, or Finnish
  • Perhaps others, but the required data would need to be added to Wikidata.
• The lexical data will be derived from Wikidata. There is an existing tool to generate appropriate lexical dictionaries for each language, and there are examples of other supported languages.

Skills

• Required: Working proficiency in English
• Required: Understanding of a language that is not already supported in Unicode Inflection.
• Required: Experience with writing software on Windows, Linux, or macOS
• Required: Experience with C or C++
• Preferred: Experience with cmake
• Preferred: Ability to edit XML
• Preferred: Ability to edit data in Wikidata.

[Muzamilkhan35713]

The title is all ready changed

[Blaeriz]

@grhoten , I would like to add support for an Indian language called Marathi. According to worlddata.info, there are 96 Million native speakers. Non-native speakers are way more than that.

I am a native Marathi speaker.

[grhoten]

@Blaeriz Thanks for your interest. It looks like the number of words on Wikidata for Marathi is too low to consider to support in this project at this time. You can look at this website for the lexical size of each language in Wikidata.

Once the number of words get larger than a 1,000 lexemes in Wikidata, it could be sufficient to consider supporting, but it wouldn't be through GSoC. Even if that number were met, there are a number of inflectional languages in modern use that provide better word coverage that would be a higher priority to consider support because they would be more likely to provide successful support. A person adding support for Bangla (10,899 lexemes) or Malayalam (67,367 lexemes) would be more likely to be successful with their final results.

Even when considering the size of the lexical coverage, the quality is also a consideration. Here are some additional considerations.

• Are the lexemes phrases or individual words? Phrases can be broken down to individual words for inflection in a repeatable pattern. Phrases are generally ignored, like idioms or sayings. This is because they provide no value for inflecting words when getting a sentence into grammatical agreement.
• Are the lemmas for a human occupation separate or unified? If the lemmas are separate, then there is no way to modify the word to grammatically agree with the gender of a person, which is a blocker from supporting in this project. Many languages will gender words like doctor, nurse, actor/actress, writer, singer and so forth. Many default a gender in a generic context, but they tend to use a specific gender when referring to a specific person with a known gender.
• Do the forms of a lexeme cover all variations of the given lexeme? For example, does it cover both the singular and plural forms? Does it cover the relevant grammatical cases and so forth?
• Are frequently used words being supported? A good way to gauge that support is to look at the potential list of words in Unilex.

If someone volunteered to add such missing data to Wikidata for a given language, they would set up this project to be successful when supporting that language.

[nciric]

@Blaeriz <https://github.com/Blaeriz> one aspect of the project is adding relevant words, see George's reply, to the lexicon in Wikidata. Contributing to Wikidata, and building a community of volunteers to do so is very important for us. Are there any good sources of data for Marathi? Online lexicons, open sourced data that you could upload to Wikidata? There is a script <https://github.com/unicode-org/inflection/blob/main/data/tools/wikidata_upload.py> that can help you with upload, but needs work - it needs to be expanded to support "Senses", e.g. description of each word added - otherwise your contribution would be rejected by the WIkidata admins. пон, 10. мар 2025. у 11:23 George Rhoten ***@***.***> је написао/ла:

…

@Blaeriz <https://github.com/Blaeriz> Thanks for your interest. It looks like the number of words on Wikidata for Marathi is too low to consider to support in this project at this time. You can look at this website for the lexical size of each language in Wikidata <https://ordia.toolforge.org/language/>. Once the number of words get larger than a 1,000 lexemes in Wikidata, it could be sufficient to consider supporting, but it wouldn't be through GSoC. Even if that number were met, there are a number of inflectional languages in modern use that provide better word coverage that would be a higher priority to consider support because they would be more likely to provide successful support. A person adding support for Bangla (10,899 lexemes) or Malayalam (67,367 lexemes) would be more likely to be successful with their final results. Even when considering the size of the lexical coverage, the quality is also a consideration. Here are some additional considerations. - Are the lexemes phrases or individual words? Phrases can be broken down to individual words for inflection in a repeatable pattern. Phrases are generally ignored, like idioms or sayings. This is because they provide no value for inflecting words when getting a sentence into grammatical agreement. - Are the lemmas for a human occupation separate or unified? If the lemmas are separate, then there is no way to modify the word to grammatically agree with the gender of a person, which is a blocker from supporting in this project. Many languages will gender words like doctor, nurse, actor/actress, writer, singer and so forth. Many default a gender in a generic context, but they tend to use a specific gender when referring to a specific person with a known gender. - Do the forms of a lexeme cover all variations of the given lexeme? For example, does it cover both the singular and plural forms? Does it cover the relevant grammatical cases and so forth? - Are frequently used words being supported? A good way to gauge that support is to look at the potential list of words in Unilex <https://github.com/unicode-org/unilex/blob/main/data/frequency/>. If someone volunteered to add such missing data to Wikidata for a given language, they would set up this project to be successful when supporting that language. — Reply to this email directly, view it on GitHub <#85 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7GEKQI4LVS3D74UGY3OTT2TXRE5AVCNFSM6AAAAABYKTZAUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJRGQ3DANZZG4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> [image: grhoten]*grhoten* left a comment (unicode-org/inflection#85) <#85 (comment)> @Blaeriz <https://github.com/Blaeriz> Thanks for your interest. It looks like the number of words on Wikidata for Marathi is too low to consider to support in this project at this time. You can look at this website for the lexical size of each language in Wikidata <https://ordia.toolforge.org/language/>. Once the number of words get larger than a 1,000 lexemes in Wikidata, it could be sufficient to consider supporting, but it wouldn't be through GSoC. Even if that number were met, there are a number of inflectional languages in modern use that provide better word coverage that would be a higher priority to consider support because they would be more likely to provide successful support. A person adding support for Bangla (10,899 lexemes) or Malayalam (67,367 lexemes) would be more likely to be successful with their final results. Even when considering the size of the lexical coverage, the quality is also a consideration. Here are some additional considerations. - Are the lexemes phrases or individual words? Phrases can be broken down to individual words for inflection in a repeatable pattern. Phrases are generally ignored, like idioms or sayings. This is because they provide no value for inflecting words when getting a sentence into grammatical agreement. - Are the lemmas for a human occupation separate or unified? If the lemmas are separate, then there is no way to modify the word to grammatically agree with the gender of a person, which is a blocker from supporting in this project. Many languages will gender words like doctor, nurse, actor/actress, writer, singer and so forth. Many default a gender in a generic context, but they tend to use a specific gender when referring to a specific person with a known gender. - Do the forms of a lexeme cover all variations of the given lexeme? For example, does it cover both the singular and plural forms? Does it cover the relevant grammatical cases and so forth? - Are frequently used words being supported? A good way to gauge that support is to look at the potential list of words in Unilex <https://github.com/unicode-org/unilex/blob/main/data/frequency/>. If someone volunteered to add such missing data to Wikidata for a given language, they would set up this project to be successful when supporting that language. — Reply to this email directly, view it on GitHub <#85 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7GEKQI4LVS3D74UGY3OTT2TXRE5AVCNFSM6AAAAABYKTZAUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJRGQ3DANZZG4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[S-Devipriya]

I have gone through the API for the "Integration of Unicode Inflection into MessageFormat 2.0" and documentation for adding new languages. I have better understood the project scope for adding new language compared to the other project, so I would like to know more about adding new language project, specially with regards to Malayalam.

I have already signed the CLA and forked the repository to better understand the codebase. Going through the codebase, I have realized the need to be familiar with few grammatical terms and I have been working on that. Proficiency in a language and grammatical awareness in the language are two seperate things, especially pertaining to native languages as it feels innate to us.

I have a couple of questions:
• Is it okay to start on this project if I don't mind whether I am accepted in GSoC? I am interested in contributing to adding inflection support for Malayalam regardless of application status.
• If so, do I get started by following the guide on marking language as supported? I found the locale code is ml-IN on CLDR website. Is that the relevant source while updating the Locale and LocaleUtils with "ml" and "ml", "IN" respectively?
• Could you provide guidance on what else to work on while preparing for this project?

I am currently looking through lexicographical data available on wikidata and understanding various steps involved in new language support.
As for actual scope of inflection support, personal pronouns and noun inflection with regards to number and genitive form is first priority as outlined in the project. Outside of that, I am not sure if there are other areas as Malayalam doesnot have adjective or verb inflection with gender or number. Verbs change according to tense but I'm unsure if that comes under inflection. Regardless, I would like to work on pronouns and nouns first.

[nciric]

пон, 10. мар 2025. у 15:19 S Devipriya ***@***.***> је написао/ла:

I have gone through the API for the "Integration of Unicode Inflection into MessageFormat 2.0" and (documentation for adding new languages)[ https://github.com/unicode-org/inflection/blob/main/documents/how_to_add_new_language.md]. I have better understood the project scope for adding new language compared to the other project, so I would like to know more about adding new language project, specially with regards to Malayalam. I have already signed the CLA and forked the repository to better understand the codebase. Going through the codebase, I have realized the need to be familiar with few grammatical terms and I have been working on that. Proficiency in a language and grammatical awareness in the language are two seperate things, especially pertaining to native languages as it feels innate to us. I have a couple of questions: • Is it okay to start on this project if I don't mind whether I am accepted in GSoC? I am interested in contributing to adding inflection support for Malayalam regardless of application status. • If so, do I get started by following the guide on marking language as supported? I found the locale code is ml-IN on CLDR website. Is that the relevant source while updating the Locale and LocaleUtils with "ml" and "ml", "IN" respectively? • Could you provide guidance on what else to work on while preparing for this project? I am currently looking through lexicographical data available on wikidata and understanding various steps involved in new language support. As for actual scope of inflection support, personal pronouns and noun inflection with regards to number and genitive form is first priority as outlined in the project. Outside of that, I am not sure if there are other areas as Malayalam doesnot have adjective or verb inflection with gender or number. Verbs change according to tense but I'm unsure if that comes under inflection. Regardless, I would like to work on pronouns and nouns first. — Reply to this email directly, view it on GitHub <#85 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7GEKSEDT4MJQNWEGY2ZFL2TYFYVAVCNFSM6AAAAABYKTZAUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJRHE3TIOBTGY> . You are receiving this because you commented.Message ID: ***@***.***> [image: S-Devipriya]*S-Devipriya* left a comment (unicode-org/inflection#85) <#85 (comment)> I have gone through the API for the "Integration of Unicode Inflection into MessageFormat 2.0" and (documentation for adding new languages)[ https://github.com/unicode-org/inflection/blob/main/documents/how_to_add_new_language.md]. I have better understood the project scope for adding new language compared to the other project, so I would like to know more about adding new language project, specially with regards to Malayalam. I have already signed the CLA and forked the repository to better understand the codebase. Going through the codebase, I have realized the need to be familiar with few grammatical terms and I have been working on that. Proficiency in a language and grammatical awareness in the language are two seperate things, especially pertaining to native languages as it feels innate to us. I have a couple of questions: • Is it okay to start on this project if I don't mind whether I am accepted in GSoC? I am interested in contributing to adding inflection support for Malayalam regardless of application status.

Inflection is an open source project and it's great you want to contribute (and you already made a first step by accepting CLA). AFAIK, any work you do now won't count towards GSoC hours later on, if you are accepted. Otherwise you are free to contribute. • If so, do I get started by following the guide on marking language as supported? I found the locale code is ml-IN on CLDR website. Is that the relevant source while updating the Locale and LocaleUtils with "ml" and "ml", "IN" respectively? Take a look at this document <https://github.com/unicode-org/inflection/blob/main/documents/how_to_add_new_language.md> (and PR#40) to see what code and data needs to change. That's incomplete PR as it's missing crucial logic on dealing with inflection rules or reading from the dictionary. • Could you provide guidance on what else to work on while preparing for

this project?

See above. Also check quality of Malayalam dictionary by downloading wikidata and building Inflection dictionary for it - see https://github.com/unicode-org/inflection/blob/main/documents/add_lexicon_data.md . I am currently looking through lexicographical data available on wikidata and understanding various steps involved in new language support.

…

As for actual scope of inflection support, personal pronouns and noun inflection with regards to number and genitive form is first priority as outlined in the project. Outside of that, I am not sure if there are other areas as Malayalam doesnot have adjective or verb inflection with gender or number. Verbs change according to tense but I'm unsure if that comes under inflection. Regardless, I would like to work on pronouns and nouns first. — Reply to this email directly, view it on GitHub <#85 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7GEKSEDT4MJQNWEGY2ZFL2TYFYVAVCNFSM6AAAAABYKTZAUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJRHE3TIOBTGY> . You are receiving this because you commented.Message ID: ***@***.***>

[S-Devipriya]

@nciric Thank you for your response. I will start by checking quality of Malayalam wikidata and building inflection dictionary. I had looked into PR#40 and that's where I was confused what locale codes would be for Malayalam for the Locale and LocaleUtility.cpp files. Are locale codes taken from CLDR (so "ml" and "ml","IN")?
As for GSoC, I fully understand. I am a beginner at open source so gaining some experience is my priority.

[nciric]

You are welcome. Yes, ml and ml-IN would be correct language and region codes. You can also look them up in https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes . пон, 10. мар 2025. у 15:55 S Devipriya ***@***.***> је написао/ла:

…

@nciric <https://github.com/nciric> Thank you for your response. I will start by checking quality of Malayalam wikidata and building inflection dictionary. I had looked into PR#40 and that's where I was confused what locale codes would be for Malayalam for the Locale and LocaleUtility.cpp files. Are locale codes taken from CLDR (so "ml" and "ml","IN")? As for GSoC, I fully understand. I am a beginner at open source so gaining some experience is my priority. — Reply to this email directly, view it on GitHub <#85 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7GEKSGPX7KGTE6A7PURQL2TYJ7PAVCNFSM6AAAAABYKTZAUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJSGAZDQNRYGA> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: S-Devipriya]*S-Devipriya* left a comment (unicode-org/inflection#85) <#85 (comment)> @nciric <https://github.com/nciric> Thank you for your response. I will start by checking quality of Malayalam wikidata and building inflection dictionary. I had looked into PR#40 and that's where I was confused what locale codes would be for Malayalam for the Locale and LocaleUtility.cpp files. Are locale codes taken from CLDR (so "ml" and "ml","IN")? As for GSoC, I fully understand. I am a beginner at open source so gaining some experience is my priority. — Reply to this email directly, view it on GitHub <#85 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7GEKSGPX7KGTE6A7PURQL2TYJ7PAVCNFSM6AAAAABYKTZAUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJSGAZDQNRYGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[S-Devipriya]

Thank you. I queried Wikidata to see the Malayalam lexemes and downloaded a copy of Wikidata lexemes. However, I am unable to parse it with ParseWikidata. On running, the parser does not do anything, there are no errors either.
I am not familiar with gradle or java so could you guide me what might be the issue here? I am not even able to see all options by running ./ParseWikidata on vscode powershell. It runs but gives no output, perhaps I am missing some environment set-up?

Any help or input is greatly appreciated

[grhoten]

Assuming that you're on macOS or Linux, and Java is available, you run the following commands from the root directory of this project.

cd inflection/tools/dictionary-parser
../../gradlew fatJar
./ParseWikidata --language ml --inflections ../../resources/org/unicode/inflection/dictionary/inflectional_ml.xml --dictionary ../../resources/org/unicode/inflection/dictionary/dictionary_ml.lst ~/Downloads/wikidata-20250305-lexemes.json

You also need a copy of a lexeme Wikidata dump from https://dumps.wikimedia.org/wikidatawiki/entities/. In my case, I decompressed it before running these commands. If you don't know how to decompress it, you can specify the bzip2 file directly without decompressing it. It's just way slower to run this tool if you run it more than once.

This only generates the data. The rest of the documentation previously mentioned will take you further for the next steps.

[S-Devipriya]

Thank you for the help. Is it required that the parser must be run from macOS or Linux? I am using VScode on Windows, but I do have WSL installed so should I reopen the project in it?

Outside of that, I have followed everything else. I have decompressed the lexeme Wikidata file as well and ensured file names match with one I downloaded. I have also made sure Java is available, OpenJDK 21.0.6 is installed.

[grhoten]

The Java code and gradle files should run fine on any OS. The primary issue for this dictionary-parser tool is building the code on Windows. You might have some inspiration on that by looking at https://gradle.org/install/ for details. I suspect that merely adding the gradle batch file from an installation to the inflection directory may be enough, but you could look into installing gradle into your Windows installation instead.

If that doesn't work, you can also consider using Docker and running a Linux container to build and run the code instead.

With that being said, the bigger issue is getting this project running on Windows. The C++ code and cmake files have not been tested on Windows. You might want to consider trying to build and run the tests before trying to create a lexical dictionary for a given language.

[S-Devipriya]

I installed gradle 8.9 on Windows. However, it is still not running. I will try again soon, perhaps by re-installing java.

Meanwhile, as you suggested, I will prepare a Linux container, build dependencies and run the code there as it will be required later as well. I will report back once that is done. Thank you, I appreciate your patience and help as I am working to understand an opne-source repo like this as a beginner.

[S-Devipriya]

Apologies for the delay in updating, there were system hardware issues (short-circuiting) requiring repair at a servicing centre.

I have installed Linux (Ubuntu). However, I am unable to install the dependency icu4c to proceed with building codebase for unicode inflection. The issue is that I am unable to run ./runConfigureICU due to permission issues. I understand that it's Linux troubleshooting and unrelated to topic here.
On further consideration of my lack of experience working on Linux or having done other code projects, I have decided to utilize the summer for learning programs to better prepare myself for GSoC next year. I am still interested in contributing to adding Malayalam support during the summer after my exams outside of GSoC.
Thank you for all the guidance and I look forward to being able to contribute once I familiarize myself with Linux and sort out the issues.