Indic LLM Leaderboard -> an open-source platform for comparing Indic LLMs across various Indic/standard benchmarks.
In the ever-evolving landscape of artificial intelligence (AI) and natural language processing (NLP), the focus on Indic languages, spoken by over a billion people in South Asia, is gaining momentum. Despite their vast user base and cultural significance, Indic languages face numerous challenges, ranging from data scarcity to the lack of sophisticated language tools. In this blog post, we'll delve into Indic Eval, a nimble evaluation suite crafted for appraising Indic LLMs across various tasks. Furthermore, we'll look into a leaderboard equipped with specialized benchmarks to assess and compare the performance of Indic Eval systems. This platform offers a holistic approach to evaluating model efficacy in the Indic language modeling sphere.
Recent advancements in Indic Large Language Models (LLMs) underscore progress, yet the absence of a unified evaluation framework complicates tracking and comparison. This challenge exacerbates existing issues like data scarcity and inadequate language tools. A unified evaluation framework with benchmarks is crucial to overcome these challenges and drive meaningful advancement in the field.
Founded by Adithya S K, CognitiveLab specializes in providing AI solutions at scale and undertaking research-based tasks. This initiative aims to create a unified platform where Indic LLMs can be compared using specially crafted datasets. Originally developed for internal use, CognitiveLab is now open-sourcing this framework to further aid the Indic LLM ecosystem.
After the release of Amabri, a 7b parameter English-Kannada bilingual LLM, CognitiveLab sought to compare it with other open-source LLMs to identify areas for improvement. Thus, the Indic LLM suite was born, consisting of two projects:
The alpha release of the Indic LLM Leaderboard and Indic Eval. These tools are pivotal steps towards standardizing evaluations within the field.
The Indic LLM Leaderboard is an evolving platform, aiming to streamline evaluations for Language Model (LLM) models tailored to Indic languages. While this alpha release is far from perfect, it signifies a crucial initial step towards establishing evaluation standards within the community.
As of this release, the following base models have been added into the leaderboard to use are reference:
meta meta-llama/Llama-2-7b-hfgoogle/gemma-7bTasks incorporated into the platform:
ARC-Easy:{language}ARC-Challenge:{language}Hellaswag:{language}For evaluation purposes, each task includes 5-shot prompting. Further experimentation will determine the most optimal balance between evaluation time and accuracy.
we are currently testing a lot more benchmarking datasets and planning to integrate with indic_eval
Datasets utilized for evaluation are accessible via the following link: Indic LLM Leaderboard Eval Suite
The decision to label this release as alpha stems from the realization that extensive testing and experimentation are necessary. Key considerations include:
Anticipate the following enhancements in the upcoming release:
Here are the steps you will have to follows to put your model on the Indic LLM leaderboard
Clone the repo:
git clone <https://github.com/adithya-s-k/indic_eval>
cd indic_eval
Create a virtual environment using virtualenv or conda depending on your preferences. We require Python 3.10 or above:
conda create -n indic-eval-venv python=3.10 && conda activate indic-eval-venv
Install the dependencies. For the default installation, you just need:
pip install .
If you want to evaluate models with frameworks like accelerate or peft, you will need to specify the optional dependencies group that fits your use case (accelerate,tgi,optimum,quantization,adapters,nanotron):
pip install .[optional1,optional2]
The setup tested most is:
pip install .[accelerate,quantization,adapters]
If you want to push your results to the Hugging Face Hub, don't forget to add your access token to the environment variable HUGGING_FACE_HUB_TOKEN. You can do this by running:
huggingface-cli loginaccelerate launch run_indic_evals_accelerate.py \
--model_args="pretrained=<path to model on the hub>" \
--language kannada \
--tasks indic_llm_leaderboard \
--output_dir output_dir \
--push_to_leaderboard <yourname@company.com> \
It's as simple as that.👍
For --push_to_leaderboard, provide an email id through which we can contact you in case of verification. This email won't be shared anywhere. It's only required for future verification of the model's scores and for authenticity.
After you have installed all the required packages, run the following command:
For multi-GPU configuration, please refer to the docs of Indic_Eval.
requirement for GPUs to run the evaluation?kannada, hindi, tamil, telugu, gujarati, marathi, malayalamWhile running the evaluation, you are given an option to push results to the leaderboard with
--push_to_leaderboard <yourname@company.com>. You will need to provide an email address through which we can contact you. If we find any anomaly in the evaluation score, we will contact you through this email for verification of results.
To foster collaboration and discussion surrounding evaluations, a WhatsApp group is being established and we can also connect on Hugging faces discord indic_llm channel
All the projects are completely open source with different licenses, so anyone can contribute.
The current leaderboard is in alpha release, and many more changes are forthcoming:
The alpha release of the Indic LLM Leaderboard and Indic Eval marks an important first step towards establishing standardized evaluation frameworks for Indic language models. However, it is clear that significant work remains to make these tools truly robust and accountable.
The decision to label this as an alpha release underscores the need for extensive testing, refinement of evaluation metrics, and determination of optimal setup parameters. The path ahead requires close collaboration and active participation from the open-source community.
We call upon researchers, developers, and language enthusiasts to join us in this endeavor. By pooling our collective expertise and resources, we can build a comprehensive, transparent, and reliable system to assess the performance of Indic LLMs. This will not only drive progress in the field but also ensure that advancements truly benefit the diverse linguistic landscape of the Indian subcontinent.
The road ahead is long, but the potential impact is immense. With your support and involvement, we are confident that the Indic LLM Leaderboard and Indic Eval will evolve into reliable and indispensable tools for the Indic language modeling community. Together, let us embark on this journey to unlock the full potential of Indic language technologies.