About the Organisation
The OpenNyAI mission, created in 2021, is a collaborative mission to catalyse exponentially better justice solutions by harnessing the power of AI. The mission’s members include Agami, Thoughtworks, Ekstep, and the National Law School of India University, Bangalore. Agami, which steers the OpenNyAI mission, is geared towards discovering and catalysing innovation in the field of law and justice.
OpenNyAI works with a diverse network of legal professionals, technologists and academic institutions. The goal is to encourage the integration of AI in the justice sector by developing the necessary tools and addressing the challenges posed by the process. By combining their network’s expertise, the partners within OpenNyAI are actively working towards establishing standardised testing methodologies currently lacking in Indigenous AI development, bridging a crucial gap in technology evaluation.
About the Project
Artificial Intelligence (AI) technologies are increasingly integrated into various sectors, including the legal field. RAG (Retriever Augmented Generation) systems are designed to enhance the accuracy of AI responses. AI often “hallucinates” or generates inaccurate answers. A RAG system comprises two main elements: a retriever and a generator. The retriever fetches information from an existing database, and the generator uses this information to generate a response. Since the database is narrow, an RAG system produces more accurate answers, reducing the possibility of hallucinations. This highlights the need for standardised testing methodologies, especially in Indigenous AI development contexts where such frameworks are absent. To address the challenges of AI evaluation, our program aims to create a comprehensive legal benchmarking dataset in India. This dataset will be used to test RAG systems on two key metrics: recall and accuracy. Recall measures the system’s ability to retrieve all relevant information, while accuracy assesses the correctness of the information retrieved. The creation of this dataset will involve a diverse cohort of students from various disciplines, including law, social sciences, gender studies, data sciences, and humanities. The goal is to foster a dataset that supports the development of Responsible AI by reflecting a wide range of perspectives and expertise.
The OpenNyAI mission is looking for volunteers to help in our legal dataset compilation. This dataset will be used to measure the accuracy of an AI framework: the RAG system.
Required
200 people
Mode
Remote
Duration
- 2-3 hours every weekday
- 20th July to 20th August
Who can Apply
- To participate in our annotating exercises, as a significant portion of the work requires legal aptitude. We are preferably looking for 2nd – 5th year law students.
- Including social sciences, gender studies, data sciences, engineering, and humanities. We aim to create a dataset that lends itself to Responsible AI, and we’re looking for as much diversity as possible.
Incentives
Participants in this program will benefit from weekly interactions with experts in different related fields through scheduled talks. They will receive certificates recognizing their contribution, and opportunities for future engagement with our communities, and participate in collaborative meetings to foster peer interaction and learning.
Program Details
Students are provided with different datasets, and a login to a portal that allows them to simulate questions, annotate parts of the document that contain the answers, and write out their answers. The Jugalbandi retriever is then run on the same question along with two other retrievers.
Students will have to compare the chunks of information pulled up by all three retrievers to determine their relevance to the question. Relevant chunks are annotated by the students, who then revise their answers based on the new information. These revised answers will be used as ground truths to measure the accuracy of the RAG.
How to Apply?
Deadline
15th July
Contact
Email: [email protected]; [email protected]
