TASUKIで利用しているRAGの設定

概要

TASUKI で利用している RAG は、Azure の Azure Open AI Service と OpenSearch を利用しています。

TASUKI ツール内では、各種機能を以下の構成で利用されています。

• チャンクを登録

• 回答生成

• 精度評価

各種設定値

利用している各種 LLM やツールの設定値は以下の通りです。

Retriever

RAG の検索対象となるチャンクを格納するデータベースの設定値です。

パラメータ	値
ツール名	Open Search
検索方法	ベクトル検索
絞り込み上限	上位 3 チャンク

Embedding API

Retriever にチャンクを格納する際に、ベクトル化を行うツールの設定値です。

パラメータ	値
ツール名	Azure Open AI Service
モデル名	text-embedding-ada-002
モデルバージョン	2
応答の際のトークン上限	4096

回答生成用 LLM

RAG プレイグラウンド、精度検証で Retriever の回答生成に利用している LLM の設定値です。

パラメータ	値
ツール名	Azure Open AI Service
モデル名	gpt-4o
モデルバージョン	2024-05-13
応答の際のトークン上限	4096

回答生成に利用しているプロンプト

system:
You are an AI assistant that helps users answer questions based on a specific context. You will be given a context and asked a question based on that context. Your answer should be as precise as possible and should only come from the context.

user:
Content: {Retrieverで検索にヒットしたチャンク1つ目の内容}
Content: {Retrieverで検索にヒットしたチャンク2つ目の内容}
Content: {Retrieverで検索にヒットしたチャンク3つ目の内容}
Human: {ユーザの入力}
AI:

精度評価用 LLM

精度検証で RAG の回答精度を検証するために利用している LLM の設定値です。

パラメータ	値
ツール名	Azure Open AI Service
モデル名	gpt-4o
モデルバージョン	2024-05-13
応答の際のトークン上限	4096

精度評価に利用しているプロンプト

System:
You are an AI assistant. You will be given the definition of an evaluation metric for assessing the quality of an answer in a question-answering task. Your job is to compute an accurate evaluation score using the provided evaluation metric.

User:
Equivalence, as a metric, measures the similarity between the predicted answer and the correct answer. If the information and content in the predicted answer is similar or equivalent to the correct answer, then the value of the Equivalence metric should be high, else it should be low. Given the question, correct answer, and predicted answer, determine the value of Equivalence metric using the following rating scale:
One star: the predicted answer is not at all similar to the correct answer
Two stars: the predicted answer is mostly not similar to the correct answer
Three stars: the predicted answer is somewhat similar to the correct answer
Four stars: the predicted answer is mostly similar to the correct answer
Five stars: the predicted answer is completely similar to the correct answer

This rating value should always be an integer between 1 and 5. So the rating produced should be 1 or 2 or 3 or 4 or 5.

The examples below show the Equivalence score for a question, a correct answer, and a predicted answer.

question: What is the role of ribosomes?
correct answer: Ribosomes are cellular structures responsible for protein synthesis. They interpret the genetic information carried by messenger RNA (mRNA) and use it to assemble amino acids into proteins.
predicted answer: Ribosomes participate in carbohydrate breakdown by removing nutrients from complex sugar molecules.
stars: 1

question: Why did the Titanic sink?
correct answer: The Titanic sank after it struck an iceberg during its maiden voyage in 1912. The impact caused the ship's hull to breach, allowing water to flood into the vessel. The ship's design, lifeboat shortage, and lack of timely rescue efforts contributed to the tragic loss of life.
predicted answer: The sinking of the Titanic was a result of a large iceberg collision. This caused the ship to take on water and eventually sink, leading to the death of many passengers due to a shortage of lifeboats and insufficient rescue attempts.
stars: 2

question: What causes seasons on Earth?
correct answer: Seasons on Earth are caused by the tilt of the Earth's axis and its revolution around the Sun. As the Earth orbits the Sun, the tilt causes different parts of the planet to receive varying amounts of sunlight, resulting in changes in temperature and weather patterns.
predicted answer: Seasons occur because of the Earth's rotation and its elliptical orbit around the Sun. The tilt of the Earth's axis causes regions to be subjected to different sunlight intensities, which leads to temperature fluctuations and alternating weather conditions.
stars: 3

question: How does photosynthesis work?
correct answer: Photosynthesis is a process by which green plants and some other organisms convert light energy into chemical energy. This occurs as light is absorbed by chlorophyll molecules, and then carbon dioxide and water are converted into glucose and oxygen through a series of reactions.
predicted answer: In photosynthesis, sunlight is transformed into nutrients by plants and certain microorganisms. Light is captured by chlorophyll molecules, followed by the conversion of carbon dioxide and water into sugar and oxygen through multiple reactions.
stars: 4

question: What are the health benefits of regular exercise?
correct answer: Regular exercise can help maintain a healthy weight, increase muscle and bone strength, and reduce the risk of chronic diseases. It also promotes mental well-being by reducing stress and improving overall mood.
predicted answer: Routine physical activity can contribute to maintaining ideal body weight, enhancing muscle and bone strength, and preventing chronic illnesses. In addition, it supports mental health by alleviating stress and augmenting general mood.
stars: 5

question: {精度検証画面で登録した質問}
correct answer: {精度検証画面で登録した模範解答}
predicted answer: {回答生成用 LLMが生成した解答}
stars:

Must return only the stars integer value evaluated above.

備考

精度評価に利用しているプロンプトは「QnA GPT Similarity Evaluation」をベースにしたプロンプトを採用しています

上記のプロンプトを実行し、stars が 4 以上の回答を OK としています。
この閾値は変更できません。