Add links to reference docs
This commit is contained in:
William FH
2023-07-18 01:50:14 -07:00
committed by GitHub
parent 3179ee3a56
commit c6f2d27789
8 changed files with 66 additions and 791 deletions
@@ -8,12 +8,12 @@
"source": [
"# Embedding Distance\n",
"\n",
"To measure semantic similarity (or dissimilarity) between a prediction and a reference label string, you could use a vector vector distance metric the two embedded representations using the `embeding_distance` evaulator.<a name=\"cite_ref-1\"></a>[<sup>[1]</sup>](#cite_note-1)\n",
"To measure semantic similarity (or dissimilarity) between a prediction and a reference label string, you could use a vector vector distance metric the two embedded representations using the `embedding_distance` evaluator.<a name=\"cite_ref-1\"></a>[<sup>[1]</sup>](#cite_note-1)\n",
"\n",
"\n",
"**Note:** This returns a **distance** score, meaning that the lower the number, the **more** similar the prediction is to the reference, according to their embedded representation.\n",
"\n",
"Check out the reference docs for the [PairwiseEmbeddingDistanceEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.embedding_distance.base.PairwiseEmbeddingDistanceEvalChain.html#langchain.evaluation.embedding_distance.base.PairwiseEmbeddingDistanceEvalChain) for more info."
"Check out the reference docs for the [EmbeddingDistanceEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.embedding_distance.base.EmbeddingDistanceEvalChain.html#langchain.evaluation.embedding_distance.base.EmbeddingDistanceEvalChain) for more info."
]
},
{
@@ -26,7 +26,7 @@
"source": [
"from langchain.evaluation import load_evaluator\n",
"\n",
"evaluator = load_evaluator(\"pairwise_embedding_distance\")"
"evaluator = load_evaluator(\"embedding_distance\")"
]
},
{
@@ -48,7 +48,7 @@
}
],
"source": [
"evaluator.evaluate_string_pairs(prediction=\"I shall go\", prediction_b=\"I shan't go\")"
"evaluator.evaluate_strings(prediction=\"I shall go\", reference=\"I shan't go\")"
]
},
{
@@ -70,7 +70,7 @@
}
],
"source": [
"evaluator.evaluate_string_pairs(prediction=\"I shall go\", prediction_b=\"I will go\")"
"evaluator.evaluate_strings(prediction=\"I shall go\", reference=\"I will go\")"
]
},
{
@@ -118,8 +118,9 @@
},
"outputs": [],
"source": [
"# You can load by enum or by raw python string\n",
"evaluator = load_evaluator(\n",
" \"pairwise_embedding_distance\", distance_metric=EmbeddingDistance.EUCLIDEAN\n",
" \"embedding_distance\", distance_metric=EmbeddingDistance.EUCLIDEAN\n",
")"
]
},
@@ -134,7 +135,7 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 6,
"metadata": {
"tags": []
},
@@ -143,12 +144,12 @@
"from langchain.embeddings import HuggingFaceEmbeddings\n",
"\n",
"embedding_model = HuggingFaceEmbeddings()\n",
"hf_evaluator = load_evaluator(\"pairwise_embedding_distance\", embeddings=embedding_model)"
"hf_evaluator = load_evaluator(\"embedding_distance\", embeddings=embedding_model)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 7,
"metadata": {
"tags": []
},
@@ -159,18 +160,18 @@
"{'score': 0.5486443280477362}"
]
},
"execution_count": 10,
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hf_evaluator.evaluate_string_pairs(prediction=\"I shall go\", prediction_b=\"I shan't go\")"
"hf_evaluator.evaluate_strings(prediction=\"I shall go\", reference=\"I shan't go\")"
]
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": 8,
"metadata": {
"tags": []
},
@@ -181,20 +182,20 @@
"{'score': 0.21018880025138598}"
]
},
"execution_count": 12,
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hf_evaluator.evaluate_string_pairs(prediction=\"I shall go\", prediction_b=\"I will go\")"
"hf_evaluator.evaluate_strings(prediction=\"I shall go\", reference=\"I will go\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a name=\"cite_note-1\"></a><i>1. Note: When it comes to semantic similarity, this often gives better results than older string distance metrics (such as those in the `PairwiseStringDistanceEvalChain`), though it tends to be less reliable than evaluators that use the LLM directly (such as the `PairwiseStringEvalChain`) </i>"
"<a name=\"cite_note-1\"></a><i>1. Note: When it comes to semantic similarity, this often gives better results than older string distance metrics (such as those in the [StringDistanceEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.string_distance.base.StringDistanceEvalChain.html#langchain.evaluation.string_distance.base.StringDistanceEvalChain)), though it tends to be less reliable than evaluators that use the LLM directly (such as the [QAEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.qa.eval_chain.QAEvalChain.html#langchain.evaluation.qa.eval_chain.QAEvalChain) or [LabeledCriteriaEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.criteria.eval_chain.LabeledCriteriaEvalChain.html#langchain.evaluation.criteria.eval_chain.LabeledCriteriaEvalChain)) </i>"
]
}
],