| --- |
| |
| pipeline_tag: text-classification |
| inference: false |
| language: es |
| tags: |
| - transformers |
|
|
| --- |
| |
| # Prompsit/paraphrase-roberta-es |
|
|
| This model allows to evaluate paraphrases for a given phrase. |
|
|
| We have fine-tuned this model from pretrained "PlanTL-GOB-ES/roberta-base-bne". |
|
|
| Model built under a TSI-100905-2019-4 project, co-financed by Ministry of Economic Affairs and Digital Transformation from the Government of Spain. |
|
|
| # How to use it |
|
|
| The model answer the following question: Is "phrase B" a paraphrase of "phrase A". |
|
|
| Please note that we're considering phrases instead of sentences. Therefore, we must take into account that the model doesn't expect to find punctuation marks or long pieces of text. |
|
|
| Resulting probabilities correspond to classes: |
|
|
| * 0: Not a paraphrase |
| * 1: It's a paraphrase |
|
|
| So, considering the phrase "se buscarán acuerdos" and a candidate paraphrase like "se deberá obtener el acuerdo", you can use the model like this: |
|
|
| ``` |
| |
| import torch |
| |
| from transformers import AutoTokenizer, AutoModelForSequenceClassification |
| tokenizer = AutoTokenizer.from_pretrained("Prompsit/paraphrase-roberta-es") |
| model = AutoModelForSequenceClassification.from_pretrained("Prompsit/paraphrase-roberta-es") |
| |
| input = tokenizer('se buscarán acuerdos','se deberá obtener el acuerdo',return_tensors='pt') |
| logits = model(**input).logits |
| soft = torch.nn.Softmax(dim=1) |
| print(soft(logits)) |
| |
| ``` |
|
|
| Code output is: |
|
|
| ``` |
| |
| tensor([[0.2266, 0.7734]], grad_fn=<SoftmaxBackward>) |
| |
| ``` |
|
|
| As the probability of 1 (=It's a paraphrase) is 0.77 and the probability of 0 (=It is not a paraphrase) is 0.22, we can conclude, for our previous example, that "se deberá obtener el acuerdo" is a paraphrase of "se buscarán acuerdos". |
|
|
|
|
| # Evaluation results |
|
|
| We have used as test dataset 16500 pairs of phrases human tagged. |
| Metrics obtained are: |
|
|
| ``` |
| metrics={ |
| 'test_loss': 0.4869941473007202, |
| 'test_accuracy': 0.8003636363636364, |
| 'test_precision': 0.6692456479690522, |
| 'test_recall': 0.5896889646357052, |
| 'test_f1': 0.6269535673839184, |
| 'test_matthews_correlation': 0.49324489316659575, |
| 'test_runtime': 27.1537, |
| 'test_samples_per_second': 607.652, |
| 'test_steps_per_second': 09.003 |
| } |
| |
| ``` |
