MiniCheck-RoBERTa-Large

Por favor, ejecute el siguiente comando para instalar el paquete MiniCheck y todas las dependencias necesarias.

pip install "minicheck @ git+https://github.com/Liyan06/MiniCheck.git@main"

A continuación, se muestra un caso de uso simple:

from minicheck.minicheck import MiniCheck
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
doc = "A group of students gather in the school library to study for their upcoming final exams."
claim_1 = "The students are preparing for an examination."
claim_2 = "The students are on vacation."
scorer = MiniCheck(model_name='roberta-large', cache_dir='./ckpts')
pred_label, raw_prob, _, _ = scorer.score(docs=[doc, doc], claims=[claim_1, claim_2])
print(pred_label)  # [1, 0]
print(raw_prob)  # [0.9581979513168335, 0.031335990875959396]

Prueba en nuestro Benchmark LLM-AggreFact:

import pandas as pd
from datasets import load_dataset
from minicheck.minicheck import MiniCheck
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
df = pd.DataFrame(load_dataset("lytang/LLM-AggreFact")['test'])
docs = df.doc.values
claims = df.claim.values
scorer = MiniCheck(model_name='roberta-large', cache_dir='./ckpts')
pred_label, raw_prob, _, _ = scorer.score(docs=docs, claims=claims)  # ~ 800 docs/min, dependiendo del hardware
# Evaluar el resultado en el benchmark
from sklearn.metrics import balanced_accuracy_score
df['preds'] = pred_label
result_df = pd.DataFrame(columns=['Dataset', 'BAcc'])
for dataset in df.dataset.unique():
   sub_df = df[df.dataset == dataset]
   bacc = balanced_accuracy_score(sub_df.label, sub_df.preds) * 100
   result_df.loc[len(result_df)] = [dataset, bacc]
result_df.loc[len(result_df)] = ['Average', result_df.BAcc.mean()]
result_df.round(1)