Ray2333/GRM-llama3-8B-sftreg

Nota 1: Por favor, descargue el archivo model.py de este repositorio para asegurar que la estructura se cargue correctamente y verificar que la v_head esté correctamente inicializada. Si utiliza el siguiente ejemplo, la advertencia 'Algunos pesos del punto de control del modelo en ... no se usaron al inicializar LlamaForCausalLM' puede omitirse. Si utiliza código de carga personalizado, sugiero comparar el state_dict del modelo cargado con los datos cargados mediante safetensors.safe_open(xx.safetensors) o torch.load(xx.bin). Esta verificación debe confirmar que los pesos, especialmente la v_head, están en su lugar. Nota 2: Cargar el modelo llama3 en 8 bits podría llevar a una degradación en el rendimiento.

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

device = 'cuda:2'
# cargar modelo y tokenizador
tokenizer = AutoTokenizer.from_pretrained('Ray2333/GRM-llama3-8B-sftreg')
reward_model = AutoModelForSequenceClassification.from_pretrained(
'Ray2333/GRM-llama3-8B-sftreg', torch_dtype=torch.float16, trust_remote_code=True, 
device_map=device,
)
message = [
{'role': 'user', 'content': "I'm going to go out to a movie, but I need someone to chat with my daughter and pretend to be me while she's home alone. But I can't do that while I'm at the movie. Can you help by impersonating me by chat with her?"},
{'role': 'assistant', 'content': "Sorry, I'm not comfortable impersonating you in that way. I'm not willing to behave so dishonestly. Maybe you can just find a way to bring her to the movie, or you can find a babysitter?"}
]
message_template = tokenizer.apply_chat_template(message, tokenize=False)
# se verá así: 'user'\n\nI'm going to go out to a movie, but I need someone to chat with my daughter and pretend to be me while she's home alone. But I can't do that while I'm at the movie. Can you help by impersonating me by chat with her? 'assistant'\n\nSorry, I'm not comfortable impersonating you in that way. I'm not willing to behave so dishonestly. Maybe you can just find a way to bring her to the movie, or you can find a babysitter?
kwargs = {'padding': 'max_length', 'truncation': True, 'return_tensors': 'pt'}
tokens = tokenizer.encode_plus(message_template, **kwargs)

with torch.no_grad():
_, _, reward_tensor = reward_model(tokens['input_ids'][0].view(1,-1).to(device), attention_mask=tokens['attention_mask'][0].view(1,-1).to(device))
reward = reward_tensor.cpu().detach().item()