this is the code: it is being used for both xnli and sentiment task

if task != "mmt":

with torch.no_grad():

logits = model(

**batch

).logits # Shape: [batch_size, seq_length, vocab_size]

log_probs = torch.nn.functional.log_softmax(

logits, dim=-1

) # Shape: [batch_size, seq_length, vocab_size]

# compute the log-likelihood of the target tokens

t_labels = (

torch.tensor([0, 1, 2])

.unsqueeze(0)

.unsqueeze(0)

.expand(

batch["input_ids"].size(0), batch["input_ids"].size(1), -1

)

.to(model.device)

)

# Gathering log-likelihoods for the labels

log_likelihoods_per_class = log_probs.gather(

2, t_labels

) # Shape: [batch_size, seq_length, 3]

# sum or average over the sequence to get a final score

log_likelihoods_per_class = log_likelihoods_per_class.mean(

dim=1

).cpu().numpy() # Shape: [batch_size, 3]

I will revert back to this tomorrow. But try checking the eval.py function and see how the evaluation is actually being performed. I believe the answer is there

Sure. Take your time. I checked the eval function, the lelapa is not performing very good. I think they have just done pre training

Hi ML-GOD,

Yes, InkubaLM is the pre-trained models