'%3e%3ccircle r='20' fill='%23008'/%3e%3ccircle r='17.5' fill='%23fff'/%3e%3ccircle r='3.5' fill='%23008'/%3e%3cg id='d'%3e%3cg id='c'%3e%3cg id='b'%3e%3cg id='a' fill='%23008'%3e%3ccircle r='.875' transform='rotate(7.5 -8.75 133.5)'/%3e%3cpath d='M0 17.5.6 7 0 2l-.6 5L0 17.5z'/%3e%3c/g%3e%3cuse xlink:href='%23a' transform='rotate(15)'/%3e%3c/g%3e%3cuse xlink:href='%23b' transform='rotate(30)'/%3e%3c/g%3e%3cuse xlink:href='%23c' transform='rotate(60)'/%3e%3c/g%3e%3cuse xlink:href='%23d' transform='rotate(120)'/%3e%3cuse xlink:href='%23d' transform='rotate(-120)'/%3e%3c/g%3e%3c/svg%3e){kind=small}
Clarification about Inference Time
Platform · 17 May 2025, 13:02 · 10
What does 'Inference must be less than 100ms per vignette' mean? Does this refer to the total time taken to generate the entire output, or just the time to generate the first token?
The whole time for a single inference.
is it possible for SLM to achieve those time constraints
I've been trying some things, quantization, ollama, ... and I don't see how it's possible, at least with qwen 0.6
The fastest I have achieved with T5 base is 280ms. A trial I am sure
T5 seems like a good choice. I think because the inference requirements, I'm not going to continue this contest.
Yes, I actually achieved a 135 ms today.
@Joseph_gitau did you achieve that on a local PC? Just curious, I've been testing performance in colab.
Yes, It's on my local pc
I achieved an inference speed of 75.7 ms today, I believe it's possible to stay Below the contraints. I might share the notebook for reference. Has a score of 0.35
Have shared the notebook under notebooks