Hello @avt_nyanja @Zindi

What is the essence of CPU inference? First you are going to get extremely slow responses, second you have to quantize your models leading to accuracy loss.

Case study based on my experiments:

• 8bit quantized model: Approximaltely 1hr per input on google collab cpu using llama.cpp
• Original model(unquantized) : on google collab free tier gpu: 500 questions , aproximately 2hrs
• Do the maths: cpu inference on two questions is approximately 500 questions on free tier collab
• So that means for 8 bit quantized model to finish inference it is going to take 500 hrs with that estimate of 1hr per one question. 500/24 thats nearly 20 days to finish inference. What!!! How are these models going to be usable or was this a research project where these models won't be used in production?

Okay, the other option quantize to 4bit. In as much as it reduces the inference time, we get to 1-3 days inference for 500 questions.

Imagine sacrificing 2hrs of gpu inference to atleast 1day inference on a 4 bit quantized model and also hurting accuracy. Top that off with these inference platforms not being mature enough. E.g Ollama which is one of the most popular platforms for cpu inference getting stuck after some runs:

Ollama stuck after few runs · Issue #1863 · ollama/ollama (github.com) which is still an open issue

So I ask the question again. What is the essence of cpu inference????

N/B: Note that these numbers are based on my experiments. So someone else can chip in the discussion with their numbers since we are all not using the same approaches

@Reacher @DanielBruintjies @AdeptSchneider22 @Saifdaoud Are you facing similar challenges on cpu inference?