Emin Temiz PRO

Yes. I meant the case where parameters in the LLM does not match the retrieved knowledge.

Retrieved knowledge may say "the white of an egg is more beneficial", whereas LLM may have the opinion "the yolk". And eventually it may produce the yolk as the answer even though the context is full of white.

Google has a benchmark for this i think "FACTS Grounding". I don't agree with its name choice but it is relevant here.

I made an LLM act as an aggregator and combine answers of several other LLMs like in a mixture of agents scenario. The aggregator does not always produce the average or median answer. It brings its own opinion.

I think Google has a benchmark for this (sticking to context and not bringing its own words).

what is the best way to get <think> </think> and the tokens in between? openAI library is removing them.. i want to run llama-server in console and talk to it using a python library that does not remove the thinking tokens.

i checked the llama-cpp-python but it does not have that.

Update to the indicator

https://huggingface.co/posts/etemiz/490866222392672

This fine tuning would score 56 and be placed 1st in the leaderboard but I didn't add it, I only include full trainings in the leaderboard or (further tunings by the same company):

https://huggingface.co/CWClabs/CWC-Mistral-Nemo-12B-V2-q4_k_m