AI Finding Logical Fallacies

There is a new test how new LLMs: Gemma2, Qwen2 and Mistral Nemo - do in logical fallacy Detection, comparing with 10+ other models

1 Test for AI

• to detect informal logic errors

New gemma2 and qwen2 are good, but in their test phi3 showed the best result/

See for details how and what was tested https://www.glukhov.org/post/2024/08/gemma2-qwen2-mistral-nemo12b/

Here is the table with the test results

2 Their Conclusion

They expected LLMs to find in sample text the Strawman fallacy, Ad Hominem, Appeal to authority and Oversimplification. The text was from some journalist article about politics.

Most new models can see in our sample text Appeal to Authority, and some see Ad Hominem, some Oversimplification. Good results showed qwen2:7b-instruct-q8_0 and gemma2:9b-instruct-q8_0.

But still, winers remain the same.

• phi3:14b-medium-4k-instruct-q6_K
• phi3:3.8b-mini-4k-instruct-q8_0

and only then go qwen2:7b-instruct-q8_0 and gemma2:9b-instruct-q8_0