Introduction

Recently, open source developers and proprietary vendors have been releasing open weight LLMs with permissive licensing. These models can run locally on a user's PC without sending private data to any vendor. There is no need to worry about token cost or quota. These coupled with the fact that LLMs now have incorporated chain-of-thought motivated me to try these models.

Models for the experiment

There is a plethora of open source/weight models to choose from. I tried the following models for the experiment:

  • gemma4:12b (Google's Gemma 4 with 12 billion parameters)
  • qwen3:14b (Alibaba's Qwen 3 with 14 billion parameters)
  • mathstral:7b (Mistral AI's mathstral with 7 billion parameters)
  • medgemma:4b-it-bf16 (Google's medgemma instruction-tuned model with 4 billion parameters)

I chose the above since they are a few of them suitable to run on my Laptop which has 32 GB of main memory and no GPU. Considering the amount of training data and the CPU/GPU resources needed to train these models, the folks who publish the open weight models deserve a lot of appreciation.

I would like to stress that it was not my intention to compare the performances of these models.

The experiment

First, I decided to put myself in the shoes of a high school/college student who is trying to solve some algebra and calculus problems. The rationale is that these problems require a number of steps and precise reasoning in each step that lead to the final solution. I thought they might be good candidates to test how these LLMs come up with the answers.

Secondly, I tested medgemma that was specifically trained on medical data including medical images and x-rays. This LLM was posed with typical questions that appear in Medical Board Examination taken by practicing doctors and medical residents. Again, many of the answers require a fair amount of reasoning.

For testing medgemma, MKSAP questions were taken from this file.

Key takeaways

  • I am quite impressed by the fact that terabytes of training data is locked into these models which take up only a few GB of memory. In simple terms, this is possible because the sum total of memory used for the dictionary of words, the parameters (weights), and the context is only a few GB. Myriads of text can be generated algorithmically (albeit non-deterministically) by using the neural network of the model. Starting with various input contexts, it is possible to unlock the training data from these models.
  • The floating point precision of the weights can be decreased (quantization) without sacrificing the performance of the LLM significantly. This reduces the main memory footprint.
  • The models run somewhat slower when there is no GPU (like my Laptop.) For most of the problems, all CPU cores were fired up with each CPU utilization in the range of 50% to 100%. The CPU cooling fan was continuously running.
  • The models generate markdown output. Interestingly, the mathematical expressions are formatted in LaTex math. They accept LaTex formatted math in the input as well.
  • Some of the symbolic mathematical operations took excessive CPU time. Perhaps, one can complement the LLMs with Model Context Protocol (MCP) to invoke sympy for symbolic math problems. I opted to take a shortcut and used function calls instead of MCP to invoke some sympy functions. The use of sympy as an external tool helped reduce the CPU time in some cases. It should be noted that mathstral does not support function calls.
  • Most of the math problems were answered correctly by the LLMs. For one problem, the LLMs were struggling to come up with a definite answer. When run multiple times, the answers were different (perhaps due to hallucination ?) For LLMs which support function calling, sympy came to the rescue except in one case where qwen passed the argument to the function incorrectly and failed.
  • medgemma was able to answer only a fraction (14 out of 25) of the questions correctly. Perhaps, models with more parameters may yield better performance.
  • Specialized LLMs ran faster than generic ones. For example, mathstral consistently ran faster than gemma and qwen since it was trained mostly on mathematical data. Similarly, for the same question "What is the amount of calcium in one cup of milk?" medgemma was able to answer faster than gemma (52 sec. versus 3 min. 23 sec. on my Laptop.)

If you are interested in the details of the experiment, you can look at my github repository.

Conclusion

Looking at the responses from the LLMs, it appears that all these models are not reasoning like a real human being. They seem to try non-deterministically different paths and verify that the answers are the same. In case there is a function that can be called, they try to take that path. As of now, for all these LLMs, human in the loop is necessary to validate the answers or make any critical decisions.

The open weight models open up a plethora of possibilities. Some of the models can be installed even on cell phones. Individual developers can create AI agents that run locally on user devices without sharing any private data with other entities. For example, see this and this blog articles. Coding agents are built using local models. For example, see this article.

Companies that are concerned about tokenmaxxing may shift to open weight models to cut down cost. For example, AT & T is shifting to open weight models.