Early 2024, I purchased a new computer with advanced specs. My fairly limited goal was to take my journal’s text files and feed them into an LLM system.
I also had some dreams of building AI engines of my own based on that body of text. I would have no worries about my ownership rights to the text. It also would make any analysis and training more personalized. Doing it locally would also help with privacy concerns.
One resource that I found online are videos by the researcher Andrej Karpathy. The YouTube playlist he has compiled is Neural Networks: Zero to Hero. (I haven’t watched them all.)
My new computer has some advanced specs such as an upper-middle range Nvidia video card with 16GB GPU memory. I built up the MSI motherboard to the max of its RAM at 192GB. The GPU is a 12 core/24 thread AMD CPU. I’m glad I bought the system when I did because that much RAM is prohibitively expensive now.
I’ve done some experimentation using Ollama and found that I can run many of the Open Source LLMs including some with up to 80 billion parameters with a reasonable performance level. Not in the same performance ballpark as a commercial service, but still workable. Many of the models I have were released by AliBaba in their Qwen series.
By running the models locally, I don’t need to worry about the expense of using commercial servers to do my experiments. Instead, the biggest cost of my experiments is the amount of time it will take to iteratively apply my different configurations. That will require me to become more disciplined in how I proceed. My internet bandwidth becomes moot. In addition, I can combine different engines concurrently to play on each other’s strengths.
I’ve used the (free) version of Gemini so far to give me a lot of coaching and writing some simple processing scripts. I also use ChatGPT as a Python tutor. I need to experiment more with OpenAI’s Codex.