Comment on How LLMs can be Assisted to do Arithmetic Correctly by Steve

In reply to KenN.

You'd fire someone because they couldn't do arithmetic in their head? That's effectively what asking an LLM to do arithmetic is. I was playing around with using a memory system to replace the whole CoT process and it was very interesting. I gave it the problem of multiplying two 10-digit numbers by doing multiplication the way I was taught in grade school. It was quite capable of correctly multiplying a 10-digit number by a 1-digit number, with 100% accuracy. It was then capable of recording the partial product in the memory system, then go to the next digit, shift and add, until it got the correct result at the end of the process. It was clearly not capable of doing that multiplication in its "head" (it could produce an answer, but it wasn't right). You wouldn't expect a person to do that in their head, you'd expect them to use scratch paper to record partial answers and add them together, the same way this LLM was able to do. Then I gave it access to a calculator program and it did it a whole lot faster. Same as a person. I don't think LLMs are sentient or intelligent, but they are more than just producing the statistical best word, once you get past the early basic models and classifiers. There is some interesting emergent stuff going on, dismissing it as nothing is naive.