Fun fact: there's a name for the phenomenon of instantly recognizing the number of objects when it's fewer than five. It's called "subitizing."
There's a pretty interesting overview of what we know about math on the brain (or at least knew as of its writing) in the book "Where Mathematics Comes From" by Lakoff and Nuñez.
Your brain finds it easy to size up four objects but not five — here’s why (www.nature.com)
Neuron activity shows that the brain uses different systems for counting up to four, and for five or more....