Fun fact: there's a name for the phenomenon of instantly recognizing the number of objects when it's fewer than five. It's called "subitizing."
There's a pretty interesting overview of what we know about math on the brain (or at least knew as of its writing) in the book "Where Mathematics Comes From" by Lakoff and Nuñez.