When we speak about formal axiomatic systems, "just being able to do arithmetic" is not that "basic", really.

The thing is, there are infinitely many different natural numbers, and the ability to represent natural numbers along with some meaningful operations on them (which, by the way, requires at least 8-9 axioms, which is, in some sense, a lot) gives the formal system access to this "power of infinity". The power of infinity is not something to be taken lightly :)

In fact, you might notice at some point that among the sophisms that I post here, all of the "really tricky ones" (i.e. not those which are based on a camouflaged mistake) are in some sense related to infinity.