You only use present tense within dialog or if the story is being narrated from someones POV.

This is not true. Charles Dickens' Bleak House, for instance, is written with half the narrative in 1st person past, the other half in 3rd person present. Coetzee's Disgrace is also 3rd person present.

There really are no hard and fast rules here, I choose tense and POV based on situation, motive, and desired effect. I don't think I'm alone in this, as I've heard the same from others.