Horses and other mounts are valuable commodities, mostly used by the wealthy for long travels. That means they are not often used on short distances or by the poor.
They are also a rare sight in combat; after all, if you've traveled 3 days by horseback, getting your horse injured or killed would mean a long, long hike back.