The honest rule is that people are not scenery. In busy tourist areas, wide market shots where no individual is the subject are generally fine, but the moment your lens focuses on one person, asking becomes the respectful default.
Context changes the calculus. In villages and traditional communities, photography can feel intrusive or even carry cultural or religious objections, so ask first every time. Photographing children requires a parent's permission, full stop. Some people in tourist hotspots pose for a living and will expect a small payment; either agree to that exchange or do not take the photo.
A few habits earn goodwill: show people the photo afterwards, buy something from a vendor you photographed, and never shoot someone in an embarrassing moment. In some countries privacy laws also restrict publishing identifiable photos without consent, which matters if you post to social media. When someone declines, thank them and move on.