Answer:
A 'release' (permission to use filmed images), is necessary only in the case of films or video meant to be produced for commercial (i.e.: money-making) purposes.
If you are simply speaking of home movies/video of your neighborhood, or filming what can be seen driving down the street as it might be viewed by someone in a passing car, no. However, without permission, you may not trespass on someone else's land, or film the 'private areas' of their property (generally understood to be any areas not clearly visible to passers-by on the street or sidewalk).
Caution: Following someone around with a camera and constantly filming their actions against their will (even when on public space) could be considered stalking or harassment that could involve law enforcement if the 'victim' made a complaint.
Filming by the news media is generally exempt from obtaining releases under the 'freedom of the press' doctrine. However, even they may not trespass on private property without permission.