Taking Public Photos

I was recently in a downtown park and started to take a shot of several small children (ages 3-5)who were feeding a rather large flock of birds. The woman nearby shouted," i prefer you didn't take pictures of the children."
I didn't take the picture, but it got me thinking just what were the rules for taking photos in a public place. I would assume that if a person(no matter the age)is in a public place they are fair game. Then there's also the issue of common courtesy. I can sorta see why, in this day and age, someone might not want their children photographed. Anyway, I would like to hear comments and ideas on how best to handle this issue. Thanx!