the RIGHT to SHOOT

(beginning of sarcasm) Perhaps they're afraid that you're scoping out the place, so that when you break out the ninja skills, you'll know where everything is and where everyone was. (end of sarcasm)

Seems kind of BS that you can't take a picture of someone or something. If it's in the public view, then what's the point? I mean... what's next, they're going to stop you from looking in that general direction? I wonder what would happen if you started to draw on site, if they'd stop you then.

People will be people (illogical and apathetic)

Too bad this happens, but in this case I'm assuming you were on private property. They're probably worried about someone who has not so good intentions taking photos. Even though your intentions are artistic, and mean no harm, someone who does mean harm could claim the same thing.
You could try to talk with someone who is in a position to grant you permission and explain your an artist, and you would like to do a painting of their property, and show them some work you have done in the past. Maybe you could turn this into an opportunity to show your work there, or a commission. It's probably a long shot, but you never know.