It’s not that Americans shouldn’t be concerned about the National Security Agency.But the fact is we’ve known that we must have more oversight of federal security operations. In fact, the first major call for oversight of federal intelligence programs was a report by the Secretary of Defense Charles Wilson in 1956.

So oversight is long overdue—but we need to have more oversight of all federal, state and local spy operations. Some experts think there may be far more spying done on U.S. citizens by all non-federal agencies than we currently know about.

In his blog, David Simon, producer of The Wire, detailed how Baltimore cops ran “passive surveillance” on phone calls in Maryland no differently from what the NSA has been doing, and there hasn’t been a peep about it. Simon, like others, believes “passive surveillance” or general data gathering, is a safer police tool than “active surveillance,” i.e., listening to your phone calls, having goons follow you, etc.

And let’s face it, passive surveillance can do a lot of good: it was instrumental in cracking the Boston Marathon Terror Bombing case. Audio monitoring fights the illegal gun trade and there are cameras at many stores, ATMs, traffic lights—hell, even Coke did a cute Super Bowl ad on fun hidden footage.

But even “passive surveillance” poses dangers for personal freedom. All this official activity leads to something called “surveillance culture,” an ethos by which security authorities try to protect their system’s functions over public safety.

The fears over loss of liberty expressed in chilling works like 1984 or the classic 1974 film The Conversation are more dramatic than real—and the real danger of too much surveillance are seen in the film The Lives of Others or Kafka’s The Trial. In those works, the authorities, using active and passive tactics, abuse the system and terrorize citizens with commonplace, even humdrum—but not secret—information.

Abusing a surveillance system can occur at any level: recently, Florida transportation officials changed the timing on red-light cameras. That violated established safety standards, leading to more accidents—even though the point of traffic cameras is to make intersections safer. New timings also led to more tickets and $100 million raised for the state’s road program but officials say that had nothing to do with the change.

And no place in America best proves the dangers of this surveillance ethos than Commissioner Raymond Kelly’s New York City. His controversial “stop and frisk” policy has allowed police officers to forcibly search 1,800 residents a day for years.

At least that’s the number cops admit to: no records are kept and most frisks are not recorded. One teen who had the foresight to record one of his many frisks, was abused by cops who said they had the power to stop any “fucking mutt” they saw.

When ABC News asked why, in a city that’s 47 white, about 87 percent of those stopped and frisked were non-white, Kelly (pictured) said his data showed only 53 percent of the stop and frisks involved African-Americans and he believed lots of black people needed to be frisked lots more because they have a higher rate of violence.

But “stop and frisk” is only one part of it: Kelly bills his “Intelligence Division” as a major strike on the “war on terror” and even credits his staff as having saved America. Alas for Kelly, the FBI finds such claims half-baked and it questions his methods.

For instance, it was paying snitches to record conversations at mosques, social clubs—even coffee shops and kabob carry-outs—while keeping a database to harass people who made questionable comments. For many Muslims, silence in public is not just golden, but safer.

If Kelly’s supporters can’t make him mayor, they are pressuring every nominee to promise to keep Kelly on the job and even boost his $1.6 billion surveillance budget by finding more corporate donors. Citibank may have given New York a bike share program, but Microsoft is making it possible to watch every street corner.

A decade ago I undertook a research assignment in which I encountered the familiar history of how the FBI’s D.C. office spied on Martin Luther King and how Alabama’s FBI spied on the KKK. But I would soon discover an almost unknown story of how Bull Connor, who had Kelly’s job in Birmingham, Alabama, spied on and disrupted civil rights activities throughout the South.

Were it not for a few files someone neglected to burn in 1969, we’d never known how Connor ordered Birmingham city cops to wiretap Vice President Hubert Humphrey’s phone or how a police car was implicated in a terrorist bombing of King’s motel.

Today’s New York City may seem a far cry from Alabama in 1963.

Still, maybe someone should look at Kelly’s files.

Just in case.

Peter Lindstrom is a political consultant and researcher. He lives in Washington, D.C.