Quick summary: The government requests for user accounts info and user data just keep on coming, particularly in the U.S. They have doubled since 2010 — 25,900 requests in the first half of 2013 compared to 13,400 in the same period of 2010. And the US now makes up 42% of all requests, compared to 32% in 2010.

On the bright side, the percentage of U.S. requests that eventually led to some data being released has been steadily declining. It was 83% in the first half of this year and 94% at the end of 2010. The type of request that had the highest chance of leading to data being released: A wiretap order (100% in 7 cases). The least likely: A court order that wasn’t a subpoena or search warrant, with 69% of cases ending with data release.

And here’s a chart we at OTB put together to show how the authorities come knocking.

And now that Google has weighed in, here is a summary of all the major companies that are now issuing transparency reports.

An internal document released last week to MuckRock showed 3,382 FOIA requests between June 6 and Sept, 14 of this year — nearly 12 times the 293 filed in that same period a year ago.

The requests have leveled off somewhat from earlier in the summer when the first media leaks appeared, although they continue to be much higher than normal. For perspective, for all of FY12 we received only 1809 requests,” the NSA said in its memo.

Based on so-called “transparency” reports released so far this year from Microsoft, Yahoo, Twitter and Facebook, US agencies between January and the end of June made, on average, up to 183 requests per day across the four companies, for a total of 33,338 requests affecting 84,597 accounts.

Google, which has been releasing law enforcement request data longer than any major competitor, has yet to release its data for the first half of the year. Microsoft first released some data early last summer after former NSA employee Edward Snowden began leaking top-secret documents about government surveillance programs involving the major internet companies.

Turkey, Germany, the United Kingdom and France followed the U.S. in the No. 1 spot for requests from Microsoft. Together, the five made up 3 in 4 of the global 37,196 requests affecting 66,539 accounts.

Just over 1 in 10 U.S. requests led to user content being released. In 2 out of 3 cases, at least some user account information such as name, gender and Zip Code was turned over. Only about 1 in 100 was rejected for not meeting legal requirements. And in just under 1 in 20 cases — 17% — no customer data was found.

“The overwhelming majority of law enforcement requests seek information related to our free consumer services used by individuals in their personal capacity such as: web-mail accounts (Hotmail/Outlook.com), SkyDrive cloud storage; Messenger, and Skype,” Microsoft said.

The company said it received more requests from foreign countries than its competitors because it has so many official offices in about 100 countries.

Microsoft’s data did not include “National Security Letter” requests, for which the government puts strict restrictions on numbers and data that can be released, and over what time period. It does include in its report that aggregate, rounded data from earlier years.  Many companies, Microsoft included, are publicly challenging the restrictions put on the national security data by the secretive Foreign Intelligence Surveillance Court.

Microsoft Summary for US, Jan-June 2013

Microsoft graphic

Most of those cases resulted in the government getting at least some data, including e-mails, photos and uploaded files.

Friday’s was Yahoo’s first-ever transparency report and it says it will continue every six months. Twitter and Microsoft released their reports earlier this summer; Google, which has been releasing reports longer than the other major players, has yet to for the first half of 2013. (Related: The Washington Post reported this morning that Google has stepped-up its efforts to encrypt that data that moves between its servers in an attempt to thwart spying).

“Democracy demands accountability, and accountability requires transparency,” Yahoo General Counsel Ron Bell wrote in a blog post on Friday. “We hope our report encourages governments around the world to more openly share information about the requests they make for users’ information.”

Yahoo reported that in about 8% of cases, either no data was found or Yahoo rejected the request. So in just over 9 in 10 cases, at least some data was turned over.

About half the time — 55% — that was “non-content data,” which Yahoo describes as “basic subscriber information including the information captured at the time of registration such as an alternate e-mail address, name, location, and IP address, login details, billing information, and other transactional information (e.g., “to,” “from,” and “date” fields from email headers).”

In nearly 2 in 5 cases, other content was turned over. Yahoo’s description: “Data that our users create, communicate, and store on or through our services. This could include words in a communication (e.g., Mail or Messenger), photos on Flickr, files uploaded, Yahoo Address Book entries, Yahoo Calendar event details, thoughts recorded in Yahoo Notepad or comments or posts on Yahoo Answers or any other Yahoo property.”

Yahoo reported about the same number of government requests as Facebook, but affecting substantially more user accounts — 40,300 vs. up to 21,000. Twitter reported 902 requests affecting 1,319 accounts for the first half of the year.

Germany, Italy, Taiwan and France filled out the Top 5 in number of requests after the U.S. Outside the U.S., requests totaled 17,026, involving 22,453 accounts.

The government requests usually involve criminal investigations and came come by way of warrant or subpoena. Yahoo says it only complies “in response to valid, compulsory legal process from a government agency with proper jurisdiction and authority.”

National security authorities also make the requests. All companies are restricted about how much they can say — even specific numbers — about requests under the Foreign Intelligence Surveillance Act. Yahoo and others have been pressing the government to allow them more freedom to divulge those details.

→ Earlier stories on transparency report data.

Countries that made the most data requests of Facebook in the fist half of 2013.

The report covers the first half of 2013 and says the U.S. made between 11,000 and 12,000 requests involving between 20,000 and 21,000 users or accounts. Facebook complied with 79% of those requests. The range for the U.S. vs. a specific number for other countries is believed to be because of U.S. requirements that requests involving national security can only be released in ranges. Facebook and several of its rivals are urging the government to allow specific numbers be released instead of ranges.

“We continue to push the United States government to allow more transparency regarding these requests, including specific numbers and types of national security-related requests. We will publish updated information for the United States as soon as we obtain legal authorization to do so,” Facebook General Counsel Colin Stretch said in releasing the data.

Facebook’s data seems to indicate a significant drop-off in the number of users or accounts from the second half of 2012. In first-ever data it released after the PRISM program details were first leaked by Edward Snowden earlier this year, it said it had received 9,000-10,000 requests involving 18,000-19,00 users or accounts between July 1 and Dec. 31, 2012. (See a table that summarizes data reports from a variety of companies).

Globally, Facebook said it has up to 26,607 requests involving 38,954 users in about 70 countries. The average compliance rate was 33%. India (3,245 requests and 4,144 users/accounts) and the United Kingdom (3,245/4,144) were No. 2 and No. 3 behind the U.S.

“The vast majority of these requests relate to criminal cases, such as robberies or kidnappings,” Facebook’s Stretch said. “In many of these cases, these government requests seek basic subscriber information, such as name and length of service. Other requests may also seek IP address logs or actual account content. We have strict guidelines in place to deal with all government data requests.”

Facebook’s compliance rate is higher than Twitter’s (it reported 67% a few weeks ago) and is about the same as Microsoft, which reported about 80% for 2012, and is lower than Google’s most recently reported 88%, which itself was down from 94%.

→ Earlier stories on transparency reports.

ON THE JUMP: A sortable chart with all the Facebook data.

For those who’ve been in eyes-half-open mode because it’s Summer and want to catch up, The New York Times today has a must-read on the politics and cross-party partnerships behind last week’s surprising oh-so-close vote in the House that would have killed funding for the National Security Administration’s telephone data collection, exposed by Edward Snowden.

Two quotes that stand out:

“There is a growing sense that things have really gone a-kilter here.”

“The time has come to stop it, and the way we stop it is to approve this amendment.”

The first, from U.S. Rep. Zoe Lofgren, D-Calif., who is helping craft a bill “to significantly rein in N.S.A. telephone surveillance,” the NYT story says.

The second is from someone who was a force behind the Patriot Act and who is helping Lofgren on that bill. — U.S. Rep. Jim Sensenbrenner, R-Wisc. His very brief (and unexpected) remarks — seven sentences — before the House vote last week were seen as pivotal.

Meantime, new research data shows the opinion shift among Americans. Almost half of those polled from July 17-21 said their “greater concern about government anti-terrorism policies is that they have gone too far in restricting the average person’s civil liberties,” Pew Research Center for the People & the Press reported on Friday.

That was a 15-point jump since 2010, when the question was last asked. “This is the first time a plurality has expressed greater concern about civil liberties than security since the question was first asked in 2004.”

Nearly 3 in 5 of those adults surveyed — 56% — said courts aren’t providing the safety net needed for government data collection. “An even larger percentage (70%) believes that the government uses this data for purposes other than investigating terrorism,” Pew said.

Politically, a clear lean toward more support for civil liberties vs. national security since 2010 is pretty much across the board, as the chart below shows. Particularly noteworthy is the shift in the civil liberties v. national security view by Tea Party Republicans.

For the media, the public remains divided over how aggressive journalists should be in reporting on secret methods used to fight terrorism — 47% yes, 47% no. That’s the same as 2006. However, there has been a partisan shift, as Republicans now are much more supportive of an aggressive media role (+17 points), while Democrats have retreated (-14 points). See chart below.

In a column today, the Guardian’s Glenn Greenwald, whose release of Snowden’s inside information along with the Washington Post detonated the current NSA and civil liberties controversy, took note of the shift in the Venn diagram of civil liberties and national security that Pew’s survey revealed.

As I’ve repeatedly said, the only ones defending the NSA at this point are the party loyalists and institutional authoritarians in both parties. That’s enough for the moment to control Washington outcomes – as epitomized by the unholy trinity that saved the NSA in the House last week: Pelosi, John Bohener and the Obama White House – but it is clearly not enough to stem the rapidly changing tide of public opinion.

(Note: If you’re interested, Pew has details on how it conducted its survey).

Under the programs, “Postal Service computers photograph the exterior of every piece of paper mail that is processed in the United States — about 160 billion pieces last year. It is not known how long the government saves the images.

“Together, the two programs show that snail mail is subject to the same kind of scrutiny that the National Security Agency has given to telephone calls and e-mail.”

Under mail covers, When asked by law enforcement, postal officials track a persons mail for a period of time. The post office rarely turns down a request, the Times says, and includes “tens of thousands” of items a year.

Mail Isolation Control and Tracking was born of the anthrax scare a few years ago. It photocopies the outside of mail — 160 billion pieces last year alone, the Times said.

“In the past, mail covers were used when you had a reason to suspect someone of a crime,” said Mark D. Rasch, the former director of the Justice Department’s computer crime unit, who worked on several fraud cases using mail covers. “Now it seems to be ‘Let’s record everyone’s mail so in the future we might go back and see who you were communicating with.’ Essentially you’ve added mail covers on millions of Americans.”

Bruce Schneier, a computer security expert and an author, said whether it was a postal worker taking down information or a computer taking images, the program was still an invasion of privacy.

“Basically they are doing the same thing as the other programs, collecting the information on the outside of your mail, the metadata, if you will, of names, addresses, return addresses and postmark locations, which gives the government a pretty good map of your contacts, even if they aren’t reading the contents,” he said.

→ Full NYT story

Twitter CEO Dick Costolo visits with ASNE in Washington (Photo from C-SPAN video).

Twitter CEO Dick Costolo joined us for lunch on Wednesday at the American Society of News Editors conference in Washington and talked about journalism and its intersection with Twitter (and vice-versa); his company and its culture; privacy; and a few other topics.

But when it came to one hot topic in DC of late and  why Twitter wasn’t included on the now-famous PowerPoint slide about companies the NSA has relationships with for the top-secret PRISM user data collection program, Costolo pretty much had precisely 140 characters of nothing specific to say. With a hashtag of #mumstheword.

Asked  by Marty Baron of the Washington Post whether Twitter was “invited or instructed by the federal government to participate in this program, whether you chose to turn the government down, and if you did that based on legal objections, what were those legal objections,” Costolo wouldn’t take the bait.

He did, however, say Twitter is very aggressive and takes a “principled approach” when it comes to pushing back against government requests for large amounts of user data.

“When we get specific, pointed legal requests that are legally valid. . . we respond to them,” he said in reply to Baron’s question. “When we receive general requests that we feel are overly broad, not valid legal requests, we push back on those. I think it’s fair to say as has been reported in other cases like Wikileaks, we will spend time and energy and money to defend out users’ rights to be informed about the information that is being requested about them. . . That’s really all i can say about it.”

Earlier in his visit, interviewer Cecilia Kang of the Washington Post said, “It feels like we’re sort of dancing around this thing that’s all in caps called PRISM. You can’t talk so much about it.  . . .  Does it bother you you can’t talk more about your relationship with the government and these sorts of requests?”

His repsonse was much the same as what he told Baron.

He did note that he has talked publicly about a rule in the UK that makes it illegal to even disclose there’s an injunction in some situations, and called such rules “Kafkaesque. . . . Those kinds of things are generally disturbing and we have called for and would love to see more transparency around these types of requests.”

I’ve extracted the PRISM-related discussion from the hourlong Costolo chat (but the content isn’t available yet for embedding). View it here:

• Part I (Kang’s questions)
• Part II (Baron’s question)
• Complete appearance