NSA Declassifies and Releases the Friedman Collection

We welcome guest blogger, Dr. Dave Sherman, Associate Director for Policy and Records at the National Security Agency, who describes below details about NSA’s recent accomplishment: the declassification review and release of records in the Friedman Collection.

During the past two weeks I’ve had the privilege of participating in a series of events marking the National Security Agency’s declassification and public release of the official papers of an individual who, were I asked to name a single person as the founder of that agency, deserves the title more than anyone else:  William Friedman.

On Monday, April 20, 2015, NSA released over 50,000 pages (or 7,000 records) of Friedman’s official papers to the public.  These newly declassified records are now available at the National Archives and Records Administration.   NSA also posted digital copies of the entire collection to its website and provided electronic copies to NARA and to the Marshall Foundation, which holds Friedman’s personal papers.  On Thursday, April 23, 2015, the Foundation hosted a symposium marking the public release and featuring speakers from both government and academia.   It also opened an excellent exhibit on the life and work of William Friedman and his wife Elizebeth – a renowned codebreaker in her own right – with numerous photographs, papers, and artifacts from its collections.   Five days later, the National Cryptologic Museum, located on NSA’s Fort Meade campus, cut the ribbon on its own exhibit honoring the Friedmans.

Friedman served in almost every one of the signals intelligence agencies which preceded NSA and at NSA itself.  He is remembered to have brought the discipline of a scientist to the making and breaking of codes and ciphers, or cryptology, a field previously left mostly to inspired amateurs and some less than noble dilettantes.   After the first U.S. codebreaking organization, the American Black Chamber, was shut down in the 1920s, Friedman carried on as the Army’s lone cryptologist.  Then, in the early 1930s, he founded the Signals Intelligence Service…with an initial staff, including Friedman himself, of six!  The SIS grew slowly throughout the 1930s, but its small size did not keep it from achieving one of the most significant feats in the history of cryptanalysis:  the breaking of Japan’s “Purple” cipher, which that nation used to protect its most sensitive diplomatic communications.  But Friedman and his team were not just codebreakers.  They also were code makers,  devising an encryption machine – SIGABA – which, unlike the German Enigma or the Japanese Purple systems, protected Allied communications through World War II and beyond without ever having been cracked…at least not that we know of.

It took NSA almost two years of work to get to the point where we were ready to publicly release the Friedman papers.  Here’s what I’ve learned in the process.

First, the declassification and public release process looks easier than it is.  What could be simpler than getting out a few thousand pages of records from (mainly) the 1950s and 1960s, right?  I would be the first to agree that the government, and especially NSA, needs to get more historically significant information into the public domain more quickly, and in a more complete and comprehensive form.  On the other hand – and I can understand why the historical community would have difficulty believing this – there remains some information that must remain secret to protect certain intelligence sources and methods still active and relevant today, even for example from the early Cold War period.  Just enough, in fact, that for a collection covering the wide range of topics which the Friedman collection encompasses we had to review much of it page-by-page to ensure we inadvertently did not endanger such activities.   But – and this is my first lesson – what we must not do is allow the complexity of the task to deter us from undertaking it.  As Public Interest Declassification Board member Bill Leary observed at the National Declassification Center’s recent public forum, declassification projects which are the most challenging to undertake are also likely to be the most historically relevant.  They should be at the top of our to-do list, and we also need to find new ways to do them quickly and comprehensively.

The second thing I’ve learned is that there is no single approach that will succeed for every declassification effort.   For the Friedman project, for example, we made a decision early that in addition to making the paper originals available at NARA we also would release the entire collection in digital form.  We did this to ensure the broadest public access to the records.  Other efforts have taken this route as well, including a smaller NSA release on Vietnam POWs and MIAs last year.  However, it remains to be seen whether our process would prove feasible for projects with millions of pages.  We also decided to conduct a line-by-line declassification review of the records, redacting still secret information when required.  More time-consuming than a simple “pass-fail” declassification review, our objective was to tell as complete a story as possible without compromising national security or fostering historical misinterpretation.  As a result, roughly 85% of the Friedman collection was released in full.  Understandably – and reasonably, in my view – we’ve been criticized a bit for our redactions, on the grounds that any excisions from a record increase the likelihood of its significance being misinterpreted.  That criticism is not unfair.  It also, at a minimum, reminds us in government to wield the redaction knife – or, as it was more colorfully described to me recently, meat cleaver – with care.  We strove to minimize the redactions on this project.  However, this approach may not be suitable for other projects.  And those of us who resort to it would be well advised to institute checks against the risk of our enabling historical misinterpretation, inadvertently or otherwise.

Finally, I’ve come to an even greater appreciation than before that declassification is a team sport and partnerships – both within the government and between the government and the public – are critical to success.   The more obvious reason for this is that, with the increasing integration of the Intelligence Community over time, many of our records contain classified information which falls under the jurisdiction of multiple agencies.  That is absolutely necessary as a matter of analytic tradecraft in order to provide the policymaker or military commander with the most comprehensive and authoritative intelligence possible.   To be effective, then, declassification will require increasing use of cross-agency teams, both at the National Declassification Center and elsewhere.  But partnerships with public institutions and individual researchers are equally important.  This is not just because, in the final analysis, the government’s records are the public’s records.   Public institutions and individuals also provide essential steerage in helping those of us in government identify records of highest interest.  They also hold their own records collections, ones which – as we have seen in the case of the Friedman release – when augmented by government materials create a full and complete documentary record ripe for historical interpretation and public understanding.  And, finally, the public, academia, and the advocacy community can and must hold us in government accountable to ensure the soundness of our declassification processes and their outcomes.

Those of us at NSA who have been involved in the public release of the Friedman collection are proud of the results.  But we are even more gratified to have learned what we have along the way.  We realize that there is much more left to do.  We also know that there is always more we can and will learn about how to make our records available to the public in ways that do not harm our nation’s security but do increase public understanding and accountability, something which is now more than ever critical to sustaining an intelligence service in a democratic society.


Posted in Uncategorized

NDC Outlines Prioritization Plan

On Friday, April 10, 2015, the National Declassification Center held a public forum, NDC Prioritization: What Secrets Do People Want to See? to discuss prioritization of its holdings as a way forward since the completion of the 351 million page backlog in February 2014.

The public forum featured remarks from the Archivist of the United States, David Ferriero, and commentary from the Director of the NDC, Sheryl Shenberger.  In her remarks, Ms. Shenberger outlined the five goals of the NDC moving forward (in no particular order):

  1. Making sure another backlog of records awaiting declassification review never accumulates at the National Archives again,
  2. Increasing public access to previously reviewed and exempted records by focusing on document-level referral review at the Interagency Referral Center,
  3. Standing up a review process for the earliest withdrawn items, particularly those withdrawn before the NDC began using a computerized data capture system in 2002,
  4. Fostering an improved and more direct relationship with researchers at the National Archives and
  5. Continuing to prioritize series of records based on researcher/government requests, the significance of the historical topic and the quality of the earlier review to provide special historical themed collections.

David Langbart, Senior Archivist, provided his thoughts on topic-based prioritization as it relates to archival processing and description.  Supervisory Archivist Martha Murphy was also a presenter, discussing how the National Archives is currently processing the remaining withheld records related to the JFK Assassination.  You can view more information about the work of the National Archives and the processing of the JFK assassination records here.

PIDB member and Acting Chair, William Leary, participated as a member of a panel discussing prioritization for declassification.  In his remarks, Mr. Leary discussed the six recommendations made by the PIDB in its 2014 supplemental report, Setting Priorities: An Essential Step in Transforming Declassification.  In this report, the PIDB advocates for a coordinated, government-wide approach to declassifying information based on those records most sought after and of most historical significance to the public.  To this end, Mr. Leary discussed the need for experts in the declassification community and requester community to work cooperatively to determine how to set priorities, acknowledging that useful models exist already, including that which drives the review of records for inclusion in the Foreign Relations of the United States series (FRUS).  He noted that ending pass/fail declassification determinations, which inevitably lead to wasteful re-reviews of records, should be a part of the adopted model for prioritization.

Mr. Leary discussed the need for improved records and information management practices.  He successfully argued that the NDC has the ability to incorporate topical declassification without compromising archival principles, including those related to provenance and the idea of original order.  Indeed, selecting topics by series as priorities is feasible, practical and in concert with archival processing.  Mr. Leary noted that this is one way, among many ways, to improve public access to high-value records, noting that agencies will need to use better risk management strategies and eliminate or severely restrict review of specific records found to have little value.  He also discussed the importance of prioritizing Presidential records as these are arguably the most complete and accurate source of information about our nation’s history and role in the world.

Mr. Leary’s remarks during the forum reflect the PIDB’s shift in focus from the quantity of records reviewed to the quality of records declassified.  Moreover, the challenges posed by electronic records and the volume of information the government now creates mean that changes in declassification processes, from a variety of standpoints, are necessary to effectively transform the system to one that is sustainable in the digital age.

To view the entire NDC public forum online, please visit https://www.youtube.com/watch?v=ABsaEa9v4ik  (best viewed via Chrome browser).

The panel discussion featuring Mr. Leary begins at 32:00, and Mr. Leary’s specific remarks begin at 1:17:00.

For more information about the NDC public forum and comments from the Director of the NDC, please visit the NDC blog.

Please continue to follow our blog, Transforming Classification, to learn more about the PIDB’s recommendations concerning prioritization.

Posted in Uncategorized | 1 Comment

PIDB Member William Leary to Participate in NDC Public Forum on Prioritization

We are pleased that PIDB member William (Bill) Leary will participate as a panelist at the National Declassification Center’s next public forum.

The forum’s theme is NDC Prioritization: What Secrets Do People Want to See?  This is an excellent opportunity for Mr. Leary to discuss the PIDB’s recent supplemental report, Setting Priorities: An Essential Step in Transforming Declassification, and offer commentary on the six recommendations in the report that support the need for new declassification policies that include topic-based declassification.

The forum will be held on Friday, April 10, 2015 from 10:00 a.m. until 12:00 p.m. in the William G. McGowan Theater at the National Archives Building (700 Pennsylvania Avenue, NW, Washington, DC).  We encourage you to attend the forum, which is free and open all who are interested in access to historical records(enter via the Special Events entrance on Constitution Ave and 7th street, NW).  You can find more information on the forum here.

Archivist of the United States David S. Ferriero will provide opening remarks and NDC Director Sheryl Shenberger will update the public on NDC prioritization practices and ongoing declassification progress.

The forum will feature a panel of experts addressing the topic “What Secrets Do People Want to See?”  Other panelists include historians from Government agencies, researchers, and representatives from Civil Society groups.  The forum will conclude with a question and answer session.

Session highlights include:

  • An overview of the role of provenance in archival holdings processing and arrangement, by Rick Peuser, Supervisory Archivist.
  • “Approaches to Prioritization” panel discussion with experts: David Robarge, chief historian, CIA; Stephen Randolph, The Historian, Department of State; Katherine Hawkins, National Security Fellow, OpenTheGovernment.org; Nate Jones, FOIA Coordinator, National Security Archive; William Burr, Senior Analyst, National Security Archive; and Bill Leary, Public Member, Public Interest Declassification Board (PIDB).

For additional information or to submit questions in advance question, contact Don McIlwain at don.mcilwain@nara.gov or (301) 837-0587.

Posted in Uncategorized

Prioritization and the National Declassification Center

In our 2014 supplemental report, Setting Priorities: An Essential Step in Transforming Declassification, the PIDB advocated for a coordinated, government-wide approach to declassifying information based on those records most sought after and of most historical significance to the public.  To this end, we believe topic-based prioritization is a viable alternative to prioritizing records simply by age and level of difficulty and effort to review.  Prioritization is one component of the overall transformation needed to sustain declassification given the growth of information across government and the dwindling resources available to agencies.

We are looking forward to participating in the upcoming NDC public forum.  This forum will focus on the topic of prioritization for improved declassification.  A member of the PIDB plans to discuss the six recommendations from our Setting Priorities supplemental report and provide comments on next steps in making topic-based prioritization a possibility in government.

The NDC completed the quality assurance review of over 351 million pages of records, commonly referred to as the “backlog,” in February 2014.  We are pleased to know the NDC is using this forum as a way to involve the public and stakeholders to improve its processes now that the “backlog” has been retired.  With this large challenge completed, there is an opportunity to rethink how the NDC and agencies operate and how they may prioritize records for declassification review so that those of most importance to the public are processed first.  We are thankful for the opportunity to begin dialog on this topic and look forward to the NDC public forum.

Please continue to follow our blog, Transforming Classification, to learn more details about our participation in the upcoming NDC public forum.

Posted in Uncategorized | 1 Comment

Sunshine Week and Open Government Progress

We are pleased to announce the Presidential appointments of Ms. Laura A. DeBonis and Mr. Solomon B. Watson, IV as members to the PIDB on March 12, 2015.  It is fitting that the newest members of the PIDB are able to participate in our executive session meeting being held today.  As we reflect on the significance of Sunshine Week and public access to Government information, we intend to use our meeting today to both review what has happened in the past year and decide on plans for the coming year.  We will take a detailed look at past year Government accomplishments to see what policymakers and practitioners have made to advance open government initiatives, particularly those committed to the transformation of the security classification system.  We will also take this opportunity to see what challenges and impediments still exist and see where we may be able to advocate for more change and modernization.

The National Archives and the Central Intelligence Agency earned well-deserved praise for the pilot projects they spearheaded at the Center for Content Understanding (CCU). We were able to view firsthand their accomplishments when we traveled to the Applied Research Laboratory at the University of Texas at Austin in September 2014.  Some of the technologies developed at the CCU are already in use at the CIA and are leading to improved efficiency and better reviews overall.  Still, we will continue to advocate for the adoption and use of these technologies across declassification programs in the Government, including at the National Declassification Center.  Although the records included in the pilot project are not yet publicly available, the results are an important step forward to declassification modernization.

Another open government commitment of particular interest to the PIDB is for change in the treatment of obsolete historical nuclear information.  We were excited to hear that the Department of Defense (DoD) created the Formerly Restricted Data (FRD) Working Group in response to this National Action Plan commitment and were pleased to learn that DoD made reviewing obsolete FRD information for declassification its flagship open government initiative.  Just last week, the DoD updated its website to show the eight facts it declassified through the working group process and in cooperation with the Departments of Energy and State.

Still, we believe there is more work to be done on both these important initiatives to wholly fulfill their commitments included in the Second National Action Plan for Open Government.  We urge senior leaders to increase actions, allow for wider implementation, and greater public access to the Reagan email collection and no longer sensitive nuclear information that is of historical interest.  Additionally, we hope to see agencies increasing public access to Government information of interest to the public, a focus of our Setting Priorities report to the President last year.

As Sunshine Week continues, we will comment more on open government initiatives. We want to thank the hardworking professionals who conduct declassification and access reviews at the agencies for their dedication to Government transparency and thank them for their work on behalf of the public.


Posted in Uncategorized

The Public Interest Declassification Board commemorates James Madison’s birthday and Sunshine Week

Sunshine Week is an annual initiative designed to raise awareness of the importance of citizen access to Government information.  This commemoration coincides with National Freedom of Information Day and James Madison’s birthday (March 16).

We reaffirm the principle that an Open Government is essential in our democracy.  An informed citizenry, actively participating in debating and discussing the actions of its government leaders, is only possible when they have all necessary access to government information.  In December, we issued our supplemental report, Setting Priorities: An Essential Step in Transforming Declassification, as an aid to government policymakers and practitioners.  This report provided six recommendations to support improved declassification policies.  The recommendations focused on prioritizing declassification to those records that are most sought-after by the public and those records that are historically significant and of interest to policy makers, citizens, historians and researchers.

We continue to advocate for new policies to implement an improved declassification system.  These new policies are necessary as government information generation increases.  We believe that technological solutions offer the only answer to the long-term challenge of managing this exponential growth of information across government.  There remains an ever-increasing amount of government records and digital information inaccessible to the public.  Prioritization will set-up this information for a public access review, but providing real access will require automated workflow tools, advanced search and retrieval capabilities, and content understanding technologies if we want to seriously amend the system and increase declassification to an acceptable level.

In 2015, the PIDB intends to focus on learning more about these technologies and how they can be used to increase and improve declassification.  The PIDB will continue advocating for their testing and implementation at the National Declassification Center and at agencies.  We have long-supported the idea that modernization requires the adoption of these technologies.  We look forward to working with the Security Classification Reform Committee, agencies and the public to advance our mutual goal of reforming our policies and practices for today’s digital age.



Posted in Uncategorized

The President Announces His Intention to Appoint Laura A. DeBonis and Solomon B. Watson IV to the PIDB

Yesterday, the President announced his intention to appoint Laura A. DeBonis and Solomon B. Watson IV to each serve three-year terms as members of the Public Interest Declassification Board.  You can find a link to the White House press release announcing the appointments here.  The members of the PIDB look forward to working with Ms. DeBonis and Mr. Watson as they continue their efforts to improve declassification and modernize the classification system.

Posted in Uncategorized

William Leary To Be Reappointed to the PIDB

President Obama announced yesterday afternoon his intention to nominate William Leary to serve a second three-year term on the PIDB.  The members are pleased to learn of his reappointment and we are looking forward to his continued help in advocating for transforming the national security classification system.  Before his retirement from Government service as the Senior Director for Records Access and Management at the National Security Council, Bill was instrumental in assisting us as we developed recommendations to improve declassification.  After his retirement, the President nominated him to serve as a member of the PIDB.  Bill helped us fulfill the President’s request that we study the classification system and make recommendations on its modernization.  Most recently, Bill led our effort to highlight and make additional reforms to the way our Government declassifies information.  This supplemental report, Setting Priorities: An Essential Step in Transforming Declassification, recognizes the need for the Government to target its declassification efforts on information of most interest to the public.

Bill is a key member of our Board and we look forward to working with him as he begins his second term.  Congratulations Bill!

Posted in Uncategorized

NEW ANNOUNCEMENT: The PIDB Releases its Supplemental Report, Setting Priorities: An Essential Step in Transforming Declassification

Today, the Public Interest Declassification Board released online its latest report recommending additional changes to transform the security classification system.  This Supplemental Report, Setting Priorities: An Essential Step in Transforming Declassification, focuses on topic-based declassification prioritization.  In it, the PIDB makes the case for the government to adopt a centralized approach to topic-based prioritization and recommends specific policy and process changes aimed at improving access to historically significant records most sought-after by the public.

With input from the public, agency classifiers, declassifiers and historians, the recommendations found in this supplemental report are meant to assist the Records Access and Information Security Interagency Policy Committee/Classification Reform Committee (RAIS IPC/CRC) in its work of evaluating the PIDB’s 2012 Report recommendations and developing a government-wide approach to transforming classification.

We concluded that automatic declassification should no longer be the sole policy driving declassification programming across government. We found that this policy no longer supports the President’s policies from Executive Order 13526, “Classified National Security Information,” as intended.  In practice, automatic declassification has fueled a risk-averse process limiting quality declassification review, brought about expensive re-reviews, and added unnecessary costs to an overburdened system.  As the volume of information continues to increase exponentially in the digital era, topic-based prioritization would ensure declassification review of records of the greatest potential for use by the public, historians, public policy professionals and the national security community itself.  It also would more closely align with electronic information management practices designed to ensure discovery and access to relevant information.

We provide six recommendations in support of topic-based prioritization by giving attention to records of greatest public interest.  They are:

  1. Topic-based declassification should be the normal process rather than the exception.
  2. The National Declassification Center (NDC), in consultation with the public and with agencies, should design and implement a process to solicit, evaluate and prioritize standard topics for declassification government-wide.
  3. End pass/fail determinations and identify necessary redactions for topic-based reviews.
  4. The government should require agencies to develop and use new technologies to assist and improve declassification review.
  5. Agencies and the NDC must improve risk management practices.
  6. Revisions to the current Executive Order are needed to lessen the burden of automatic declassification on agencies in support of topic-based declassification review.

Our Supplemental Report details the reasons our recommendations and makes the case for these needed changes.  The report also includes a list of topics solicited from the stakeholders inside and outside government.  This list should provide a suitable starting point for government policymakers to begin designing and implementing a prioritization process.  We recognize that change will be difficult, but know the consequences of inaction will be far more negative than will steps made in the overall effort of transformation.

Posted in Uncategorized

Retirement of Miriam Nisbet, Director of OGIS

On behalf of the members of the Public Interest Declassification Board, I would like to congratulate Miriam Nisbet on the eve of her retirement from Federal  service.  Throughout her Federal career, she served with distinction as a tireless advocate for transparency and access to government records.  We first met Ms. Nisbet in her role as the first Director of the Office of Government Information Services (OGIS).  We were impressed with her vision of OGIS as the Freedom of Information Act (FOIA) ombudsman to provide mediation services to resolve disputes between FOIA requesters and Executive branch agencies.  Those familiar with open government and transparency advocacy regard Ms. Nisbet as a trusted advocate for the proper administration of FOIA, as Archivist of the United States, David Ferriero noted in his announcement Ms. Nisbet’s retirement.

I want to thank Ms. Nisbet personally for her support of our work to develop recommendations to modernize and transform the security classification system.  Many similar challenges exist that impede the declassification process and the administration of the FOIA at agencies, and we are grateful to Ms. Nisbet and the staff at OGIS for recognizing that limiting secrecy to the minimum necessary for the national security assists both the agencies and the public in their efforts to access and manage government information.

The members join me in thanking her for her wise counsel and her work to increase transparency and access to government records.  We wish her all the best in her retirement.

Posted in Uncategorized