FDA IRAI INFORMATION REPOSITORY REPORT

Stephen A. Weitzman, J.D., LL.M.

Principal Investigator

 

With improved access to the information on the FDA website our intention is to improve, streamline the research, teaching, training and compliance.

(With an initial focus on students in academic institutions who for the first time are being exposed to FDA issues, needs and challenges).

I.                   Executive Summary

Over the last two years as part of our contract with FDA, we have studied the FDA website, its construction, organization of data, search and retrieval systems and content of over 500,000 individual files, about 60 databases, and spreadsheets based upon extrapolation of document content, audio visual content, and FDA’s related content on other government websites.

Our conclusion - the FDA website content is the most valuable primary source of information for industry, as it regulates food and drugs and more importantly it provides the public with information and knowledge building so that each of us can make an informed decision about our health.  However, the website has the major flaw of attempting to service multiple audiences which detracts from its ability to meet the needs of those stakeholders who come to learn.  It is our opinion that this is true especially in efficiently educating and training of both students and industry.

The identified problems, as discussed below, we believe may be solved with 1) proper augmentation, 2) better organization, and 3) a stronger information management system that permits effective browsing of reorganized content for the entire site and 4) one that will enable a more efficient user-friendly exploration and ultimately findings, of this critical resource.

Under Contract with FDA, with limited funding, we have built a new system called the FDA Information Repository for Academic Institutions (IRAI) that meets much of the criteria specified above. It is our opinion that these problems may be even better addressed by the formation of an FDA Advisory Committee from academia and industry that will work with the Office of Public Affairs, the FDA Library, the Centers and the Office of Regulatory Affairs as an Advisory and Editorial Board. 

As a library, the site needs proper organization by subject, based on a clear overall introduction to subject areas and a logical information flow and classification system which follows the course of product development, submissions, approval, and post market requirements.  As a journal, it needs editorial board management with expertise in all subject areas that may help in the organization of the website, and website archiving policies.

The editorial board may also help to identify content that is not on the FDA website though routinely generated within FDA as well as foster the posting of information requested under the Freedom of Information Act (FOIA) which is not regularly posted.

We also believe that with minimal support, the FDA IRAI website can be maintained as an integral part of FDA offerings for its publics.

II.                 Introduction

Background

Two years ago based on Stephen Weitzman’s experience of 45 years with information management systems and years of formal classroom and seminar teaching in all FDA areas (excluding Tobacco) we discussed the FDA website as the library for CERSI (Centers for Excellence) institutions being funded by FDA and the need to make available new tools that would enable students and researchers to more easily navigate and search FDA’s primary resources, locate relevant information even in unexpected places, and present that information in an easily accessible  format, with the highest of integrity. This capability would be an invaluable resource for the academic institutions that, with FDA’s encouragement, are implementing regulatory science degree programs and training the future workforce for FDA and regulated industry as well as FDA, state and local officials.

Thus the goal of the FDA Information Repository for Academic Institutions (IRAI) was to give students and faculty the latest software and tools with an array of user-friendly features to provide students and educators more efficient online access to the extensive repository of relevant FDA and related-agency materials.

Findings

ORGANIZATION What we found was that the website lacks an overall cohesive organization that reflects how FDA is organized, its statutory functions, and its official regulatory requirements embodied in the Code of Federal Regulation (CFR). We also found that that the breakdown of main webpages (home pages) linked to the product subject areas identified in its webpage banner header are themselves haphazardly organized.  This lack of organization mitigates against browsing materials in an orderly fashion and force people to search for materials using the page’s elementary search function based on keywords which may yield an endless lists of results.

ERRORS IN METADATA We also found that in recent years FDA has attempted to add metadata to documents so that the “endless list of results” may be parsed by subject areas. However, we found that the metadata was inaccurate and that using the subject filter eliminated relevant documents. For example, documents that were applicable to devices or biologics were erroneously limited to drugs.  

FDA IMPLICITLY RECOGNIZES PROBLEMS WITH GOOGLE We also found that to get away from the lengthy Google results lists the agency has created over 60 databases (presumably Oracle databases). However, in at least one instance a database of documents only indexed titles so that relevant documents that did not have the keywords in the title were not retrieved using the search template provided.

LACK OF A COHESIVE NARRATIVE There is no overall descriptive narrative of FDA and how it regulates products and enforces compliance to assure that products are safe and effective.  That used to be fulfilled by the very useful pamphlet “Requirements of Laws and Regulations Enforced by the U.S. Food and Drug Administration” published and updated until 1998.  While we reproduced the 1998 version of this “Blue Book” in FDA IRAI, we supplemented it with narrative monographs of the Congressional Research Service that describe regulation of specific product subject areas and statutory amendments since 1998. We strongly believe that Blue Book should be updated.

THE CFR, FEDERAL REGISTER DOCUMENTS, AND COMMENTS While there is no full text version of the CFR on the website, there is a searchable database to the CFR. This however does not facilitate browsing of relevant sections or subsections preceding or following the section or subsection “hit.” Also the site itself lacks the Federal Register preamble to all proposed and final regulations in the CFR. Further, since about 2008 FDA links FR document titles to the GPO website and no longer posts comments to those FR documents.  As a consequence, these documents are not fully indexed for retrieval purposes. Indeed, the comments to FDA collected on the GPO site are merely listed and not indexed.

Also, in order to search CFR sections, there must be consistency in citation and spacing between section numbers and subsections in all FDA-generated documents.  This will enable better searching and retrieval of documents by CFR number. In fact, it could yield the equivalent of an annotated CFR just like what we have with the West System of case law and legislation.

TRANSPARENCY FDA has made a substantial effort to be transparent in part, in an effort to open its decision making process to scrutiny and at the same time foster compliance by its regulated industry which can learn from regulatory actions against others who did not comply with FDA regulations causing a recall, notation of defects in an inspection report, warning letter, adverse or other required defect report.  Unfortunately, not all these documents or other documents available to individuals that request them under the Freedom of Information Act (FOIA) get posted on the FDA website in part due to the lack of mandatory requirements, unless the document was requested three times. But the only way to know that a document exists is to subscribe to the FOIA Log (which is not posted on the FDA website) or see a news story that prompts a request for documents behind that story.

The administrative problems with FDA’s FOIA system that impedes transparency is the FOI Office’s lack of an archive of previously released documents from its office or directly from FDA field offices.  The database management system for documents contains minimal information and does not link to an archive of documents so that a second request must be processed just like a first request.  Thus even if FDA wished to release all documents previously released in the last ten years, there is no ability to do so. 

EDUCATION FDA does a remarkable job of producing educational videos (A/V) through organized seminars or individual topical webinars. Just about all of the video presentations are accompanied by a slide presentation. Unfortunately, there is no organized catalog to all these presentations. Not all of the presentations have transcripts so that they too are not indexed so retrieval is based upon the title or the content of the slides. Further, FDA internal teaching and training materials are not on the website.  Just former FDA employees have knowledge and access to them but the general public does not know of their existence.

OLD & NEW DOCUMENTS Since the advent of PDF documents the agency no longer posts documents in Microsoft Word or Power Point.  However, older documents in Word or Power Point have not been updated.  In some cases, slide presentations remain in Windows 95 or old Windows Explorer versions and are difficult to use.

ARCHIVE POLICIES We found that as guidelines or guidance documents are withdrawn or replaced the old versions are deleted and no longer maintained on the FDA website.  The same goes for warning letters from 1997 to 2004. The FDA has removed many pages from the Safety>Recalls, Market Withdrawals, & Safety Alerts > Major Product Recalls areas. If you do a Google search for “Hydrolyzed Vegetable Protein Product Recalls” you get this page: http://www.fda.gov/Safety/Recalls/MajorProductRecalls/HVP/ucm2022497.htm which  does not have a list of the recalls. And when you search the noted recall database on the right of the page you get “Page not Found.”

THE FDA DIRECTORY There is no complete FDA Directory on the FDA website. Indeed, one must use the HHS directory which provides minimal information about FDA employees. There are biographies on the FDA website of top employees and descriptions in AV presentations of the credentials of a speaker.

RELATED MATERIALS ON OTHER GOVERNMENT WEBSITES In addition to FDA requirements, regulated industries must comply with requirements of other government agencies.   In particular companies which are covered by the Office of Medical products have to comply with NIH clinical study requirements.  There is also other relevant information on the NIH/NLM website including Notices of Decision (Court Actions) of FDA. These items really need to be included on the FDA website so that they may be easily browsed, indexed and relevant items retrieved in a single search.

TECHNICAL PROBLEMS OF THE FDA WEBSITE The contents of the FDA website are housed in two major server areas: www.fda.gov and www.access.fda.gov. The www.fda.gov server houses the HTML (text) pages and access.fda.gov houses files linked to those pages in non-html format. The files in the www.fda.gov are organized in folder and subfolders which are redundant (example there are 2 “Drugs” folders and a CDER folder.) In addition, documents in subfolders that are unrelated and should be divided and put in separate subfolders are not separated. This leads to the inability to create a true site map of the FDA site that could be easily browsed and navigated.  

We also found that web pages have been moved to another subfolder without correcting the page to which it was linked, yielding a page not found error (we estimate as many as 15,000 pages).

There is also a compliance issue with the 508 requirements where the title of a PDF is not properly entered in about 80% of the PDF documents.  What we found is that not only is the title field not entered or erroneous and the same goes for the subject field, creator field, and date fields.

III.                RECOMMENDATIONS

Several improvements to FDA’s dissemination of information are self evident from the findings above.

More specifically, based upon these findings we make the following recommendations:

A.      CONTENT

1.       Revive and update the FDA Blue Book

2.       Integrate the entire 21 CFR and relevant portions of 42 CFR beyond the sections related to biologics.

3.       Add the FDA FOIA Log and all matters released under FOIA as the next step in the Transparency Initiative

4.       Reorganize FDA web pages linked from well organized tables of contents.

5.       Have an outside expert with both information systems and FDA regulatory expertise audit the document metadata.

6.       Separate out content specifically for consumers and healthcare providers following the example of the new Food Safety website.

 B.      REPAIR TECHNICAL PROBLEMS ON THE FDA WEBSITE

 1.       Write SOPs for writing and creating documents which standardize Statutory and CFR number systems as well as filling out of metadata especially in PDF documents.

 2.       Reorganize the FDA Folder and subfolder systems

 3.       Create an FDA thesaurus to standardize search terms like Medlars.

 C.      EDUCATIONAL ADVISORY AND EDITORIAL BOARD

 1.       Create an Advisory Committee under 21 CFR with the function of advising on teaching and training as well as editorial functions regarding content of the FDA website.

 2.       Populate the committee with academics and industry regulatory and training personnel.

 3.       Involve the FDA librarians who are the experts in information organization and research.

 4.       Initiate recruitment for such an advisor committee by holding a conference on Transparency and needed improvements.  Speakers should also include news media writers, trade and professional associations, and corporate special librarians.

 D.      DATA SYSTEMS FOR DOCUMENT MANAGEMENT

 1.       Completely update the database system for FOI so that all documents provided under FOIA are cataloged and linked to that database.

 2.       Establish a parallel database for all internally generated FDA documents so that documents requested can be more easily found and provided expeditiously to any FOI request.

 3.       Establish a procedure for posting all document requests under FOIA and all documents submitted in response as well as denial letters.

E.       MAINTAIN FDA IRAI AS AN FDA RESOURCE

 1.       FDA IRAI gives an organized “satellite” view of the entire FDA website making it easier to navigate.

 2.       It has a structured table of contents.

 3.       It has a better browsing and search and retrieval system than Google (even as implemented and improved by FDA). For example, with one search one can see everywhere on the FDA website where there are hits in each category of the table of contents and the number of documents with hits in each category.  This capability enables one to find documents where they did not expect to find relevant documents; the Serendipity effect.

 4.       It has more relevant content for training and education than the FDA website from NIH and other sources.

 5.       It has the ability to add other relevant content that may then be simultaneously searched with all the public domain materials such as student training materials created by academic and medical institutions.

 6.       IRAI was built without any technical or other support from FDA, though it was jump-started with BAA.  It can be enhanced and maintained for under $1.0 mil. per year especially if FDA Offices fully cooperate.

 7.       FDA IRAI can also become a Public/Private Partnership by providing a platform for trade and non-profit organizations that wish to share relevant information (secondary materials, including password-protected proprietary materials) and have them searched simultaneously with public domain data.

 8.       We have Support Materials including a description of the key information system features and a user manual.

 F.       SUPPORT IRAI AS A PLATFORM FOR SHARING AMONG NONPROFIT EDUCATIONAL INSTITUTIONS

 One advantage of IRAI is it ability incorporate secondary materials which may be simultaneously searched along with all public domain materials.

 This means that IRAI can function as a platform for adding course curriculum and materials for faculty at institutions to share with their colleagues.

 To this affect we have an agreement with the Association of Food and Drug Officials to share their materials.  We are now creating a section in IRAI Called AFDO materials.

 We are also in discussions with CTSAs to use IRAI as a means of exchanging and sharing course materials.  Further, a number of key faculty at CTSA institutions have agreed to serve on the IRAI Editorial Advisory board.

 The content of the platform can be further extended to include materials of other non-profit professional and trade associations as part of a one-stop-and shop concept.  Materials that are deemed public would be free and materials that are deemed proprietary can identified as relevant to a search, password protected and offered for sale under control of its owner.