2001 Wisconsin Avenue, N.W.
Telephone: (202) 334-2934
April 30, 1999
The Honorable Jolene M. Molitoris
Federal Railroad Administration
400 Seventh Street, S.W.
Washington, D.C. 20590
Dear Administrator Molitoris:
The TRB Committee for Review of the Federal Railroad Administration (FRA) Research and Development (R&D) Program held its third meeting in Washington, D.C., on March 4-5, 1999. The purpose of this meeting was to conduct an annual review of the FRA R&D Program. The enclosed committee roster indicates the members who attended the meeting.* Following an open session with FRA staff, the committee met in executive session to deliberate on the information presented and to develop this report.
On behalf of the committee, I want to thank James McQueen, Arrigo Mongini, Steven Ditmeyer, Claire Orth, Magdy El-Sibaie, and Robert McCown of FRA and Robert Ricci of the Volpe Center for participating in the meeting, and for providing the committee with program updates and plans for fiscal year 2000. The committee also appreciates the ongoing interest of congressional staff in this program review.
In its letter report of January 25, 1999, the committee recommended that FRA take a global risk assessment approach to identifying the most serious hazards in the rail system. Such an approach would make it possible to develop a strategic analytic framework for the R&D Program, establish priorities relative to the seriousness of the risks involved, and map those priorities to projects that hold promise for improving safety. Prior to the meeting, the committee requested that FRA provide specific information related to the 10 major program areas that could serve as a first approximation of a program-level risk assessment. The committee appreciates the efforts of FRA and Volpe Center staff in preparing and presenting this information and realizes that doing so was no small task. Following are the committee’s comments on this material with regard to various issues related to the FRA R&D Program.
EVALUATION OF FEDERAL RESEARCH PROGRAMS
Before addressing specific issues related to FRA’s R&D Program, the committee would like to call attention to a recent publication of the Committee on Science, Engineering, and Public Policy (COSEPUP) of the National Academy of Sciences, National Academy of Engineering, and Institute of Medicine. That committee, comprising prominent members of the research community, was asked to review the issue of how to measure and evaluate federal agency research programs for compliance with the requirements of the Government Performance and Results Act (GPRA). Although the present review of the FRA R&D Program is not being done specifically in compliance with GPRA, this committee is struck by the relevant conclusions of the COSEPUP report, in particular:
Conclusion 1: Both applied research and basic research programs supported by the federal government can be evaluated meaningfully on a regular basis.
Conclusion 3: The most effective means of evaluating federally funded research programs is expert review. Expert review¾ which includes quality review, relevance review, and benchmarking¾ should be used to assess both basic research and applied research programs.
This committee’s ongoing review of FRA’s R&D Program is clearly in the spirit of the research review process for federal agencies. Evaluation of research programs is an idea whose time has come, and TRB is pleased to be participating with FRA in this activity.
SAFETY INFORMATION AND DATA
In direct response to the committee’s January letter report, the FRA staff, with assistance from the Volpe Center staff, undertook a significant effort to develop a process that could be used to establish priorities for the R&D Program in relation to associated risks. In attempting to measure the relative risk associated with each program area, the FRA and Volpe Center analysts had to rely on existing data sources, all of which proved to be inadequate for the task. The committee fully recognizes that these database deficiencies are beyond the control and responsibility of the FRA Office of R&D, and that there are broader issues associated with changes to data collection procedures that need careful examination.
The analysis drew on the following four data sources:
Before accident data can be used to identify required research, causes of accidents and contributing factors must be carefully reviewed and validated to support the expenditure of R&D resources. Each of these data sources is inadequate for identifying the multiple contributing factors involved in accidents. Each focuses more on descriptions of what occurred (e.g., the conductor failed to throw a switch) than on underlying factors (e.g., distractions or lack of training). Nor is information available on "incidents" (or near-accidents). Analysis of incidents can be useful in determining mechanisms that helped prevent an incident from becoming an accident, as well as in identifying new trends and developing countermeasures. Companies have such information, but generally do not share it with government agencies. It might be possible for a neutral third party to serve as a repository for this information, with company and individual identifiers being removed (similar to the Aviation Safety Reporting System, administered for the Federal Aviation Administration by the National Aeronautics and Space Administration through its contractor Battelle Memorial Institute).
Recommendation 1: The committee recommends that the FRA Administrator, in coordination with the Office of Safety and the Office of R&D, take the necessary steps to improve FRA’s data collection so that the multiple contributing factors involved in an accident can be correctly identified and analyzed and the sequence of events characterized. One of the numerous benefits of more accurate and complete accident data would be the ability to conduct R&D in closer balance with actual safety risks. This effort could be initiated with a research project that would define the need for improved safety data and develop a taxonomy of causes. (A random sample of actual accidents could be analyzed to provide a basis for identifying root causes. For example, in conjunction with the American Public Transit Association, FRA is conducting a full causal analysis of a sample of low-speed commuter rail accidents.) Consideration should be given to collecting data on incidents (near-misses), in addition to accidents, that could indicate areas in which accidents might be avoided or prevented (recognizing the limitations of voluntarily reported data). To the extent that they are not fully exploited now, additional data sources that could be used to determine accident causes include National Transportation Safety Board reports, Office of Safety railroad audits, and FRA dossiers on individual accidents.
The committee is pleased that the FRA staff has made such good progress in approaching the concept and philosophy of risk assessment in relation to the major program areas, in spite of being hampered by deficiencies in the available accident data. We recognize the difficulty of this task. The FRA staff has made a major and commendable step forward. The committee believes the process of categorizing the program in terms of risk measurement should be carried forward and offers the following comments to assist in outlining and undertaking the next steps, building on the good work done to date.
Relating Risk Assessment to Program Areas
FRA’S intended R&D focus is based on risk as measured by the range of accidents from infrequent but severe events to those with higher frequency but lower severity. Drawing on the above four data sources to measure risks, FRA and the Volpe Center assembled historical data on fatalities, injuries, and damage associated with 7 of the 10 program areas. The difficulties encountered in matching the data with the program areas clearly reveal the deficiencies in these sources.
With respect to revising and expanding the draft methodology presented by FRA and Volpe Center staff, the committee believes that while the correlation between budget allocations and the "risk index" is a good first cut, improvements can be made. Calculating the risk index for each of the 10 program areas required significant assumptions, some of which were related to the inadequacies of the available data. These assumptions included the following:
Recommendation 2: The correlation between budget allocation and total risk should be improved by including all sources of loss (or harm).
Relating Risk Assessment to Individual Projects
In response to the committee’s recommendation to demonstrate an explicit process for evaluating individual R&D projects, FRA has undertaken a significant effort to develop a methodology and to apply it to currently active projects. The process is based on three rating criteria: safety ratings, regulatory ratings, and a project’s likelihood of success. Final ratings are based on staff consensus. This project evaluation process could be described as a "microscopic" approach, as opposed to the "macroscopic" approach used to create a risk index for each program area (based on accident statistics as described above). Because the safety ratings do not appear to capture the risk index, however, there is a disconnect between the results of the two approaches.
Recommendation 3: As the next step, FRA should assess how to connect the separate approaches used for risk assessment of the program areas and for project evaluation.
Recommendation 4: Once the above improvements to the risk assessment process have been made, the committee recommends that FRA begin using this process to assess the R&D projects slated for the FY 2001 budget.
SYSTEMS PERSPECTIVE AND PERFORMANCE-BASED SAFETY STANDARDS
The committee is encouraged by the positive steps FRA is taking in adopting a systems perspective and moving toward performance-based safety standards. For example, FRA has expanded one major program area from "Safety of High-Speed Rail" to "Improved Railroad System Safety." This is an important area in which FRA can take a more proactive approach. FRA has agreed with the recommendation in the committee’s January letter report regarding the need to move toward performance-based safety standards and to undertake important research that would facilitate that transition. Furthermore, FRA proposes to focus initial activity in this area on new performance-based regulations for electronic and software-based systems for braking and train control technology.
Recommendation 5: The committee recommends that FRA continue to use the regulatory analysis outlined by the R&D staff at the meeting to encourage R&D efforts in support of the move toward performance-based standards and procedures. With respect to Recommendation 1 above, accurate safety data are also critical to the development of metrics and measurements needed to support performance-based standards. (For example, in the commuter rail analysis mentioned above, most of the accidents have been attributed to wheel/rail causes. The data generated by this analysis could be used to support the development of performance-based standards.)
ALLOCATION OF FUNDING TO R&D PROGRAM AREAS
From the program review presented, the allocation of funding to program areas is reasonable, with two possible exceptions: human factors and grade crossings.
The proposal for increased FRA funding for human factors research is a commendable step in the right direction. In fact, it appears that even more funding in this area may be justified for two reasons. First, the committee questions the assessment of risks associated with human factors as presented in the program review. Accidents involving human factors causes are probably underreported, partly because of the limitations associated with assigning each accident a single cause when, in fact, mishaps tend to evolve through the conjunction of several factors. Almost all accidents, including those whose primary cause is an engineering failure, have a human factors element (e.g., an inspection being missed or done sloppily). The committee is not convinced of the appropriateness of allocating injuries and fatalities from the human factors category to other program areas, as presented in the draft analysis. Better accident data are needed before more accurate and appropriate allocations can be made, as discussed above (see "Safety Information and Data" and the related recommendation).
Second, the committee believes FRA’s ongoing and proposed human factors research does not adequately address the new cognitive demands likely to result from the increased automation of tasks and functions occurring in the railroad industry. Experience in other complex domains has shown the benefits of automation, such as increased precision and efficiency of operations. At the same time, however, automation tends to impose new knowledge and attention demands, and creates the potential for new risks and problems related to breakdowns in the interaction between humans and machines. These breakdowns are often related to the communication with rather than the operation of the machines. Effective operator communication and cooperation with advanced automated systems are critical because many of these systems operate at a high level of autonomy and authority. They can initiate actions with no immediately preceding operator input (autonomy), and they are capable of modulating or overriding user input (authority). Therefore, the operator must maintain a high level of awareness of the automation status, behavior, intentions, and limitations. Railroads can and should learn from research in other domains or modes of transportation and build on the existing knowledge base in human factors and cognitive engineering. As an example of how this committee might provide guidance and support at a more detailed level than the program areas, Annex A reviews possible specific changes and additions to the human factors research agenda.
Recommendation 6: As discussed in Annex A, a number of issues on the human factors research agenda have already been addressed in other domains. The committee recommends that FRA engage in discussions with researchers in these other fields (e.g., aviation) to a greater extent than is currently the case in order to utilize existing knowledge and avoid replication of available research.
The committee does not agree that grade-crossing deaths and injuries should be excluded from the risk analysis used to identify research priorities, considering that they represent such a large percentage of all railroad-related deaths and injuries. Admittedly, there is an argument to be made that the most appropriate means of reducing risks at grade crossings¾ such as grade separation¾ are already known, but it is also the case that the available means are not being as widely implemented as would be desirable because of their high cost and/or lack of public acceptability. The committee believes research to identify less costly or more acceptable solutions for reducing risk at grade crossings (and preventing trespassers) should continue vigorously. The committee’s concerns in this regard have subsequently been heightened by the multifatality Amtrak grade-crossing accident in Illinois.
The committee supports the funding in the R&D Program for evaluation of some of the concepts developed in the Next Generation High-Speed Rail Program. The committee also supports FRA’s decision to consolidate the relevant grade-crossing projects into the human factors program area in FY 2000, particularly research into driver psychology and behavior. Policy research might also be highly productive at this point to improve understanding of what policies work, such as grade-crossing closures and standards for grade separations. Differences in fatality rates among states suggest that there are policies and approaches that contribute to lower rates in some states. The committee recognizes that policy research may fall outside the purview of the Office of R&D, but should be encouraged by FRA in conjunction with Federal Highway Administration (FHWA).
Recommendation 7: The FRA Administrator, in coordination with the Office of R&D and the Office of Policy and the appropriate offices within FHWA, should develop a plan for policy research related to grade crossings and aggressively press for research related to standardization of grade separations and crossing elimination. This research should draw on successful practices in individual states, as well as other countries. These efforts would not necessarily be costly, and should involve both state officials and researchers exploring new concepts and approaches. The committee encourages the initiation of such research as soon as possible, with a project proposal being included at least in the FY 2001 budget.
NEXT GENERATION HIGH-SPEED RAIL PROGRAM
Progress is being made under this program. The development of a nonelectric locomotive is particularly encouraging.
With respect to the development of positive train control (PTC), particularly the design of the Illinois PTC project, the committee is pleased to see FRA’s continued, vital role in this project. Given the number of parties involved and the variety of technical and institutional issues being addressed, it is appropriate for FRA to be an active participant.
This project represents the third effort at developing PTC, and the results of the two prior efforts were not implemented. The committee is concerned that the current process appears similar to the Advanced Train Control Systems (ATCS) project2, and that given the stated (and rather modest) project goals of defining an interoperability specification and performing a demonstration involving three or four trains, the current process may not succeed for the same reasons. The committee cannot comment on the project’s technical accomplishments, because none were reported at the meeting. It appears that most effort to date has been invested in developing a process for involvement of the various parties and for research management. The committee would like to review this project in greater depth at its next meeting in order to explore these concerns.
Recommendation 8: The committee urges that in the future, the staff of the Next Generation High-Speed Rail Program provide the committee with more complete reports on program developments, including technical progress reports.
The committee looks forward to continuing a constructive relationship with the FRA R&D staff and to providing advice on strategic directions and appropriate goals for the R&D program. Specifically, the committee will be happy to provide guidance on the recommended evolution of the safety regulatory process and on the anticipated benefits of better understanding risks, implementing new technology, and improving safety performance.
The overall size of the FRA R&D Program has been significantly reduced over the years, and the budget is now less than half of what it was in the late 1970s and early 1980s. If FRA is able to build on its excellent work in developing a risk-based rationale for budget allocations, and if there is continued movement toward performance-based standards, the committee can envision increased demands for R&D and would support requests for corresponding budget increases.
The committee would like to call attention to a recent publication, Transportation Research Circular 490, Research Problem Statements on Intercity Passenger Rail, developed by three TRB standing technical committees that have strong interest in the thorough analysis of intercity passenger rail issues. We believe this publication may be of value to FRA in shaping its R&D and policy programs.
We also note that the committee held a preliminary discussion with FRA and Volpe Center staff about the possibility of holding an interim committee meeting at the Volpe Center. We will be in further contact about this possibility.
Joseph M. Sussman
Chair, Committee for Review of the FRA Research and Development Program
cc: The Honorable Ted Stevens
The Honorable Robert C. Byrd
The Honorable Richard S. Shelby
The Honorable Frank R. Lautenberg
The Honorable C. W. Bill Young
The Honorable David R. Obey
The Honorable Frank R. Wolf
The Honorable Martin Olav Sabo
1 The "harm index" in the draft methodology is actually an annual average (for a 6-year period) of the monetized values of fatalities and injuries related to each program area.
2 The ATCS project was initiated by the Railway Association of Canada and the Association of American Railroads in 1984 and ended in 1993. Its purpose was to develop a series of comprehensive and advanced operating systems for train control for improved railroad safety, productivity, and energy efficiency.
* As is standard policy for NRC committees, the members of this committee meet in executive session at the outset of each meeting to discuss any potential or perceived conflicts of interest that might have arisen for any of them. The committee has agreed to abide by TRB policies for dealing with conflicts of interest that may arise in the bidding for or winning of FRA contracts by firms with which members are associated. In the interest of full disclosure, we note the following FRA-related activities.
First, FRA funds a research program from its Next Generation High-Speed Rail (HSR) Program that TRB administers on FRA's behalf, as described below. TRB has established policies and procedures to ensure that this committee can evaluate the HSR R&D program independently of any impact its evaluation might have on TRB or the National Research Council. The Innovations Deserving Exploratory Analysis (IDEA) Program for intelligent transportation systems (ITS) includes a component that supports projects on ITS applications for the development and deployment of advanced rail system technologies, including HSR systems. The funds are used to encourage researchers to develop potential innovations in train control and risk reduction at grade crossings. Funding is provided by FRA at a level of about $500,000 annually. An additional $500,000 was provided from FY 1997 funds for an HSR- IDEA program that will solicit innovations for non-ITS technologies related to HSR safety. The total funding provided to TRB in FY 1997 was thus $1 million, which represents about 3.5 percent of the HSR R&D expenditures. The IDEA programs are administered by the Special Programs Division of TRB. IDEA investigations explore the feasibility of innovative and unproven new concepts or evaluate novel applications of advanced technologies from defense or industry to ITS or HSR practice. An IDEA award is a pass-through of funds to provide one-step, short-term support.
Second, individuals with the expertise and experience necessary to review the FRA R&D Program generally have some prior or ongoing relationship with the sponsor. For example, the U.S. Department of Transportation has a standing arrangement to fund research projects at the Massachusetts Institute of Technology through a Basic Ordering Agreement (BOA). In a project contracted through this BOA, Joseph Sussman and colleagues developed an analytic procedure for risk assessment, which could have application in future FRA safety regulatory analyses. A draft report for this $100,000 project was submitted to FRA in the summer of 1998 and is pending final acceptance by FRA. This contract has been renewed, effective March 1, 1999, for a 1-year period at $100,000 for corridor risk assessment projects. In addition, Alan Bing’s employer had two contracts with FRA in HSR vehicle crashworthiness and one in finite element analysis of crashes, with which Bing had only peripheral involvement. Bing has been leading an additional project for FRA, involving a benefit/cost analysis of crashworthiness improvements, funded at $105,000 and now close to completion. Finally, John Samuels is chairing the Positive Train Control (PTC) Management Committee for the Association of American Railroads/Transportation Technology Center, Inc. (TTCI)/FRA/Illinois DOT train control project, which is partially funded by the FRA Office of R&D and is being managed by TTCI staff with guidance from the management committee.
Possible Changes to Human Factors Research Agenda
Ongoing and proposed FRA human factors research activities address a number of important issues: problems related to fatigue and vigilance decrements (e.g., studies on engineer napping strategies, engineman vigilance monitoring, and dispatcher and high-speed operator fatigue); information flow, management, and representation (e.g., research on knowledge display interfaces, information management and control in railroad operations, and digital communications); operator training; and teaming/crew resource management. Since these issues have already been addressed in a number of other domains (in particular, aviation), the committee recommends that an attempt be made to consider the existing knowledge base and seek greater involvement of researchers in these fields in order to build on rather than replicate their work (see Recommendation 6 in the body of this report). For example, the committee finds it appropriate that FRA is funding a project on "Human Factors and Automation." It is not clear, however, that the project objective should be to "identify the human factors issues related to new technology." Those general issues are quite well established and understood. It appears more important to apply and adapt the available knowledge to existing and predicted railroad-specific problems and technologies, such as the human-centered design of displays for modern high-speed rail operations.
The committee agrees with FRA’s approach of attempting to identify specific problems and research needs through an analysis of accidents in terms of their multiple contributing factors. As noted in the body of this report, equally important is analysis of incidents. One of the factors involved in a mishap can be an erroneous action or assessment by an operator (as opposed to deliberate violations of rules or procedures), as correctly identified by FRA. It is important to keep in mind, however, that the discovery of human error should not be the end but rather the starting point of an analysis. Human error is not the cause of an accident, but the symptom of some underlying problem, such as improper training, inadequate interface design, or procedural difficulties. To prevent the error from reoccurring, one must look beyond its observable outcome, such as "failure to apply handbrake" or "failure to release handbrake." Common underlying factors and processes need to be examined to determine why the error occurred and what measures can be taken to prevent its recurrence. For example, failure to apply or to release the handbrake involves the same type of error¾ a lapse or error of omission. Lapses are most often the result of missed or mistimed attentional checks, and they are known to be more difficult for the operator to detect than other forms of error, such as slips. Failure to apply or release the handbrake could potentially be addressed by the same countermeasure, such as the introduction of a reminder function provided by the system.
The above example illustrates how a large number of accidents and incidents that may at first appear dissimilar can be reduced to a more tractable number of relevant R&D topics by abstracting from their surface appearance. The committee understands and appreciates the difficulties faced by FRA in this process given the absence of sufficient data on accidents and incidents noted in the body of this report. Therefore, we encourage FRA to make the development of effective means of collecting the needed data a major priority, as discussed under Recommendation 1.
Dr. Joseph M. Sussman
JR East Professor and Professor
of Civil and Environmental Engineering
Massachusetts Institute of Technology
3/4 & 5
Ms. Anna M. Barry
Director of Railroad Operations
Massachusetts Bay Transportation Authority
3/4 & 5
Mr. Ronald G. Markon
International Brotherhood of Electrical Workers System Council No. 16
Mr. John G. Bell
High Speed Trainsets
National Railroad Passenger Corp.
3/4 & 5
Dr. John M. Samuels
Operations Planning and Budget Operations Division
Norfolk Southern Corporation
Dr. Alan J. Bing
Arthur D. Little, Inc
Dr. Nadine B. Sarter
University of Illinois at Urbana-Champaign, Institute of Aviation-Aviation Research Lab
3/4 & 5
Dr. Sherwood C. Chu
Mr. Thomas P. Schmidt
3/4 & 5
Mr. Thomas M. Downs
Mr. Louis S. Thompson
The World Bank
3/4 & 5
Mr. Nazih K. Haddad
Finance & System Development Administrator
High Speed Rail Program
3/4 & 5
Mr. Warren D. Weber
Rail Program Manager, MS-74
California Department of
3/4 & 5
Mr. William Weinstein
Principal Member of the Technical Staff
The Charles Stark Draper Laboratory, Inc.
3/4 & 5
Mr. Steven R. Ditmeyer