Appendix

B

THE RESEARCH PROCESS - DETAILS AND EXAMPLES

Introduction

 

The purpose of this appendix is to provide greater detail, and a number of examples, of some of the activities described in Chapter Three.  Most of the sample documents have been taken from the National Co-operative Highway Research Program.  These documents can be modified relatively easily to suit the requirements of other programs.

 

Problem Statement Development

 

The format of the problem statements used by AASHTO to select projects for inclusion in the National Co-operative Highway Research Program is as follows:

 

·         Problem Title

·         Problem

·         Objective

·         Key Words

·         Related work

·         Urgency/Priority

·         Cost

·         User Community

·         Implementation

·         Effectiveness

 

The title should be succinct.  A title, which is vague, may result in a project not receiving due consideration.  A title that is too long not only obscures the problem but also complicates cataloguing and databases, which frequently limit the number of characters in each field.


 

The Problem section justifies the need for the project. The initial justification of the project is generally only the beginning in the understanding of the research problem.  All research activities should be couched within “Further understanding the problem or problems.” This concept provides the “Big Picture” that justifies and unites all the activities and purposes of the research project. Without this concept, research programs run the high risk of failure from investing in solutions searching for problems, or allocating resources to problems that are already well understood. Problems needing research should be redefined and better understood as a matter of course throughout a successful research project. Without constant refinement of the research problem itself, finding effective solutions and implementable recommendations is elusive.

Researchable problems may be discovered in the problem statement phase. Some initial problem descriptions may not turn out to be researchable. Some may not be true problems. Many requests may be referred to operations or planning units for assistance. If a problem is thought to be worth research investment, hopefully, after further discussion, literature syntheses, surveys, pilot studies, experiments, development, etc. the problem can be better understood, and a useful result can still be offered. It is for these reasons that some successful research projects employ multiple lines of investigation, hoping that at least some of them will be useful for different aspects of the problem.

The problem section typically consists of one to three paragraphs describing the problem, and explaining why it should be solved.  The section should include background information on current practices and why they are deficient.  It is important to provide information on the magnitude and extent of the problem.  For example, whether the problem is a serviceability issue or a safety issue, and whether it occurs at the local, regional or national level.  Wherever possible, it is useful to provide baseline data, e.g. number of accidents, or cost of failures.  It is often appropriate to place limits on the scope of the project by indicating what form the solution should take, e.g. a specification, report, test method, design procedure, computer program, or a piece of equipment.  A very useful test of whether a problem can be solved by research is that, if the form of the solution can be defined, the problem is researchable.

 

The purpose of the Objective section is to state very clearly what products are expected from the research.  The ability to define products that will resolve the problem, are attainable, and can be implemented, has a major impact on the likelihood of success.

 

Key words are used for indexing, and also for conducting a literature search to determine if related research is underway elsewhere.  The section on related work provides an opportunity for the proposer to identify other work that is in progress or proposed in other research programs.  Frequently, a research project will build upon the results of a project completed recently.  It is unlikely that the results of the project will have been recorded in electronic databases, and would not be discovered by reviewers unless the proposer draws attention to the study.

 

Urgency/Priority provides an opportunity for the proposer to explain why the research needs to be initiated soon, and the consequences of a delay.

 

In the NCHRP process, problem statements include the cost, but not the duration of a project.     Cost is required to estimate the benefit-cost of projects during the project selection phase.  Other considerations, including cost, are discussed in the section ‘Request for Qualifications and Proposals’.  Some agencies do include an estimate of time required for the work in the problem statement.  This also provides an indication of the anticipated scope of the work, especially if fieldwork is envisaged, and permits annual budgets to be prepared for multi-year projects.

 

The User Community section is intended to identify groups who would benefit from the research and be affected by implementation of the findings.  The Implementation section is necessary to identify how the results will be implemented.  It is essential that an implementation strategy be developed before research work is initiated.  This allows those who will be affected by the implementation to make necessary plans and ensures that, if successful, the results of the research will be in the form required by the users.  The statement of effectiveness describes the anticipated state-of-the-practice after the results of the research have been implemented.  In other words, it describes the benefits of undertaking the research so that those involved in the selection of projects can estimate the benefit-cost ratio.  Where possible, the anticipated benefits should be expressed in the same units as the baseline data in the Problem section.

 

Because of the size of the program, and its large constituency, the NCHRP uses a two-stage process for developing problem statements.  The first step is the documentation of ideas.  These are then processed by NCHRP and FHWA staffs to provide feedback to the submitters in the form of a literature search, and identification of work in progress and related problem statements.  It is the responsibility of the submitter to determine whether to prepare a second stage proposal, which must contain all the sections, described above.  An example of a completed research problem statement is given in Figure B-1.

 

 

Figure B-1: An Example of a Research Problem Statement

 

 

Project Selection

There are a number of ways of ranking individual projects for the purpose of developing the research program.  In many agencies, the responsibility rests with an advisory committee, comprising representatives of the client groups.  Typically each member of the committee will review all the research problem statements independently, discuss them with colleagues, and rank them according to some or all of the criteria given in Chapter Three.  The ranking may be complex, involving weighting the criteria, or it could be simple three (0,1,2 or low, medium, high) or four (0,1,2,3, or no need, low, medium, high) point ranking.  This simple approach works well when there are a large number of problem statements to consider, and the size of the program is such that most will not be selected.  In situations where only a small number of problem statements are to be considered, a ranking procedure involving a greater number, and weighting, of the selection criteria may be justified.

 

Requests for Qualifications or Proposals

 

As noted in Chapter Three, a Request for Qualifications (RFQ) is normally used in one of two situations:

 

1.       When the work activities are known and the sponsor wishes to hire qualified people to complete the work.

2.       As the first step before issuing a Request for Proposals (RFP).

 

An example of the first situation is given in Figure B-2.  The RFQ provides an introduction describing why the work is being undertaken and the specific duties of the contractor.  It also provides details of the format required for the response, selection criteria, time and schedule, and administrative details.

 

The format of an RFQ issued to screen respondents prior to issuing an RFP is very similar to that used in the first situation.  The RFQ normally states the nature and scope of the work in general terms, and requests recipients to describe the qualifications and availability of key personnel, facilities, and prior experience in related work.  It is common to limit the number of pages for the response.

 

An example of a Request for Proposals is given in Figure B-3.  The RFP is an amplification of the research problem statement, with the addition of a schedule and administrative details.  Because the NCHRP utilizes project panels, comprising individuals who are themselves technical experts in the field of study, RFP’s usually provide details of the tasks which the panel views necessary to complete the research successfully.  Other research programs are less specific, and define the problem and solution desired, without stating the approach that is expected.

 

 

Figure B-2: An Example of a Request for Qualifications

 

 

 

 

 

 

 

 

 

 

 

 

Figure B-3: An Example of a Request for Proposals

Research Proposals

Most sponsors have specific requirements for the content and format of proposals.  This ensures that all the required information is provided.  Review of the proposals is simplified if a standard format is used because the reviewers know where to find specific items.  The NCHRP requires that the following information be presented in order:

 

1.   Cover

2.   Summary Page

3.   Table of Contents

4.   Research Plan

5.   Qualifications of the Research Team

6.   Accomplishments of the Research Team

7.   Other Commitments of the Research Team

8.   Equipment and Facilities

9.   Time Requirements

10.  Itemised Budget

11.  Co-operative features (if appropriate)

12.  Appendices (if appropriate)

 

Complete details of content and format are provided in the booklet ‘Information and Instructions for Preparing Proposals’.  The following is an excerpt of the requirements (items omitted are self-explanatory or satisfy administrative requirements).

 

Research Plan

The research plan shall detail completely the prosecution of the research, including the submission of an acceptable final report.  The plan ultimately becomes a part of the contract by reference of the proposal; therefore, it should describe, in a specific and straightforward manner, the proposed approach to the solution of the problem described in the project statement.  It should be concise, yet include sufficient detail to describe completely the approach to the solution of the problem.  Research methodology shall be described in sufficient detail to permit evaluation of the probability of success in achieving the objectives.

 

The research plan shall be subdivided into the following sections:

 

Introduction

The introduction to the research plan should provide a concise overview of the proposer’s approach to conducting research.  It should describe the manner in which the expertise and experience of the proposed team will be used in the research, and the application of special data, facilities, contacts or equipment should be presented.  The Introduction should highlight the linkages of the proposed team’s capabilities to the project tasks and the manner by which the proposed plan will satisfy the objectives.

 


Research Approach

This section will be used to describe how the objectives will be achieved through a logical, innovative, and rational scientific plan.  The plan shall describe each phase or task of the research to be undertaken.

 

Anticipated Research Results.

The research plan for each proposal shall contain specific statements describing the anticipated research results.  The results are expected to be presented in terms of the language and working tools of the practicing engineer or administrator so as to be immediately applicable to practice.  Consequently, there must be specific statements of the manner in which the desired results would be reported; e.g., mathematical models, design techniques, field or laboratory test procedures, or recommendations for changes in AASHTO, FHWA, or standard highway specifications.  If the nature of a project is such that it is known initially that the results will not be amenable to immediate implementation into practice, the research plan must include recommendations for the additional work necessary to reach the implementation stage.

 

Applicability of Results to Highway Practice. 

The research plan shall include a section captioned “Applicability of Results to Highway Practice” that clearly describes how the anticipated research results can be used to improve highway engineering practices.  Clearly indicate the expected audience for the research results.  The writer should present the strongest case possible to convince the reviewer that the results of the research will be practical and implementable.

 

Qualifications of the Research Team

Name. Address, telephone number, and pertinent background information must be provided for the principal investigator bearing primary responsibility for the project.  The same information is required for other research team members participating to a significant degree.  The proposals must describe how the research team members’ academic, industrial, and/or research experience relate to the project to be undertaken.

 

Accomplishments of the Research Team

Proposals shall contain a summary of the past accomplishments (“track record”) of the research team in the same, or closely related problem area of the project to be undertaken.  This summary is to include full particulars concerning all known instances of application to practice of the agency’s research results.  If no such knowledge exists, it should be so stated.

 

Other Commitments of the Research Team

Proposals shall contain a listing of current organization and personnel commitments to other work in sufficient detail to indicate that the organization and all of the individuals assigned to the proposed project will be able to meet the commitments of the proposal.  Man-hour commitments and percentage of time committed to other work for each member of the proposed research team shall be specified.

 

 

Equipment and Facilities

This section shall include a description of the facilities available to undertake the research and an itemization of the equipment on hand considered necessary to complete the research.  In the event that use of the facilities or equipment is conditional, the conditions should be described.  In the event that certain facilities or equipment are considered necessary to undertake the research but are not on hand, that fact should be presented.  The proposer should identify any arrangements that will be made to borrow or rent necessary equipment.  In the case where it is contemplated that additional equipment will be purchased under project funds, be certain that the budget item “capital equipment” indicates this.

 

Time Requirements

The time required to complete the research project shall be specified.  Proposals will not be rejected if the proposed time does not match the time specified in the project statement.  However, the agency must justify any difference.  In addition, a schedule shall be included that shows each phase or task of the work, when that phase or task will begin, how long it will continue, and when it should end.  The timetable should clearly delineate the points in time where project deliverables and reports are planned.

Itemized Budget

The estimated cost for the project should be based on the proposed performance period.  The budget shall reflect phase and/or task costs.  Lump sum estimates are not acceptable; budgets shall be itemized in accordance with the following cost categories where appropriate:

(a)  Salaries and Wages.  Each employee to participate in the performance of the project shall be identified by name, with role, level of effort, and cost presented in the format specified in the terms of Figure B-4.

(b)     Borrowed Personnel.  Reimbursement to other employers for salaries and wages paid by them to their employers released for, and directly engaged in, the performance of the subject research, plus federal and state payroll taxes and related employee benefit plan costs.

(c)  Consultants.  Costs for services of independent consultants deemed necessary for accomplishment of the research.

(d)  Subcontract.  Costs for services deemed necessary for performance of a portion of the research.

(e)  Capital Equipment.  Items with a value in excess of $500.00 per article or assembly required for the conduct of the research.

(f)   Materials and Services.  Materials, supplies, and other articles, including the cost of processing, testing, rental of apparatus and equipment from others, preparing, editing and reproducing reports, including the final report.

(g)  Communications and Shipping.  Long-distance telephone calls, postage, freight, etc.

(h)  Travel.  Transportation costs plus reasonable actual subsistence expenses

(i)   Employee Benefit Plan Costs and Payroll Taxes.  Costs of group insurance and employees’ pension and retirement plans federal and state payroll taxes for employees working directly on the subject research.

(j)   Overhead.  An allowance for overhead costs determined in accordance with the research agency’s usual method of accounting and generally accepted accounting principles.

 

 

Figure B-4: NCHRP Requirements for Tabulating Level of Effort in Proposals

 

 

Cooperative Features

If assistance in the form of personnel, data, or equipment is required from other agencies, public or private, describe the plans for obtaining such help or information.  In the case where such cooperative features play an important part in the conduct of the research, a letter of intent from agencies agreeing to provide cooperative features should be included in the proposal.

 

Appendices

The appendices may include such things as statements concerning previous work on this problem or related problems, abstracts of related projects. A bibliography or list of references, or descriptive brochures or materials describing the agency’s organization and capabilities in general terms.  Any other material not specifically mentioned previously and felt to be relevant for purposes of the proposal may be included as an appendix.

 

 

Review of Proposals and Contractor Selection

The selection criteria used by different programs are very consistent, and typically consist of:

 

1.   Understanding of the Problem - Has the investigator grasped what the problem and issues really are and not just reiterated the contents of the request for proposal?

2.   Research Approach - Has the investigator stated clearly a research methodology with “whys”, “whats”, “hows”, and “whens” in sufficient detail, in terms of both work and budget, to assess the probability of success and monitor progress against prescribed tasks and milestones?

3.   Staffing - Are those who will work on the project identified and their capabilities documented by reference to qualifications, experience and publications?  Is the role and responsibility of each contributor clear, and is the contributor available for the time committed in the proposal?

4.   Facilities and Equipment - Does the research team have access to all the facilities required to complete the work?

5.   Application of Results - Does the proposal provide a realistic appraisal of the prospects for satisfying the objectives?  Will the results be reported in a practical manner and is a feasible implementation plan provided?

 

Forms often provide space for the reviewer to make notes or comments, in addition to entering a score. This is very useful in enabling the reviewer to recall the strengths and weaknesses of individual proposals when they are discussed in committee, sometimes weeks after the review was made.  The form used for the initial rating proposal by panel members in the NCHRP is shown in Figure B-5.  Weights are assigned by the individual reviewer.  In most cases, the greatest importance is attached to the caliber of the research principals and the adequacy of the research approach.

 

Figure B-5:  An Example of a Form for the Initial Rating of Proposals


Research Work Plan

 

In the absence of specific requirements for the format and content of the research work plan, the following constitutes a useful outline:

·         Title

·         Introduction

·         Objectives

·         Research Approach

·         Project Team

·         Equipment and Facilities

·         Budget

·         Schedule

·         Anticipated Results

·         Implementation

 

The introduction provides background information taken from the Problem Statement or the Request for Proposals.   It is included because the work plan should be a “stand-alone” document, i.e. the reader should not need to refer to other documents.  The objectives also should be a re-statement of the objectives from the earlier documents.  The ‘Research Approach’ is the largest section in the work plan.  It should present the hypothesis, and describe and justify the strategy, which is being employed to satisfy the objectives.  For instance it could be a desk study comprising literature review and analysis, laboratory studies, field studies, or a combination of approaches.  It is usual to divide the work into tasks, each of which constitutes a milestone in the completion of the work.  For example, if the study involves writing a state-of-the-art report, the tasks might be:

Task 1                    Literature Survey

Task 2                    Questionnaire

Task 3                    Field Visits

Task 4                    Preparation of First Draft

Task 5                    Preparation of Second Draft

Task 6                    Preparation of Final Draft


For each task the activities should be described in detail.  In Task 1 the search strategy and the databases should be described.  In Task 2 the recipients of the questionnaire should be identified.  If it is known which agencies will be included in the field visits in Task 3, they should be identified, otherwise it should be made clear how the decision would be made.  In Task 4, the scope and intended audience should be identified, and a tentative outline of the report provided.  The work plan should also state who would review each of the drafts.

 

In the case of a project that involves experimental work and developing the plans for subsequent fieldwork, the tasks might be as follows:

Task 1                    Literature Survey

Task 2                    Laboratory Studies

Task 3                    Exposure Plot Studies

Task 4                    Development of Field Study Design

Task 5                    Preparation of Final Report


Task 1 would be described as in the previous example.  Task 2 will probably consist of sub-tasks, each comprising a series of experiments.  The experiments should be described in detail.   This allows the principal investigator to ensure that the necessary people, equipment and laboratory facilities will be available to perform the work, and to calculate the cost of the work.  This will usually be an iterative process that involves determining how many experiments can be performed within the available budget.  It is essential that the principal investigator be confident that the experiments being performed will be sufficient to satisfy the objectives for the task.  It is also good practice to identify a strategy to be followed if the experimental work is not completely successful, e.g. are there other test procedures that could be used?  The work plan should describe clearly what would have been accomplished before Task 3 begins, the way in which the work in Task 3 will be influenced by the results obtained in Task 2, and the manner in which the results will be reported.  The activities in Task 3 should be described in the same way as for Task 2, except that they will be more tentative because it is likely that many details cannot be finalized until results from Task 2 become available. Because it is a plan that is being written, it is inevitable that less detail can be provided for the later activities. One of the key roles of the principal investigator is to recognize when changes in the work plan are necessary, and to modify the plan accordingly.  The good researcher will maintain flexibility in the work plan to deal with setbacks, and to seize opportunities, while planning in sufficient detail to estimate resources, and to make arrangements for activities which require a long lead time.

 

In the example given, Task 4 consists of developing a design for a field study to verify the results of Tasks 2 and 3 under real world conditions, not performing the field work itself.  The specific details will depend on the results of Tasks 1 to 3, but the plan should contain the following sections:

·                     hypothesis to be tested on field studies

·                     criteria for selection of the study site

·                     potential sites

·                     equipment and other facilities required

·                     data collection and analysis procedures

·                     staffing plan

·                     schedule

·                     budget

The final report prepared in Task 5 should provide full documentation of the problem definition, experimental methods, results, conclusions and future activities, including implementation.  Further information on the content and preparation of reports is given in Chapter Five.

 

In the section describing the project team, the team members should be listed, along with the responsibilities and time commitment for each person.  This information is required to ensure that each person will be available to work on the project when required, and to establish costs.  The key members of the team should be identified, but for junior positions where the work could be done by one of a number of technicians, it is sufficient to identify the person by rank, e.g. senior technician @ 200 hours, 3 junior technicians @ 300 hours each.  The key members must be identified because if a substitution becomes necessary, management and the sponsor will need to satisfy themselves that the replacement has the necessary credentials and experience to perform the research.  Curriculum vitae are not required in a work plan.

 

Equipment and facilities are listed for the same reason as personnel, i.e. to ensure that they will be available when needed and that all the costs are captured in the budget.  This also forces the researcher to verify that all the equipment necessary to perform the planned experiments is available, or identify equipment and supplies that must be purchased.


 

If the research approach, project team, and the equipment are documented in sufficient detail, calculation of the budget for the project is relatively straightforward.  Other items such as travel, overhead and profit must, of course, also be included.

 

A detailed schedule for the work is required to establish a completion date and to provide a mechanism for management to measure the progress of the work.  It is also needed to identify when resources will be needed, and to calculate cash flow requirements.  For projects that involve no more than about 10 tasks, and about an equal number of sub-tasks, a chart similar to that shown in Figure B-6 is perfectly adequate.  Using the second study described above as an example, the chart shows the start and completion dates for each sub-task, and the anticipated progress of each sub-task, as a cumulative percentage.  It also shows the interrelationship of the tasks and sub-tasks because it is clear which activities are underway concurrently, and which are sequential.  The schedule allows one month for the review of the interim report, and two months to review the final report.  The anticipated progress on the overall project can be calculated from a knowledge of the amount of effort required for each task and sub-task.  The amount of effort required to complete each sub-task, expressed as a percentage of the effort required to complete the project, is shown in the third column. Taking the end of June 1996 as an example, the anticipated overall completion is:

 

Task 1 + 25% of sub-tasks 2a, 2b, and 2c

= 10% + 0.25 x (16% + 16% + 16%)

= 22%

 

By plotting the values from the row “Overall Completion” in Figure C-6 against duration of the project, a chart of anticipated progress is developed, as illustrated in Figure B-7.  Using the same technique, planned expenditures can be plotted, as shown in Figure B-8, from knowledge of the budget and cash flow projections for each sub-task.  Figures C-6 to C-8 are used to measure the progress of the research as described in the section “Execution of the Research”.

 

The Gantt chart provides a powerful visual representation of the work schedule and quite large research projects, typically up to $1million budget, can be managed in this way.  Larger projects, especially those involving many semi-autonomous contributors, or requiring access to expensive equipment that is used on many projects, may require a more sophisticated work schedule to be developed.  This can be done by preparing a critical path analysis or using computer software designed for project management.

 

The final section of the work plan, Implementation, describes how the findings of the research will be implemented.  This enables the researchers to involve those who will be responsible for implementation in the research, and to allow the plans for the implementation activities to be developed while the research is in progress so that there is no delay between completion of the research and implementation of the findings.


 

Figure B-6:  Example of a Work Schedule Using a Gantt Chart

 

 

 

  Figure B-7: Planned Progress

 


 

Figure B-8:  Anticipated Expenses

 

 

Execution of the Research

 

As noted in Chapter Three, technical progress and financial expenditures must be tracked, and monitored, throughout the period of execution of the research.  Charts similar to those illustrated in Figures B-6 to B-8 are suitable for this purpose.  As an example, Figures B-9 to B-11 illustrate the status of the project described in the previous section as it might have been reported in December 1996.  The data indicate that there were delays in beginning Task 2, but, except for work on the performance tests (Task 2c), the project was back on schedule.

 

Where a greater level of detail is required, as for example, where different budget codes must be reconciled, comparisons between planned and actual expenditures can be made by category, e.g. salaries, travel, and equipment.  In this way, variations can be detected early in the process, and steps taken to bring the project under control.

 

 

Figure B-9: Completed Progress Schedule for December, 1996

 

 

Figure B-10: Overall Progress to December, 1996

 

 

Figure B-11: Expenditures to December, 1996

 

Implementation and Dissemination of Research Results

 

An NCHRP study of technology transfer methods and implementation practices included a survey of transportation agencies and an analysis of case studies (10).  Much of the information in this section is based on the data collected in the study, and from which the recommendations presented in Chapter Three were developed.  Implementation “success” was defined in terms of timeliness, effectiveness, and scope of use.  The study examined success from three perspectives:

·                     Characteristics of the research results

·                     Characteristics of the implementing organization

·                     Characteristics of the implementation process

 

From each perspective, the relative importances of factors that promote implementation, and those which act as barriers to implementation were rated, by participants in a workshop, on a scale of 1 to 5. The results are given in Tables B-1 to B-3 (the data were obtained from figures contained in reference 10).  Barriers to implementation have been expressed on a scale of -1 to -5 (-5 being the greatest barrier).

 

Characteristics of Research Results

Some attributes of the research output can impede implementation.  Most obviously, if the research does not meet the needs of potential users, the results will not be put into practice.  Also, if users do not see evidence that a new product or process has been adequately tested, they may not want to be guinea pigs.

 

On the other hand, research results are more likely to be put into practice rapidly and effectively, if the researchers have accounted for real-world situations.  In many cases, pilot or demonstration projects are useful steps toward full implementation.  Their purpose is to evaluate the new products or processes under operational conditions, and make any necessary improvements, before full implementation on the in-service system.  This is important to reduce the risk of failures.  Even when full-scale implementation is adopted, the product or process may be designated experimental for a prescribed period to ensure that there is a systematic evaluation of the how well the changes are working.

 

Site selection for demonstration projects is much more complex than it might appear to be.  The objective is to evaluate the new product or process under conditions, which are representative (with respect to site conditions, difficulty of work, contractor expertise, time of year) of the anticipated future applications while minimizing the risk of delay or failure.  Frequently the work must be incorporated in contracts already awarded, or about to be tendered.  Each has its disadvantages.  Changes to existing contracts are often expensive and the contractor may not have the necessary expertise.  Contracts not yet awarded are subject to shifts in award schedules, often resulting in projects being moved from one construction season to another.  Rarely will there be an ideal candidate and the selection must be based on compromises between ideal and available conditions.  The longer the time available to plan a demonstration project, the greater are the chances of locating site meeting a larger number of the selection criteria.

 

Table B-1: Factors Pertaining to the Characteristics of Research Results

 

Table B-2: Factors Pertaining to the Organization

Table B-3: Factors Pertaining to the Implementation Process

 

Characteristics of the Organization

The most important institutional barriers to implementation include organizational inertia, risk-averse behavior, management discomfort with change, and inadequacy of resources.  As noted in the section describing the characteristics of research results, new products and processes entail a degree of risk.  Some will prove not worth their cost, and some may not function effectively.  The organization must be prepared to encourage staff to innovate, and to support them even when new ventures are not successful.  This includes a commitment on the part of senior management to provide resources for implementation efforts and the willingness of senior officials to champion innovation.

 

Characteristics of the Implementation Process


Collaboration between the researchers and the users throughout the research and implementation phases is the key to the successful use of research results.  Many agencies involve the researchers, in-house or contract staff, in the implementation efforts and technology transfer.  Commonly, this involves giving seminars or training sessions to agency staff.  Depending upon the circumstances, it could also involve evaluating the success of demonstration projects or results designated experimental.  However, some agencies deliberately limit the involvement of researchers in the implementation process.  The main reasons for doing so are that the contractor is not familiar with the internal process of the agency, and not necessarily expert in implementation efforts.  It is also recognized that the implementation effort is substantial, and often different disciplines from those needed to conduct the research.  In cases where the researchers are not involved directly in the implementation efforts, it is important that they be accessible.

 

Cost is frequently a major impediment in implementing new products, and procurement rules that prohibit sole-sourcing can be a major obstacle.  Targeted funding for implementation, and flexibility to incorporate experimental features in construction contracts are effective ways of dealing with these problems.

 

Evaluation of Research Projects

 

This section describes the tools that are available for use in the evaluation of research projects.  Most of the information is taken from reference 11.

 

Expert Opinion

This method involves obtaining the opinions of people who are knowledgeable about the subject of the research and the impact of the project being evaluated.  The experts can be internal or external to the agency.

 

Expert opinion is often solicited in the project selection phase.  The opinion can be provided on an unstructured basis, or structures using performance indicators such as the quality of the research, level of innovation, or economic impact.

 

User or Client Opinion


This method involves obtaining the opinions of clients, often through technical advisory committees or project panels.  Like expert opinion, client opinion is most commonly sought when problem statements or proposals for research projects are reviewed.  Surveys are a useful method of obtaining client or user input in situations where there are numerous users, or the users are widely separated geographically.

 

Cost-Benefit Methods

Cost-benefit analysis seeks to assess the project in terms of the economic and social benefits generated for a particular “referent group” (e.g. an agency, a region or society as a whole) as well as the economic and social costs incurred by the referent group to carry out the research and implement the results.  The definition of the referent group is important because it defines the bounds for the analysis.  There are a number of general principles associated with conducting cost-benefit analyses of transportation-related research projects.

 

·         There are three main types of costs associated with a research project that must be taken into account in the analysis.  These are the costs of generating the research results, the costs of introducing and supplying the results to the end users, and the costs incurred by the end users to implement the results.

·         The benefits of the research project are valued at the price society is willing to pay for them.  The assessment of benefits includes not only those for which prices are paid, but also benefits associated with, for example, reduced environmental damage, better air quality and improvements in health and safety, even though such benefits may be difficult to quantify.

·         Once they have been identified, all the benefits and costs associated with the project are compared using a common measuring system of constant dollar values.  Allowances can be made for future changes in relative real prices, but no adjustments are made for the future changes in the general price level as a result of inflation or deflation.  All costs and benefits must be discounted to their present values through a social discount rate before they can be compared to determine the net benefit of the project to society.  The selection of the discount rate is controversial, and has a major impact on the results of the analysis.  The discount rate is intended to be the social discount rate, which represents the present generation’s weighting of benefits and costs to be borne by future generations.  In the private sector, the discount rate is sometimes taken to be the difference between the prime interest rate and the rate of inflation.


 

 

 

 

 

The net present value is given by the formula:

 

 

where    Bi = the benefits attributable to the project in year i

Ci = the costs attributable to the project in year i

r = the discount rate

N = the number of years over which benefits and costs (compared to the

“base case”) can be estimated, usually 5 to 10 years.

 

·                     The research project being analyzed must be incremental, and the benefits must be attributable to the results of the research.  “Incremental” means that the results would not have been available in the absence of the project.  “Attribution” deals with the extent to which the benefits are actually due to the research results.  This is extremely important in defining the “base case” which is rarely the situation before the project was undertaken, but more properly it reflects benefits that would have occurred even if the project had not been undertaken.  This includes the incorporation of developments made elsewhere, and changes that would have occurred from sound management strategies.  Given the uncertainty of research, many evaluators find cost-benefit analysis impractical, except possibly for projects which have been completed for some time, and which have well-defined benefits.  On the other hand, cost-benefit analysis can be a useful method for deciding which potential projects are most attractive, and for demonstrating the value of investing in research.

 

Further guidance and examples of the application of cost-benefit analysis to research projects are given in reference 11.

 

Case Studies


Case studies involve a detailed and thorough analysis of research projects, and seek to track and document the evolution of the impacts associated with the research.  The advantage of case studies is that they afford the best opportunity of identifying the relationship between the research study and the impact of the research results.  They are particularly useful for evaluating projects that benefit the public good, rather than using cost-benefit analyses that assign dubious values to some impacts such as the value of a human life.  Because the results are usually qualitative, it is difficult to use case studies to compare the value of projects, or to aggregate the results to demonstrate the value of research programs.

 

Performance Indicators

There are two main categories of performance indicator methods, partial indicators and integrated partial indicators.  The first method involves the collection of information items, each of which provides some insight into the project such as cost, time, risk and potential outputs.  The method of integrated partial indicators involves the collection of the same kinds of information, except that it is integrated in some way.  The method usually involves summing the partial indicators to arrive at a “bottom line score”.  The most common approach is to evaluate each project with reference to a specific set of criteria (e.g. cost, feasibility or risk, likely impact, relevance to the agency mandate), assign a score and a weight to arrive at an overall score for the project.  This method is often used to select individual projects, and to evaluate responses to requests for proposals.