for Engineers and Researchers
- Starting a Study
- Background Research
- Project Scope
- Climate Model Output
- Baseline Analysis
- Future Analysis
- Communication of Findings
- Next Steps
This section describes the preliminary administrative steps required to start a pilot study. Pilot studies are used when looking to gather a basic understanding of a general trend on a specific subject. When investigating a question with the intent of doing further study, conducting a pilot study can answer two questions about research before too much time and effort are invested: is it feasible and does it matter? Pilot studies can determine if this research can even be conducted, or if there is no practical way to continue. They can provide insight as to whether preliminary results justify performing a more extensive study.
Pick a Question
The scientific question that you want to answer or the hypothesis that you want to test must be specific enough to narrow down variables, a timeframe, and a location(s) in order to determine a conclusive answer.
Example: Will climate change have a significant effect on the freezing and thawing of roads in Madison, Maine, in the mid century and end of century as compared to present day?
Acquire a Team
Assemble a team of interdisciplinary specialists and/or students to help you work on the study. This team may involve those in academia, practice, or some combination. Designate team member roles. Keep in mind the limited scope/time of the pilot study. You may want a small team actually working on the pilot study, but a larger “team” that gives you feedback or reviews the pilot work and then may be a part of a larger study.
File Storage System
Choose a file storage system for your team to compile all of the documents and datasets necessary for and created during the pilot study. Organization at this phase will help tremendously if the pilot study is extended to a full study.
This section describes the background research needed before starting the planning and analysis stages of a pilot study. Background research on both the subject matter and on the specific study or project approach is necessary.
Research the subjects and concepts you need to understand prior to data collection. Expect to find an overwhelming number of climate change studies, but relatively few studies specific to climate change and transportation infrastructure. Rely on refereed papers that have been peer-reviewed and agency reports on the topics. Refereed papers can be found via Web of Science, JSTOR, Google Scholar, and the TRB database.
Research similar studies to focus your research questions and to ensure that you are building on previous research. To do this, conduct a literature review of peer-reviewed papers. In addition to searching the transportation sector, consider looking at methods applied by other sectors with a longer historical of climate change analyses including water resources, agriculture, and energy.
Keep in mind that most published articles are of in-depth studies and a pilot study will be less complex and have less data being analyzed. However, this background research still gives you a good idea of past practices and a general idea of the components to use.
Develop a reference database at this early stage to track your references. Consider using a reference database tool to help track citations such as EndNote, Mendeley, and Zotero.
This section reviews the key steps in creating and finalizing your project scope. It is important to complete a consultation with specialists who have the knowledge and experience with the climate research process and are able to provide input on your plan.
Scope of Work
Create a detailed project scope that outlines the tasks required to accomplish your goal.
When planning your study keep in mind that the idea of a pilot study is that it can be conducted in a reduced time frame and by one or a few people. These proofs of concept generally take on the order of a year (6 to 18 months is typical). Well-designed pilot studies help create networks, lay the groundwork for a proposal of a larger study, and provide a learning opportunity for the research process.
If there are gaps in your team, consider reaching out to the ICNet climate scientists and infrastructure engineers for feedback. Make revisions to your research plan or project scope based on the input you receive.
- Scope of work example
- Contact list of ICNet Climate Scientists and Engineers
- Summary of findings from ICNet Climate Scientist discussion
This section covers the collection of output from your desired source. If your team chooses to use the Bureau of Reclamation site, as recommended in the Federal Highway Agency (FHWA) guides, refer to the detailed instructions on how to gather those data provided on the ICNet and the FHWA.
Caution: There are a tremendous number of GCMs, emission scenarios and source for model output as well as analysis methods. Do not let your pilot study get away from you and become closer to a larger study. You can transition what you have learned from this pilot project to a more comprehensive research experiment with a different goal and plans.
Desired Variables, Resolution, Geographic Area, and Timeframe
Decide on the following:
- The variables you want to investigate (e.g., temperature, precipitation, etc.)
- The time and space resolution for your climate model output if using gridded output, or whether station information is more desirable for the study.
- The geographic area or the individual station sites of the study (coordinates)
- The timeframes for the analysis including
- Baseline: historic period for which you have both climate model output (later referred to as baseline output) and documented observations (later referred to as observed data) of your environmental variables (e.g., temperature data from 1950 – Present)
- Future: the period you are interested in studying (e.g., temperature data from Present – 2100)
When choosing an emissions scenario(s) for a pilot study, there are two options. You could choose a single scenario or two scenarios. Choosing two scenarios better incorporates the variability of the future; it is recommended to choose one high and one low emissions scenario. However, scientists and researchers do not usually use RCP 2.6 anymore because we have already exceeded that trajectory. If you choose a single scenario it is best to choose the highest one (RCP 8.5) to get an idea of the worst possible scenario.
Source of Output
Deciding where to obtain climate model output is based on the emissions scenarios, variables, resolution, geographic area, and timeframe you choose, and whether or not you want downscaled data. Not all models provide output for every emissions scenario, so this step can be based on the emissions scenario decision. This choice is based on whether model output source or emissions scenario is more important to your study.
The ICNet has a spreadsheet tool that summarizes some of the leading available climate model output sources. See the Summary Sheet in the resources tab below.
Climate Models and Model Runs
Pilot studies are intended to be on a smaller scale so they generally use one to three models (though that is inadvisable for large-scale studies) and only a single model run. To help guide the choice of which models to use in these cases ICNet members have developed a new tool. This tool classifies the models provided by every source of climate model output based on their reliability and usefulness. For more information, see the “Summary Sheet” in the resources tab below.
The Wiley Online Climate Model Genealogy from Knutti al.’s (2013) study (see resource link below) can help to pick one-three models based on your needs. It is recommended that you select models that seem to capture the most “branches” on this tree. If you only want 3 models you move from right to left until you have 3 subgroups, then you can select a model that ties back to that subgroup.Use both of these tools to make the best model selection. It is recommended that pilot studies use just use one run of the selected models.
Translation to Engineering Variables
The output from the climate models can then be taken and used as an input into other models or mathematical formulas that produce the final variables to be examined in a study.
Downscale if Needed
A baseline analysis (section 5) can help to determine if further downscaling is required to achieve the resolution required for the pilot study.
Remember, models are not designed to match the day-to-day variability of the observations, but the overall statistics, such as mean, standard deviation, trend, etc. should be similar over a twenty to thirty year period.
Set aside time to review all of your climate model output decisions. Changes may need to be made to get the best quality output for your research. Consider your time and resources and take caution to limit the data analysis. Estimate the actual amount of data that your studies will yield and compare it with your scope of work and timeline to ensure that the study is logistically feasible. It is easy to get overwhelmed by the plethora of climate model output available but do not let the study develop past the scope of a pilot study.
Follow the instructions provided by your chosen source to input your decisions and collect output. If you are using CMIP3 or CMIP5 data from the Bureau of Reclamation site, review ICNet’s instruction guide for data collection and data processing without using the interface.
- Climate Change Analysis Technical Guide
This section covers how to run and collect output from the “baseline” or “historical” section of your timeframe. It also reviews how to conduct a baseline analysis to assure the selected models are replicating historical conditions and will produce credible information for future analyses.
If you are running the climate model output through an engineering model, the baseline analysis should run both the observed data and the baseline climate model output through that model to produce the desired variables. If you are not running the output through your own model, use the observed and baseline climate variables (e.g., precipitation or temperature).
The goal of the baseline evaluation is to determine if the climate model output is reasonable as compared with historical observations. A baseline analysis should be relatively quick and simple. When analyzing the baseline output, use a 20-30 year window.If any of the models produces results that are drastically different than the observed, consider removing those models from your analysis.
If many of the models differ, then your output may need to be downscaled further to better reflect the climate in that area. This typically occurs if you are using GCM model output or using extreme values (e.g., annual maximums). For a pilot study, downscaling adds a level of complexity that may not be warranted. An alternative to downscaling is to compare model output from the baseline period to that from the future period to identify changes or “deltas” in your results. Before proceeding to your future analysis, you might get advice from the climate experts on your team.
This section outlines how to conduct the “future” portion of your research. This methodology follows a “top-down” approach to climate and infrastructure research, but other methods including the “bottom-up” approach are also available. The top-down approach is applicable when considering how a variable changes over time. The bottom-up approach is applicable when attempting to characterize the vulnerability of an asset. It typically examines where or when a parameter of interest exceeds a particular threshold in a larger region.
The future analysis should use the same climate models that were identified as reasonable in the baseline analysis. Run the time series or summary parameters of the climate model output through the engineering model. Each combination of climate model, emission scenario, and model run should be analyzed individually initially.
Compare the model output for the future period to the observed data or model output from the baseline period. During the evaluation process, questions often arise about when it is and is not permissible to combine results from multiple models during analysis. It is never permissible to average output from the climate model before running your engineering model because averaging will remove the extremes from the climate parameters. However, it is permissible to average the final variables after the output from the climate model has been run through your own engineering model, and the final results have been reached.
- Freeze Thaw future output comparison to observed (appendix)
- IPCC on climate models
- Does climate adaptation policy need probabilities?
This section outlines recommendations for communicating your research results. Regardless of the findings, the results from your research as well as information about your research process will add to the relatively limited information available on conducting climate change and infrastructure research or understanding infrastructure vulnerability.
Depict your desired final parameters visually. These visuals may include line graphs, bar charts, and box plots. Choose the best graph to display each category of the parameters. For example, magnitude is usually depicted as line graphs and bar graphs whereas dates are usually depicted with line graphs and box plots.
The technical communication should document your research questions, methods, and findings. For the methods, create a table listing the climate models and emission scenarios used in your study as well as the source of the climate model output. Document the process used for the model output (e.g., models were removed from the study, was additional downscaling applied). Briefly depict your baseline analysis results. For your future results, choose the visuals that best depict your overall findings.
Pilot studies are often not publishable, as they are not extensive enough to provide any conclusive results, however a summary of your findings may be useful to others in the future. However, it is still crucial to document your results and get them out into the public so that others can learn from your research or at the very least, not repeat what has already been done.
ICNet Research Page
ICNet maintains a database of pilot studies and offers a means to communicate your findings and to help others find your work. Even if you do not anticipate writing a paper based on your findings or the results of your study are inconclusive, a summary of findings may still be useful.
- On how to make graphs readable: Visual Analysis Guidebook
- On prose and visualizations: Visual Communication—Document Design, Figures, and Tables
- On visualization: Graphical Display for Effective Communication of Research Results
- The ICNet Guide to Climate Communication
- Guide to creating visuals in Plot.ly
- Submit your research to [email protected]
- Case Study Research Report (coming soon)
This section outlines what to do after a pilot study has been completed. If the findings are inconclusive or the research team decides it is not necessary to conduct a more in-depth study then the communication of findings (Step 7) concludes the project. However, if a pilot study shows that there is an impact that needs to be examined in a given system, a more fleshed out study or full-fledged research project can be conducted and the steps to do so are given here.
Conducting a larger study might mean expanding your variables, adding more models (from climate or infrastructure), expanding the number of locations, reframing your question, and checking for other factors that might influence the study outcome. For more information on creating a full-scale study, read our guide to advancing your research in the resource link.