Judge’s Commentary
Moody’s Mega Math Challenge 2017
Kelly Black, Ph.D.
Department of Mathematics
University of Georgia
Download this paper.
Introduction
The student teams that took part in Moody’s Mega Math Challenge 2017 were asked three questions about five parks administered by the U.S. National Park Service (NPS). The questions required students to estimate future sea levels at each of the five parks, estimate the relative vulnerability of each park, and then recommend how to allocate scarce resources to best protect all of the parks. The questions posed are difficult, and researchers and policy makers at NPS have struggled to answer them. While this may have been the most difficult set of questions ever posed in the Challenge, the students’ submissions continue to impress the judges—a larger percentage of the papers read made significant progress compared to previous years.
Between the improvements in the students’ preparation and the increased difficulty of the questions, the judges needed more time to read the student submissions this year compared to previous years. One side effect of the difficult questions was that every paper had at least one significant flaw in either the writing or the mathematics. It was a difficult task to compare the teams’ papers, and the judges had to decide how to balance the students’ outstanding efforts with the unfortunate mistakes associated with exploring a hard problem in a short time.
To figure out how to balance the considerations we recognized that the Challenge consists of three important parts. These include the development of the model, the analysis of the model, and the documentation of the model. The observations about this year’s event are split into three parts to mirror these three aspects. The first part of the commentary explores a number of observations about the students’ modeling efforts. The second part explores different aspects about the mathematics associated with the analysis of the models. The last part offers some observations about the students’ writing and the presentation of the students’ results.
Modeling
Three important aspects of the process of creating mathematical models are examined. The first aspect includes the important considerations required in the development of mathematical models. The second part explores the role of units in modeling. The final part focuses on the role and use of regression techniques.
Modeling Considerations
One step in creating a mathematical model is to decide which phenomena have the greatest impact. This year the primary focus is the changing sea levels associated with climate change. Unfortunately, climate scientists do not agree about which phenomena has the largest impact on sea levels. Some student teams assumed that melting ice has the largest role in changing sea levels while others assumed that the change in density of ocean water due to thermal expansion plays a larger role. [2]
The judges did not adopt any one standard for this important consideration. It was more important that the teams clearly state their position and provide some justification. A citation with a reference was a sufficient level of justification. Teams that clearly stated their assumption and provided some minimal level of evidence to justify their decision were given wide latitude as long as their resulting model was consistent with their assumptions.
Another important consideration is the local topology of an area within and surrounding a park. The gradient of the shoreline is important in determining the impact of changing sea levels around a given park. For example, a park with a large amount of land with a small change in elevation around the shoreline will likely suffer greater changes in land loss with a rising sea level compared to a larger park that has steep terrain around its shoreline.
The teams that addressed this important aspect generally received greater consideration with respect to their modeling efforts. Teams considered the local terrain in a variety of ways. Some examined the average angle of the shoreline with a horizontal plane, some made use of the sine of the angle or the tangent of the angle, and some teams made use of an average slope. Each of these approaches were considered appropriate as long as the results were integrated into the final model in a consistent manner.
A third consideration is the methodology a team used to compare different variables. The questions required teams to balance the impact of a wide range of factors such as sea level change, wild fires, and other phenomena. A number of teams struggled to determine ways to balance the impacts of the different phenomena. One common approach was to convert the impacts into dollar costs. For example, to compare the impact of land loss to the impact of damages due to fire many teams calculated the cost of the land loss as well as the costs associated with the damage caused by fires. The judges looked for a consistent way to compare different impacts and expected the teams to carefully describe how they made their comparisons.
The final consideration is the interplay between the three questions. In previous years the three questions were somewhat independent of one another and allowed students to construct different models for each of the three questions. This year, the questions were closely related, and the teams had to demonstrate a broader understanding of the modeling process. In general, the modeling process usually proceeds with the construction of a series of models.
The questions posed in this year’s Challenge required teams to build on and add to their models. The first question was relatively straight forward. The second question required teams to adapt their first model and add to its complexity. The third question also required teams to build upon their prior results, and the teams had to continue to include more complexity within their models.
The third question explicitly required the teams to make use of their results from the second model. This is a difficult task and requires a strong understanding of the models as well as an intuitive understanding of the process of mathematical modeling. It is a sophisticated procedure, and most teams struggled with this aspect of the Challenge. Teams that were able to adapt their models from the first two questions and extend their prior models for the third question were more likely to advance further in the judging rounds.
Units
Every year the topic of units is something that judges explicitly discuss when trying to compare submissions. The topic of units took on an even greater role in the exchanges between judges this year because many teams struggled with ways to compare the impacts of different physical phenomena.
Among professional modelers, a comparison of units is generally the first step taken when reading other people’s work. In general, a good model should have units that are consistent, and variables should be directly compared only if they have the same units. For example, it does not make sense to compare a variable whose units are meters with a variable whose units are a currency. Also, it is difficult to attach a meaning to a variable whose units are an odd or unfamiliar combination.
Unfortunately, the importance of this issue was greatly magnified this year. A number of researchers have adopted methods to calculate risks that involve adding terms with different units. For example, one widely used calculation made use of the Coastal Variability Index (CVI) [4]. Their definition of the CVI makes use of a linear combination of a wide variety of phenomena with widely varying units.
To make matters worse, another research group has defined a variable with the same name [5], the Coastal Variability Index, and their calculation makes use of a geometric mean of a large number of other variables. Their approach appears to be reminiscent of a Cobb-Douglass production model [3] which is generally used in economics due to its ease of use in calculating coefficients using standard regression methods. The resulting units from this alternative calculation of a CVI represents an even more difficult study in the comparison and abuse of units.
Unfortunately, in this situation there is little that a professional modeler can do other than carefully cite the source and try to provide insight and analysis into the results. Teams that made use of these existing models and provided proper citations and references were granted a great deal of leeway. The unfortunate side effect was that it was quite difficult to compare different papers using a predefined rubric. The papers that were examined in the final rounds of judging offered a wide variety of approaches, and the judges required a great deal of time to read through and discuss the various approaches. Careful consideration led to ways to achieve a balance and decide on a fair way to compare the students’ efforts.
Regression
Another modeling topic that comes up every year is regression. In previous years the use of regression to construct models based on data declined, but this year a larger number of teams made use of regression to construct their models, and in many cases it was justifiable and appropriate. Furthermore, many teams took explicit steps to justify their use of regression prior to determining the values of parameters.
The models that the teams created for the first question varied a great deal. Many teams used a linear relationship between time and mean sea levels while other teams argued that the rate of change must increase because of the strong evidence that the average global temperature is not increasing at a constant rate. These teams commonly chose to use a quadratic model, but many teams decided to use an exponential model. Another common model was a logarithmic model. A small number of teams chose to use a logistic model arguing that the mean sea level must eventually level out.
The judges were generally accepting of all the different models as long as the teams provided good reasons for their choices. The difficulty, though, was how to use the model to predict what would happen in the distant future. For example, if a team chose to use a quadratic or exponential model, then it is difficult to justify that the model’s prediction will be valid 100 years in the future. Unfortunately, this aspect of the question was ignored by the large majority of the teams, and many teams that did consider it explicitly said that it is not a problem to use their models to predict what will happen after a long time period. The creation of a model requires mathematicians to be critical and to understand the underlying nature of the model as well as be able to point out important shortcomings of the model.
This leads to one of the more difficult aspects of the use of regression. Answering the questions required the teams to use extrapolation for predicting future events. Extrapolating is a difficult task, and it is the subject of intense interest for many statisticians. Many teams struggled to establish prediction intervals, and it was clear that they did not have the tools and vocabulary to address a topic that is also difficult for many scientists.
A large number of different ways to predict the possible range of values for the mean sea level were attempted. One common approach was to try to determine the distribution of the mean sea level and predict the range of values based on the distribution. Many teams referred to this as being a confidence interval, but because it is found in the context of regression it is not the same thing. The judges recognize that this is a difficult topic and the majority of students are not familiar with the notion of prediction intervals. We tried to be as gentle as possible in assessing the students’ description of their calculations. The judges are sympathetic to these struggles and note that even trained professionals may not understand the approaches associated with this topic [1].
Mathematics
One of the thrilling aspects of this event is to see the growing sophistication of the students’ ability to construct a model, describe it, and provide a good analysis of the model. At the same time, the students are making use of a wider variety of mathematical techniques and skills. Some mathematical ideas used by teams and how the teams attempted to make use of the ideas for the Challenge are explored in this section.
In particular, three different aspects are explored. The first aspect is the way probability was used to create mathematical models. The second aspect is the way statistical techniques were used to estimate parameters and generate predictions. And finally, the difference between total change and average rate of change is examined.
Probability
A number of teams used a probabilistic approach to create their models. The questions posed explore the kinds of hazards associated with different locations, and the use of ideas found in probability is a good way to quantify risk. Part of what makes the questions difficult is that they involve low frequency events. Many teams made use of Poisson distributions to create estimates of probabilities. The majority of these teams correctly identified the importance of estimating the parameter associated with the Poisson distribution, and they commonly made good decisions on how to make use of their results.
One common problem, though, was to mix up the use of a geometric distribution and a Poisson distribution. This is a mistake we commonly see undergraduate students make, and given the nature of the two distributions it is easy to see the source of confusion. For the most part, though, the student teams demonstrated due diligence and provided appropriate analyses. The nature of the questions that were asked this year differed from previous years, and it was encouraging to see that many teams were able to adapt and react in an appropriate manner.
Statistics
As noted above, the proper use and description of statistical techniques is a difficult topic. In previous years it was common to see mistakes in the use and interpretation of statistical results. Given the extensive and appropriate use of regression this year, the correct and appropriate use of techniques associated with regression moved to the forefront. In particular, a number of teams struggled to determine either the extremes in the possible values of the mean sea level or to determine a prediction of the mean sea level. The teams commonly referred to their calculations as being confidence intervals when their results were something different.
One common task was to construct a model, make use of regression to estimate parameters, and then use the resulting functions to predict what would happen far into the future. I was not aware of any paper that made explicit mention of the prediction interval for the mean. This is understandable because it is an advanced topic, and we do not expect students to be able to make use of such a technique. On the other hand, it should not be referred to as a confidence interval since that is a term with a specific definition.
We made every attempt to be understanding since it is a difficult idea with many common misconceptions. Fortunately, a number of teams provided good descriptions of what they were trying to do. A team that mislabeled their calculation as being a confidence interval but went on to clearly describe their goal and what they did was more likely to make a favorable impression. A team that was able to clearly express their goals in a coherent way and was respectful of the mindset associated with statistical calculations demonstrated that they had a good understanding of the possible pitfalls and limitations of statistical analysis.
Another common task was to try to estimate the possible range of values to be expected given the prediction of the mean sea level at a point in time in the future. The students generally did a good job of describing the quantities that they were trying to determine. It is a difficult topic, but many teams did convey a sense that they understood that the sea levels followed a distribution, and the teams struggled to define and describe the distribution. Teams that provided insight into their goals and conveyed a sense of understanding of the idea of the distribution were generally given the benefit of the doubt.
Total versus Rate Of Change
Another recurring issue is the confusion between total change and rate of change. This year was no exception, and it was exacerbated by the confusion over units. A large number of teams calculated a model to determine the change in the mean sea level given a time in the future. Many teams were not careful to identify if their model was for the change in mean sea level per year or the overall change in sea level from some standard benchmark.
When it is not clear what quantity is being calculated it can become difficult to determine what calculations make sense. If those calculations are not performed in a consistent manner or are not clearly defined then it can be quite confusing when reading a paper. For example, a team may define a linear model to model the mean sea level over time. In one part of the paper they may use the model to estimate the mean sea level but in another part of the paper they may add the function over a number of years to get a total change. When reading a paper a judge can easily get confused and may have to start going back and forth between different sections to decide what quantity has been defined. When a judge is in this position, it is often an indication about the readability of the paper and raises questions about the model itself.
This issue of units and the meaning of the model can be illuminated through the use of graphs and figures. Graphs should be clearly annotated and the axes labeled. If a question comes up about the meaning of a particular definition then one common way to help sort it out is to go and look at any relevant plots. It is expected that the units will be clearly stated on the plots, and the team’s description of the figure within the text should help make it clear how to interpret the quantity being calculated.
Technical Writing
As noted above, it took much longer to read a given set of papers compared to previous years. This was partially due to some of the issues noted above, but also due to improvements in the students’ writing, as well as seeing a larger number of teams able to make good progress on the problem. Overall, the teams’ writing and mathematical abilities continue to improve, and we are grateful for the coaches and teachers who continue to adapt and help their students raise their level of achievement.
In previous years, I have included a section on writing, and my observations have been mixed. This year, however, the majority of my observations are positive, and many of the good things that we saw this year are noted. In particular, the improved use of citations is discussed. Next, the overall flow and structure of the documents have seen improvement. Also, the executive summaries have seen dramatic improvement. Another notable advancement is the improved integration of assumptions as well as justifications in the students’ reports. Finally, a few notes on the how the students’ analysis of their models has changed.
Citations and References
The percentage of papers that include both citations and references continues to increase. In the past a paper with both stood out from the other papers. This may have been the first year that papers that include both were not seen as being out of the ordinary, and this year papers that had references at the end without citations within the text were more likely to be seen as missing an important writing component. It is likely that in the future the expectation will be for papers to include both citations and references while other papers will be working from a deficit.
One thing that was missing, though, was the use of proper citations and references for the software that was used. For example, a large number of teams simply said that they used a spreadsheet to calculate coefficients using regression. It would be better to be specific and state the particular software and version used.
Overall Structure
The papers tended to be more complete this year, and the overall structure of the papers seemed to be easier to identify. Many teams continue to include a table of contents which is helpful. The general structure of the papers tended to be more consistent with current academic practices. This is a trend that will be helpful for students making the transition from high school to universities, and it is a place where the high school coaches and teachers are well ahead of their colleagues in the universities. In my personal experience, students at the university level have a difficult time translating what they learn in their writing courses to their mathematics and science courses, and this is an excellent example of how the preparations for the Challenge are providing direct help in preparing our students.
There is one particular thing to note about the structure of the papers this year. It seemed like it was more likely for each section of the paper to have a short summary and a brief outline of the section. This is another welcome change, and it is something that makes it much easier to read and understand the students’ work.
Executive Summaries
We have seen steady progress in the teams’ executive summaries every year. I was expecting this trend to begin to level out, but there was clearly a greater number of teams able to use their cover page to give a brief overview of the problem, provide a general idea of their approach, and also give specific details. The inclusion of specific results stood out this year, and it was more common to see papers that provided conclusions in the executive summary.
The trend is quite clear with respect to the executive summaries. In prior years a paper with a good executive summary stood out from the others. This year I came to expect to read a strong executive summary, and a paper with a summary that did not include the three aspects noted above was at an immediate disadvantage.
Assumptions and Justifications
One of the biggest surprises this year was the increase in papers that included justifications for the teams’ assumptions. This is a nice development that again reflects well on the teams’ mentors. We understand that the time limit is a difficult burden, and the inclusion of justifications for the assumptions is a welcome addition to the teams’ reports. Adding the justifications does require a non-trivial amount of time, but it does convey a sense of the thought that a team is giving to this important aspect of the modeling process.
Most teams included a separate section for the assumptions or placed them at the start of their modeling section. This year it was also common to see additional assumptions and justifications interwoven into the discussion of the models. This is an impressive feat that is difficult for even professional modelers to do well. It is another reason that the judges spent a longer amount of time reading each paper. To read through and parse the complex writing was much more difficult and time consuming this year. This is an example of how a greater difficulty for the judges is an encouraging and welcome development! Many teams were able to provide more nuanced insights into their models and, more importantly, into their motivations and thoughts about the process that guided them.
Analysis of the Models
One of the key elements of modeling is the editorial process. The basic idea is to start with a simple model that captures what you think are the important factors. The second step is to analyze and provide critical insight of the model. The third step is to go back and update the model with the goal of making it a little better. The whole process is then repeated until the result is something that provides reasonable insights into the phenomena of interest.
The severe time constraint makes it difficult to demonstrate the recursive nature of the modeling process. The difficulty in going back and updating the models for the Challenge amplifies the importance of the second step. It is vital to analyze and pick apart a model as a way to demonstrate the steps necessary to go back and make important changes.
One important part of the analysis of a model is to evaluate which parameters have the greatest influence with respect to the sensitivity of the model. It is becoming increasingly common to see papers that have a discussion about sensitivity. Unfortunately, there seems to be a good deal of confusion surrounding this difficult subject. While the relative sensitivity of various parameters and assumptions is important, it is a disadvantage for a team to mislabel their analysis.
It was not uncommon to come across a paper with a section about sensitivity but to find out upon closer reading that the team’s discussion is based on a different topic. It is better to skip a discussion on sensitivity if the team does not have a good understanding of this difficult topic. Having also judged the Mathematical Contest in Modeling (MCM) [6], which targets undergraduates, it is my opinion that the teams’ efforts in M3 Challenge to address this commonly misunderstood topic exceed that of the teams in the MCM.
On the plus side, one of the changes we saw this year was that it was more common for teams to provide their insights within the discussion of the narrative rather than as part of a separate section. This was yet another factor that contributed to the increased times required to read the papers. As noted above, this is a difficult task and is an impressive development in the evolution of the Challenge. It marks an increased level of sophistication and is further testament to the dedication and efforts of our coaches and teachers who continue to improve the students’ preparation and help make the Challenge an important milestone in our students’ experience.
Conclusions
The questions posed in the 2017 Moody Mega Math Challenge were some of the most difficult tasks that have been posed in the Challenge in its 12-year history. Despite the difficulty, the student teams did an incredible job of creating good models and offering important insights and analyses of their models. The teams continue to improve, and we are inspired and grateful for the tireless work of the students and of the teachers and coaches who help guide the students.
The question this year focused on the impacts of changing sea levels on five parks maintained and administered by the National Park Service. The teams were asked to estimate how sea levels will change at each park, estimate the impact of a variety of related events at each park, and provide guidance about how to allocate limited funding across each of the parks.
The teams continue to demonstrate improvements in the way they develop mathematical models. Their use of appropriate mathematics and their ability to express sophisticated ideas also continue to improve.
The coaches and teachers that help guide and mentor the student teams continue to make remarkable strides, and they take the leading role in helping lift their students so that they can achieve remarkable results under immense time constraints.
Acknowledgments
I am grateful for the help provided by Kathleen LeBlanc and Michelle Montgomery at the Society of Industrial and Applied Mathematics. Their insights and direct help were instrumental in making this a better document.
Bibliography
[1] Altman, Douglas G., and J. Martin Bland. "Improving Doctors' Understanding of Statistics." Journal of the Royal Statistical Society. Series A (Statistics in Society) 154, no. 2 (1991): 223-67. doi:10.2307/2983040.
[2] J. Church et al., in Climate Change 2001: The Scientific Basis: Contribution of Working Group I to the Third Assessment Report of the Intergovernmental Panel on Climate Change, J. T. Houghton et al., Eds. (Cambridge Univ. Press, Cambridge,2001), pp. 639–693 (2001).
[3] Cobb, Charles W. and Paul H. Douglas.. A Theory of Production. The American Economic Review, 18.1: 139-165 (1928).
[4] Vivien M. Gornitz, Richard C. Daniels, Tammy W. White and Kevin R. Birdwell, The Development of a Coastal Risk Assessment Database: Vulnerability to Sea-Level Rise in the U.S. Southeast, Journal of Coastal Research, Special Issue No. 12. Coastal Hazards: Perception, Susceptibility And Mitigation, pp. 327-338 Published by: Coastal Education & Research Foundation, Inc. (1994). Stable URL: http://www.jstor.org/stable/25735608 Accessed: 06-04-2017 12:51 UTC
|
[5] Thieler, E.R., and Hammar-Klose, E.S., National Assessment of Coastal Vulnerability to Future Sea Level Rise: Preliminary Results for the U.S. Atlantic Coast. U.S. Geological Survey, Open-File Report 99-593, 1 sheet ( 1999). |
[6] Mathematical Contest in Modeling, Consortium for Mathematics and Its Applications, https://www.comap.com/undergraduate/contests/mcm/ (Accessed April 2017)