Download this perspective [131K PDF]
Kelly Black
Department of Mathematics and Computer Science
Clarkson University, Potsdam, NY
1. Introduction
The topic for the 2009 Moody's Mega Math Challenge was the American Recovery and Reinvestment Act of 2009. The participants were asked to decide which elements of the bill will yield the largest increases in employment, how long it will take to produce results, and whether or not a second stimulus bill will be required. The student teams interpreted the questions in different ways, and the rich variety of the responses added to the outstanding quality of the submissions.
We are grateful for the effort and insights provided by the students who took part in the competition. We are also grateful for the work of the coaches who helped to prepare the student teams. The dedication shown by these advisers and students continues to augment the success of the Challenge.
In response to the efforts of the teams, we hope to provide some insight into this year's competition. Here we first provide an overview of the judging process in section 2. Next we discuss some of the negative aspects of some of the entries in section 3 and then discuss the positive aspects in section 4. Finally, we offer some general observations about modeling and the presentation of results in section 5.
2. The Judging Process
There are two phases of the judging. The principle goal of the first phase is to determine which papers require a more careful reading and should proceed to the second phase. The judges have limited time to read each paper, but two different people read every submission. If there is disagreement on a particular paper, it is read by a third judge. Because of the limited time available for each paper, the administrators and judges make every effort to err on the side of caution and assume that a paper should be given another reading if it is not clear how to treat it in this early stage.
For a paper to pass through the first round, it must be well written, and the judges must be able to quickly determine the approach and conclusions of the student team. It is vital that a paper have a wellwritten summary to be able to make it through this round. The summary should include an overview of the strategy adopted by the team as well as specific conclusions. Judges are not always able to evaluate all of the details of the modeling activities, but they are able to determine if the team has submitted a complete solution.
The second stage of the judging takes place about two weeks after the first stage is complete. In the second round, a new group of judges is assembled including a small number of judges from the first round. The judges are sent a copy of the problem statement prior to meeting, and one of the first things we do is discuss the problem and determine what is being asked and what the important aspects of the question are.
The second stage of the judging includes many rounds. The judges are able to spend more time reading each paper as the rounds progress. The goal of the second stage is to determine which papers offer the best responses and can compete for the highest prizes.
For a paper to do well in this second stage, the writing must be of the highest caliber, and the team must offer a complete and detailed solution. The judges are able to spend much more time reading the papers and the technical details are more important. A paper must be well balanced and consistent to be able to survive to the last rounds of this second stage.
The second stage of the judging takes place about two weeks after the first stage is complete. In the second round, a new group of judges is assembled including a small number of judges from the first round. The judges are sent a copy of the problem statement prior to meeting, and one of the first things we do is discuss the problem and determine what is being asked and what the important aspects of the question are.
The second stage of the judging includes many rounds. The judges are able to spend more time reading each paper as the rounds progress. The goal of the second stage is to determine which papers offer the best responses and can compete for the highest prizes.
For a paper to do well in this second stage, the writing must be of the highest caliber, and the team must offer a complete and detailed solution. The judges are able to spend much more time reading the papers and the technical details are more important. A paper must be well balanced and consistent to be able to survive to the last rounds of this second stage.
3. Negative Reactions
Some comments about this year's event are discussed. Specifically some of the things that elicited a negative reaction to an entry are given below. The first two issues discussed are relevant to the first round of judging, and these two issues usually prevented an entry from proceeding into the second round. The final two issues are relevant to both stages of judging, and these two issues made it difficult for a paper to move forward but not impossible.
3.1. Repeating Dogma
A number of the entries simply declared whether or not the bill would achieve the stated goals, and the supporting arguments were based on the authors' personal views. Some of the entries that fall in this category supported the efficacy of the bill while others did not. Teams that presented arguments with little or no mathematical support were not likely to make it past the initial round of judging. Regardless of the personal views of the team, they should have constructed a mathematical model describing the economic impact of the bill. The teams are expected to provide some justification for their model, and proceed with an analysis of this model. Any entry that simply stated conclusions without mathematical analysis was unlikely to receive much credit from the judges.
3.2. Regurgitation and Detailed Discussion
The bill in question has come under intense scrutiny, and there is a large amount of information available. Some teams simply restated conclusions of other sources. This is closely related to the issue in the previous subsection, and a number of entries took this approach.
Another group of teams merely repeated the mathematical analysis of other sources. Many did provide mathematical models, but their subsequent analysis relied solely on work done by others. In the worst cases, the teams simply repeated the predictions found at various U.S. government web sites. Teams that took this approach were not likely to receive high marks in the first round of judging. Some of the teams did do a good job of interpreting the analysis and relating the various graphs that are available. Teams that took this extra step were more likely to receive a higher score, but they were not likely to proceed very far in the later rounds of the judging.
3.3. Citations
There is a large amount of information available that helped provide insight into the problem this year. The information comes in a variety of forms ranging from basic economic data to the basic economic principles used to develop the subsequent mathematical models. The way the teams cited their sources varied widely.
A number of entries provided very little in the way of citations. Some teams even provided graphs and figures copied directly from various websites with no attribution. Papers that provided few citations or footnotes were likely to make a negative impression on the judges, especially in the later rounds.
Another group of entries included an extensive reference section. The text of the discussions, however, included little or no citations. It was a bit like reading a mystery novel wondering where the teams found each piece of information. These papers were seen in a more favorable light than those without any references, but the majority of these papers did not fare well when directly compared with those that did include specific citations.
The papers that received the highest ratings tended to include extensive references and citations within the text. The citations could come in any form including footnotes or endnotes. As long as the citation style was consistent, the judges tended to react more favorably to the paper.
3.4. Comparisons
Another set of papers based their mathematical models on extrapolation from other situations. For example, a number of papers simply extrapolated from the New Deal from 1933 to 1938. Other papers extrapolated from Japan or Sweden and Finland in the 1990s.
The reaction of the judges varied from paper to paper. In some cases, the comparison was done in a way that made a positive impression. In other cases, the comparison was done using a simple extrapolation. For example, a number of papers noted that Roosevelt's New Deal took two to three years to see a turn around in the unemployment rate. The team then concluded that the same must be true for this situation.
Teams that took this approach tended to do okay in the early rounds of judging. They did form a model and many performed an analysis of their model, which was enough to warrant a closer examination in subsequent rounds. However, close examination in later rounds could be problematic for some of these entries. If the team did not recognize that there are fundamental differences between the situations and try to account for those differences then the team's entry was less likely to move forward as the judging progressed.
4. Positive Reactions
Here we look at aspects of the entries that tended to elicit a positive reaction from the judges. The first four are general observations. The last aspect is more specific to this year's problem.
4.1. Models and Analysis
The judges looked for a well-developed mathematical description of the economic impacts of the bill. This often came in the form of a system of equations that related the various interrelated parts of the economy. It is a difficult task to describe the motivations for a system of equations and then form the subsequent relationships in a coherent manner. Teams that were able to explicitly form their model and describe it in concise straightforward language were very likely to make a positive impression on the judges.
The judges are not looking for sophisticated and complicated models. We look for descriptions that seem plausible and are well formed. More importantly, we focus on the subsequent analysis of the model once it has been determined. We want to know what the team can conclude based on their mathematical description.
Once a mathematical representation is formed, the judges want to know how deeply the team is able to explore the model. The judges also want to know if the team has thought about the model itself. Every model has deficiencies, and it is important for a team to be able to look at the model and understand which parts of it are good and which are problematic.
A team that explicitly acknowledges the deficiencies in their model is more likely to make a positive impression. We do not expect perfect solutions and consider the short time available. We do expect that the teams have a basic understanding of their model and demonstrate that they thought about potential pitfalls and problems. It is not possible to address every problem, but it is possible to acknowledge and provide some likely approaches for future consideration.
An important aspect to an exploration of a model is an examination of the sensitivity of the model. For example, in this year's contest the teams had to assume the values for a number of parameters. One question that arises is what would happen if one of the parameters were off by a small amount. Teams that examined the impact of a range of values for several of their parameters were more likely to impress the judges and move forward as the judging progressed.
4.2. Assumptions and Justifications
A list of assumptions is an essential and vital part of the modeling process. The majority of papers did a good job of explaining the basic assumptions, which often came in the form of a bulleted list. Some of the teams went further and provided brief justifications for their assumptions. The teams that included well-reasoned justifications for each of their assumptions tended to come across more favorably than those that simply listed the assumptions. The best entries in this respect included brief justifications and citations.
4.3. Different Aspects of the Bill
The question explicitly asked that the students examine various parts of the bill and discuss the impact with respect to each of those parts. A large number of entries did an exemplary job of addressing this issue. The judges appreciated when a team recognized this explicit requirement and acknowledged that this is a time consuming activity.
4.4. Writing
Most of the teams were able to interweave equations and relationships within the text and use proper punctuation. The judges recognize that this is not an easy task, and we were impressed with the overall quality in the writing.
This issue and the previous issue, given in subsection 4.3, reflect well on the team advisers. It was obvious that many of the teams were well supported by their advisors, and their ability to pay attention to the details of the question and carry through with strong writing is a credit to the coaches who helped prepare them.
4.5. Indicators to Watch
Another explicit requirement given in the problem statement was to determine whether or not a second stimulus bill will be necessary. The majority of entries did a good job of addressing this part of the question and teams interpreted it in a wide variety of ways.
Some teams made specific predictions while others offered detailed instructions on how to make the decision and what to expect. This part of the question was open to interpretation, and the judges were amenable to the different ways that a team chose to address this part of the problem. In general, the teams that provided detailed instructions on what factors to watch and offered detailed instructions on how to determine the size of a new bill tended to be seen in the most positive light. The process of describing the procedures tended to highlight the team's understanding of their model. It also tended to highlight their ability to describe abstract information.
4.6. Comparisons
Finally, a number of the teams provided a comparison to the New Deal from 1933 to 1938 or made comparisons to Japan or Sweden and Finland in the 1990s. This approach is included as both a positive and a negative. As mentioned previously, some teams did little more than make simple comparisons. Other teams took a deeper look and made multiple connections. Some teams also recognized that there were fundamental differences between the situations.
The teams that recognized the distinctions and adjusted their model based on those differences were more likely to make a positive impression. In such cases, the teams were able to demonstrate a fundamental understanding and provide insight into the problem.
5. General Comments
Some general comments are included here. These comments apply to the entries this year, but they will most likely be applicable in subsequent years. Some of the comments are addressed in the materials that are sent out prior to the Challenge but should be re-emphasized.
5.1. Summary
The guidelines for the contest explicitly acknowledge the importance of the summary. This is the first thing that the judges see, and the summary can set the tone for how the judges read the rest of the team's entry. A complete, well-written summary immediately indicates that the team paid attention to detail, and the entry is worth the most careful consideration.
The summary should include an overview of the problem. It should also inform the judges what approach the team decided to take to explore the problem. Finally, it should include specific results. The summary should prepare the judge for what to expect. After reading the summary, the judge should have a good idea of the organization and approach, and there should not be any surprises.
5.2. Computational Models
Mathematical models can be complex and commonly require some kind of approximation or series of calculations. The calculations can be done in a spreadsheet or using a program written in JAVA, Python, or other computer language. This was the case in this year's competition, and a number of teams submitted results that came from the output of a spreadsheet or program.
Some teams included a copy of the program in an appendix. This is acceptable, but if the program requires a nontrivial algorithm, the judges are not going to sit down and try to decipher the program. Rather, they will closely inspect the mathematical model and ask whether or not the results make sense.
It is absolutely vital that the team's paper include details about the mathematical model and a brief discussion about the algorithms used to implement the model. For example, a number of entries discussed the mathematical model in very broad terms without offering specifics, and they stated that the team members wrote a program and included the output from the program. Simply stating that a program was written or even providing the code is not an adequate description of the mathematical model.
5.3. Regression
A number of teams made use of regression based on data from past years to estimate some of the parameters in their models, which can be appropriate, but is not a substitute for the development of a mathematical model. Teams that developed a model and used regression in an appropriate way were seen in a positive light. Some teams went further and those that conducted an appropriate hypothesis test also impressed the judges.
However, some entries concluded that the model should be linear, quadratic, or cubic solely by looking at the data. This is a good first step, but the burden is then on the team to explain why they should get a particular order to the polynomial based on an economic theory. Simply concluding the order by looking at the data alone is not adequate, especially when the team does not examine a residual plot and note the patterns in the residuals.
6. Conclusions
Moody's Mega Math Challenge 2009 was an exploration of the American Recovery and Reinvestment Act of 2009. The problem is a current area of intense interest, and a wide range of material is available. The student teams did a tremendous job of synthesizing the material and creating a mathematical description of the American economy.
The judging of the entries took place in different rounds, and the strength of an entry was based on different aspects as the judging progressed. The papers that were examined in the final rounds of judging had the best overall balance with respect to the writing, the mathematical model, and the subsequent analysis of their model.
The quality of the submissions continues to impress the judges. The students are able to convey complex information with impressive writing skills. The teams are also receiving excellent support from their coaches, and the students show a great deal of discipline in bringing together their papers while also carefully addressing all of the required tasks.