2010: Judging Perspective

Download this perspective [49K PDF]

Kelly Black
Department of Mathematics and Computer Science
Clarkson University, Potsdam, NY 13699-5605 U.S.A.

1. Introduction

The questions for this year’s challenge come from topics relating to the 2010 census. Three questions were given:

Should the census figures be adjusted for the undercount? If so, how? If this solution introduces errors of its own, estimate how large they are, compared to undercounts.
What methods should Congress select for apportioning the House of Representatives? Why is this method superior to others for dealing with this issue?
What recommendations should be made to the states to ensure that Congressional districts are fairly drawn? Justify these recommendations.

Once again we were impressed with the overall quality of the student teams and are grateful for the efforts of our student teams. Additionally, we thank the teams' advisers who are a vital aspect of the challenge. The efforts that take place in preparation for the challenge and the activities that take place once the challenge has started help make this a special event.

With this perspective we will try to provide some insight into the judging process. In section 2, some of the specifics about the judging process are discussed. In section 3, some of the things that elicited a negative reaction are discussed. Finally, in section 4, some of the positive aspects are discussed.

2. The Judging Process 

The process of assessing the student papers took place in several stages. The first stages of the judging are designed to decide which papers require closer scrutiny. In the later stages of the judging the process evolves to focus more on deciding which papers offer the best solutions.

Initially, each submission is read at least twice by two different college or university faculty members or industrial mathematician. Papers that receive consistently high scores are passed on to the later rounds. Papers with inconsistent scores are read by a third reviewer.

Papers that are well written and clearly address the specific questions are more likely to receive higher scores in the earlier rounds. The judges do not have extensive time to focus on every paper, and the importance of the team’s summary is magnified. Also, the judges’ impression of the quality of the writing has a greater impact in the early rounds. Details such as clear citations within the narrative and delineation of the assumptions and the question being addressed have a larger impact on the judges’

During the initial stages the judges’ principal focus is on whether a paper should be read more closely in later rounds. There may be some doubt, but if it is not clear without closer reading, the judges try to err on the side of the students and pass the paper on for a more detailed reading.

After the initial rounds the judging changes. The papers are read in a more focused environment, and fewer judges take part in the process. This is done to ensure more consistent scoring. As the rounds progress, the judges spend more time on each paper. To be successful in these later rounds, a paper must be well written and also include a strong modeling component.

Once the judging moves to the final rounds, the judges become more critical, and the papers are read by more judges. To make it to the final rounds a paper must appeal to the majority of judges and must address each part of the problem in some way. Additionally, the paper must be consistent with respect to the mathematics and the exposition.

In this year’s event there were three difficult questions. We did not require that a paper provide a strong analysis for each question. However, to make it to the final round, each paper had to provide some insight into each problem. A paper that did not provide a strong analysis to one of the questions was expected to provide extraordinary insight into one of the other questions.

3. Negative Reactions 

Here we examine some aspects of the submissions that tended to elicit a negative response from the judges. Some of the issues appear to be specific to this year’s problem, but they can be generalized to issues that are seen each year. For example, the use of linear regression techniques without providing technical motivations for the specific relationship is something that we see each year.

3.1. Should the Census Be Adjusted?

The first question can be broken up into several parts. The first part is to determine whether or not the census should be adjusted. A large number of papers examined techniques to adjust for an undercount but provided little motivation as to why it might be necessary. Of those that did, many simply stated that the undercount results in less money for a particular state or district.

This aspect of the problem offers a rich area for conveying an understanding the statistical and mathematical issues associated with the problem. It is not obvious that extensive efforts are required to adjust for the undercount. Few teams recognized that the overall budget is limited and asked whether a change in the undercount would result in a change in the amount of resources disbursed by the federal government. For example, if every district has an increase in the number of people the same, fixed amount of funds would still need to be distributed.

Some student teams explored the disparity between socioeconomic class and race with respect to the undercount. Most of these explorations were limited to repeating estimates from other sources. Very little analysis was done on the implications of the disparities. For example, it was not clear that there was a statistically significant difference in the disparities, and few teams explicitly explored the specific differences that would result for different disbursement patterns given the limitations of the federal budget.

3.2. Simply Repeating Other Sources

A large number of papers this year simply repeated results from other sources. Some of the student teams were able to bring different sources together in ways that offered new insights, but student teams that simply repeated what they found were not likely to elicit a positive reaction in the later rounds of the judging.

An additional problem is the way that student teams made use of references. The majority of teams offered good, complete reference sections. Very few offered
consistent citations within the narrative. Simply putting a list of references together at the end of the paper is not an adequate way to cite other works.

3.3. Trusting Past Census Results

The majority of papers put immense trust in the undercount estimates from prior years. Given the nature of the problem, there is a great deal of uncertainty surrounding these figures. Simply accepting themfrom previous years seems to imply that there is no problem and that it does not need to be addressed.

Magnifying this problem, a number of papers offered predictions based on extrapolation from past censuses. Little or no reason was given as to why past trends will continue, especially in light of the question’s implicit goal to explore new ways to reduce the undercount.

Furthermore, many of the predictions are based on a polynomial fit of the data with little or no physical motivation as to why a particular polynomial should be used. Part of the goal of mathematical modeling is to provide insight into the problem. Simply choosing a polynomial to fit the data absent any fundamental understanding of the process does not offer a way to provider deeper insight into the basic issues.

3.4. Defining the Terms

The problems in this year’s challenge have a long history and have been studied by people from a wide range of backgrounds. A number of technical terms have been developed to refer to specific aspects of the problem. A large number of papers adopted the language used in this field of study but offered little in the way of defining the terms.

For example, a number of papers used the terms “imputation,” “undercount,” and“undercount percentage.” Very few student teams offered a detailed explanation of the meaning of these terms. The few teams that made the effort to discuss these terms and explain their differences made an immediately positive impact on the judges.

4. Positive Reactions 

Here we discuss some of the aspects of the papers that were more likely to elicit a positive response from the judges. All of these aspects are general in nature and can be generalized to future events. These are the things that help a submission stand out and help the judges focus on the details of a paper. When a judge is able to focus on the aspects of a paper that set it apart it generally results in a better experience for a judge and will result in higher scores for the paper.

4.1. Assumptions Clearly Indicated

A mathematical model is derived from a simplified and idealized view of the physical problem. Prior to the derivation of a mathematical model, the basic assumptions and important aspects of the situation must be identified. Student teams that explicitly identify their assumptions prior to the discussion of the mathematical model provide a more coherent framework for their discussion.

Also, it is important that the narrative reflect the basic assumptions throughout the discussion, and the conclusions and derivations should remain consistent with the basic assumptions. This can be difficult to do with the limited time available, especially if the student teams divide the tasks among themselves. It is vital that the student teams devote some time and effort to ensuring that their derivations and conclusions remain consistent with the assumptions.

4.2. Table of Contents

Providing a table of contents is a small feature that offers big dividends. This year, most of the submissions included a table of contents, which provided a road map and was greatly appreciated. It makes reading the paper much easier for the judges, especially in the early roundss.

4.3. Summaries

The summary is the first thing that a judge will see. The instructions for the contest explicitly state that this is an important aspect of each submission, and the judges are aware of this. A judge will read the summary first, and it sets the tone for the subsequent reading of the submission.

A good summary should briefly indicate the question being asked. The summary should indicate the techniques and approach adopted by the student team. Finally the summary should provide a brief statement of the team’s conclusions. There should be no surprises for the judge upon reading the rest of the submission.

It is difficult to bring these three parts of the submission together in a single page. Fortunately, this is one thing that a student team can practice before hand. Taking the time during the event to ensure a good summary will make it more likely that a submission will do well in the earlier rounds of the judging.

4.4. Equations, Tables, and Graphs Integrated Into the Text

The majority of teams do a terrific job of integrating equations, tables, and graphs into their narrative. Every equation should be integrated within the flow of the text and be
properly punctuated. Every table and every graph should be included in the discussion, and the narrative should indicate exactly what is important and what is missing from every table and graph. Student teams should not place a graph on the page and hope that the judges figure out what is special about the figure or how it relates to the team’s results.

A submission that does not do an excellent job of tying together the equations and describing every table and graph will not fare well. It will immediately suffer in comparison to the other submissions. A submission will not move beyond the initial rounds if this important part of the writing is overlooked.

The extent that student teams are able to do so well on this difficult task speaks well of their preparation and the efforts of their advisers.

4.5. Citations and References

This year the student teams did a good job of including references. We do not look for any particular style of providing a reference but do expect the style to be consistent within a given submission. Every source that a student team found helpful should be included in the list of references.

One thing, though, that set some teams apart is the proper use of citations. While most teams offered a good set of references at the end of their submission, few teams offered consistent citations within the narrative. Student teams that provided
citations within the text made an instantly positive impact and immediately stood apart from the other submissions.

4.6. Sensitivity

We expect teams to conduct some sort of sensitivity analysis. Approximations and assumptions about the values of certain parameters must be made in any mathematical model. An important question is how small changes in those parameters impact the results and conclusions.

In the context of this event, the judges do not expect an extensive or complicated analysis, but some attention should be paid to this aspect of the problem. An analysis of the sensitivity can be performed by simply changing the values of some of the parameters by some small amount.

If a small change in the value of a parameter results in a large change in the results, then that indicates an important feature of the model that needs to be emphasized. The team should make it clear which parameters are most important and require the greatest attention.

It is also important to let the judges know that a team took a critical approach to their model and can identify the weakest aspect of the model. When the model is sensitive to small changes in one parameter, that indicates how conclusions based on the model might be subject to potential errors. Every model is sensitive to some degree, and an examination of which parameter is due the most scrutiny is an important part of the modeling process.

5. Conclusions 

This year’s contest focused on the census that is currently under way in the United States. Three specific questions were asked. Each student team was expected to address each question. Every team focused on some aspect of the problem to different degrees, and the judges made every effort to provide a balanced view of the overall paper and tried to balance the strength of each paper in relation to the others.

The judging for this event takes place over several stages. The initial stages are designed to decide which submissions deserve closer scrutiny, and we try to err on the side of the students. As the judging progresses we become more and more critical and demanding. A paper that is successful in all stages must be well written, have a strong summary, and include excellent mathematical modeling combined with appropriate analysis of the model.

There are a number of different aspects that the judges must balance when comparing different submissions. The papers should convey a sense of some of the existing work that has been done on the problem but should also go a little beyond or offer some new twist to set it apart. The paper should make it clear what the student teams consider the principal aspects of the situation and incorporate them into the mathematical model.

Overall, the submissions were impressive and speak well of the motivation and talent of the student teams as well as the dedication and preparation provided by the team’s advisers. The continued success of this event is dependent on the student teams and advisers. Thank you all for your efforts!