Apparent Anomalies in Reported Data for the Field of Music in the Report of the National Research Council,"Research-Doctorate Programs in the United States: Continuity and Change"

NOTICE: All material in this document and the documents reached by the links below is ©1995 by the author, David W. Fenton. This text and that of all of these documents may not be re-distributed or re-used in any form without including this notice.

It is my duty to repeat, in all fairness to the authors of the NRC report, that I have not had the opportunity to examine the full report, only the selected data made available by the NRC on the Worldwide Web.(1) The conclusions made here are offered subject to verification that the available data have been correctly interpreted.

Contents

• The Comparison Data from the CMS Directory.
• Table 1: Significant Discrepancies in Numbers of Faculty Reported.
• Implications of the Comparison
• Numbers and Proportions of Faculty Awards — includes Table 2: Discrepancies in Reported Ratios of Faculty Awards.
• Numbers of Students and Ph.D.'s Awarded. — includes Table 3: Discrepancies in Reported Numbers of Students and Ph.D.'s Awarded.
• Student Assistantships. — includes Table 4: Discrepancies in Student Assistantships.
• Appendix A: Complete Faculty Data.
• Appendix B: Complete Student Data.

I. Comparison Data

This discussion is chiefly concerned with a comparison between the reported numbers of faculty in the report and a count of music faculties made from listings in the College Music Society's Directory of Music Faculties in Colleges and Universities, U.S. and Canada for the period 1992-94 (the NRC's report was for 1993). (2)

The NRC tabulated its faculty counts from "Institutional Coordinator Response Data" provided to the NRC by representatives of each of the institutions surveyed. The numbers to which the "Institutional Coordinator" data are compared come from a manual count of members of music faculties listed in the Directory using restricted definitions of both "Research-Doctoral Programs" and "faculty" — only faculty members listed in the Directory as holding full-time appointments in the following research-oriented fields were counted:

• Theory & Analysis (code 15 in the Directory)
• History & Literature (20)
• Musicology (21)
• Ethnomusicology (22)
• Western Hemisphere (55)
• History of Jazz (56a)
• Systematic Musicology (59)

In order to ameliorate any negative effects of this narrow definition, the fields of Music Education and Composition are listed separately in Table 1 if the programs award the Ph.D. Instructors in doctoral programs not awarding the Ph.D. (including the DMA and DMus) have been classified in "Other" (Table 1, column 5).(3)

In the tabulation of data from the CMS Directory, "faculty" members were defined as those holding the ranks of associate, assistant and full professor. Part-time instructors, adjunct instructors and emeritus professors were excluded on the assumption that they would not be teaching Ph.D. candidates.

Implications of the Comparison

The conclusion to be drawn from this comparison is quite obvious: the numbers in the NRC report are inaccurate because Insitutional Coordinators did not use consistent criteria for reporting the numbers of faculty. Even were one to abandon these "narrow" limitations and count all professors involved in the instruction of Ph.D. candidates, the data as reported would still be inaccurate, as can be seen by totaling columns 2 through 5 of Table 1 and comparing the result to column 1. Whereas the NRC report explicitly limited itself to Ph.D.-granting programs, it is clear from the Directory data that some ICs must have reported all faculty, regardless of specialization, and regardless of the proportion of students involved in Ph.D. programs,(4) while others reported only faculty in a limited number of fields and programs. Others apparently included faculty for all doctoral programs, including DMA, DMus and EDD programs.(5)

Regardless of the question of the intended scope of the survey, the fact of this inconsistency alone wholly invalidates any comparisons based on that data, and consequently invalidates any conclusions about apparent correlations between so-called "quality" and faculty size.(6)

In short, there is no consistent definition of "faculty members in Research-Doctoral programs in Music" that could produce the data on faculty numbers collected by the ICs. That such inconsistent data were collected suggests that:

• The instructions from the authors of the study to the "Institutional Coordinators" were not sufficiently detailed to insure consistency of reporting from institution to institution.

An individual in the field of music would neither neglect to distinguish between research and performing faculty, nor fail to notice the peculiarities in the numbers reported. For instance, the largest music school in the country, Indiana University, is listed as having only 6 faculty members, while the State University of New York at Stony Brook (a much smaller program) reports 32 faculty members. It is clear that Indiana University reported only professors of Music History and Musicology, while Music Theory & Analysis and the Ethnomusicology program (which is part of the Anthropology Department) were omitted. On the other hand, SUNY-Stony Brook appears to have counted the entire faculty (including some emeritus or part-time faculty). And, even worse, Temple University, which has a Ph.D. program only in Music Education with only six faculty members specializing in the field, reported 38 faculty members. The numbers from these three institutions alone are surely not comparable, for they represent entirely different things.

Additionally, not all the programs in all the ranked institutions were accounted for. While the figures for UCLA encompass the faculty of three different departments within the University (Dept. of Music, Dept. of Ethnomusicology and Systematic Musicology, and Dept. of Musicology), and those for the Eastman School of Music cover an institution with nine different doctoral degrees (DMA in Composition, Conducting, Music Education, Performance, Accompanying; Ph.D. in Composition, Historical Musicology, Music Education, Music Theory), other programs are not so fully reported. Yale University's Department of Music, ranked number 5, awards two Ph.D.'s in academic fields (Historical Musicology and Music Theory), but the Yale School of Music, which awards DMAs in both Performance and Composition is absent from the rankings. Similarly, New York University's Department of Music (three Ph.D.'s, in Musicology, Ethnomusicology and Composition & Theory) is included, but New York University's Department of Music and Music Education in the School of Education is omitted entirely (two DAs, in Music or Music Therapy; the EDD; and the Ph.D. in Applied Music, Composition, or Music Education).

But the flaws in the data are not limited to the numbers alone. Far worse is the fact that comprehensive schools of music are compared as a whole to small-scale academic departments. This "problem of comparability" manifests itself most strikingly in cases where the survey compares small "boutique" programs such as Wesleyan's Ethnomusicology program, which awards one Ph.D. in one specialized field, to sprawling, comprehensive music schools such as Indiana University, which offers doctoral programs in at least seven fields other than Ethnomusicology, including the DMus in four fields (Composition, Conducting, Music Literature and Performance, Music Literature and Pedagogy), and the Ph.D. in three (Musicology, Music Education and Music Theory). Attempting to rank these two programs on a single scale is futile, regardless of how accurately the programs are described — the very area in which Wesleyan exclusively specializes is not even present in the data on the program at Indiana to which it is compared.

For any ranking of "Music Programs" to be at all meaningful, similar programs must be compared. In the field of music, this would undoubtedly mean that doctoral programs in performance should be compared to other performance programs, while academic programs (Musicology, Theory and Ethnomusicology) should be compared to other academic departments. Quite clearly, music schools should not be ranked in comparison to music departments, except in the case of departments which aspire to the same comprehensiveness as full-fledged schools of music. In this light, the reported numbers suggest quite strongly that:

• Neither the designers of the study nor the ICs themselves were sufficiently knowledgeable of the field of Music to be aware of the variety of degrees awarded in music.

This in turn leads to the third conclusion — that the inconsistencies in the data went unnoticed suggests that:

• No one tested the data for accuracy once they were collected.

The heart of the study was the National Survey of Graduate Faculty in Spring 1993. The data collected in this survey produced the rankings of all the evaluated doctoral programs. However, in all cases, the lists of faculty and the lists of respondents were acquired from the ICs:

Survey forms were sent to a sample of faculty raters chosen from lists provided by ICs in all 41 fields included in the study. Each rater received a questionnaire with approximately 50 programs in their field selected at random from the roster of participating programs. For each institution they were asked to rate, raters were given a faculty roster provided by the ICs. . . .
(from the "Executive Summary," Selected Findings — The National Survey of Graduate Faculty, paragraph 1; emphasis added)

The demonstrated inconsistency in the faculty numbers reported by Institutional Coordinators casts doubt on the accuracy of all data collected by Institutional Coordinators — if the numbers of "Doctoral-Research" faculty in Music were inconsistently reported by the ICs, it follows that the faculty rosters must have been correspondingly inaccurate, as well, since it is highly unlikely that ICs would report an accurate faculty list while simultaneously reporting an incorrect count of those instructors. Furthermore, since survey respondents were chosen from data from the same flawed source, the respondents themselves must also have been inconsistently chosen. Since the data about Music programs from the "Institutional Coordinators" appear to have been used without testing their accuracy or validity, and since the whole structure of the study is built upon the data inconsistently and inaccurately collected by the ICs:

• Any rankings made on the basis of data collected by the ICs must be discarded.

If the base survey data and its rankings must be discarded, any proposed correlations with data drawn from the "Doctorate Records File," Federal Agencies, and from "Associations and Organizations Administrating [sic] Prestigious Awards and Honors" are also invalidated. Therefore:

• Few (if any) of the adduced correlations referred to in the "Executive Summary" are valid for the field of Music.

At the very least, the comparison to the numbers from the CMS Directory raises sufficient doubt that the rankings of music programs must be set aside until such time as the apparent anomalies in the NRC's data are addressed.

The report is not limited to the faculty rankings and numbers. A substantial amount of additional data collected from various other sources is presented as the basis for comparison with the faculty rankings. These data are not verifiable in the same fashion as the faculty numbers, but significant anomalies appear nonetheless, particularly in the data on faculty awards and on student populations. These anomalies are detailed in Sections II and III.

Since correlations between data from these outside sources and program rankings in the survey are adduced, should these outside data prove inaccurate, the correlations would need to be discarded even if the survey rankings themselves were to prove valid.

II. Numbers and Proportions of Faculty Awards

The number of errors is not great, but it is puzzling. It may be that the discrepancy originates in some factor not clear from the downloadable data tables.

More significant anomalies appear in the actual numbers of awards reported, which, unlike the data on faculty size, cannot be independently confirmed. However, the numbers suggest the possibility that, once again, data were inconsistently collected, specifically:

1. What "prestigious" awards were counted?
2. How long was the period under consideration?

In the absence of Appendix G, which details the data sources (but was not made available for download), the data may be evaluated only for their general sense. If the period covered is the year 1993 only, then it seems remarkable that the faculty of the University of Chicago held prestigious awards at a ratio of three awards to every four faculty members, and Harvard at nearly two for every three. A ten-year period (1983-1993) would make more sense. However, in the case of New York University at least, the stated number of awards (1) would be understated by at least one for 1993 alone, and by a factor from five to ten between 1983 and 1993 for NEH grants, Guggenheims, ACLS and I Tatti Harvard Fellowships alone, not to mention numerous other awards which are prestigious within the field of music, if not well known to the general public. Regardless of whether the period covered is one year or ten, the numbers from institution to institution appear to be irregular.

Once again, inconsistencies probably reflect inadequacies in the data sources upon which the report relies. If so, any correlations between faculty awards and "quality" ranking must be discarded on the grounds of invalid data.(7)

III. Numbers of Students, Ph.D.'s Awarded and Assistantships Reported

A. Students and Ph.D.'s Awarded

The data in Table 3 were collected from ICs. Presumably, the numbers for students are for the year 1993, while the number of reported Ph.D.'s is for the cumulative period since the previous report (1983-1993).

Although it is not possible to test the accuracy of these numbers, it is difficult to imagine that the large numbers of students reported for the departments listed in Table 3 truly represent students in "Research-Doctoral" programs only. In fact, it seems likely that the numbers reported by many of the programs include all students, in both masters and doctoral programs, as well as students in DMA, DMus and EDD programs. Even in cases where only students working towards a degree in "Research-Doctoral" fields were reported, it is probable that graduate students not yet admitted to the Ph.D. program were counted in some instances.

The numbers of Ph.D.'s awarded also appear suspect, particularly in comparison to the numbers of students reported for one year. One would expect some reasonable proportion of students to earn the doctorate each year. Numbers such as those reported for CUNY, Illinois, Columbia and U. Texas suggest either that large numbers of students never complete their degrees, or that the reported number of current students includes large numbers of students who are not working toward the Ph.D.

Such mixed numbers are not only not comparable, but wholly inconsistent with the stated scope of the report. Therefore, any conclusions based upon them must be discarded.(8)

B. Student Assistantships.

Several interesting points are raised by the data shown in this table. The numbers of students and Ph.D.'s reported by New York University are completely inconsistent with the assistantship percentages, because the percentages would require less than one whole student receiving support. Either the numbers of students reported by the ICs are incorrect or the assistantship percentages are inaccurate.

Further, it is not at all clear whether these percentages represent the support of the Ph.D. candidates claimed to be the subject of the NRC study, or if the percentages apply to all graduate or all doctoral programs at the institutions in question. And, finally, no data are reported on fellowships, which generally carry neither research nor teaching duties. Since the study reports correlations between student financial support and time to completion of the degree as well as between degree of dependence on teaching assistantships and "quality" ranking, this omission appears significant.(9)

The potential inaccuracy of these data together with the failure to account for all methods of student support cast serious doubt on both of these correlations.

Appendix A gives the data on faculty for all 65 rated programs.
Appendix B gives the corresponding data for students.

Return to DWF's home page.

NOTES:

1. The full text of the report is not available on the Worldwide Web (http://www.nas.edu/nap/online/researchdoc). The following two Appendices are the basis for the present discussion:
   • Appendix Table J (an Excel 5 file) - Selected Characteristics of Research-Doctorate Programs in Music (all programs at http://www.nas.edu/nap/online/researchdoc/appendix_j.html)
   • Appendix Table P (Excel 5) - Relative Rankings for Research-Doctorate Programs in Music (all programs at http://www.nas.edu/nap/online/researchdoc/appendix_p.html)
   
   The "Quality" (HTML) and "Effectiveness" (HTML) data for the Humanities in general were also consulted (Appendix Tables H (Excel 5) and I (Excel 5).

2. There are several caveats about the data tabulated from the Directory. First, these numbers represent a manual count of the printed faculty lists, so that any single data point is subject to normal human error. Second, since each of these lists was provided by the departments involved, there is some variation in the terminology used to report faculty rank. Additionally, the accuracy of the reported fields of specialization is unknown, although I noticed no errors in the faculties known to me. Since the Directory is widely available, these data are subject to verification and correction. However, notwithstanding these potential weaknesses in the compiling of the data from the Directory, the discrepancies with the NRC's reported numbers are too numerous and too large to originate solely from incidental inaccuracies in the numbers collected from the Directory.

3. Composers on the faculties where Ph.D.'s in Composition are awarded are listed separately only when they are not already included in the narrowly-defined research-doctoral fields by virtue of also teaching Music Theory, Musicology or some other field included in column 2 (see notes to Table 1).

4. Temple has no Ph.D. programs in any field except Music Education. Therefore, the number reported for Temple is not for faculty in "Research-Doctoral" programs as it is defined here.

5. It is interesting to note that at least three large music schools awarding only DMAs at the doctoral level (New England Conservatory of Music, The Juilliard School and the Manhattan School of Music) are omitted from the survey entirely. This suggests that the study's authors truly intended to include only Ph.D. programs and not all doctoral programs after all.

6. "A strong positive correlation between the number of faculty and its reputational standing has been demonstrated in the past but has not been explored thoroughly. From data collected by the committee, the size-"quality grouping" relationship was found to be the strongest in the Biological Sciences and weakest in the Arts and Humanities. By and large, however, top-rated programs in most fields tended to have a larger number of faculty and more graduate students than lower-rated programs." (From the "Executive Summary," Program Characteristics Associated with "Quality," paragraph 1).

7. "To explore the relationship between "quality" scores in the Arts and Humanities and "scholarship," the committee compiled a list of awards and honors using a variety of sources. The list was matched against a list of faculty members provided by the ICs. From this analysis, the committee observed that a larger share of faculty in top-rated programs in the Arts and Humanities were likely to have received a prestigious award than faculty in lower-rated programs. This relationship was most evident in the fields of Classics, Comparative Literature, Philosophy, and English Language and Literature, reflecting in part the sources of information that were used to compile this listing of awards and honors. . . ."
   (From the "Executive Summary," Program Characteristics Associated with "Quality," paragraph 5; emphasis added.)

   Note once again the reliance on faculty lists supplied by ICs.

8. ". . . By and large, however, top-rated programs in most fields tended to have a larger number of faculty and more graduate students than lower-rated programs." (From the "Executive Summary," Program Characteristics Associated with "Quality," paragraph 1.)

9. ". . . Another factor [in causing graduates in the Arts and Humanities to take longer than graduates in other fields to complete their degrees] is thought to be differences in patterns of student support, in which greater dependence on teaching assistantships (TAs) than on research assistantships (RAs) may account for the time it takes a student to earn a degree. From data collected by the committee it was observed that:

   — Graduates from lower-rated programs in many fields tended to utilize TAs as a primary source of student support at a greater rate than graduates of higher-rated programs."

   (From the "Executive Summary," Selected Information About Program Graduates, paragraphs 2-3.)