My NRC Correspondence
This file includes my entire correspondence with the NRC.
Since I have no permission to print the correspondence from the NRC, I have edited out the name and e-mail address of their representative and replaced it with "[correspondent]", as well as replacing his words with a paraphrase in my own words. I was v
ery careful not to use his words, but was still careful to relay the exact sense of his statements line by line.
The messages listed here are:
From dwf4930@is2.NYU.EDU Thu Sep 21 14:40:04 1995
Date: Thu, 21 Sep 1995 14:40:03 -0400 (EDT)
From: "David W. Fenton"
To: news@nas.edu
Subject: Doctoral-Research Report
Message-ID:
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: RO
X-Status:
I would like to know if there is an E-Mail address for anyone with whom I
might discuss extreme inaccuracies in the data reported about doctoral
Music programs in the report "Research-Doctorate Programs in the United
States: Continuity and Change."
This is the only E-Mail address I have found on your WWW page. I hope it
is the correct one to request this information.
Thank you.
David W. Fenton
New York University
dwf4930@is2.nyu.edu
http://pages.nyu.edu/~dwf4930
From [contact]@nas.edu Fri Sep 22 09:49:55 1995
Received: from darius.nas.edu by is2.NYU.EDU; (5.65/1.1.8.2/23Sep94-1121PM)
id AA29781; Fri, 22 Sep 1995 09:49:54 -0400
Received: from nas.edu (chariot.nas.edu [144.171.1.14]) by darius.nas.edu (8.6.11/8.6.10) with SMTP id JAA13319 for ; Fri, 22 Sep 1995 09:49:54 -0400
Received: from cc:Mail by nas.edu
id AA811788509; Fri, 22 Sep 95 09:42:00 EST
Date: Fri, 22 Sep 95 09:42:00 EST
From: "[contact] <[contact]@nas.edu>
Encoding: 18 Text, 31 Text
Message-Id: <9508228117.AA811788509@nas.edu>
To: "David W. Fenton"
Cc: [person1]@nas.edu, [person2]@nas.edu
Subject: Doctoral-Research Report
Status: RO
X-Status:
[says that my request has been forwarded to the appropriate people, and that if I don't hear from them within a week, that I should call a telephone number and speak to someone directly]
[quotations deleted]
From [correspondent]@nas.edu
Date: Mon, 25 Sep 95 08:46:00 EST
From: [correspondent] <[correspondent]@nas.edu>
To: dwf4930@is2.NYU.EDU
[the correspondent at the NRC responded by inviting my inquiring via e-mail or phone]
From dwf4930@is2.NYU.EDU Mon Sep 25 19:56:20 1995
Date: Mon, 25 Sep 1995 19:56:18 -0400 (EDT)
From: "David W. Fenton"
To: [correspondent] <[correspondent]@nas.edu>
Subject: Research-Doc Study
In-Reply-To: <9508258120.AA812044379@nas.edu>
Message-ID:
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Status: O
X-Status:
I have not been in a position to see the full report (I hope to in the
next few days), but I have examined the data available for download
across the Internet.
There are very serious problems with the reported numbers of faculty in
the field of Music. I have checked the report's numbers against the
listings of music faculty in the College Music Society's Directory for
1992-94. Apparently, the numbers reported by the Institutional
Coordinators were not all based on the same criteria for inclusion. If
the numbers are wrong, this suggests that the faculty lists that formed
the basis for the survey were also wrong. At the very least, it suggests
that unlike departments were being compared to each other (full-scale
music schools vs. small, one-program academic music programs). It would
also seem that some music programs counted _all_ graduate faculty, while
others counted just those involved in teaching doctoral students, while
still others limited the count to those teaching in Ph.D. programs, while
still others limited it to selected Ph.D. programs.
The summary of my investigation reads as follows:
---------------
Summary: One of the areas of data reported by the National Research
Council in its report "Research-Doctorate Programs in the United States:
Continuity and Change" is the number of faculty in the surveyed programs.
A comparison of the report's numbers for the field of Music and the
College Music Society's Directory of Music Faculties in Colleges and
Universities, U.S. and Canada for the same period reveals numerous
discrepancies with the numbers reported by the NRC.
The data on faculty used in the National Research Councils's report were
collected from "Institutional Coordinators" at each of the schools
surveyed. It appears that each institution used different criteria for
reporting its faculty, therefore invalidating any comparison based on
these data. Since these same "Institutional Coordinators" also provided
the lists of faculty members used in the survey which constituted the
thrust of the National Research Councils's study, the accuracy of the
information on which the survey was based is also called into question,
consequently casting grave doubt upon the validity of the survey results
as a whole.
In addition, anomalies appear in other information included in the
report, including data on faculty awards and data on student populations.
Since these data form the basis for numerous hypothesized correlations
with program rankings in the survey, should these data prove inaccurate,
the correlations would need to be discarded even if the survey rankings
themselves were to prove valid.
Since the data from the "Institutional Coordinators" about Music programs
appear to have been used without testing its accuracy or validity, the
conclusions of the entire report must be put aside for the field of Music
until the data on which it is based is proven accurate.
-----------------
My own department here at NYU is under pressure from the University about
why we have dropped in the rankings. The stakes are rather high -- we
could stand to lose funding, or even faculty. It is very important to know
whether the Research-Doc Study was sufficiently well-conducted in the
field of Music to justify such pressures.
Once I am able to examine the full report, I will be happy to forward my
critique.
However, in the meantime, I look forward to your prompt response.
David W. Fenton
New York University
dwf4930@is2.nyu.edu
http://pages.nyu.edu/~dwf4930
From dwf4930@is2.NYU.EDU
Date: Tue, 3 Oct 1995 22:36:21 -0400 (EDT)
From: "David W. Fenton"
To: [correspondent] <[correspondent]@nas.edu>
Subject: Doctoral-Research Study
I wrote to you a while back about problems I have found in the study for
the field of music.
As I have not heard a response to my reservations, I have gone ahead and
written them up. You can view them at:
http://pages.nyu.edu/~dwf4930/nrc_report/nrc_rept.html
A small group of people is reviewing this document, and, once corrections
are made, I will be making it publicly available from my home page, and I
will announce its existence to the Internet mailing lists of the American
Musicological Society and of the Society for Music Theory (to which well
over 500 individuals subscribe).
The allegations are serious, and you should have an opportunity to answer
them before I go public.
Should you desire to review my report in some more accessible form (it is
Netscape optimized), please don't hesitate to ask.
Will I be hearing from you?
David W. Fenton
New York University
dwf4930@is2.nyu.edu
http://pages.nyu.edu/~dwf4930
From [correspondent]@nas.edu
Date: Wed, 04 Oct 95 08:52:00 EST
From: [correspondent] <[correspondent]@nas.edu>
To: dwf4930@is2.NYU.EDU
Subject: Re: Doctoral-Research Study
[the NRC representative apologized for not getting back to me sooner, but he was reviewing the response form for Music, and checking the info. in the '82 study.]
[he asserts that the numbers for faculty members are correct insofar as they represent the numbers reported by the ICs. He says that the NRC had no way of knowing if they submitted long or short lists, and asks if the CMS Directory is for resear
ch faculty only. He defends the IC data on the grounds that the institutional administrations should be the ones best able to determine who their research faculty are. He advises me that I should check with the NYU representative to see why he reported th
e numbers for NYU that he did. He also
says that it would have been possible to have consulted with all 4,000 of the surveyed programs, and that they relied on the data from the ICs.]
[he says that the NRC in fact is aware of the mix of theory and performance specializations, and clearly requested only research faculty on the lists from the ICs. In response to my point about Juilliard, he says that they did not participate this time
, possibly because after the 1982 report they were identify as one program which was not really
a research-doctorate program, and asks if this could be a reason why the numbers are not the same]
[he says that the other information in the report (the data sources and the way the data were collected) is explained in the full report. He asks if I have found any specific errors there so that he could confirm their procedures]
[he quotes a paragraph from my web page, saying that he could not get to my main report. This was my fault, because I had not set access rights correctly]
[he says he will say more once he's been able to read my report]
[quoted material deleted]
My response follows.
Click here to read a second attempt at justification.
From dwf4930@is2.NYU.EDU
Date: Wed, 4 Oct 1995 17:43:45 -0400 (EDT)
From: "David W. Fenton"
To: [correspondent] <[correspondent]@nas.edu>
Subject: Re: Doctoral-Research Study
Sorry the web page is not accessible -- there were errors in it, anyway.
I'll try moving it up one directory and see if that helps.
I'll also forward the HTML files to you directly with this message (all
of them are attached -- dump them into any directory accessible with your
WWW viewer, and you should have no trouble with them).
On Wed, 4 Oct 1995, [correspondent] wrote:
[he asserts that the numbers for faculty members are correct insofar as they represent the numbers reported by the ICs. He says that the NRC had no way of knowing if they submitted long or short lists, and asks if the CMS Directory is for resear
ch faculty only. He defends the IC data on the grounds that the institutional administrations should be the ones best able to determine who their research faculty are. He advises me that I should check with the NYU representative to see why he reported th
e numbers for NYU that he did. . .]
The numbers of faculty at NYU are essentially correct.
The numbers of students, reported Ph.D.'s and assistantships are grossly
inaccurate.
[. . .He also says that it would have been possible to have consulted with all 4,000 of the surveyed programs, and that they relied on the data from the ICs.]
But this is precisely my point. You did not check to see whether the data
reported by the ICs was in fact the data you requested. You collected the
data and based your entire survey on it, apparently without ever having
verified its correspondence to reality.
I'm not saying you should have contacted the departments directly, only
that you should have had some instrument in place for testing the
accuracy of the data on which you so heavily relied. Is that not standard
procedure for a survey of this nature? Was no sampling done to see if
there was any degree of error in the reports from the ICs?
The representative at NYU who supplied the data was clearly in error.
Why? I do not know. The error remains. Should the authors of the survey
have detected it? If it were the only error, of course not -- that would
be asking far too much of what was already an ambitious undertaking.
However, the weaknesses in the rest of the data suggest that you accepted
the data entirely without verification. If that is true, I don't see how
you can claim any validity for the survey results at all.
[he says that the NRC in fact is aware of the mix of theory and performance specializations, and clearly requested only research faculty on the lists from the ICs. In response to my point about Juilliard, he says that they did not participate this time
, possibly because after the 1982 report they were identify as one program which was not really
a research-doctorate program, and asks if this could be a reason why the numbers are not the same]
But some of the ICs clearly reported _all_ faculty, including
performance, while others did not. This negates any comparison. It also
shows that, despite your best intentions, the data reported by the ICs
was not what you had requested. This apparently went undetected until now.
Given that NYU's School of Education Dept. of Music (which awards a Ph.D.
in Music Education) was omitted (while the GSAS Music Dept. was included),
it is not clear to me what constitutes a "research-doctorate" program,
particularly given the fact that Temple University is included in the
survey but awards only one Ph.D., in Music Education. If Temple should be
in the survey, why is NYU's SEHNAP Music Dept. omitted? If the latter was
correctly omitted, why is Temple included? Where is the consistency in
reporting?
My point is not to defend/correct the data about NYU. I notice the
mistakes at my home institution only because that is the one with which I
am most familiar. My point is that the IC data are completely inconsistent.
If the ICs didn't get the _numbers_ right, it follows that the faculty
lists on which the entire survey was based must be correspondingly wrong.
[he says that the other information in the report (the data sources and the way the data were collected) is explained in the full report. He asks if I have found any specific errors there so that he could confirm their procedures]
I am in the peculiar position of not having seen the full report. I
thought that I was going to get access to a copy last week, but that
didn't work out.
I do not relish the thought of having to spend $65 only to be in a
position to point out the errors in the survey.
However, even without the main body of the report, my points about the
inconsistency of the faculty numbers are sufficient to cast doubt upon the
survey rankings, since they, too, were based on data collected from the
demonstrably unreliable ICs.
Please review the HTML documents. I think you will see the logic of my
position, at least in regards to my doubts about the survey for the field
of music.
David W. Fenton
New York University
dwf4930@is2.nyu.edu
http://pages.nyu.edu/~dwf4930
[Attachments deleted]
From dwf4930@is2.NYU.EDU
Date: Thu, 5 Oct 1995 19:10:37 -0400 (EDT)
From: "David W. Fenton"
To: [correspondent] <[correspondent]@nas.edu>
Subject: Re: Doctoral-Research Study
I have made some relatively minor corrections to my report, and fixed the
access rights. You should be able to see this updated version at the
original URL that I gave you.
David W. Fenton
New York University
dwf4930@is2.nyu.edu
http://pages.nyu.edu/~dwf4930
From [correspondent]@nas.edu
Date: Fri, 06 Oct 95 14:41:00 EST
From: [correspondent] <[correspondent]@nas.edu>
To: dwf4930@is2.NYU.EDU
Cc: [copyright person]@nas.edu
Subject: Re: Doctoral-Research Study
[says I need to get the National Academy Press's permission to link to their home page from my material on the Research-Doctorate Study, as well as to use the data presented in my tables. I can get this permission by contacting the person CC'd in his E
-Mail message]
[quotations deleted]
From dwf4930@is2.NYU.EDU
Date: Fri, 6 Oct 1995 19:32:08 -0400 (EDT)
From: "David W. Fenton"
To: [copyright person]@nas.edu
Cc: [correspondent] <[correspondent]@nas.edu>
Subject: Permission to link to and quote from Doctoral-Research Study
On the advice of [correspondent], I write seeking permission to:
1. place links on my home page to the Doctoral-Research Study
documents on the NAP WWW server
(http://www.nas.edu/nap/online/researchdoc) as well as to
its component sub-documents, specifically the "Executive
Summary" (http://www.nas.edu/nap/online/
researchdoc/summary.html).
2. use the data reported in these downloadable Appendices to the
report:
- Appendix Table I (Selected Characteristics of
Research-Doctorate Programs in Music)
- Appendix Table P (Relative Rankings for
Research-Doctorate Programs in Music)
I ask for this permission in order to publish on the Worldwide Web my
critique of the NRC report. The document is available for you to
review at:
http://pages.nyu.edu/~dwf4930/nrc_report/nrc_rept.html
Should you find other areas which I have not mentioned above in which you
would prefer an explicit request for permission, please do not hesitate
to inform me.
This URL will not be publicly distributed until such time as I have
received the permissions I request here and until such time as I have
received a response indicating the NRC's desires as to whether they would
like the opportunity to address my objections.
If you have any questions, please do not hesitate to contact me.
Thank you.
David W. Fenton
New York University
dwf4930@is2.nyu.edu
http://pages.nyu.edu/~dwf4930
From dwf4930@is2.NYU.EDU
Date: Fri, 6 Oct 1995 19:37:28 -0400 (EDT)
From: "David W. Fenton"
To: [correspondent] <[correspondent]@nas.edu>
Cc: [copyright person]@nas.edu
Subject: Re: Doctoral-Research Study
On Fri, 6 Oct 1995, [correspondent] wrote:
[says I need to get the National Academy Press's permission to link to their home page from my material on the Research-Doctorate Study, as well as to use the data presented in my tables. I can get this permission by contacting the person CC'd in his E
-Mail message]
Thank you for this information. As I mentioned before, at present I have
not made this URL public, pending a response from you on the issue of my
main criticism. If you will check the URL below, you will see that,
although I have announced my report, it is not yet available from my home
page.
I think that there can be no copyright violation in establishing a link to
a WWW page, regardless of whether it contains copyrighted material, but to
demonstrate good faith, I will disable the link to the report. I will,
however, include the _text_ of the URL for the home page of the Press (not
the report in question).
Should I decide to publicize the information on the Worldwide Web, I will
not do so until such time as I have received the permission you request
that I obtain.
However, I believe that I am under no legal obligation to do so, as my use
of these materials clearly falls under the concept of "Fair Use,"
particularly with regard to "Quotation of excerpts in a review or
criticism for purposes of illustration or comment;" or "Quotation of short
passages in a scholarly or technical work for illustration or
clarification of the author's observations" (Copyright Office document
FL102, quoting the 1961 Report of the Register of Copyrights on the
General Revision of the U.S. Copyright Law). This applies both to my use
of quotations from the "Executive Summary" published by the NAP on the
WWW, and my presentation of tables of data extracted from the downloaded
data files made available by you for download from your FTP site. All of
these fully acknowledged uses constitute "fair use" by virtue of their
being necessary for the purpose of criticizing the report. Without the data
in my tables (not used unchanged, by the way), I would be unable to
demonstrate the pattern of error I am alleging.
But, as I said, in a spirit of fair play, I have already requested
permission to quote these materials. Indeed, I would feel better if I
could change my copyright notice to acknowledge the NRC's permission to
use the materials.
As it stands, I still await a response from you in regard to my criticism.
Are you in a position to be able to say when you will answer my
objections? Or will you forgo the opportunity to make your own case? If
you feel you would like to evaluate the CMS _Directory_ yourself, I would
be glad to fax some pages of it to you. And, should you respond in
writing, I would be happy (if you approve) to make your unedited response
available alongside (and linked to) my own report. In fact, should you
choose to publish your own response on the WWW, I would unquestionably
include a link to it from my own report. I believe that would be only
fair.
David W. Fenton
New York University
dwf4930@is2.nyu.edu
http://pages.nyu.edu/~dwf4930
From [copyright persons]@nas.edu
Date: Mon, 09 Oct 95 09:47:00 EST
From: [copyright person] <[copyright person]@nas.edu>
To: dwf4930@is2.nyu.edu
Cc: [copyright person]@nas.edu, [correspondent]@nas.edu
Subject: Permission
PERMISSION TO REPRINT
NATIONAL ACADEMY PRESS
2101 CONSTITUTION AVENUE, NW
WASHINGTON, DC 20418
October 9, 1995
David W. Fenton
New York University
Dear Professor Fenton:
Thank you for your communication dated October 6, 1995.
You requested permission to reprint the following information:
Quotations and information, including tabular data, from
RESEARCH-DOCTORATE PROGRAMS IN THE UNITED
STATES, National Academy Press, 1995, for use in URL posting.
NOTE: We grant this permission with no fee based on your
description of "fair use" of materials from the published book
and the WWW site. Thank you for your interpretation of "fair
use." We do not disagree with your interpretation, but we do
reserve the right of the publisher in determining what is and
what is not fair use of our copyrighted materials.
___X___Your request is granted with a fee of $00.00* for
materials originally published in the title cited above and provided
that full credit is given to the original publication.
Please use the following citation:
Reprinted with permission from RESEARCH-DOCTORATE PROGRAMS
IN THE UNITED STATES. Copyright 1995 by the National Academy of
Sciences. Courtesy of the National Academy Press, Washington, D.C.
[you may edit as long as a copyright notice is carried with the site]
_______ The Academy cannot grant your request for the following reasons:
Please made check payable to NATIONAL ACADEMY PRESS and return
with a copy of this permission form to RICHARD E. MORRIS, at the above
address.
Richard E. Morris
Permissions Manager
National Academy Press
FAX: 202-334-2793
PHONE: 202-334-3335
FED ID: 53-019-6932
From dwf4930@is2.NYU.EDU
Date: Tue, 10 Oct 1995 17:19:18 -0400 (EDT)
From: "David W. Fenton"
To: [correspondent] <[correspondent]@nas.edu>
Subject: Further Questions About the Full Report
I have finally had access to a hard copy of the NRC report.
First let me say that the data collection forms sent to the ICs appear to
me to have been adequate to have insured correct information. It's pretty
clear what is wanted -- I don't see that they appear open to
interpretation. So, in the end, responsibility for any errors has to be
lodged with the Institutional Coordinators. Nonetheless, this still leaves
the NRC responsible for not having tested the accuracy of this data.
Having read these seemingly well-designed forms, it has occured to me for
the first time that schools may have purposely mis-represented their
faculty lists in an effort to improve their standing in the survey. This
possibility makes the ommission of any testing of this data an even more
serious flaw.
What follows is a compendium of comments drawn from my note-taking file.
I'll try to edit them and re-order them so they make the most sense.
----------------------------------------------------------------------
First, a typo: The table that shows which schools produce 80%, 90% and
100% of the A & H Ph.D.',s (C-2, p. 81) has an error in the 100% column.
It reads "2,57" when it must be something like 2,57X (given that 90% is
2,337).
----------------------------------------------------------------------
Second, general comments:
1. Apparently, "research-doctoral" simply means "awards a Ph.D." There
seems to be nothing in Ch. 2 ("Study Design") which explicitly suggests
that any other degree is relevant. Likewise, there is no consideration of
the possibility that not all Ph.D.'s are awarded in "research" fields.
What was the working definition for determining what degrees were
eligible?
2. Given my citation of the omission of Juilliard as an example of a
school that issues doctoral degrees (DMAs) but no Ph.D.'s, it is
interesting to note that, even though Juilliard was selected as eligible
to participate (p. 17), they themselves declined to participate (p. 28, n.
3.). Possibly Juilliard did a better job of interpreting the survey scope
than those designing the study?
3. After having seen the program response form, it is difficult for me to
imagine how NYU could have reported only 2 students for my department. The
form asked for the number of "Ph.D.'s (or equivalent research-doctorates)"
which were awarded in each of the years from 1987-88 to 1991-92,
specifically asking "Approximately how many full-time and part-time
graduate students enrolled in the program at the present time (Fall 1992)
intend to earn doctorates?" (App. D., p. 95). Maybe this was erroneously
interpreted at NYU to mean students who would earn it in the present
year? Apparently "intend" is the key word that must have been
misunderstood. How could this have happened if the scope and purpose of
the study was clearly explained to the Institutional Coordinators?
Exactly where does the apparent incompetence lie?
4. Appendix Table C-1 shows production of Ph.D.'s in the Arts &
Humanities, 1986-90 (p. 81). It shows 2,573 Music Ph.D.'s!! Surely this is
all doctoral degrees.
5. The description of the survey is somewhat ambiguous in respect to who
evaluated whom:
"Owing to the fact that the sample was drawn from faculty lists provided
by Institutional Coordinators crossing departmental boundaries,
respondents occasionally indicated that they did not consider themselves
qualified to rate programs in a certain disciplinary area." (App. F, p.
116)
Does this mean related but different fields? Or different fields entirely?
The letter to raters on p. 118 suggests that it was within a field, and
that the "interdisciplinary" problem occurred in fields with
well-delineated specializations. Music would fall into this class, which
may be why Music was one of four fields which required a followup survey
(App. F, p. 116).
If this is the case, it would tend to support my hypothesis that the full
range of sub-specializations in Music was not foreseen by the designers of
the survey.
7. Survey respondents had to indicate their one area of specialization.
What were the specializations listed for Music? (App. F, p. 123).
8. The Character of the Doctorate Records File
The "Doctorate Records File" is created from date collected in the Survey
of Earned Doctorates, conducted by the NAS & NRC (p. 144). I wonder if
there has ever been any test of the accuracy of the data collected in this
survey, and if there has ever been an attempt to compare different fields
to see if the data collection instrument might be better suited to
capturing profiles of graduates in the sciences better than in the
Humanities?
9. The study finds that "Degree recipients in the Sciences and Engineering
are more likely to have received research assistantship (RA) support. . .
than students in the Arts and Humanities. . . ." (p. 51). The authors then
go on to describe differences in how "research assistantships" relate to
doctoral research. However, they clearly do not understand the role played
in the Humanities by full-support fellowships, which require neither
teaching nor research duties, and allow the student to concentrate on her
course-work and research. Apparently, the DRF data does not include any
category for this.
They also consider the TA/RA juxtaposition to be one between teaching and
no teaching. This is a false comparison for the field of Music. They also
don't seem to delve into what the DRF data actually mean. How well do the
requested categories of data actually fit the way programs in various
disciplines really work? The DRF data may very well report what graduates
put on their forms, but what if the categories don't fit a graduate's
situation? What do the reported numbers mean then?
----------------------------------------------------------------------
Third, examples drawn from the study which indicate that the field of
Music tended to be an exception to a number of trends:
1. Sample selection: In Music, 260 raters were selected to rank 65
programs. The sample was adjusted to 255 who were considered qualified. Of
the mere 107 completed questionnaires received, one was completed by
someone other than the originally selected rater, giving a usable rater
percentage of 42%. There were 129 non-respondents with no known reason, 19
actually gave reasons.
Music had the lowest response rate in the A & H. Only the Biological
Sciences tended to fall in the same range as Music (App. F, Table 1, p.
134), and this is an area identified in the study as one in which there
were a number of cross-disciplinary problems, leading to large numbers of
respondents feeling unqualified to respond. This suggests the possibility
that the same may have been true of the field of Music.
2. In the SELECTED FINDINGS (Ch. 2), modern European language programs are
identified as the ones tending to have the fewest faculty (10 to 15,
generally; p. 34). Had they correctly classified Music programs (truly
only research programs), they would have found something similar, I think.
In fact, here are the average numbers of faculty reported:
Spanish/Portuguese 13.35
French 12.98
German 11.26
Music 22.36
For Music, my calculation of the average of the NRC's numbers comes out to
21.84 (still rounds the same). However, for the programs where I
re-counted the faculty, the average is 14.62. If you put back all the
schools which I did not recount, the average comes to 16.71. This is much
more in line with my general impression of the size of programs that the
scope of the report suggests should have been compared.
3. Music, Philosophy and Spanish/Portuguese are the only Humanities
programs where the top quarter does not have the largest average faculty
size. There are no other fields in which this is the case (p. 35).
Together with my knowledge of the field of music, this suggests to me that
the top-rated programs tend not to be the same _kinds_ of programs -- more
large, general music schools fall outside the top quarter, and
smaller-scale academic programs tend towards the top. Most of the
large-scale programs do not belong in the survey at all, for most of the
doctorates are not "research" doctorates. This is only an impression, but
it would explain the departure from the norm for the field of Music, at
least.
4. A point is made of the fact that Faculty awards cluster
disproportionately in programs in the top quarter (p. 41). Music is
singled out as being one of the fields in which this is most plainly the
case -- in fact, Music has a greater drop-off from the first quarter to
the lower quarters than any other field in the Humanities. But, they also
suggest that this pattern may be indicative of "limitations of the data
base. . . ." (p. 41).
5. Music was one of the programs in which average number of faculty is
identified as having experienced the most growth (p. 44). Although the
data are not directly comparable, the CMS Directory figures run like this:
Year Faculty Institutions Average
1980-82 unknown
1982-84 24,796 1,536 16.143229
1984-86 23,392 1,538 16.509752
1986-88 unknown
1988-90 26,666 1,545 17.259546
1990-92 29,663 1,745 16.998853
1992-94 30,582 1,808 16.914823
1993-94 31,138 1,783 17.463825
1994-95 32,124 1,830 17.554098
(Apparently, the CMS went from bi-annual to annual in 1993. I have faxed
CMS Publications and posted a message on the AMSList seeking the missing
numbers, but have not yeat received them).
These numbers don't really support or contradict the NRC finding, but it
would be interesting to know if any of the faculties that experienced the
greatest increases in faculty size since 1982 were those which I have
identified as having misreported their faculty.
6. Overall, in A&H, the top programs produced fewer degrees in the period
compared to the 1982 report. Linguistics is pointed out as one field where
"degree production" increased, but it also increased for some quarters in
Art History (lower two), Music ( top and third) and Spanish and Portuguese
(bottom two) (Figure 3-6, p. 45). Why is Music so anomalous on this (the
only one where adjacent quarters differ so widely)? Doesn't it suggest
that the numbers for Music could be wrong?
----------------------------------------------------------------------
Well, I know that's quite a lot, but it reflects reservations that I have
which have emerged from examining the report in full.
I do not know if you will choose to respond to these questions or to any
of my previous ones, but I will be publicizing my report (along with my
new questions) on the WWW sometime this week.
I would very much like to hear from you.
David W. Fenton
New York University
dwf4930@is2.nyu.edu
http://pages.nyu.edu/~dwf4930
From jvoytuk@nas.edu
Date: Wed, 11 Oct 95 09:38:00 EST
From: Jim Voytuk
To: dwf4930@is2.NYU.EDU
Subject: Re: Further Questions About the Full Report
[thanks me for my comments. Acknowledges that the IC data are important, but asserts that they are the ones who are in the best position to collect the data. Says that they couldn't check the data because they requested research facul
ty and program faculty instead of department faculty. Even if there were some place to verify the IC data for the impossibly large number of programs, who would make the judgement call in the case of discrepancies?]
[thanks me]
[quotation deleted]
My response follows.
From dwf4930@is2.NYU.EDU
Date: Wed, 11 Oct 1995 10:48:22 -0400 (EDT)
From: "David W. Fenton"
To: [correspondent] <[correspondent]@nas.edu>
Subject: Re: Further Questions About the Full Report
On Wed, 11 Oct 1995, [correspondent] wrote:
[thanks me for my comments. Acknowledges that the IC data are important, but asserts that they are the ones who are in the best position to collect the data. Says that they couldn't check the data because they requested research faculty and program fac
ulty instead of department faculty. Even if there were some place to verify the IC data for the impossibly large number of programs, who would make the judgement call in the case of discrepancies?]
At no point have I ever questioned your assertion that the ICs are in the
best position to supply the data. That is eminently sensible. The
question is one of asking exactly what quality of data they will collect.
I am not trained in this field, but from what I have read it seems to me
that in cases where data are too voluminous to verify completely, random
sampling is usually used to test their validity. That is, you choose a small
subset (maybe 5%) and you contact them directly to see if the results you
have collected match reality. It's not perfect, but it should at least
uncover any large-scale problems with the data collection process. It
might not have uncovered the Music problem, for 5% would be only 3
programs, but at least you would have done something to test the data that
would be reasonable given the extent of the survey.
Further, could the NRC not have had at least one or two
knowledgeable people in each of the fields eyeball the data for any
obvious inconsistencies, like the report that Indiana University has only
6 faculty members involved in doctoral-research programs? _Everyone_ in
my field who has seen the numbers for music has immediately pointed out
that number, while also wondering why some of the other departments list
such _large_ faculties (e.g., if Indiana reports 6, why does Illinois report
74?).
Maybe I'm just repeating myself, but, did the NRC do _anything_ to even
_try_ to test the data after they were collected? No matter how general
and unscientific it may have been?
Wasn't it crucial to the survey that the IC data be valid? Wasn't it
important to see if the collection instrument in fact produced the data
the survey required?
I don't think these are trivial, 20-20 hindsight types of questions. I
would have thought that any survey conducted by professionals would consider
both ends of the process, rather than taking on faith that a seemingly
valid data collection process will actually produce good data (let alone
throwing one's hands in the air and saying "But what could we have done?
There was too much data too verify").
The admission that the NRC did not verify the data to any level of
certainty seriously undermines the credibility of the study as a whole.
And in the case of music, where I have demonstrated likely inconsistencies
and inaccuries in the data, the survey result should have been discarded
entirely, because the inaccurate data make the result too unreliable to be
released. Given all the special cases in the treatment of the Music data
that I quoted from the report in my previous message, it might have been
reasonable for someone to have considered the possibility of inaccurate
data.
If I'm reading between the lines correctly, it sounds like a resource
problem -- maybe you did not have the funds to manage a study of this
magnitude at the level you would have liked. In that case, it's truly a
tragedy, for the aims and structure of the study appear to me to be
laudable.
I would be very interested in the answers to my voluminous questions, but
I understand that you may not be able to respond to each in any timely
manner. Perhaps you will be able to respond on the issues you deem most
important.
Thank you.
David W. Fenton
New York University
dwf4930@is2.nyu.edu
http://pages.nyu.edu/~dwf4930
[ Back to Information on Copyright Issues. . . ]
[ Back to NRC Report introduction. . . ]
[ To Overview of NRC Report Critique. . . ]
[ Back to DWF's home page. . . ]
[ Back to previous
page. . . ]
Contact David Fenton
©1995-96, David W. Fenton (last modified Wednesday, April 3, 1996)