Last modified: Tuesday, June 29, 2004
Federal college access studies mislead policymakers, experts say
Statistical errors downplay financial burden
BLOOMINGTON, Ind. -- Over the last 10 years, much of the federal research charged with gauging the impact of financial barriers on access to college for low-income students and families has seriously misled the public and policymakers, according to a new study of this research.
The results of the study are published in a book titled Public Policy and College Access: Investigating the Federal and State Roles in Equalizing Postsecondary Opportunity, edited by Edward St. John, professor of education at Indiana University in Bloomington. Released last week by AMS Press Inc., the book is the latest volume of Readings on Equal Education, an annual book series that for 20 years has examined issues of equity, opportunity and access to education.
As a result of statistical errors, studies by the National Center for Education Statistics far too often falsely portray low-income high school graduates -- even those who by federal definition are college-qualified -- as under-prepared, unambitious, ill-informed and lacking in a will to complete preparatory courses, according to the authors. The authors also found that the considerable financial barriers these students face -- abundantly evident in the same database NCES analyzed -- were never considered a causative factor in their behavior. NCES is the nation's primary source of information on education trends.
"Breaking the rules of elementary statistics in study after study, the NCES misused a rich longitudinal database created at huge taxpayer expense to make faulty inferences, generate questionable findings and potentially misdirect public policy," said St. John, who brought together public and private sector experts in higher education, educational policy and economics for this volume.
Part I of the volume also includes findings by the Advisory Committee on Student Financial Assistance, a panel appointed to advise Congress and the U.S. Secretary of Education on student financial aid policy. St. John said the advisory committee initiated an external review of the NCES studies in light of the serious methodological weaknesses. One of these weaknesses, the authors found, is that the NCES reports failed to consider how the decline in federal grants over the past 20 years influenced the decline in enrollment rates for low-income, college-qualified students. Instead, NCES used correlations to argue that poor preparation was the reason.
Excluding from these analyses financial aid -- known to be critically important to the likelihood of enrollment at all types of colleges -- led to a serious underestimate of the true effect of family income on enrollment, concluded William E. Becker, professor of economics at Indiana University in Bloomington. Without including relevant financial aid measures in studies of college access, he said, and without adequately controlling for sample selection in the college-going decision, conclusions based on regression analyses about the importance of other explanatory variables cannot be taken seriously.
"Put more bluntly," St. John said, "the NCES studies found that finances did not matter without ever truly considering finances, an artful and mistaken dodge of an important policy question."
The findings were mixed in Part II of the volume, where St. John along with Don Heller, associate professor at Pennsylvania State University's Center for the Study of Higher Education, and others examined the impact of state education and finance policies on academic preparation and college enrollment. Heller documented the expansion of merit-based aid in many states. St. John and his IU colleagues found that both need-based and merit grants were associated with higher college enrollment rates in the states, but that merit grants also were associated with higher drop-out rates for students. Another study found that many state education reforms -- like implementation of math standards and exit exams for high school students -- were associated with higher SAT scores but lower graduation rates.
"In combination, these studies indicate that states have used misguided standards and merit aid to improve access, but these policies had some unintended effects," St. John said. "They discouraged many students from completing high school."
They also found well-documented approaches that states can use to improve both preparation and access. For example, Glenda D. Musoba, an education policy analyst at IU, found that Indiana's Twenty-first Century Scholars Program actually has improved college applications and enrollment for minorities in the state. Addressing both high school curriculum and financial access, these approaches take an alternative, comprehensive path pioneered by Indiana and a few other states, St. John said.
Given the shortcomings of federal policy and research, states should carefully evaluate how their policies influence educational opportunity, St. John said. He said the message that researchers and policymakers should take from this volume is clear: "The failure to acknowledge and correct the errors permeating recent federal access studies is a bad omen for future access research and the nation's access policy as well."
Two other examples of what St. John and the other contributors found as flawed research are:
- NCES studies abandoned previous federal definitions of equal access and employed a new definition.
- NCES studies ignored their own multivariate estimates that attributed only modest effects to parents' education in the presence of other factors and used high bivariate estimates to exaggerate its importance. The strong impression thus was created that the access problem facing low-income students, which the studies were initially charged to assess, neither required nor could be alleviated by increases in financial aid.
Contributors to Public Policy and College Access: Investigating the Federal and State Roles in Equalizing Postsecondary Opportunity include Brian Fitzgerald, staff director of the Advisory Committee on Student Financial Assistance, and John B. Lee, president of JBL Associates, a higher education consulting firm. Where appropriate, the findings were reviewed by an advisory panel that included some of the nation's leading education economists.
For more information, reporters can contact St. John at 812-855-0107 or estjohn@indiana.edu;Becker at 812-855-3577 or beckerw@indiana.edu; Heller at 814-865-9756 or dheller@psu.edu; and Musoba at 812-855-1604 or gdroogsm@indiana.edu.
Information about student financial aid can be found at the Advisory Committee on Student Financial Assistance Web site at https://www.ed.gov/about/bdscomm/list/acsfa/edlite-index.html.
