Student Outcomes Research Project

The Student Outcomes Research Project was developed and conducted by the Division of Adult & Continuing Education at the Office of Academic Affairs of the City University of New York under a contract awarded by the Literacy Assistance Center.

            The Project was intended to determine the changes in students’ lives which result from their participation in adult literacy and language programs funded through the New York City Literacy Initiative (NYCALI). In addition, the project was intended to identify program components which have the most significant impact on students’ lives.

            While the Initiative has been in existence for more than ten years and several hundred thousand individuals have enrolled in programs over the course of those years, a review of archival materials from the Initiative revealed that little of a definitive nature concerning program impacts has been established.  Indeed, that review raised a number of important issues concerning the ways in which student achievements have been assessed, outcomes determined, and data collected.  

            In order to collect more detailed information about the relationship between students’ lives and their classroom work, the project staff surveyed current program directors, conducted student focus groups and individual student interviews and collected case studies authored by literacy practitioners.  (A mail survey to former NYCALI students was initially planned but could not be completed for reasons explained in the Project Description.)

            We began our research with the conviction that the significance of the changes in students’ lives was not transparent.  Outcomes and impacts are often hard to see and even harder to link directly to literacy instruction.  At the same time that literacy educators may underestimate some of the benefits students achieve as a result of their participation, some policy pronouncements overstate the extent to which literacy education can affect the complex social and economic realities which determine whether someone removes him or herself from public assistance or gets a substantially more secure or better paying job. 

            In an attempt to capture an appropriately nuanced measure of actual outcomes, we wanted our study to be framed by an understanding of what students desired to be achieved and the ways in which programs supported these goals.  We wanted, whenever possible, to show the relationship between students’ initial goals, expectations and program practices and the impact that literacy or language education had on their lives.  Our various research activities were designed to allow us to sift through the data from multiple sources and offer a textured summary of:

            1.  student characteristics, goals and expectations;

            2.  program practices;

            3.  student outcomes.   

            The Final Report describes our findings in these three key areas through the Archive Review and our own research activities.  Some of the major findings of each section are highlighted here.

Archive Review

Student Characteristics, Goals and Expectations

Characteristics

In several important ways, the characteristics of individuals enrolled in NYCALI programs have remained quite similar:

• Women have consistently outnumbered men among students.

• The educational backgrounds of students have been consistently varied–some individuals have had little or no schooling while others, typically among ESOL students, have completed college-level studies.

• Many students have been enrolled in other adult education programs.

• The number of students reporting that they were on public assistance remained fairly constant.

On the other hand, there have been important changes:

• The enrollment of students below the age of 25 has fallen from 34% to 19%.  Currently, most students are between ages 25-44.

• The number of foreign born students has been increasingly steadily.  In the 1992-1993 program year, it was reported that 84.3% of all students were foreign-born.

• In recent years, more students have been unemployed than were during the first four years of the Initiative, and, for those who are working, more are employed on a part-time basis.

Enrollment patterns have been stable for the last five or six years:

• Enrollment in basic education and ESOL classes is approximately equal; however there is a substantial enrollment of students with limited English proficiency in basic education classes.

• Approximately forty percent of BE students and seventy-five percent of ESOL students have been enrolled in the two lowest levels of instruction.

• Hours of average attendance increased over the first few years of the Initiative and then stabilized.

• Persistence (defined as continued enrollment over the course of a year or from year to year) appears to be modest.

Goals and Expectations

Students’ goals have consistently been varied and complex; they have wanted to improve their skills and acquire knowledge; they have wanted to improve their job prospects; they have wanted to be better able to provide for their children; they have wanted to be more actively involved int he affairs of their communities; they have wanted to overcome feelings of doubt about their ability to learn.

There was little information concerning students’ expectations in the documents reviewed.

Program Practices

Our review of Final Reports made clear that programs have developed a rich array of practices in response to the needs of students and the changing requirements of the funding agencies.  It was also clear that practitioners had invested many hours in scrutinizing their work, in borrowing from work done in other educational arenas and in learning from each other.

Many of the most important practices reflected both continuing and expanding understandings of program priorities.  Thus, programs have maintained a commitment to providing adult students with opportunities to acquire knowledge about a broad range of topics.  But, at the same time, they have attempted to be responsive to students’ concerns .

Programs have administered and reported results from a number of different tests and test forms required by the Initiative’s funders.

Programs have also been asked to count other types of outcomes, such as the acquisition of specific skills or various kinds of life changes (such as getting a job).

A good deal of effort has gone into the development of alternative assessment measures to identify outcomes not readily accessible by standardized testing.  Early on, program staffers realized students had made considerable progress in areas that were not being accounted for or captured in any regular way.  This work has lead to a series of program based alternative assessments used to assess students and measure progress.  These alternative practices have not been systematized or adapted for the entire Initiative.

Outcomes

The test results reported for both Basic Education and ESOL students have usually suggested modest, and oftentimes inconsistent, achievements (although the use of averages may mask more significant accomplishment by some students).  The analysis of those test results has also characterized by a number of significant methodological problems–having to do, variously, with the somewhat arbitrary changes of the definition of gain, the comparison of non-equivalent scores, and the unavailability of alternative test forms.

There has been a good deal of confusion and discontent over testing matters.  Numerous criticisms articulated concerning the usefulness of the required tests have been articulated by program staff.  In response, the policy makers have tended to acknowledge many of the problems, to urge programs to develop other approaches but, at the end, to insist on the continued use of what apparently all involve appear to agree are flawed instruments.

There is no valid or reliable evidence concerning patterns of student achievement available through the use of the tests required.  In the absence of any alternative Initiative-wide evidence, we are unable to independently confirm the extent of skill development across the Initiative’s many programs.  We do not know if only a few or many students make significant progress.  However, we do know that most of the students who are queried about their participation indicate that it has been beneficial to them and that teachers report evidence of significant progress on the part of at least some of their students. 

Over the years, numerous reports have included testimonies by students to the effect that they were satisfied with their experiences in programs funded through the Initiative, thought that they had learned a good deal and reported that they had used their newly acquired literacy and language skills to attend to important matters in their own lives.  Perhaps most convincingly, the report of results from a CUNY survey of more than 7,000 students conducted in the spring of 1995 concluded that the great majority of students perceived considerable benefits resulting from their participation.

The various outcome counts have left unexplored the simple, but nonetheless fundamental, issue of the existence of any causal relationships between participation and those various outcomes.  For example, the mere fact that a student has obtained a job is not evidence of an outcome associated with participation or achievement as is typically suggested in the reports using a headcount approach.  Such achievement may or may not be related to participation.  We must be able to tell the difference.  Follow-up surveys of representative samples of students might very well be able to illuminate the matter but the available formats of the data files do not readily lend themselves to that purpose.

Current Research

Student Characteristics, Goals and Expectations

Characteristics

We neither saw nor spoke to many young people in the course of our research.  The majority of the students in our study were between the ages of 25-59.

The majority of students in our study reported that they had been born outside the United States.

More than half of the students we spoke with had a native language other than English.

Some of the ESOL students in our study had substantial prior educations; others had very little.  Most of the English-speaking immigrants had also had little.  The native-born English speakers had often completed elementary school, if not some of high school.

Many of the students we spoke with were working.

We did not collect data on students’ public assistance status.

All told, just over forty percent of the students in our study were enrolled in ESOL classes but a good number of the BE students had previously been enrolled in ESOL classes.

While all of the students in our study had been enrolled in their respective programs for at least 100 hours, many had been enrolled for much longer.

Goals and Expectations

Students report a wide variety of goals connected to the following broad categories: acculturation, educational development, the management of everyday affairs, employment, parenting, changing life directions and, simply but profoundly, becoming more literate.

Students rarely come to programs for one reason–most students articulate what will, in most cases, be relatively long-term goals (such as obtaining a GED diploma, enrolling in college, getting a better job) as well as goals that are more immediate or short-term (such as being able to handle everyday affairs more independently).  The goals are usually multiple and interrelated.  For example, at the same time that ESOL students want to learn how to speak better, they hope that being better educated will lead to increased job opportunities.

Students indicated that there were a number of expectations that they had prior to enrollment which, in almost every case, makes the subsequent accomplishment more impressive.  They were afraid that they would be embarrassed or humiliated and they thought it likely that they would not be able to learn much because the teaching methods would be similar to what they had experienced as children or in other programs.

Program Practices

In general, programs are committed to the provision of opportunities for broad educational achievement rather than specific skill acquisition.

Programs utilize a rich variety of teaching and learning approaches and to provide as much in the way of support services as their limited resources allow.

Programs rely extensively on teacher judgment and student input in the evaluations of student achievement and programmatic effectiveness.

There appear to be some differences among program directors concerning the usefulness of the required testing instruments.

Outcomes

Students appear to achieve significant outcomes in a number of areas:

• They report greater proficiency and versatility as readers, writers and speakers.

• They report being better able to more independently take care of themselves and their families by feeling confident enough and skilled enough to fulfill day-to-day responsibilities.

• They report being able to perform work tasks more effectively.

• They report being able to participate more fully in community and family activities (even to the extent of helping others with language and literacy tasks).

• Many of the students we talked to felt that although they had seen tangible outcomes from their participation, they had still not attained many of their initial goals.

• The achievement of highly valued goals, such as a high school equivalency diploma or English language fluency, requires substantial investments of time and energy.  For example, three of the case studies revealed that moving from the beginning levels of BE instruction to GED classes is a long and effortful journey.  The apparent reality is that relatively few of those students who have the longest road to travel stay on the road long enough.  It is not clear why this is so.

• Although we were not able to conclusively establish relationships between particular program practices and outcomes, program practices which foster the development of a community of learners appear to be quite effective in engaging and sustaining students.  In addition, some literacy and language-related outcomes can be fostered by the inclusion of supplementary activities such as work internships.

• Categorizing and counting outcomes in aggregate sometimes mask the important ways literacy instruction does or does not relate to significant changes in students’ lives.  For example, students expressed a desire to better provide for their children which sometimes meant explicit instruction on how to help with homework, but more often meant making more money to provide for them better.  That literacy and family issues are related is clear; what’s less clear is what a program needs to do or can do to assist students’ desired outcomes in these complex areas.

• Frequently mentioned claims of enhanced student self-esteem seem to relate intimately to the development of literacy and language skills in particular. Students in both basic education and ESOL classes talked extensively and enthusiastically about the ways in which their class participation had allowed them to learn about other people, to become accustomed to speaking out and asking questions, to develop points of view on complicated topics, and to take care of day-to-day activities that required reading, writing, and speaking independently.  In brief, they described the many ways in which they had become better able to use language–in both its written and spoken forms–for a wide variety of purposes and goals.  It may be that participation in other non-educational activities could similarly result in a greater power to use language proficiently and with some versatility but–as was evident from our survey of program directors and from the responses of students in the interviews, focus groups and case studies–the promotion and proficiency of and versatility with language is at the very center of adult education in New York City.

Reflections on the Results

            Most of the evidence we will offer for these outcomes comes in the form of student self-accounts.  That is not unlike the results of previous efforts to explore outcomes.  Over the years, numerous reports have included testimonies by students to the effect that they were satisfied with their experiences in the adult education programs funded through the Initiative, thought that they had learned a good deal and reported that they had used their newly acquired literacy and language skills to attend to important matters in their own lives.  Perhaps most convincingly, the report of results from a survey of more than 7,000 students conducted in the spring of 1995 concluded that the great majority of students perceived considerable benefits resulting from their participation.

            The students we interviewed, the ones who participated in focus groups and, for the most part, the ones profiled in the case studies consistently reported that they were significantly benefitting from their participation.  In the interviews, substantial majorities of students reported skills improvement and a wide variety of other accomplishments.  In the focus groups, participants spoke eloquently about the ways in which participation had improved their lives.  And the authors of the case studies profiled a group of quite diverse students–some who committed themselves to the hard work involved in learning and changed themselves and their lives and others who, for a variety of reasons, appeared unable to follow through with the work necessary to advance. 

            In the absence of valid data concerning overall patterns of student achievement, we are unable to independently confirm the extent of skill development across the Initiative’s many programs.  We do not know if only a few or many students make significant progress.  We only know that most of the students who are queried about their participation indicate that it has been beneficial to them.

            In most cases, the outcomes that students report are consistent with the goals they had upon enrollment.  While some such goals reflect distinctive personal characteristics, circumstances and experiences (such as sex, age, country of origin, native language, prior schooling and employment status), it is also possible to enumerate broad motivations– to be more independent and self-confident, to be better educated, to be more involved, to overcome their own reluctances.  In that regard, sustained participation in a program should perhaps be seen as an accomplishment itself since many students report that they were quite worried about what they would encounter.

            Along with the evidence of accomplishment, there is also evidence that some students do not accomplish what, in their mind, would count as “enough” and additional evidence that teachers do not witness student effort or accomplishment commensurate with their stated goals.  This lack of accomplishment is evident in reports of insufficient effort, uncompleted assignments, erratic attendance, and premature withdrawals.  At the same time, it is clear–especially from the case studies–that the achievement of some highly valued goals, such as a high school equivalency diploma or English language fluency, requires substantial investments of time and energy.  The apparent reality is that relatively few of those students who have the longest road to travel stay on the road long enough.  It is not clear why this is so.  As a result, there is much to investigate and learn about the complex intersection of student expectations, program practices and external demands.

            When we began our research, we were especially concerned to probe beneath the frequently mentioned claims of enhanced student self-esteem and to determine the actual extent of the dependence of such enhancement on literacy and language skill development.  In other words, would students have benefitted as much or more if they had been involved in some activity other than adult education?

            Although we hardly have the last word on the matter, we have come to think that the outcomes associated with successful participation in adult education are quite distinctive and meaningful.  Students in both basic education and ESOL classes talked extensively and enthusiastically about the ways in which their class participation had allowed them to learn about other people, to become accustomed to speaking out, to develop points of view on complicated topics.  In brief, they described the many ways in which they had become better able to use language–in both its written and spoken forms–for a wide variety of purposes and goals.  It may be that there are other forms of participation that could similarly result in a greater power to use language proficiently and with versatility but–as was evident from our survey of program directors and from the responses of students in the interviews, focus groups and case studies–the promotion of proficiency and versatility with language is at the very center of adult education.

            Although we were not able to conclude much about the impact of discrete program practices, we believe that the reports by students–of fears overcome, anxieties relieved and of a kind of support and attention so unlike their earlier school experiences or experiences in other settings suggest that a constellation of program practices results in the establishment of productive communities of learners and has far-reaching positive effects.

            Unfortunately, as mentioned above, the definitive documentation of what we believe to be the outcomes associated with participation has been made very difficult by the very procedures customarily used to record outcomes and by inconsistencies in and inaccessibility of the Initiative’s primary data base.  The procedures have emphasized the reporting of different types of test scores for literacy and language students, ostensibly to document patterns of achievement, and the counting of an array of presumably beneficial consequences–such as obtaining employment, registering to vote and so forth. 

            Far-reaching criticisms have been articulated and borne out concerning the validity of all the test instruments.  In addition, the counting of outcomes has left unexplored the simple, but nonetheless fundamental, issue of the existence of any causal relationship between participation and/or achievement and those various outcomes.  For example, the mere fact that a student has obtained a job is not evidence of an outcome associated with participation or achievement.  Such an achievement may or may not be related to participation.  We must be able to tell the difference.  Follow-up surveys of representative samples of students might very well be able to illuminate the matter but the available formats of the data files do not readily lend themselves to that purpose.

            We believe that this project has yielded deeper understandings, new insights and useful suggestions for the future work of the Initiative.  Specifically, we believe it essential that the Initiative’s policy makers address the issues related to test validity, outcome identification and reporting and data base usefulness.  These issues are not new ones.  Indeed, the archive review revealed that they were being articulated from the very earliest days of the Initiative.  However, they have not been satisfactorily addressed.  That failure casts shadows on the authentic accomplishments of programs and students and must be remedied.