How is tvaas calculated

Make informed, data-driven decisions about where to focus resources to help students make greater progress and perform at higher levels. Align professional development efforts to the areas of greatest need. Student expectations are not built on a single score but on a multi-year, multi-subject measure so high or low performers are expected to perform in line with their own history of performance.

When teachers are effective, their students will not lose ground, even if their students are initially high-performing. The converse of this can be true as well. Search for:. TVAAS is a powerful tool because it measures how much students grow in a year, relative to other students across the state that took the same assessment that year. Although one might reason intuitively that two scores are better than one, let us explore the scientific reasons for it. Usually, they are administered once a year.

In Tennessee the tests, although equivalent, are different each year. Even if the same tests were given again and again, students would undoubtedly experience learning and fatigue.

Therefore, the score for any single student for any single testing situation is unlikely to be the exact score that is the true measure of his or achievement. Instead, the scores that students receive from a single testing experience reflect their actual level of achievement mitigated by elements of chance or luck. It is assumed that measurement error is associated with any test score. The standard error of measurement is an estimate of the amount of error to be expected in a particular score from a particular test.

This statistic provides a range within which a student's true score is likely to fall. Therefore, an obtained score should be regarded not as an absolute value but as a point within a range that probably includes a student's true score.

This phenomenon causes difficulty when we try to place students into achievement groups by using only one test score. The students who scored close to their true level of attainment would form the majority in each group, but some of their lucky and unlucky classmates would also appear in the high and low groups. The luck factor shows up as bias in their scores. Experience has shown that these very lucky and very unlucky students are likely to score closer to their own true scores the next time they are tested.

This means that extremely lucky high scorers will tend to score lower the next time and extremely unlucky low scorers will tend to score higher. All of this may seem fairly obvious, but it lies at the root of the problem with the interpretation of raw scores. TVAAS does not rely upon single scores to calculate gains. Students are followed longitudinally over a period of three to five years, and the variance of their scores is entered into the determination of system, school, and teacher effects.

When scores are analyzed this way, it is possible to strip the bias from the individual scores and furnish unbiased estimates of gains. Although TVAAS employs complex computational and statistical methodologies, an individual may use averages of at least two scale scores as a simple way to mitigate an important part of the bias inherent in student scores.

The means teachers use to determine whether students have achieved their goals may be called indicators of learning. They range from simple observation to group-administered standardized tests, from daily homework to complex laboratory experiments.

Test scores, documented performance, and portfolio artifacts are all indicators of learning. Determining which indicators are best suited to specific purposes is the core of the student assessment debate. The determination of whether learning has taken place depends upon what questions are asked and how they are asked.

Regardless of the subject or grade level, there is an infinity of indicators that can provide information about the action or subject under consideration. The precision of the measurement depends on the means used to gather data and the extent to which data are collected.

To put it another way, a meaningful evaluation depends largely upon the quality of the indicators utilized. Frequently, several different indicators may be considered.

Statistics can easily determine the correlations among indicators. By knowing the capacity of the indicators to assess the subject and the correlation between various indicators, it is possible to form inferences from one indicator to another. If the indicators are highly correlated, then it is no longer a question of which is better, but which is more cost effective.

For example, if the eighth grade TCAP language arts achievement test is highly correlated with the eighth grade writing assessment, then it is not necessary to use both from an evaluation perspective.

Both may be needed for other reasons, e. On the other hand, a total lack of correlation between indicators indicates one of two things: they are not measuring the same things, or at least one of them is a poor measure of the subject. Absolute accuracy in any type of measurement is impossible.

To disregard cost, time, and the impact on the subject would be irresponsible. The point is that statistical correlations can be extremely useful to educators in checking the validity of measurement devices and in providing valuable input for making cost effective decisions.

In education, the assessment of learning is generally built around the demonstration of competence in certain domains. These domains and the goals and objectives that address them are formalized in curricular frameworks and course outlines. Teachers design courses of instruction based on the curricular guidelines and, generally, even though teaching may take any number of forms, there is a high correlation between what is taught and the formal curriculum.

It is this correlation that makes large-scale assessment possible. Indicators can be developed that measure learning along the articulated curriculum, and because of the correlation between instruction and the stated curriculum,.

What are some issues important to teachers that are addressed through mixed-model methodology? First, the mixed-model methodology used in TVAAS makes it possible to use all the data available on each child. This is important because, as everyone knows, children sometimes miss tests. Other models that use test data for assessment must either eliminate all sets of incomplete data or must somehow "impute" data to fill in the blanks. By using mixed-model methodology, TVAAS can utilize all the available data without imputing any data.

TVAAS does this by weighting complete records more heavily than partial records, so the records of children with fewer years of data or scores for fewer subjects count less in the determination of educational effects than do the records of children for whom more data are available.

Second, by using longitudinal data, TVAAS is able to produce more reliable estimates of the school, system, and teacher effects on the academic gains of students than other assessment systems. Because students are followed over time and because several years of data are used to determine these estimates, more data are utilized to determine the effects, making them more reliable than "one shot" assessment models. Third, TVAAS contains a methodology that insures that no teacher will be misclassified as extremely good or extremely bad due to chance.

The "shrinkage" estimate that is an integral part of TVAAS prevents this misclassification from occurring. In TVAAS, all teachers are considered to be at their system's mean until overwhelming data "pull" their estimates away from that mean. Since all teacher estimates are measured against their own system's mean gain, a teacher must be found to have gains significantly different from this system mean to be classified above or below average.

Fourth, other assessment systems based on standardized testing have depended on simple raw scores. TVAAS, on the other hand, has dealt with the same evaluation problems by focusing on the measurement of academic progress. TVAAS data have shown that academic progress of students cannot be determined by knowing the economic or racial composition of a school. This means that all students can be expected to make comparable gains, regardless of race or level of affluence, when taught in schools, systems, and classrooms of equal effectiveness.

Fifth, experts in the field of educational statistics and highly respected theoretical statisticians, who have studied TVAAS, have found the process sound and appropriate for the assessment of educational effects. Mixed-model statistics were pioneered outside the field of education in genetics , and though the statistical concepts have been around for several years, they have not been widely used until more recently because of the hardware and software requirements.

Matrix algebra is used and thousands of equations must be solved simultaneously. Even now, one can find little in the literature on the use of mixed-model statistics in the social sciences. Tennessee is on the cutting edge of this methodology, and that is exciting.

The cutting edge is never found in the comfort zone, but it is not necessarily in la-la land either. The huge TVAAS data base, currently containing nearly four million student records, has sometimes been underemphasized in previous value-added explanations. This data base stores up to five years of test scores "on-line," allowing calculations to include a historical profile of each student's scores.

TVAAS could not function without such a data base. The computer that handles this huge data base requires one gigabyte of random access memory RAM. Typically, achievement test scores have been reported annually, and all calculations involved the current year's data only. The next year the whole process began anew. Only in Title I programs or in small school districts was one likely to find multiple-year data, and those annual merges were typically performed at the central office level.

Moreover, gain scores were calculated only for students with matched pre-test and post-test scores. With TVAAS, individual student scores are retained on-line for up to five years, and mixed-model statistics allows all individuals to be included in teacher, school, and district effects calculations, including those with only one year's scores. TVAAS addresses the issue of fairness in calculating individual teacher effects by capitalizing on what is known about normal behavior of both students and test items and measuring the magnitude of any significant deviations from that normal behavior.

Such deviations are aggregated over time at least three years for each teacher. Deviations inconsistencies among a few students are "normal" and may be attributed to many different causes. Deviations by a majority of the students a given teacher has taught flags that teacher as being different from the norm-positively or negatively, depending on the direction of the deviations. Read on to see how this works. Any student taking achievement tests more than once tends to make similar scores on them.

We call this consistency. After we have taught students for a little while, we even label them. Whether we should or not is another question, but we do. We say John is an "A" student, and Mac is a "C" student.

We do this because over a period of time we find that John normally performs at a higher level than Mac on tests or any other criteria we might use to judge their work. This consistency of performance is found whether we are looking in the suburbs or in the inner city, whether we are looking at high achievers or low achievers. Turning to the TCAP achievement tests, we find that individual test items tend to be answered the same way by similar students time after time.

We call this reliability. Because of TVAAS's test score history on each student the longitudinal data base , each student can be evaluated for consistency. Remember that the test items are reliable or they would have been thrown out by the test publisher.

When a given student's profile test score history is found to contain inconsistencies , there has to be some reason for it. TVAAS data clearly show that gain scores are not sensitive to socio-economic differences or racial differences. When the student inconsistencies deviations from the norm are counted and aggregated, the greatest difference always comes up the same: who the teacher was.

Incidentally, the teacher effects calculations in TVAAS were intentionally designed very conservatively to prevent any teacher from being mis-labeled. When a teacher's effects deviate significantly from the average of all teachers, one can be almost certain it is not a fluke.

Should good teachers be worried about teacher effects? Good teachers have nothing to worry about. Here are words of comfort for the great majority:. The law does not speak of consequences at the teacher level at all. The State teacher evaluation process has stipulated for several years that student data may be included in one's data sources.

Without TVAAS, however, there was no way to filter a number of confounding variables and ensure fairness. Because of safeguards included in computing the teacher effects, they will almost always be more positive than the corresponding school effects. Yes indeed. If none of us had ventured out there, we would all still be waiting for lightning to start our camp fires. TVAAS has taken us to the cutting edge in the use of student achievement data in educational evaluation, and that is exciting.

The TVAAS development team has created customized software, brought mixed model statistics from other disciplines, assembled what may be education's largest longitudinally merged student data base, utilized a good norm-referenced achievement test, and with state-of-the-art computing power, has resolved some educational evaluation problems that had previously defied resolution. We are witnessing creative problem solving and technology on the move in educational evaluation.

We expect and accept innovation and progress in other fields; why do some of us consider it impossible in our own? What is the historical perspective for the fuss over these two types of tests, and why are the TVAAS critics confused? TVAAS has been a pawn in a battle it did not start or need-a fight that is irrelevant to the success or failure of value-added assessment in Tennessee.

Here are the facts: In order to function as it was conceived, TVAAS needed a set of scaled tests that are reasonably related to the curriculum and that contain questions of varying difficulty in order to adequately discriminate among a wide range of achievers typically found in most classrooms. TVAAS can also function with properly constructed criterion-referenced tests, refer to the criteria just listed , as we will see with the high school subject matter tests, beginning with five mathematics courses in Turning to the historical perspective: In the beginning there were norm-referenced tests.

Like most things, they had their advantages and their disadvantages. Then, about 30 years ago a new kind of test was proposed, at least partially to address some of the perceived problems of norm-referenced tests. The first skirmishes of the war were fought between the proponents of the new CRT's and the defenders of the established NRT's. It soon became obvious that the new CRT's had some disadvantages of their own, one being that they could not, by their very nature, furnish the same kind of information that was available, and needed, many said , from norm-referenced tests.

Subsequently, a third party was formed with a platform suggesting that both types of tests were needed. By this time, however, battle lines had been drawn deep in the sands for some people, and some of those folks continue to fight even after the war has ended. While searching for tests with properties which would meet their specifications, the developers fell into this now well-worn testing controversy.

Originally, circumstances led them toward NRT's-for quite logical reasons, incidentally. The good news is that TVAAS, with its mixed-model statistics and longitudinal student data base, has solved or by-passed almost all of the historical disadvantages of norm-referenced tests. The bad news is that hardly anyone knows it, and those who do have not been very successful, so far, in convincing those who doubt it. These critics are busy trying to shut down the entity that has solved some of their most perplexing problems-irony at its zenith.

An easy way to distinguish between norm-referenced and criterion-referenced tests is to compare and contrast critical points as outlined below:. An average student is expected to correctly An average student is expected to correctly. A lot of teachers seem to be more comfortable with criterion-referenced tests. This is probably because CRT'S more nearly resemble their own teacher-made tests, and the test items tend to be very course specific and limited in difficulty level to average or below.

What many do not yet understand is that through mixed model statistics and a longitudinal data base, TVAAS can accomplish with an. Again, irrespective of labels, TVAAS needs an achievement test series with 1 a continuous scale, 2 items related to the curriculum, and 3 some items both above and below grade level. Check out the topic, The cutting edge , on Page National norm gains may also be called target gains or expected gains.

One of the first things you should know is that there is nothing mysterious or secretive about national norm gains. They are derived from the norming process which is typically planned and directed by the test publisher prior to the introduction of a new or revised achievement test series.

National norm gains remain constant for the duration of a particular edition of a test. Each of the five TVAAS subjects has its own set of national norm gains, so if you are interested in all subjects, you will have not one, but five sets of expected gains. If you prefer, you may compute the national norm gains that TVAAS uses and construct your own graphs to illustrate them. Take a piece of ordinary graph paper and put time on the horizontal axis in the form of years actually, grade levels, beginning and ending anywhere you would like between kindergarten and the twelfth grade.

On the vertical axis put scale scores. For the entire span of grades on the TCAP's, the scale will begin with 1 and end with At each grade marked off on your horizontal axis, plot the scale score which corresponds to the 50th percentile.

Find a student that ranked at the 50th percentile for each grade on your graph, and look to see what that student's scale score was. To avoid clutter you should do each subject on a separate graph. Use a ruler to connect the points and you will have the normal growth "curve" for the grades you chose to plot. Improving performance on the TCAP achievement tests norm-referenced. Should teachers expect higher TCAP scores if they could obtain an item analysis of their previous year's test results and drill their students on items most frequently missed?

Strangely enough, the answer to that specific question is, "No," if it's a norm-referenced test NRT. The question of how to improve test scores legally and ethically, of course is a legitimate one. The answer is a little complicated. Let's break it down and take one piece at a time. This includes generic skills such as familiarity with test session logistics, test format familiarity, handling separate answer sheets, bubbling, anxiety relief, mental readiness, physical readiness, and practice with timed written exercises.

Because NRT's sample content from domains of knowledge that are simply too broad. For example, a topic covered on one year's test may be absent from the next year's test-replaced by a related topic with different specific objectives. It is simply not true, as most test bashers claim, that multiple choice tests consist mainly of items that ask for the recall of facts.

Anyone who doubts this statement should obtain a copy of a practice book for the ACT or SAT and work through one of the practice tests. Almost all the items require higher order thinking skills; almost none rely purely on factual recall. Low achievers in your classroom will not be ready to be taught some of the more difficult concepts. What, then, is the teacher's solution for improving student gains?

Do not worry about specific little content skills. Teach the child, not the test. Begin where the child is. Teach all the children. Remember that TVAAS gains resulting from good teaching will be reflected irrespective of where the child may rank among other children. TVAAS does not suggest or prescribe a particular method for encouraging academic growth because how teachers help students learn is, and should be, a highly individual decision based on teacher expertise and the needs of students.

Typically, students perform well on norm-referenced achievement tests whenever good teachers, day after day, promote scholarship and make sound instructional decisions. The content validity of the TCAP achievement tests is good. Second, the CRT items were written by Tennessee teachers to intentionally match the Tennessee curricula in.

They design mainstream tests, i. They have no interest in producing a test that will sell only to a narrow market. We believe the curricula in Tennessee are as close to the average national curricula as those in any other state.

In other words, Tennessee has no significant curricular deviations from the norm. Fourth, TVAAS calculations prove a sufficient relationship between the TCAP NRT's and the Tennessee curricula, because the gains demonstrated all across the state would simply not exist if the tests and the curricula were not sufficiently related. Reliability of the TCAP is also good.

Achieving the necessary reliability for a given test is a matter of applying appropriate technical expertise to the test construction process. Again, the publisher is more interested than anyone in producing a reliable test. Since test reliability can be easily demonstrated statistically, the figures are available to show it for the TCAP's. Whether it is time to change achievement tests is a matter of opinion. Our opinion is probably not,.

First, the three or four major publishers of tests are all reputable and qualified to produce good tests. The recent experience the current publisher has had with the Tennessee testing program, however, gives them a slight edge. Second, no test is perfect, and there are always individuals who are dissatisfied with whatever they have; we believe many of the critics of TCAP fall in that category. Third, we believe most of Tennessee's teachers would prefer to remain with the known than be faced with a new test format.

TVAAS can adapt to any of the major achievement test series, but in our opinion, there is no compelling reason to do so at this time. Enemies of TVAAS would like for you to think that standardized testing is going down the tubes in every other state-in favor of authentic assessment or alternative assessment or performance assessment, but that is simply not true. Stay with us on this one, and we will try to sort out this very complicated mess.

There are, of course, other student evaluation strategies besides standardized tests, and there has been a lot of interest in alternative assessments recently. Much of this interest has been sparked by persons who have been dissatisfied with standardized tests. The most rabid of these test bashers would do away with standardized testing entirely. Others take a more moderate position-frequently concluding that both types of assessments are needed.

Before going any further, it would probably be wise to pause to discuss terminology, loose as it is. Alternative assessment seems to mean any alternative to testing. If authentic assessment does not imply that everything else must be un authentic, then we are missing something. Performance assessment seems to mean that the student will do something perform that can be observed and.

Portfolios, writing assessments, research projects, and collaborative assignments seem to be examples of alternative authentic assessments. None of these methods of demonstrating proficiency was invented yesterday. They are all legitimate instructional tools, but attempts to use them as alternatives to standardized testing have met with difficulty because of reliability problems.

Attempts to enhance repeatability and inter-rater reliability have not been totally successful while tending to drive costs beyond fiscally responsible limits. Some states that have embraced alternative assessments are now reversing gears and going back to norm-referenced tests or at least including them in a blend of evaluative tools.

On the other side of the coin, TVAAS has enhanced the results of standardized tests by focusing on gain, employing a longitudinal data base, and using mixed model statistics to analyze the scores. One might be at least partially correct to conclude that the TVAAS development team and the test bashers are trying to achieve the same goal-to enhance student assessment.

The TVAAS team would do it by bringing new technology to the analysis of test results, while the severest critics would do it by eliminating standardized tests altogether and substituting alternative assessments. Tennessee leads the nation with an educational accountability system based on student improvement as measured by standardized tests. To the best of our knowledge Tennessee's Value-Added Assessment System includes the largest student data base of test scores in the world. The statistical analysis which uses mixed-model statistics to report district, school, and teacher effects is the most sophisticated system in use anywhere.

It solves traditional measurement problems associated with norm-referenced testing and, therefore, attributes credit to instructional programs and personnel fairly and without confounding variables such as socio-economic status of communities. A few other places around the globe are beginning to focus on value-added concepts in various types of educational evaluations. Most are using hierarchical linear modeling HLM , which is less sophisticated and less efficacious.

When a prototype is being designed, there are no models to go by. The endeavor is a bit more risky, but when it is successful, it is very exciting, and by contrast the alternative, stagnation, is very dreary. Since it is possible to do funny things with figures, this response needs to be very clear and precise. William L. Sanders, at the University of Tennessee. Those funds are used for the analysis of test scores, specifically, for developing and refining the necessary customized software; hardware upgrades after the original, one-time computer purchase ; annual merges of test data; annual data analyses; and other minor operational costs.

If TCAP achievement tests and value-added assessment applied to all grades, which they do not, the State-wide cost would have been approximately 0. Most achievement test scoring in Tennessee has been done by the State Testing and Evaluation Center for the last six decades, usually at a lower cost than similar services provided by test publishers.

Educators should view statistics for what they are: interesting and useful tools which can help teachers and principals make better educational decisions. Unfortunately, a few people think of statistics as incomprehensible and impractical nonsense-if they think at all.

TVAAS statistics can be explained, if not simply or in a few words. Anyone willing to take the time to develop a knowledge base in statistics can come to understand the TVAAS model. Moreover, there is nothing incomprehensible about mixed-model statistics, but the subject is complex, and without spending some time and energy on it, one will probably have to go with faith.

Incidentally, there is nothing wrong with faith, and it is certainly preferable to ignorance and prejudice. Although some TVAAS opponents may cite the complexity of the model as a basis for scrapping the concept, it is not necessary to be a statistician in order to benefit from the use of statistics. In the nineties a contractor may use both a tape measure and computer software for estimating the quantity of building materials needed.

A physician may use both a tongue depressor and sophisticated imaging equipment to make a diagnosis. Should contractors suspend construction until they are competent at computer programming? Should physicians delay diagnoses until they can build their own imaging equipment? Men and women in many professions effectively utilize information derived through statistical processes they may or may not fully understand.

It is a disservice to educators and their students to promote panic over the complexity of the TVAAS model rather than to encourage and support its effective use. To statisticians all statistics are estimates. An achievement test score is an estimate of a student's true achievement. One must understand that a given student's achievement test score does not represent. The "real thing" exists in theory only. A good estimate is as good as it gets except that the more estimates test scores one has, the closer one can come to estimating the truth.

The current state of the technology does not yet permit the direct measurement of student learning like taking one's temperature with a thermometer , but indicators of learning can be used to proxy direct measurement.

If one can begin to think of test scores as estimates, one can begin to accept the notion that last year's test scores estimates can be legitimately revised when this year's scores become available.

The new information makes it possible to calculate a better estimate of last year's "truth" as well as this year's "truth. In the past, attempts to use student achievement scores for educational assessment have been confounded by a great number of factors including socio-economic status, race, gender, educational attainment of parents, etc. These factors biased the results, because they were associated, to greater or lesser degree, with the scores children made on tests.

TVAAS uses a sophisticated methodology to partition the effect of these factors from the effects of educational entities. Furthermore, the determinant of educational effectiveness is no longer the score a child makes on a given test but the gain a child achieves from year to year. Three years of state-wide TVAAS reports have conclusively shown that mean gain scores cannot be predicted by the racial composition, percent of students on free and reduced-price lunches, or the location of the school or system.

Neither can gains be predicted by previous academic attainment. Students of all backgrounds and achievement levels can make appropriate gains if they are taught from the level at which they enter the classroom.