Assessment and Evaluation
If a teacher made statement that all students performed above average this year on a test, why is this false?
there has to be students who made average
-statistically not possible
On criterion reference test, items will be more general or specific?
specific
On a criterion reference test, items will be of the same or varying difficulty?
same difficulty
On criterion reference test, why are items more specific and of the same difficulty?
because objectives are written at certain levels, only vary 1 level at the most
What is the idea behind criterion reference testing?
content mastery
Give example of a condition in instructional objective?
Given a map, calculator, globe, etc…
What is the main function of a test blueprint?
determines if items match instructional objectives
What are the components of a well-written objective?
conditions, criteria, behavior/performance
What can we do with correlations? what can we NOT do?
make predictions, cannot tell cause and effect
Which correlation is more useful for making predictions: large negative correlation or large positive correlation?
neither
what is the correlation between test scores and criterion called?
validity
If a teacher-made test has items that match her objectives, then the test should have high…
content validity
If you compare a newly constructed rating scale that measured honesty to an older scale that also compared honesty, what validity are we looking for?
concurrent validity
-not construct, even though it deals with emotions
Why should you be more confident in reliability of a test if it is given to the whole school, not just one grade level?
you ensure the variability with a heterogeneous (mixed) group
What validity is it when SAT scores are correlated with GPA?
predictive validity
In test-taker groups, the more similar the population, the less the…
reliability -doesn’t show true population if everyone thinks the same
what is a criterion referenced test?
Compares to predetermined standards rather than to other students
-measures mastery
-standardized or teacher-made
-shorter and narrower
-reflects extent to which goals are being met
How many options are available when making a checklist?
Two
-yes/no, observed/not observed, etc
If you test the same people with the same test at a later time, it is what reliability?
test-retest
If you only have 1 test, what reliability can you calculate?
Internal ONLY
-item to item comparisons
What kind of validity is most important for regular teachers?
content validity
-because what we test must match up to our standards
What is construct validity?
how well test scores represent behavior predicted by idea/theory/etc
-unobservable traits (such as honesty, depression,etc)
If a company selects a new hire based off a test, what kind of validity?
Predictive validity
What is a norm-referenced test?
compare current test takers to past test takers (the norm group)
-standardized test
-results are “relative” to the norm group
“Marie is above average” is norm or crit ref test?
Norm reference
compares her to the norm group
“Marie is in top 25% of her class” is norm or crit ref test?
Norm referenced
“Marie got 97 out of 100 questions correct” is norm or crit ref test?
criterion referenced test
-measures mastery
Does norm or criterion reference usually have more variability of scores?
norm reference
How many items per objective are usually on Norm-ref tests?
1 or 2 per objective
-covers many objs
-purpose is to rank child
what does a NON-overlapping band mean in band interpretation?
-possibly a REAL difference in scores or ability
-overlap would mean it is most likely from chance
What is the difference between a true score and an obtained score?
Error
-obtained does not take error into account (true does)
What is “Alternate Form” reliability?
reliability b/w 2 diff forms of test given to the same person at 2 different times
-indicatess equivalency of forms (such as test form A & B or retake exams)
What are some sources of error?
-test itself (errors)
-test conditions (setting, etc)
-test taker (sleepy, not studied)
-scoring errors
What is a true score?
-value representing a score that is free of error
-impossible to find because all tests have errors
-score that takes error into account
What is content validity?
how much a test matches/measures the teacher’s objectives
-can also apply to state standards
what is the highest level of Bloom’s taxonomy?
evaluation
What are the ranges of SD?
1 SD = 68%
2 SD = 96%
3 SD = 99%
What is test reliability?
-measures dependability and consistency over time
-should yield same measurement of same variable
-3 kinds: test-retest, alternate forms, internal consistency
Which measures level of mastery: Norm or Crit?
Criterion
Can we observe someone “defend”?
yes
How do you make a performance assessment more of a direct measure?
include a variety of mediums
Is it possible to produce a test that is completely without error?
No
-there will always be SOME error
Is norm or criterion referenced tests more useful in classroom decision making?
Criterion, because you look at mastery rather than comparison to other students
What is SEM?
-Standard error of measurement
-variability of error scores
-NOT the difference between true score and obtained score
-as reliability goes down, SEM goes up
Jared has obtained score of 86, so we can be 68% sure his TRUE score lies between __ and ___
83 and 89

1 SD x 3 = 3
86 – 3 = 83
86 + 3 = 89

What kind of test will any test in the content area be?
Achievement test
-content validity
“Reliability goes up, SEM goes down” is pos. or neg. correlation?
negative
-they go different directions
What are the advantages of a performance based assessment?
-Take different forms
-Assess different kinds of skills
-Easily worked in to lessons
-Higher cog effect needed
-active learning
Jared has obtained score of 86, we can be 95% sure his true score lies between __ and ___ (SEM 3)
80 and 92 (2 SD x 3 = 6 86 + 6 = 86 86 – 6 = 80)
What are the 3 types of validity?
Content validity
Criterion-related validity
Construct validity
What is the most important validity for classroom teachers?
content validity
What can you NOT tell from correlation between two variables?
cause and effect
(can only predict)
What are you looking for when interpreting a band chart?
seeing which bars overlap and which do not
What is test-retest reliability
2 scores from same person on same test at diff times
-indicates stability
What are the 3 characteristics of a good obj?
specific, observable , measurable
What is standard deviation?
most accurate measure of variability
-includes all scores in a distribution
-estimate of variability
How many items per obj does crit. ref usually have?
3+ per obj
-emphasis on figuring out where response is lacking
-shows mastery
What is curvilinearity?
two variables move same at first, to a point, then show negative relationship

Ex: drinking and feeling good, to a point, then you feel sick
so (+)alchohol = (+)feelings
then (+)alchohol = (-)feelings

We Will Write a Custom Essay Specifically
For You For Only $13.90/page!


order now
What validity is it when you compare test items to objectives?
content validity
What are the 3 different ways to grade a performance assessment?
Holistic (don’t need to know)
-Checklist
-Rating scales
What is “obtained score”?
= true score (+) and (-) error score

difference is error

The less variability in a test, the lower the…
reliability
-shorter tests have lower reliability
-not enough items to measure more accurately
How do you get away from multiple choice tests?
performance assessments
(informal)
What is criterion-related validity?
how well tests scores correlate to other tests of the same thing
-requires correlation coefficient to be computed
What validity is it when SAT scores are correlated with GPA?
predictive validity
What is negative correlation?
2 variable move in opposite directions

-as A increases B decreases or vice versa
-think of negative as BAD relationship leading to a breakup

What can you tell if you calculate less error on one test than another?
less error = more accurate measure
Is there a “one size fits all” test in regards to validity and reliability?
No

-validity and reliability may only be appropriate for specific population, when administered by competent user

What is positive correlation?
both variables involved move in same direction

-as A increases, B increases

think of positive as a good relationship since they move together

In band interpretation, how do you find if a difference in scores is related to chance?
when there is overlap
If my watch is on 2 oclock but is broken, then the watch is…?
Reliable but not valid
(always know it will be on 2oclock, but it is only true 2 times each day)
Jared has obtained score of 86, we can be 99% sure his true score lies between ___ and ___
77 and 95

3SD x 3 = 9
86 + 9 = 95
86 – 9 = 77

Which is narrower? Norm or crit referencing
Criterion ref

covers few objects because you want to know how well they mastered each one

If you give a condition in an objective, where else must you give that condition?
in the test
-same condition, same materials, same accuracy
What are some disadvantages of performance assessments?
-unReliabile
-subjective due to outside influences
-time consuming
What is the rationale behind performance assessments?
It provides a direct measure of abilities.
Is “with 100% accuracy” a condition?
No, it does not provide a context like “with a map” or something.
What type of referencing is most appropriate for broad objectives?
Norm referencing
If student has raw score of 80, and SEM is 3, and want to be 99% confident of Trues score range, what would that range be?
89-71 (3 SD’s, times 3, then plus and minus)
When is it okay to put opinion in a True/False question?
If you attribute it to a source.
“according to so and so…”
What should you avoid when writing a T/F question?
-using always/never
-opinion
What is the best way to test organizational thinking skills?
Essay
What kind of test item is best for assessing high level thinking skills?
Essay items
What kind of test item are students most likely to guess on?
True/False and Multiple choice
Which test item is the easiest to score?
True/False
What level is appropriate for restricted essay items?
Anything lower than application
-because they do not have free range on the topic, they are restricted
How can you more objectively grade essays?
Grade one criteria at a time, make the essays anonymous
What is the mean of 81, 83, 82?
82
What is the median of 10, 12, 8, 9, 7?
9
What are the measures of central tendency?
Mean, Median, Mode
Which measure of central tendancy is repeated the most?
mode
Which measure of CT divides a distribution in half?
average
Which measure of CT is the average?
Mean
Which measure of CT is the 50th percentile?
Median
Which measure of Central Tendency is the most stable?
Mean, because it takes every score into account
Student scores 49 on vocab test, and the mean for the class is 40, with an SD is 3. What is the z-score and what percentage of the class scored higher?

z-score = 3

less than 1% scored higher

;

z=x-m

;;; SD

;

z=49-40

;; 3

;

z=3 then look at bell curve

What is variability?
How spread out the scores are in a distribution.
Which measure of a variable is the most dependable?
Standard Deviation
-Takes all into consideration
If a set of scores has a variance of 0, what can you conclude?
Everyone in the distribution has the same score
What happens to SEM when you decrease SD?
When you decrease reliability, you have a more accurate test
-SD is variability between scores
How can you change the level of a multiple choice item?
Change the distractors
-make them more plausible
-in stem, have them choose “best” answer
Should you grade one whole essay at a time (on the same test, by the same person)
No, grade by criteria for all essays
What does the “scoring criteria” for short answer and essay items tell you?
how many points it is worth, and what you will accept as correct
In norma dist, approx what percent of scores lies between T-scores of 40-80?
84%
NCLB requires assessment at what grades?
3-8
What does NCLB require?
that students be tested annually
According to the text, what is the real argument against High Stakes Testing?
Using one score to make high stakes decisions
-that score is only one snapshot in time
-it also is biased, produces narrow scores, and causes teachers to “teach to the test”
What kind of discussions should teachers have with students about High Stakes Testing?
Simple and positive
What are the 12 conditions for a HST program according to the American Educational Research Association?
1. don’t use a single score for high stakes decisions
2. everyone should have same resources and learning opportunities
3. validation for each intended separate use (don’t use same score to tell graduation, promotion, financing, etc)
4. tell users the possible negative consequences of HST programs
5. test and curriculum are aligned
6. validity of passing scores and achievement levels (what the scores mean)
7. remediation available to those who fail the HST
8. attention to language differences
9. attention to disabilities
10. stick to rules about who will and wont take test (don’t tell low-performing students not to come to school that day)
11. sufficient reliability researched for each intended use
12. ongoing evaluation of intended and unintended effects of HST
What is Reconstitution?
Moving teachers around or not renewing contracts because of test results
What are some test-taking strategies you can teach students for HST?
-sleep, breakfast, study
-follow directions carefully
-read each item, passages, information carefully
-manage test-taking time
-easier items first
-eliminate options before answering
-check answers after completing test
What does a positively skewed distribution tell you about the scores?
majority of scores fall below the middle of the score distribution
-there are many low scores, but few high scores

[image]

positively or negatively skewed?

[image]

positively skewed

 

-many low scores, few high scores

-most scores fall below the middle

-tail is toward the positive end of curve

 

[image]

positively or negatively skewed?

[image]

negatively skewed

 

-many high scores, few low scores

-scores lump above middle

-tail is toward neg. end of curve

 

[image]

|    |   |

            1.   2.  3.

 

what distribution, and label mean, median, mode

[image]

             |    |   |

            1.   2.  3.

positively skewed

1. mode 2. median  3. mean

[image]

                |    |   |

                  1.   2.  3.

 

what distribution, and label mean, median, mode

[image]

                 |    |   |

                 1.   2.  3.

negatively skewed

1.mean 2. median 3. mode

Which measure of CT is most frequently used?
mean (average)
Which is not affected by extreme scores, the median or the mean?
median
-represents the middle better when scores are skewed
What are 2 modes in a distribution called?
bimodal
what are 3 or more modes in a distribution called?
multimodal
If each score in a dist. occurs with equal frequency, what is the mode?
no mode (not 0)
What is the least stable measure of CT?
mode
(a few scores can influence significantly)
In a normal distribution, which measure of CT has the most value?
none, they are all the same value (all reach the same highest points on the bell curve)
If the mean is 47, the median is 54, an mode is 59, what is the shape of the distribution?
negatively skewed
goes in same order they are listed
What does the semi-interquartile range do?
prevents extreme scores from influencing the sensitivity of a range (such as if everyone scores in the 40s but one person scored a 90)

only the middle 50% is computed, top and bottom 25%s are left out

What does each quartiles in SIQR mean?
Q1 is the point below which 25% of scores lie
Q2 is the median, or 50% of scores
Q3 is the top 25% percent of scores
What is the most commonly used estimate of variability?
Standard Deviation
What is the most accurate measure of variability?
Standard Deviation
-includes all scores in a distribution
What does SD tell you?
how much a single estimated score actually describes all scores within the range
What do large and small score values represent in SD?
large SD means more variability, smaller SD means less
How can you tell the strength of a correlation?
how close the numbers are to -1.0 or +1.0
What does the sign (-/+) tell us about a correlation?
whether it is negative or positive
What does a correlation of .00 mean?
there is no correlation at all, high and lows in one are associated with highs and lows in another
How do you tell pos or neg correlation from scatterplots?
the direction of the slope
-from left to right is negative (pointing down)
-from right to left is positive (up)
“Does the test measure what it is supposed to test?” is asking about…
validity
“Does the test yield the same or similar score scores consistently?” is asking about…
reliability
“Does the test score closely approximate an individuals true level of ability, skill or aptitude?” is asking about…
Accuracy
Why must a test have validity?
To show that it measures what it says it measures
(such as a 3rd grade math test actually testing 5th grade math, or even a math test actually testing reading skills)
If someone inspects test questions to see if they correspond to what should be covered by the test, they are looking at what kind of validity?
content validity
What is the problem with content validity?
-hard to tell if construct tests are valid because they are abstract
-only tells if the test LOOKS valid but it might be measuring something else (such as guessing ability,reading skills, etc)
What are the three main forms of validity?
content validity
criterion-related validity
construct validity
If matching test to the standards, what validity is it?
content validity
-making sure what is meant to be testing is actually being tested
What is concurrent validity?
The correlation between two tests that measure the same thing when given to the same people
What is the difference between Construct validity and other forms of validity?
Content, Concurrent, and predictive all measure against something else (standards, other tests, future scores) while Construct measure something that is hard to test or hasn’t been tested before
What evidence is each of these compared to?
Content validity
Concurrent validity
predictive validity
Content: test compared to standards or objectives
Concurrent: test compared to other tests of same thing
Predictive: test scores compared to future scores or abilities
Which type of validity is evidence for alignment with curriculum?
content validity
-making sure test matches the standards and curriculum
What does reliability tell you?
consistency with which a test yields the same rank for individuals who take the test more than once (think broken watch example)
What are the types of reliability?
Test-Retest
Alternate forms
Internal consistency
Leave a Reply
Your email address will not be published.
*
*

BACK TO TOP
x

Hi!
I'm Colin!

Would you like to get a custom essay? How about receiving a customized one?

Check it out