« Bosnia v. Serbia | Main | Unwise Crowds and the Oscars »

March 10, 2006

TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8341c031153ef00d834af957c69e2

Listed below are links to weblogs that reference The Politics of Testing:

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

The Law Fairy

From what I've heard of complaints about SATs and other standardized tests, the problem is not with the grading but rather with what is tested and how it is tested. Obviously if there is a multiple choice question for which one answer is "correct" it's tough to argue that the grading will produce "errors" -- but I think that most people believe that the test does not in fact test what it supposedly is meant to test: aptitude. Academic aptitude reveals itself in many forms, and some people are simply bad at multiple choice tests, particularly when cultural differences are taken into account (this is part of what underlies more recent charges that the SAT is racist, for example). Popular test-prep courses by institutions such as Kaplan and The Princeton Review adopt the mantra that the SAT (or LSAT) only tests how well you take the test. Their ability to significantly raise test scores suggests that they may be on to something.

The grading errors aren't the issue for many -- rather, they stand as further problem with the test, conveniently enabling its opponents to point to yet more reasons to abandon it.

Roach

I think people don't like standardized tests (or at least uncomfortable with their importance) because they are in effect IQ tests and IQ is largely fixed. The stuff about bad test taking conditions and what-not is a chimera. No one will get an 800 on Monday a 1500 on Friday and an 1100 the Tuesday next. That's not how it works. We're talking a 50-100 point variance at most. Someone whose hard-work can bump them into honor student territory in HS finds himself largely unable to move his SAT score, and certainly unable to move it anywhere near a standard deviation.

IQ is an unpleasant fact of life for many. Our successes are related to our God-given brain power and we cannot do much to alter this. We succeed or fail at least in part due to something that we cannot improve, cannot effect, and have little responsibility for.

IQ cuts at the heart of the "noble lie" at the heart of democratic regimes: "all men are created equal." IQ testing and SAT scores make it plain that they are not, at least with respsect to their intellectual endowment.

slevmore

Agreed. Of course a good deal of high school and college grades amounts to aptitude testing of a sort. If one criticism is that the test is an IQ test, then that is true of many high school and college grades as well. And if another criticism is that we are testing things other than pure aptitude for the task (college or law school attendance, say), then surely one could criticize the coverage of many courses as well. I liked Stanley Kaplan's book about testing, by the way, but I got the feeling that he could also train thousands of people to get much better high school (and college) grades. The best defense of some of these tests is that they are circular; they simply try to produce the best predictor (along with earlier, available grades) of something, as with the LSAT offering a prediction of 1L law school grades. It is more difficult to state a predictive goal for the SAT, but then it is also more difficult to state a test for the accuracy or predictive quality of high school and college grades.

andrew smith

Refer to the February 2006 Recruitment and Retention in Higher Education (Vol 20, Issue 2 p.8). In May the College Board will consider splitting the SAT into separate sittings as you suggest. While this remedy may increase student confidence in their results and improve the image of the testing industry, it will require not only additional transaction costs but increased administrative costs. This will compel the College Board to increase the registration fee and further strain low-income test takers.

One way to alleviate some of the stress and fatigue, which is a common complaint of students, associated with the test would be to push it back a few hours. College Board’s website suggests that students arrive at 7:45AM in preparation for their 8:00AM test. I remember my own difficulty in concentrating during the beginning of the SAT and LSAT. Researchers suggest that it takes the body up to two hours to overcome the cognitive impairment associated with waking up in the morning (Effects of sleep inertia on cognition. Wertz, A. T., Wright, K. P. Jr., Ronda, J. M., Czeisler, C. A., The Journal of the American Medical Association 2006 Jan 11;295(2):163-164.) This may be an even more profound impairment for high school students, who often skip breakfast and don’t have healthy sleeping habits. While this option does not address the issue of improving student confidence in grading accuracy, it does offer a way to improve the general student perception of college aptitude tests.

anyone

Colleges use the SAT and balance it with grades, recommendations, extracurriculars, etc. Why the bogus claim that the tests are unfair? If anything, they are undervalued as part of the overall package of how a high schooler is presented to the college.

anyone

Colleges use the SAT and balance it with grades, recommendations, extracurriculars, etc. Why the bogus claim that the tests are unfair? If anything, they are undervalued as part of the overall package of how a high schooler is presented to the college.

The Law Fairy

anyone, do you have the stats about how much, percentage-wise, the SAT is weighted as part of the admissions consideration?

If anything, I suspect it's over-valued. It is a tough argument to make indeed that a one-day monolithic multiple choice test ought to have anywhere near the weight of the sum total of a person's achievements, advancements, and experiences over a four-year period.

reliance

What about those students who based their decisions about which colleges to apply to on their erroneously lower scores? These students won't get the benefit of college admissions offices revisiting their applications, because with scores 300 or 400 points below Harvard's average, these students decided to adjust their reach schools and instead aim for Tufts and BU. Shouldn't these students be compensated in some way? Given the realities of the college admissions process, and the standard industry advice given to high school students about determining their "reach schools," I don't think the College Board can legitimately assure that no students will have been hurt as a result of the errors.

josh

Roach (like many) argues that SATs and other standardized tests are inherently merit-based, and therefore critics are just unhappy for ending up lower on the food chain. I disagree.

As a transfer student to the Law School, I could sit for the LSAT 500 times and never get the score that would warrant admission. I remember a former dean of students of u of c telling me my extensive work experience and other merits probably wouldn't overcome my weak test score because u of c wants people "in a certain range." period.

well, one year after grading better than 99.7 percent of my class at a second-tier law school, I strangely became an appropriate candidate for admission into the Law School as a transfer student. in terms of merit or sound predictability, i think the LSAT is generally bunk, or at least should be looked at with skepticism by administrators.

Rich

As a former test-prep instructor for two different companies, and a self-professed geek who has taken standardized tests for fun on boring days, I disagree with most of what Roach says and some of what Dean Levmore argues.

There is a huge variance on standardized tests. In my own experience, I helped students in the 20th percentile jump to the 95th percentile on more than one occassion. My average score improvement for the SAT, when it was still a 1600 point test, was around 200 points - a jump in tiers of schools for admissions purposes. The LSAT was the same way. And from the first time I took a practice test cold until I started prepping for the exam I had some significant movement myself, even as an experienced test instructor.

I think that it is true that many people, if not most, have a score range in which they will ultimately fall, regardless of how much prep work is done. But these ranges are fairly broad, mostly because test-taking strategies are worth a large number of points on each and every exam. And because these skills are not necessarily portable (guessing strategies for multiple choice exams don't help one very much in answering a law school issue spotter), the predictive value can be pretty low.

I think the answer to this problem is, counterintuitively, secrecy. Eliminate as much of the test prep market as you can and provide almost no information about an exam ahead of time. Standardizing such an exam would be difficult, but not impossible. This is basically the philosophy of some schools internal admissions exams (like Stuvyesant in NY). By eliminating test prep (or drastically reducing it, as there are still some companies that will take parents money to prep un-prepable tests), tests will test subject matter and not test taking strategies. That should make them more predictive, less biased toward those with higher incomes, and generally less of a hassle to the median student.

Michael Machen

I think the relevant comparison is to professional testing, like the bar exam or medical boards. The SAT, LSAT, MCAT, and GRE are all 'high stakes' tests, because they affect whether little Johnny goes to Harvard or Arizona State. This is more like the bar exam, because it has a huge impact on Johnny's future. Once at Harvard or ASU, whether he gets a B+ or a B- only slightly affects his future opportunities.

My understanding is that the state bar associations get sued frequently by people who fail the bar, and I would imagine that medical boards do as well. However, it is harder to quantify the loss of Johnny at ASU than it is a law school graduate who is not allowed to practice law.

TJ

One very easy explanation for the outrage difference between standardized tests and school grading, despite the latter's unreliability, is that the magnitude of the error in standardized tests is very easily quantified, compared to the usual disputes about class grading that are highly subjective. I mean this in two senses: (1) the difference between the given score and the "correct" score is easily determined; and (2) the consequence (e.g. ASU instead of Harvard because of a 200 point SAT difference) can be inferred with some confidence.

Dean Levmore, if the registrar's office had a computer glitch that lowered 10% of the student body's law school GPA by 1 point (which cost someone their Supreme Court clerkship), I am certain that there will be considerable outrage.

The Law Fairy

TJ -- but if it was the top 10%... well, that could be a fun and interesting lesson for those with superiority complexes.

And, for those who think grades are less reliable than tests: in this case, why so very very much emphasis on law school grades? As TJ notes, a difference of a point or two can be the difference between having your dreams come true, or having them crushed under the soulless foot of the legal powers-that-be.

Roach

I think the emphasis on law school grades is partly rational and partly a holdover from an earlier error. These differences are less meaningful than they're typically credited. 78 80 76? Are these differences that meaningful considering how little these seem to reflect individual students' sense of their own knowledge and performance and the relatively narrow IQ swath represented in the law school? I think this is more a holdover from the earlier error of less efficient IQ sorting in admissions when a significant portion of the law school's students could expect to fail out and a range representing LSATs of 145-180 could be realistically represented in a particular law school class. Like HS and also college grades, in that era law school graddes were a good proxy of any given students' intellect and abilities.

Second, though no longer particularly useful as a proxy for students' IQ, grades are a useful proxy for work ethic and efficiency, which do matter a great deal on the job. Roughly, the school may be dividied in to the top 10% stars, the good performers, and the bottom 10% who lack the required ethic to succeed at most law firms (though they may succeed in other enviornments). Unfortunately, particularly in regional markets, huge swaths of the law school are cut off from jobs in spite of our high front end selectivity. In TX the bottom half is typically excluded at the top firms. This is too selective considering the false precision of the data.

In an ideal world, LSATs or IQ scores would be front and center on resumes. Then the star students at crummy schools with 163 and lower LSATs would not so easily be able to make positive comparisons to average students at Chicago to employers.

The Law Fairy

Roach, but there are big problems with IQ tests as well. It's hard to make any kind of argument about IQ without defining exactly what you're talking about. Intelligence is impossible to measure accurately, and what does it *actually* mean to say someone with an IQ of 145 is "smarter" than someone with an IQ of 135? How does this intelligence manifest itself? What does intelligence mean?

I don't think relying on IQ tests will get us very far -- particularly when you keep in mind that there are very different kinds of intelligence. For instance, a person with high mathematical intelligence may have an IQ score as high as a person with high linguistic intelligence, but this doesn't mean at all that they're equally suited for any particular job. And it's not enough to assume that people will be naturally attracted to what they are good at, as some people's personalities are geared toward overcoming challenges -- in fact, given the high work ethic of someone with that sort of personality, it might make more sense to give a job to someone with a lower IQ simply because that person will work harder and be more reliable.

Things like LSAT scores and IQ tests have extremely limited utility. That's why measuring aptitude is notoriously difficult.

Roach

How about this: IQ is whatever it is that IQ tests measure? And this measured thing that is sort of like intelligence is highly correlated with all of the things laypeople think of intelligence as coordinated with: analytical ability, memory, problem-solving ability, the ability to notice relationships and patterns, vocabulary, logical abilities, mental quickness, reading comprehension, etc.

Whatever it is that IQ tests measure is highly correlated with lots of important things and it is highly predictive. Obviously one wouldn't expect the 169ers to beat the 168ers every time. But a 175 LSAT is a different kind of intelligence than a 165 or a 155 or a 145. And, more important, for obvious reasons people with 800s on their SATs probably shouldn't be going to law school. At that level of extreme difference, it should be obvious there is something like IQ, it's measurable, and it's important. The literature in this field is actually pretty sophisticated, but comforting egalitarian myths continue for some reason.

MMJ

One big problem with the LSAT as IQ indicator is its similarity over time. No matter what IQ test you're referring to, if you take the same exact test repeatedly, your score will improve. If you take very similar tests repeatedly, your score will improve.

I'm counting on this strategy come June. I intend to take every practice test released, (and I'll pay for the privilege). I've already taken 22 timed practice tests, and my score has improved 14 points. I guess you could say I now have a brighter future in my first year of a law school that mostly gives timed exams. But, you certainly couldn't say my "intelligence" has somehow increased.

Kimball Corson

Viewed as virtual tickets to the good life, those who do not do well on SAT, LSAT, GRE etc. exams are always going to find something to complain about. Many complaints boil down one or another form of ‘they did not ask we about what I know.’ However, the current fuss is different, ETS screwed up and objectively so. So that now, all those with the more amorphous, subjective complaints feel vindicated and can vent their rage. It may not be rational, but it sure is human.

Kimball Corson

Josh,

I think your situation is readily and sensibly explainable. Your LSAT may not have let you go directly to U of C's law school, but you had the smarts to do well where you were and the savvy to then move up the ladder. Your LSAT score consequently became irrelevant. There was no need for a predictor of how well you would do in law school, because you had a good year of law school to demonstrate that. You handled the situation well and your admission was reasonable.

Kimball Corson

Apropos of the IQ, SAT, LSAT, aptitude, work-ethic discussion, I saw a study years ago that the CEO’s of major corporations typically have IQs in the range of 120 to 135 or approximately the second standard deviation above the mean, because below 120 they lack the smarts to get the job done well and above 135, their work ethic, sociability and general demeanor tends to suffer.

With this as background, it can clearly be said that good law firms surely want more than just smarts and grades. Looks, social standing, personality, manners, dress etc. all matter too at the better firms. Introverts with hunched backs and limps need not apply. That so, are the law schools too fixed on grades and LSAT scores and not sufficiently considerate of these other qualities? If so, are they then filling admission slots with too many smart unemployables?

MMJ

Kimball, you're right. Maybe they should have a fashion critic on admissions committees. I think Harvard Law already does. Isn't that why they admitted Elle Woods?

Kimball J. Corson

No, I think Elle got in spite of her dress and accessories. Notice how both improved markedly when she got into the firm as an associate, except of course for the atrocious regression and the mutt’s reappearance in the court room scenes. Believe me, dress matters. I once had a super wealthy client declare my $700 suit (late 70’s) was inadequate and take me shopping, saying I needed to “spruce up” if I wanted to be seen in "his circle." Another broke my piggy bank redecorating my office and buying art. If these people had not found me oddly engaging or at least project worthy, I would have never gotten my foot in their doors or earned the money I did from them.

The Law Fairy

Seriously??? I work for a large firm in Los Angeles and the dress code is business casual every day. I don't own any item of clothing that costs more than $500, and it's waay too fancy for work. A friend of mine has *actually* brought her chihuahua to work (hid him in her desk drawer, but still). And, come on, if you've watched the movie it's OBVIOUS the main reason they pick Elle is because they think she's hot. If the creepy admissions committee men weren't sitting down you'd be able to see their creepy erections. It's obvious from the movie that after they watch her admissions video (featuring her in skimpy clothes and bikinis) they're more open to her resume, whereas beforehand they were pretty dismissive of someone who majored in fashion.

Kimball Corson

Law Fairy,

But that is just the west or more specifically the West Coast, where everything is casual or hugely and usually tastelessly overdone. The top end of LA and, for example, the top end of Manhattan (or Paris) are light years apart. As Neil Simon the playwright once put it, 'There are six million interesting people in New York, but only 72 in LA.’ Very tasteful dress is one way to express being interesting and to open doors toward interesting people. I once saw a women lawyer in a high rise elevator in Phoenix years ago beautifully dressed and wearing the most gorgeous, tasteful pair of medium heels I have ever see. I commented on them. She thanked me, confirmed they cost a small fortune and said that many of her better clients noticed them too. She and I still keep in touch. Great dress entails elements of art and has its own aesthetic. Also, great, well-tailored clothing feels differently as well. While clothes may not really make the man, they definitely affect his or her behavior, I believe. And I say this while traveling around the world now dressed only one step up from a hobo at times, while at others, I am much better turned out. The difference in people’s reactions to me based on dress is spectacular. Dress matters.

P.S.

I found Elle likeable enough but certainly not “hot,” except perhaps by West Coast standards. which are not mine. I would never go out of my way to start a conversation with someone even remotely dressed like Elle. And since when are erections creepy. I have know lots of women who really like them, even if sometimes only as expressions of interest. But you are as right on the admissions video as I am on her dress and mutt. She should have had really great LSATs to overcome those problems.

The Law Fairy

Kimball, I have to respectfully disagree. It's an East Coast bias that deems the West "tasteless." If Easterners find Westerners so distasteful, why do they help make our movie moguls kajillionaires? While you're correct that well-made and tailored clothes fit better and are higher quality, that's hardly a reason for what I would call a snobbish insistence that clothing cost more than, say, $700. It's one thing to have your respect for someone kicked up a notch because he or she is a snappy dresser. It's quite another to insist he or she wear Ferregamo and Armani to be deemed worthy of one's respect.

Also, simply because you deem something "tasteful" does not make it so. While law firms are currently run/dominated by people who may perhaps have a more "traditional" view of dress, more and more the balance of power is shifting to people who are not so demanding on what, ultimately, is not the most important thing in the world. To insist that people dress in expensive clothing is snobby, period.

The comments to this entry are closed.