The questions we ask our students (and the ones they answer)

The accreditors are coming around to our campus again soon, so assessment is on the march. We held a two-day writing assessment workshop on campus over the summer, and I participated in scoring essays written by first-year students the previous fall. I came away just as skeptical about the quantitative assessment of college writing as I have always been, but I nonetheless found my self shaken by how much the exercise showed me about the pedagogy of college writing.

Recognizing the limitations of giving everybody the same prompt, detached from any connection to course content, the framers of our assessment project—a group of skilled and thoughtful people—gave the teaching faculty some directions about framing their writing prompts but left room for tailoring them to each class. This approach represented our effort to avoid the Scylla and Charybdis of writing assessment: the distorting artificiality of standard exercises, on the one hand, and, on the other, the inability of standardized questions to capture the kind of context-specific scholarship that we most want our student to practice. I was on my first committee trying to navigate those waters in about 2002; I haven’t yet seen anyone find safe passage.

In this latest assessment exercise, the variation among the faculty-written prompts was dizzying. Some were detailed, to the extent that they sounded like guidance for writing full-length scholarly articles. Some consisted of a single sentence inviting the student to analyze two writers, period. Some asked for summary followed by analysis. Some asked students to respond to passages that we faculty had trouble understanding out of context. My point is not that the prompts were bad but that they were so varied that it would be hard to imagine them producing writing that we could assess with a consistent set of criteria.

The real surprise came from reading the students’ essays. In crucial ways, their writing revealed that the students often had not read the prompts carefully, and they were right not to do so. The prompts asked for different kinds of writing, but the students responded in largely uniform ways. They understood the assessment exercise. Most of them have done similar things throughout their elementary and secondary educations: they knew they were supposed to write a short essay, conventionally structured, with some quoted evidence sprinkled in.

And indeed, that’s exactly what we assessed. With our rubrics and inter-rater reliability training in place, we were almost always able to score the essays in a straightforward way because the students knew to rely on the skills that had been praised and rewarded so often in their educations, no matter what their teachers tried to tell them on a given assignment.

The students’ ability to perform assessment-ready writing humbled me in two ways. First, it reminded me that students have often deduced my expectations when I have not explained everything that they need, even though I tend to explain a lot. The assessment exercise showed me how much we all lean on unstated expectations. Second, a gained a new way of thinking about how difficult I have found it to try new kinds of assignments, even with students who are curious, creative, and ambitious. Now I see such assignments in this light: every time I take a step away from an assignment that boils down to “Write an essay of length X on topic Y,” I remove some of my students’ confidence that they know what implicitly earns rewards in academic writing, even if the explicit requirements are incomplete or difficult to understand.

I still want to push my students and myself to break away from conventional essay assignments. I want them to become capable editors as well as readers, to give presentations that deploy ironic as well as explanatory slides, to work productively as members of creative teams that must evaluate their own work and choose how to share it. As I ask them to learn these skills, however, I will do so with a renewed awareness of how much I am requiring them to leave behind the techniques and assumptions that have gotten them to this college in the first place, and I need a similar sense of humility as I encourage colleagues to try new techniques and assignments. I have been thinking especially about the dynamics of classroom authority, race, gender, sexuality, class, and disability: it is easier for some of us than others to ask students to step away from expectations they know they can meet.

I am just beginning to turn from these thoughts to building a structured sense of how to respond constructively to them. From conversations I have had so far, I suspect that my thinking will draw heavily on the methods of my colleagues in the creative arts, for whom it is nothing new to ask students to express vulnerability, to judge one another’s work constructively, and to work in teams whose members have complementary skills. More to come.

More on Jane Austen and stylistic signatures

Ted Underwood responded to my post on Jane Austen’s style–pointing out the prevalence of adverbs, “to be” constructions, and terms of certainty–by raising the issue of baseline comparisons: “I’d like to know whether this is something about Austen in particular, or whether it’s a characteristic feature of a period/genre. I don’t intuitively know which is more likely.”

Let’s explore! I’m again using Ted’s corpus and software, comparing a given author’s work to the whole corpus. This file is a transcript of the commands and output I’m interpreting below.

I thought the most conventional guess of an author to produce results similar to Austen would be Maria Edgeworth. Here’s the list for her:

 WORDS OVERREPRESENTED BY MANN-WHITNEY RHO 
1	understand     	0.937	271	
2	recollect      	0.923	309	
3	talking        	0.916	127	
4	know           	0.916	523	
5	could          	0.913	754	
6	provoking      	0.912	41.9	
7	nonsense       	0.911	62.3	
8	perfectly      	0.905	119	
9	explain        	0.903	192	
10	continually    	0.889	95.4	
11	tired          	0.888	76	
12	going          	0.888	205	
13	do             	0.884	586	
14	dear           	0.88	792	
15	sorry          	0.879	79.5	
16	satisfied      	0.879	93.8	
17	yesterday      	0.879	48.9	
18	liked          	0.875	48.1	
19	spoiled        	0.874	19.6	
20	directly       	0.869	77.2	
21	quite          	0.869	136	
22	please         	0.868	182	
23	you            	0.868	2467	
24	repeated       	0.868	233	
25	decide         	0.866	101	
26	afraid         	0.864	148	
27	repeating      	0.862	52.7	
28	thank          	0.862	115	
29	manage         	0.86	44	
30	guess          	0.86	97.8	
31	sure           	0.859	290	
32	ashamed        	0.857	35.4	
33	put            	0.856	140	
34	admiration     	0.855	90.5	
35	disappointed   	0.855	44.8	
36	surprised      	0.855	75.6	
37	tiresome       	0.853	37.2	
38	especially     	0.853	76.3	
39	not            	0.853	802	
40	reading        	0.853	80.1	
41	dressing       	0.852	9.04	
42	said           	0.852	2783	
43	formerly       	0.851	50	
44	understanding  	0.851	103	
45	possible       	0.85	157	
46	because        	0.85	261	
47	really         	0.85	125	
48	any            	0.85	632	
49	saw            	0.85	183	
50	think          	0.85	173	

My unsystematic eyeballs see no forms of “to be” and far fewer adverbs than populated Austen’s list. Terms of cognition seem especially prominent:

 WORDS OVERREPRESENTED BY MANN-WHITNEY RHO 
1	understand     	0.937	271	
2	recollect      	0.923	309	
4	know           	0.916	523	
9	explain        	0.903	192	
25	decide         	0.866	101	
30	guess          	0.86	97.8	
44	understanding  	0.851	103	
50	think          	0.85	173	

What about Charlotte Lennox? Her list has “extremely” and “wholly” in the first and sixth places, but only one other “-ly” adverb (“instantly” at #29). Lennox’s vocabulary emphasizes the dynamics of sociability. Highlights:

 WORDS OVERREPRESENTED BY MANN-WHITNEY RHO 
2	civility       	0.97	117	
7	amiable        	0.959	353	
8	accompany      	0.959	55.8	
11	conversation   	0.957	258	
12	behaviour      	0.954	419	
13	mortified      	0.949	34.6	
14	mortification  	0.948	113	
15	received       	0.945	119	
18	amusements     	0.939	32.3	
19	entreaties     	0.937	54.9	
20	apprehensions  	0.937	89.4	
21	attentions     	0.936	70.9	
27	conduct        	0.929	195	
28	insisted       	0.928	80.6	
29	instantly      	0.927	209	
30	countenance    	0.925	123	
31	situation      	0.924	260	
33	visit          	0.923	107	
35	arrival        	0.922	83.5	
36	acknowledged   	0.92	53	
37	reception      	0.92	46.8	
38	circumstance   	0.919	98.7	
41	relations      	0.917	84.3	
42	letter         	0.916	312	
43	politeness     	0.916	110	
44	shocked        	0.914	89.2	
45	accident       	0.913	74.1	
46	inform         	0.913	74.8	
47	acquaintance   	0.912	131	
50	ordered        	0.91	66.6	

Walter Scott’s list of 50 (using only his fiction for the sake of comparison) includes only three adverbs, none in his top 30, and the highest-ranking is an adverb of action: “hastily.” Scott’s list evokes military contexts and especially hierarchies of authority:

1	answered       	0.958	2519	
4	warrant        	0.944	501	
8	risk           	0.93	263	
13	permit         	0.914	247	
14	trusty         	0.913	169	
19	weapon         	0.905	235	
22	boot           	0.902	127	
23	followers      	0.898	505	
27	domestics      	0.897	122	
30	commanded      	0.895	222	
32	courtesy       	0.894	262	
33	quarrel        	0.893	183	
34	kinsman        	0.892	432	
35	assistance     	0.892	248	
37	saddle         	0.891	109	
43	displeasure    	0.89	123	
44	attendance     	0.889	162	
47	willingly      	0.889	170	

Hannah More’s list (again, using only her fiction) is unsurprisingly packed with religious terminology, and I see little overlap between her list and the others.

If you want motion in your novel, open your James Fenimore Cooper:

 WORDS OVERREPRESENTED BY MANN-WHITNEY RHO 
1	movements      	0.979	903	
3	movement       	0.97	576	
4	direction      	0.961	579	
6	commenced      	0.958	374	
8	companion      	0.952	645	
18	distance       	0.915	552	
20	quest          	0.913	190	
21	returned       	0.913	829	
27	companions     	0.902	268	
37	disappeared    	0.894	137	
38	preparations   	0.893	93.3	
39	placing        	0.893	74.7	
40	position       	0.892	168	

At this point, I think we have at least a preliminary answer to our question: the prevalence of adverbs and so forth in Austen’s works is indeed characteristic of Austen herself, rather than her period or genre.

This little exploration was great fun for me, as the results returned a mix of new insights–particularly about Austen and Edgeworth–and reassuring common-sense confirmation that the tool identifies the characteristic thematic emphases of Scott and More. In a follow-up post, I’ll offer some quick thoughts about other uses of this kind of word-frequency analysis, from the perspective of a beginning user with a pedagogical emphasis.

Jane Austen and contemporary prose style

I’m on leave this semester to do work in the Digital Humanities, so I’ll be posting a lot about that. My interest in DH is not–or has not been–quantitative, but I am expanding my range by dabbling in quantitative methods, currently with the help of Ted Underwood’s wonderful introduction to the topic.

At the end of Ted’s post, he provides a dataset and a program he wrote to find groups of words that form something like stylistic signatures in authors and genres. I’ve been playing with the program, with fascinating results. I’ll share one here. This is the list of overrepresented words in Jane Austen’s works according to one of the measures Ted uses:


WORDS OVERREPRESENTED BY MANN-WHITNEY RHO
1 very 0.985 3283
2 wishing 0.984 154
3 staying 0.982 176
4 satisfied 0.977 188
5 fortnight 0.975 152
6 herself 0.973 1553
7 agreeable 0.973 350
8 be 0.971 2645
9 smallest 0.971 182
10 any 0.971 1112
11 really 0.968 555
12 acquaintance 0.967 462
13 excessively 0.967 91.8
14 nothing 0.967 639
15 assure 0.965 268
16 settled 0.964 261
17 marrying 0.964 196
18 much 0.964 841
19 attentions 0.962 212
20 encouraging 0.961 51
21 directly 0.96 290
22 deal 0.96 329
23 warmly 0.96 96.3
24 must 0.96 1141
25 sorry 0.958 198
26 certainly 0.957 323
27 not 0.957 2023
28 tolerably 0.957 95.9
29 handsome 0.957 136
30 quite 0.956 765
31 been 0.956 899
32 exactly 0.955 248
33 invitation 0.955 194
34 being 0.954 699
35 obliged 0.954 280
36 seeing 0.954 206
37 always 0.953 470
38 pleasantly 0.952 37.8
39 delighted 0.951 107
40 talked 0.95 342
41 perfectly 0.949 283
42 distressing 0.949 61.5
43 solicitude 0.949 89.7
44 comfortable 0.948 167
45 walking 0.948 129
46 continuing 0.947 39.1
47 engaged 0.945 120
48 enjoyment 0.942 122
49 dislike 0.941 86.7
50 talking 0.941 194

The list is interesting in many ways, especially in comparison to the corresponding lists for other authors, but I want to emphasize a side point. “Very” tops the list, and it may also top the list of words I discourage my students from using in their papers. (Mark Twain: “Substitute ‘damn’ every time you’re inclined to write ‘very;’ your editor will delete it and the writing will be just as it should be.”) And that’s not all: I push students to minimize adverbs, intensifiers, terms of certainty, and “to be” constructions. Such words infuse Austen’s list:


WORDS OVERREPRESENTED BY MANN-WHITNEY RHO
1 very 0.985 3283
8 be 0.971 2645
11 really 0.968 555
13 excessively 0.967 91.8
21 directly 0.96 290
23 warmly 0.96 96.3
26 certainly 0.957 323
28 tolerably 0.957 95.9
30 quite 0.956 765
31 been 0.956 899
32 exactly 0.955 248
34 being 0.954 699
37 always 0.953 470
38 pleasantly 0.952 37.8
41 perfectly 0.949 283

I’ve thought many times about writing a handout on style that outlines the conventional guidelines of modern, essayistic style with counterexamples from great literature. (What would Hamlet do without “to be”?) But this list encourages me to take such thinking a step further: Austen’s case alone could become the foundation of a unit on voice, style, and convention.

My Stanford online course

When I took one of the newly offered online Stanford courses last fall, I had no idea that they would become central to a highly charged debate about elite institutions of higher education and online learning–not to mention a key locus of the debate over the forced resignation of my alma mater’s president.  I simply wanted to sharpen my command of MySQL.  Now, the Stanford courses—or, strictly speaking, courses offered by Stanford faculty but not Stanford courses but part of “A Stanford School of Engineering Initiative” (got that?)—have been cited by the columns read by the anti-Sullivan faction at UVA: one by David Brooks, one by John Chubb and Terry Moe (“Stanford, for instance, offers a free online course on artificial intelligence that enrolls more than 150,000 students world-wide”), and one by Ann Kirschner (“Stanford University professor Sebastian Thrun’s free course in artificial intelligence drew 160,000 students in more than 190 countries”).  UVA alumnus and donor Jeffrey Walker cited “the hugely successful online course at Stanford” in an email to Visitor Mark Kington (who has now resigned his position), who sent it along to Rector Helen Dragas.

Therefore, I want to reflect on the online course I took, Introduction to Databases.  First, about those amazing enrollment numbers: though I have done some searching, I have not found a count of how many students completed the artificial intelligence class.  For my databases course, I remember hearing initial enrollment numbers in the 80,000-90,000 range; the professor, Jennifer Widom, later wrote, “This past fall my enrollment was a whopping 60,000. Admittedly, only 25,000 of them chose to submit assignments, and a mere 6500 achieved a strong final score.”  When seeing statistics from these courses–and it’s clear the anti-Sullivan faction saw them repeatedly–we should keep in mind that the number of students completing a given course might be smaller than the number of enrollees by an order of magnitude.  And in most educational environments, anything like a 10% retention rate for one semester is far from “hugely successful.”  I don’t mean that in a snarky way (if you want snark, see this tweet) but rather to note how weird our current thinking about “success” is when courses with substantial costs, no revenues, and little ability to keep the students they attract become the go-to model for emulation by elite universities.

That said, the Stanford course was successful for me, and I’m grateful for it.  I was motivated to succeed in the course: it offered almost exactly the skill set I wanted to develop; I needed those skills to accomplish larger goals for my job; I did not need Stanford-backed credit; and I enjoy situations where I am given information and am left to work through it on my own, at least in the introductory stages.  The course’s online lectures and quizzes, the latter cleverly designed to be repeatable with variations, along with a well-produced discussion board for peer-to-peer interactions, allowed me to work on roughly my own schedule.  That freedom was constrained by generous but real deadlines and aided by (usually) well-calibrated discussion board tips from fellow students.

Those comments from fellow students were crucial to the functioning of the course.  A handful of talented students became, in essence, volunteer TAs, combing the discussion board to find flailing fellow-students and helping them out.  Like almost everyone else in the class, I was a consumer rather than a provider of this help.  The helpers’ spirit of volunteerism fit well with the tone Widom set for the course, which she described as a grass-roots experiment in online education.  I wonder how well these voluntary peer interactions would function in the venture-capitalistic frameworks now being developed for online courses.

I see other challenges as well.  There are the obvious ones: I don’t see a way for this model to work for humanities education, except in a very basic way, and even in technological fields, advanced undergraduate work requires a kind of interaction with peers and mentors that my course did not attempt to offer.  I came out of my course with no peers with whom to work (or study, or joke about the course), no faculty with whom to hash out ideas for new projects, no mentor to write a letter of recommendation.

Credentialing will pose a deeper challenge as well.  My course would have been extremely easy to cheat in, as Widom occasionally pointed out.  But cheating was not a big problem because the stakes were low: Stanford gave no credit for the course, and I doubt many organizations counted it for much, either.  If these courses become means of awarding credits in a way that trades on the reputation of the sponsoring school, however, cheating may become a huge problem; for the introductory skills courses that work best online, back-channel networks can easily distribute answers that will earn credit, and such cheating could quickly devalue the credential, thus removing the incentive to pay for courses.

My conclusions come very close to those recently attributed Theresa Sullivan: I see the best near-term potential in encouraging incremental, grass-roots efforts to test the potential and limits of online learning.  Contrary to some of the rhetoric surrounding references to the Stanford courses, the best parts of the course I took embodied that spirit of grass-roots creativity.

Key document 3: Chubb and Moe in the WSJ

This concerns the third of the three documents supporting the view of online education that contributed to the forced resignation of Theresa Sullivan as president of UVA: “Higher Education’s Online Revolution,” by John E. Chubb and Terry M. Moe, in The Wall Street Journal on May 30th.  Board of Visitors Rector Helen Dragas emailed this piece to fellow Visitor Mark Kington and then commended it to alumnus and donor Jeffrey Walker, in response to Walker’s email about online education.  Walker asked, “How might [online education at UVA] lower our costs, improve productivity and link us to a group of students we couldn’t afford to serve (maybe more kids from the state to please the legislature)…maybe more second career grads?”

Chubb and Moe do present online education as a major disruption to higher education, and, like Brooks, the put forth a fundamentally optimistic view.  They also address the uncertainties and drawbacks of the current initiatives, however: they note that the Harvard-MIT edX initiative has no revenue stream, is paid for by $60 million in university funds, and has no business plan going forward.

Their acknowledgement of the limitations of online courses leads Chubb and Moe to envision a blended model of higher education:

In this way, college X might have its students take calculus, computer science and many other lecture courses online from MIT-Harvard (or other suppliers), and have them take other classes with their own local professors for subjects that are better taught in small seminars. College X can thus offer stellar lectures from the best professors in the world—and do locally what it does best, person to person.

What I find most interesting about this piece, in light of Dragas’s endorsement of it, is that it could easily support either side of UVA’s contest between Dragas’s wish for a major, top-town initiative or Sullivan’s advocacy of an incrementalist, grass-roots approach to online offerings.  Chubb and Moe write that “[e]arly stumbles and missteps (which edX may or may not be) will show the way toward what works, and what is the right balance between online and traditional learning.”  Not many institutions can currently afford to put tens of millions of dollars into learning from stumbles and missteps–and those are the risks built into the optimistic view of two Hoover Institution fellows.

Key document 2: the David Brooks column

As I said in my last post, there were three key readings supporting the view of online education that contributed to the forced resignation of Theresa Sullivan as president of UVA.  The second of these was David Brooks’s “The Campus Tsunami” of May 3rd.

Brooks says that the key recent shift in online education is the entrance of elite schools into the online arena:

[O]ver the past few months, something has changed. The elite, pace-setting universities have embraced the Internet. Not long ago, online courses were interesting experiments. Now online activity is at the core of how these schools envision their futures.

Brooks presents an optimistic view.  Note that, like Kirschner before him, he cites Clayton Christensen to make his point:

In a blended online world, a local professor could select not only the reading material, but do so from an array of different lecturers, who would provide different perspectives from around the world. The local professor would do more tutoring and conversing and less lecturing. Clayton Christensen of Harvard Business School notes it will be easier to break academic silos, combining calculus and chemistry lectures or literature and history presentations in a single course.

Brooks implies that elite universities will gain from online education but only by changing their practices dramatically; his is the sunnier articulation of Kirschner’s gloomy take on the status quo.  Brooks closes with this:

My guess is it will be easier to be a terrible university on the wide-open Web, but it will also be possible for the most committed schools and students to be better than ever.

I am not ready to hazard many guesses about the future of elite institutions, but I would take the other side of a bet on Brooks’s first proposition.  The deployment of high-quality introductory courses online will make it very much harder to be “a terrible university on the wide-open Web” and for that matter to be a community college or other institution other than the most selective.  The more the education you deliver is about building skills rather than establishing credentials, the more of a challenge the new model will be.  I will explain this opinion in light of my experience with the Stanford online course: stay tuned.

Key document 1: Ann Kirschner in the Chronicle

In the wake of UVA President Theresa Sullivan’s forced resignation, the student newspaper The Cavalier Daily filed a FOIA request that revealed the centrality of online education in the Board of Visitors’ move to remove Sullivan from the presidency.  Jeffrey Rossman of the Charlottesville Daily Progress provided a useful summary of the emails.

Rossman outlines the three readings about online education that informed the conversations among the two key Visitors, Helen Dragas and Mark Kington, and alumnus (and major donor) Jeffrey Walker.  The first of these pieces is Ann Kirschner’s column “Innovations in Higher Education? Hah!” from the April 8th issue of The Chronicle of Higher Education.  Kirschner argues that institutions of higher education have been too slow to respond to the disruptive innovation in online learning and have therefore undermined their own efficiency:

Online courses are an important component of higher education’s productivity tool kit, one of the few that offers an intellectually rigorous, measurable, and fiscally responsible way to serve more students while making better use of physical space. We could have tremendous impact by shifting first-year, entry-level courses wholly or mostly online, developed cooperatively but taught locally. Sounds radical, but it’s a pretty old idea, put forth by Carol A. Twigg in 1999, and validated by trial programs over five years with 30 two- and four-year institutions. Her research documented that when institutions redesigned their large lecture courses, retention and learning outcomes improved, and costs went down. It is akin to hospitals discovering that cleanliness reduces bacteria and saves lives.

Kirschner’s invocation of Clayton M. Christensen’s book The Innovator’s Dilemma would strike businesspeople and investors with special force: Christensen’s influential argument concerns the susceptibility of seemingly dominant businesses to competitive challenges from smaller upstarts that, because they are smaller upstarts, can adjust to fast-changing environments without the pre-committed resources and organizational inertia that can afflict their seemingly invincible predecessors.

For me, this piece explains the actions of Dragas and Kington better than any other.  Seeing UVA as the established behemoth, beset not only by institutional inertia (and anybody who knows UVA understands the prominence of that inertia) but also by funding cuts and the expenses of an aggressive financial aid program, Dragas and Kington could have perceived three more years as too long to wait for a new president, and they could have justified their change of direction as a response to a radical shift in online education even in the two years of Sullivan’s term.

Crucially, in this view, public outcry from faculty, alumni, and even other members of the Board of Visitors was a predictable outcome that further reinforced the logic of the decision.  Sullivan’s popularity among these constituencies shows Sullivan to be on the wrong side of the contest resulting from disruptive innovation.

Dragas’s comments to date, while expressing regrets about process, express exactly this view of the public opposition to the substance of her her actions.  We are now seeing a contrary wave of pieces such as this by Johann Neem taking on the logic of disruptive innovation.  I anticipate the same debate playing out in my faculty meetings and, if applicable, yours over the coming year.

UVA, Stanford, and online education

I’ve been watching with great interest the unfolding tumult resulting from the ouster of Theresa Sullivan as the president of the University of Virginia, with her cautious approach to implementing online education apparently one of the main motivations driving the conspiracy against her.  When the rector of the Board of Visitors, Helen Dragas, issued a statement on the forced resignation, she included this point:

2. The changing role of technology in adding value to the reach and quality of the educational experience of our students. Bold experimentation and advances by the distinguished likes of Stanford, Harvard, and MIT have brought online learning into the mainstream, virtually overnight. Stanford’s president, John Hennessy, predicted that “there’s a tsunami coming”, based on the response to online course offerings at Stanford (one course enrolled an astounding 160,000 students). Michigan, Penn, Princeton, Yale, and Carnegie Mellon are all taking aggressive steps in this direction. The University of Virginia has no centralized approach to dealing with this potentially transformational development.

I have many thoughts about that paragraph: I am working on my own implementations of digitally-inflected teaching; I’m an alumnus of UVA, where I first heard about implementations of the digital humanities in office-hour conversations with Jerome McGann (who played a large role in making UVA a true leader in digital innovation); and last fall, I took one of the Stanford online courses, ballyhooed by Dragas, that seem to have caused much of the fuss at UVA and elsewhere.  This will be the first of a series of short posts about the current happenings.