From "How To Build A Table." Click on the image to see the original website.

Hi John–thanks for stopping by and leaving such a long and thoughtful comment. Yes indeed, you should be blogging, man!  I’d read it, and I’d link to it, too. The blogosphere’s magical that way. Just saying. So here’s my response. Next time, I hope there’s a blog on your end so I can do some pingbackin’. Srsly.

First, thanks for the kind words about my leadership. One clarification: I’m currently Chair of the Board of Directors of the NMC. To say I’m “Chair of the NMC” makes my role sound bigger than it really is. Also, while I do hope I’m making some valuable contributions to the conversation about higher education, I take greatest pride and satisfaction in the students I’ve worked with over the years. I estimate, conservatively, that I’ve had over 3500 students come through my classes since I began teaching full-time back in 1990. I’ve tried to pay attention to what worked and what didn’t in the courses I led. I hope one of my own “instructor effects” was to encourage my students to take responsibility for their own learning, just as you say. But even there, I find, there’s an art to this endeavor, mostly in the manner and contexts in which I as a teacher try to encourage my students. I’m constantly thinking about the effect my best instructors had on me, and constantly trying to weave that into the tapestry of my own teacherly imagination. I had some utterly magnificent teachers. They were all different, except for the clear dedication they all showed to helping me find and nurture my best self. In my own journey, I keep trying to make myself worthy of the love (sometimes tough love) and commitment they gave to me.

I’m thrilled, of course, to hear of the successes of problem- and challenge-based learning in the introductory CHEM courses. This is great news in an area that sorely needs it. Of course it’s a great thing when a problem is noticed, the extent of the problem is demonstrated, and a solution is found. I’m not anti-research or anti-numbers by any means (and neither was Carl Brigham). In the talk I gave at the Fashion Institute of Technology last January, I had fairly sharp words for some of my Miltonist colleagues regarding their unhelpful sneers at quantitative data in the humanities. I so wish I were a neuroscientist–at least, one like Hillary Blakeley. 🙂 My own “APGAR for Class Meetings” is a quantitative metric, and every day I used it I would calculate mean, median, and mode in front of the students–because it was fun, and because it offered three different portraits of how well the class had prepared. Is that analytics? If so, fine. But I understood “analytics” to mean something more specific, something along the lines of “business intelligence for academia”–a kind of data-mining of narrowly defined and measured behaviors in students, behaviors that as you note are only proxies for what we’re trying to investigate (and in my view, dangerously misleading proxies). *That* kind of analytics I have serious concerns about, as I’ve already explained in my blog posts. Are those data entirely useless? No. Do they carry the great risk of making mistaken assumptions about learning seem to be “facts”? Yes. When Chris Dede says our assumptions about learning are fundamentally flawed, what light does that shed on these questions? Yes, we know that time on task correlates well with better grades in most circumstances. But what tasks? And to what end? No offense to David Wiley, who’s done fine work in open education,  but I confess I was not delighted with that waterfall. I was, however, greatly nourished by Randy Bass’s presentation on “the problem of learning in the post-course era,” which analyzed the complexities of cognition much more successfully, in my view, especially in the light of our current cultural moment.

You say that my critique is widening. I don’t think so. I think the species of what I object to are proliferating, but they belong to the same genus.  What I object to, as I’ve explained, is a move away from cognitive and social approaches to learning and assessment, and a move toward more behaviorist models. I don’t object to course web sites. I object to the idea of “learning management,” just as I object to the widespread adoption of get-em-through Computer Aided Instruction, for all the reasons Ted Nelson outlines in “Computer Lib / Dream Machines.” I think people adopt behaviorist and “learning management” models because they yield more easily quantified results (the research is more focused, less messy, and thus more “convincing”) and can drive institutional decision-making more readily. These are not good reasons. These are reasons not connected with learning, at least as I understand the process. People may adopt them with the best of intentions, and genuinely care about student welfare. But in my view they’re also risking premature standardization and a kind of self-validating meaninglessness. In the midst of the “largest increase in expressive capability in the history of the human race” (Shirky), Blackboard demonstrated that what too many folks in higher education really wanted was a closed, neat, easily monitored environment that would preserve the worst of the transactional elements of education. These systems used to be called “course management systems,” but that wasn’t grand enough for Blackboard, so they became a “learning system” and then tried to assert (as I understand it) that they owned the patent on assigning different roles and permissions to various participants in a website.  EDUCAUSE itself protested when Blackboard sued Desire2Learn, a courageous stance given all the parties that Blackboard has helped to fund over the years.

I think a password-protected course website that helps to manage documents within a course has its uses, though I’d never say that such a site “manages” learning. I don’t think learning can be “managed”–as I’ve explained in my posts, it’s the wrong metaphor, and it does matter what we call things. What I see, though, is that such websites *become* the online presence for every aspect of the course, and thus furnish data on “student involvement” that form the basis for “analytics” that measure with fantastic precision an activity that occurs within, and perpetuates, a brutally reductive paradigm of learning. Some of my faculty colleagues resist working online because they’re Luddites or mulish or whatever, sure. (And faculty mulishness has its good side, too, though that rarely gets discussed.) But I also have colleagues who resist working online because “working online” means “using a ‘learning management system.'” Once they understand the other possibilities open to them, they get interested. Cole Camplese and the folks at PSU are exploring those other possibilities in ways I too admire. A large part of what I admire comes from their willingness to build within non-management paradigms of learning and expression. Obviously UMW Blogs is also leading in this area (and has also been an inspiration for PSU, as Cole will tell you himself).

Student cynicism about school breaks my heart, because that cynicism (except for the strongest, most rebellious of them) becomes cynicism about their own lives. Yet what I hear when I talk to students these days is a tremendous amount of cynicism. They know the game. They know the drill. Their “attention” is focused, all right; it’s focused on “getting through.” Stockholm syndrome comes next.

To cite Papert again: “Before the computer changed school, school changed the computer.” If you want to know why we haven’t gotten to the honeymoon (or even first base), that’s why. The promise of teaching and learning technologies, for me, involves changes in how we think about school. I’ve documented my thoughts in this area pretty widely over the last few years, so I won’t repeat them here. I’m not sure how to answer your question about my D or F students. I have had a few of those students, sure, and I try my best to reach them. I want all my students to succeed, to grow as learners and to attain the cognitive fluency that comes from hard work with intellection (which includes memory), experimentation, and articulation. The richness you kindly describe in my presentations comes from that desire, and the students’ answering commitment. I’m not sure what the control group would be for my “R&D,” or that it’d be ethical for me to design a class that deliberately impoverished the learning experience so I could get harder evidence of the effectiveness of my methods and the work we do together. (To be fair, I don’t think you’re asking me to do that–but the “control group” is a perennial problem in experimental design in education.)  I do know that I am regularly astonished by the quality and intensity of work students can do when they stop trying to “figure out what the teacher wants” and learn that the teacher wants them to be their best selves in a particular learning context. If you want more specifics on how to teach a huge intro-level course with those goals in mind, Mike Wesch would be the one to talk to. I’ve learned a huge amount from him, and I am particularly grateful for the example he sets of stubbornly insisting that the right kind of “instructor effect” can make a huge difference.

"Noise Professor" Zachary Dowell's cover for a book I keep trying to write....

When I starting talking about “love analytics” during an interview at ELI 2011, I was thinking of Mike’s beautiful story of his wife’s telling him to love his students and they would love him back. I am also inspired by what Mike has been saying about Erich Fromm’s book on the art of loving as a teaching/learning paradigm. Mike’s a social scientist who’s not skittish at all about data of any kind. But like James Fernandez and Grant McCracken, Mike foregrounds creativity as a mode of knowing, and has no truck with what Fernandez memorably calls “administered intellectuality.” Mike is also demonstrating how we as educators might come to grips with the principle of plenitude that Plato described long ago, a principle at the heart of transformative learning. Here’s how McCracken memorably imagines what might happen if Plato were alive today:

Plato, let’s say, returns to walk among us.   He becomes, inevitably, a figure of  controversy.  The talk show circuit demands his presence.  (“Today on Geraldo:  Plato—architect of Western culture or dead white male?  You decide!”)  There are doubts, of course.  Production assistants do not warm to elderly men who must be talked out of the wonder-struck examination of a parking meter.  (“You’re telling me any citizen may make a claim against this space by inserting a coin?  That there’s an implicit contract between the ‘motorist’ and other members of the polis?”)

But Plato is not entirely astonished by the contemporary world.  He has seen some aspects of our world before.  He would have no difficulty, for instance, with the blooming, buzzing quality of contemporary life.  He wouldn’t blink at poetry too diverse for a common theme or fashion dizzy with pluralism

Plato accepted the world as a place that bloomed and buzzed….

(Grant McCracken, Plenitude 2.0, Book One of Culture By Commotion. Available as a free “drafty book” download here. Don’t miss what McCracken says about “drafty books” at the end, as it’s the sort of thing Kathleen Fitzpatrick, HASTAC, NMC, and others have been working on in other emerging forms of scholarly communication. Also, God save me from such “production assistants” as McCracken describes above–and also from ever becoming one myself.)

If  “analytics” means trying to assess whether something has worked or not, of course I’m fine with that–as long as we keep the questions of “what is that ‘something’?” and “what do we mean by ‘worked’?” and “are our measures really adequate to what we want to know?” as rich and complex as they need to be. From what I see and hear, that’s not happening. A disturbing amount of the talk I’ve heard about “analytics” simply ignores those rich and complex necessities. You write, “Higher ed needs to get more students through successfully.” Through what? And what constitutes success? The getting through? That seems to me like a tautology. You write, “we need evidence, not anecdotes of instructional technology’s effectiveness to get a seat at the resource allocation table.” I love the word “anecdote.” It’s such a polite cuss word. 🙂 What about a learner’s self-report? An auto-ethnography? A work like Papert’s that tells the story of his own journey as a learner–this, mind you, a mathematician’s journey, a mathematician of the highest caliber who spent most of his career working on computers and education at MIT? Are these “anecdotes”?

Really, if the stories of transformative learning are not admissible evidence at the “resource allocation table,” then maybe we need to get our tools together and build a new table.

Norbert Wiener

I came across a striking sentence yesterday in one of the books I’m reading, Norbert Wiener’s Invention: The Care and Feeding of Ideas.

For a great period of invention, the artisans must become philosophers or the philosophers, artisans.

I think Wiener is right, and I have several thoughts following that statement:

• The moments of insight that characterize deep learning have, for the learner, the flavor, feel, and energy of invention. In other words, when learners “realize” something, they do not simply memorize the connection that the teacher has made for them. They feel, and rightly feel, that they have made this connection themselves–which means they feel as if they themselves have invented the idea or connection. The arts of intellectual seduction (as Bruner puts it) are closely linked to the arts of temptation and elicited curiosity, not as a mode of pandering to students, but in simple acknowledgement of the fact that the “a ha” moment does not mean “a ha, now I see what you have shown me” (though one may use such words) but “a ha, I have made a breakthrough, I have invented a new thing.” Of course the learner may or may not have “invented a new thing.” If not, then of course the learner should credit other learners and not cherish the illusion that he or she has in fact invented the wheel. But it is the feeling of having done so that matters, and that separates the pursuit of insight from mere studiousness. It’s important to have the discipline to be studious, but it’s more important to understand that every moment of deep learning feels to the learner like an innovation or an invention, and (thus) to frame the learning experience in such a way as to make that experience more likely. Repeat-after-me is antithetical to the experience of insight or innovation, though it may be a useful stage of preparation, especially if it’s in the context of play, not scolding. Otherwise, as Wiener writes (with the masculine pronoun that, alas, reflects 1954’s biases), “the scholar-workman is bound to a perpetual subordination to a prearranged order of things.” (Sounds rather like our current “curricular” strategies that culminate in “learning management” and teaching-to-the-test, but I digress.) Weirdly, I find that many people seem to think the feeling of invention I’m describing is relevant only to a) mavericks or b) very gifted students (and to the combination of a and b, of course). My argument is that this feeling of invention characterizes all deep learning, and is therefore relevant to all learners; all learning experiences should be designed and carried out with this in mind.
• The artisan/philosopher connection is at the heart of what we think and talk about in the New Media Faculty-Staff Development Seminar.  As the editors of The New Media Reader put it:  “Understanding new media is almost impossible for those who aren’t actively involved in the experience of new media; for deep understanding, actually creating new media projects is essential to grasping their workings and poetics.” Or as Richard Feynman said, “What I cannot create, I do not understand.” Or as Alan Levine insists, it’s all about being there, and creating out of that being. Tanya Roscorla has captured this ethos very well indeed in this article in Converge magazine, for which my heartfelt thanks.
• The artisan/philosopher connection is at the heart of what Jim Groom and Martha Burtis are doing, brilliantly, with ds106.  I am frankly in awe of their conceptions and efforts, and equally in awe of what the students have created in response.

Michael Faraday at the Royal Institution

Finally, to time-travel backward just a bit, the artisan/philosopher connection was reinforced when the Royal Institution abandoned its plans for a separate stairway and entrance for the sweaty makers whose labors furnished the scientists with their instruments. The initial idea was to separate the artisans from the gentleman scientists. Thank goodness the Institution members thought twice, and thought better.

I spoke to this change of heart last January, in a talk at the Fashion Institute of Technology in New York City. The occasion was the opening faculty convocation at the beginning of the spring term. The topic for the entire year had been “Faculty of the Future,” though I think it could equally well be described as “Faculty for the Future.” My hosts were extremely warm and generous. The audience was perceptive and receptive. The State of the Union address just the night before, which I watched in “enhanced” mode on the Web as I sat in my NY hotel room, gave me some key insights to share the next day. And some credit-where-it’s-due there, as well, since my daughter Jenny was also watching that enhanced version, and used a Twitter backchannel to let me know she was right there with me on that meta-level of understanding. That knowledge, in turn, inspired me to further invention.

For the other truth about invention is a mystery: it feels singularly individual, and in many ways it is, but at the same time it is fostered most completely in a society of mutual respect and support. Like a family. Like a community of best-selves whose highest pitch of being emerges from a great whole. Like the Beatles. Like a fellowship of invention. With all the agitation about education these days, I sometimes feel like Frodo, who in his small but stubborn naivete insists that if we carry the ring, we will find the way.

With thanks, then, to my ace librarian Alice and my hashtag artist Jenny, here’s the talk I gave at F.I.T. in January, 2011.

Podcast: Play in new window | Download (38.0MB)

My friend and colleague John Fritz commented on my last post at some length. My response to his comment grew and grew,  so I decided to make it a post instead.

I know you’re as passionate about these issues as I am, which is no doubt why your initial question comes out more like a peremptory challenge than an inquiry. Nevertheless, there are important issues here, and I will take a stab at speaking to them.

Of course I believe in evaluating the quality of student learning, both what they’ve learned and the conditions we imagine and provide to foster that learning. But now we’ve got not one but at least three things to assess:

1. the student’s orientation toward learning (attitudinal, cultural, cognitive). One big difference between a rat in a Skinner box and a student in a learning environment is that the student brings memory, affect, expectation, etc. to the moment. What the cog-psy people call “appraisal” becomes crucial. And as Donald Norman points out, human beings infer intent and indeed the nature of other minds from the design of what they see and use. Schooling often sends very dismal messages indeed about the other minds who have designed such a deadening experience.
2. what the student has learned–and now we have to think about what we mean by “learning.” Memorization? Insight? Creativity? Cross-domain transfer? “Going beyond what is given” [Bruner]? Mastery? Life-long self-directed learning and re-learning? All of the above? I choose “all of the above,” which means that “assessing student learning” must be complex, multi-source, longitudinal, and constantly revised in terms of what we educators are learning about brain science, learning environments, social aspects of learning, etc. etc. Doesn’t mean the assessments can’t be done, but I’ve yet to see an analytics paradigm that’s answerable to that complexity, and I suspect the paradigm itself is simply too limiting, too behaviorist in its model of mind.
3. finally (or at least “finally for now”), we have to consider the very structures of schooling itself. While certain human concerns persist, or appear to (I’m not sure school really wants the disruption of true insight to dominate the experience, but maybe I’m just cynical this morning), the conditions and organization of schooling have changed over time, and not always for the better. Clark Kerr’s book The Uses of the University is very interesting in this regard. I also recommend, very highly, Seymour Papert’s The Children’s Machine, one of the most sensitive and poignant examinations of the uses of computers in education that I’ve ever read. Maybe *the* most. Right now I’m reading Norbert Wiener’s posthumously published Invention: The Care and Feeding of Ideas and learning a great deal from his thoughts on the technologies of education (including technologies such as funding, degrees, environments, etc. etc.). I just finished a fascinating article in Scientific American on “cognitive disinhibition” that suggests we should think about the role of disorder and eccentricity in education. The book Falling For Science: Objects In Mind also examines the oblique paths to deep learning, and the sometimes counter-intuitive ways in which the design of learning environments can encourage the learner to discover and explore those paths. I think also, with great admiration, of Chris Dede’s work on learning-as-bonding, and of Diane Ravitch’s newly awakened opposition to so-called high-stakes testing. For me, books and articles like these ought to frame the conversation. When I hear a speaker at a national conference say that supermarkets know more about their customers than schools know more about learners, I think the conversation is on the wrong foot altogether, and dangerously so. What a supermarket knows about my buying habits is not at all an apt analogy for what I want to know about my students’ developing cognition.

If you got the impression from my blog post that I don’t think we should assess student learning, please read the post again. The problem I mull over is the one that occupied Brigham: premature standardization and a “testing industry” (or, mutatis mutandi, an “analytics industry”) in which financial stimuli interrupt the necessary and messy process of ongoing research. Blackboard, for example, got enormously wealthy by giving higher ed a way to avoid dealing with the World Wide Web in any serious or innovative way. I remember being regaled with tales of improvements in everything from menu design to customer service while also listening to scornful commentary on “frills” like wikis, avatars in discussion boards, ingestion of RSS feeds, and of course all competing products. I also heard a lot about adoption rates, as if the very fact of widespread use was a reliable and complete measure of worth. As a sales technique, such talk was undeniably effective. As evidence of better opportunities for learning? Not so much.

Unless and until we acquire the patience, humility, and appetite for complexity that it takes to think and talk about learning, all other questions–allocating resources, evaluating teaching/learning technologies, etc.–are secondary. To assert a final answer to the question of resource allocation before we have suitably rich and complex questions about learning, let alone about assessing that learning, is to “do more widely those things that are now being done badly,” in my view. The huge danger is that resources will be allocated in the direction of anti-learning, or thin and superficial learning (they really amount to the same thing for me). For example: can we safely assume that grades in a course tell us everything we need to know about student learning, so that if grades go up, there’s been an improvement in learning? The grades tell us something, but what? And do they always tell us the same thing, across or even within a course? I don’t advocate eliminating grades as a measure of successful learning. I do advocate that we not design an entire system of assessing learning technologies around that single measure of success, when that measure itself begs so many questions about the nature and purpose and quality of learning.

On the macro scale, the degree completion stats also need more complex and nuanced thought, in my view. Reverse engineer it: if we find a way to nudge more students over the “C” line in more courses so that they pass the courses faster and thus finish the degree, what have we accomplished? Even if we assume that a “C” means the same thing in every class, and that a “C” is an acceptable outcome–and I do not assume these things by any means–what will we have when we have a nation of “college-educated” people who have squeaked by in factory-like classes based on memorize-and-repeat models of “learning” and “assessment”? Not the nation I’d want to live in.

The Internet has been transformational. College can be, too, but not by using metaphors of “management” in the way it thinks about fostering cognitive development. The honeymoon of edtech’s potential is almost over? What honeymoon? I don’t think higher education has progressed much farther in its relationship with interactive networked computing than awkward conversation on opposite ends of the sofa while the parents look on with disapproval.

I continue to marvel at Ellen Condliffe Lagemann’s An Elusive Science: The Troubling History of Educational Research. That book and Clark Kerr’s The Uses of the University (which I’ve now read twice) have been astonishing experiences for me this term. I wish I’d found them both earlier, but I’m glad I’ve found them now.

I wrote about Lagemann’s book in my last post. I want to continue with another, more focused look at a section called “Developmental Perspectives.” Here Lagemann tells the story of the rise of behaviorism as the fundamental paradigm of educational research, a paradigm that devolves into a kind of  “social bookkeeping.”  (The phrase immediately brings to mind some of the extremes in the new craze for web-based “analytics.”) Yet in that rise, even when it was happening, there were dissenting voices, warnings, even temporary halts in the headlong rush to reductive measures and models of human learning. One such warning came at the very moment the Educational Testing Service was about to be founded. As Lagemann tells the story, “the original proponents of such an organization were William S. Learned and Ben D. Wood, the directors of the Carnegie Foundation’s Pennsylvania Study.” They wanted to keep academic standards high, a laudable aim to be sure, but their models of cognition were narrow and simplistic. Like the miasma theorists in Steven Johnson’s The Ghost Map who thought cholera was caused by bad air, not water-borne bacteria, these experts were well-intentioned but working from a paradigm of fixed innate ability and stimulus-response learning whose basic assumptions were wrong.  We are still living with the dire consequences in many ways, including systems of educational “assessment” that use commodity methods to produce commodified learners.

Carl Brigham tried to intervene. He was not anti-testing. In fact, Lagemann tells us he was a psychometrician who had helped to develop the SAT and “was working to improve the SAT and other tests.” (More context: earlier in his career, Brigham had espoused racial theories of intelligence that he later disowned. Brigham’s break with his earlier views shaped many of the concerns he later expressed about uncritical adoption and use of standardized testing. You can read some of his story in this fascinating Frontline interview with Nicholas Lemann.)  What Brigham opposed was not testing, but a testing industry that encouraged schools to adopt these instruments uncritically and use them crudely, without an adequate understanding of the complexities of learning, particularly the social aspects of learning. Here’s how Lagemann describes Brigham’s effort, and his rationale:

In an article published in School and Society, as well as in correspondence with J. B. Conant, whom Learned and Wood had enlisted to help their cause, Brigham had expressed grave concern about two matters. The first was “premature standardization”–developing norms to give meaning to test results before the full significance of what had been tested was fully understood. The second concern was that there had been a lack of research into questions that were essential if tests were to be meaningful. As Brigham explained, “the literature of pedagogy is full of words and phrases such as  ‘reasoning,’ ‘the power to analyze,’ and ‘straight thinking,'” none of which is understood. Unless there was more research into such fundamental processes, Brigham insisted, testing would interfere with efforts to develop reasonable objectives for education [my emphasis]. Claiming that the demands of the market and the claims of  “educational politicians” had stunted the development of a valid science of education, Brigham feared that sales would overwhelm the research functions of a large permanent testing service. As he put it, “although the word research will be mentioned many times in its charter, the very creation of powerful machinery to do more widely those things that are now being done badly will stifle research, discourage new developments, and establish existing methods, and even existing tests, as the correct ones.”

Brigham’s words could have been written yesterday. His warnings are still urgent, perhaps even more so than when they were first written. Yet they haven’t been heeded, and the results have not been pretty, either. When Campbell’s Law kicks in, true insight disappears:

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

Under these circumstances, the “powerful machine to do more widely those things that are now being done badly” will also shape the entire schooling experience so lopsidedly that whatever the original test sought to measure, even imperfectly, can no longer be measured at all.  Instead,  the practices begin to measure themselves, untethered from complex realities, and to distort, even eliminate, the contexts in which deep learning can occur. Yet we will have self-validating data to make us feel we’re making progress, and a steady market for more feature-laden varieties of (proprietary) porcine lipstick.

Lagemann tells us that Brigham was right:  “the very existence of ETS helped perpetuate existing educational practices,” and “for a time turned scholarship in education away from the progressive purposes that had been so central to it during the interwar era.” The consequence was a shift from trying “to improve the effectiveness of instruction” toward the different goal of  “perfecting instruments of selection,” a shift that persisted until the “cognitive turn” of the 1960’s.

And now here we are in 2011, with a system that continues to appear to distinguish “academics” from “education.” Have we now come to the point in higher education at which the high-stakes testing world of NCLB and its kin, amplified by the worst models of computer-aided instruction, has concealed from us the choices we are making by selling us perfected instruments of selection in the guise of improved educational effectiveness? I often think so, and the thought frightens me. We’re being sold miasma meters to wave around instead of accepting the challenge of thinking hard about complex questions and designing our systems to be elastic enough to prevent the “vendor lock-in,” literal and metaphorical, of institutionally palatable patent medicines that will forever stunt our capacity for intellectual growth.

What could be more disastrous for a democracy?