My friend and colleague John Fritz commented on my last post at some length. My response to his comment grew and grew, so I decided to make it a post instead.
I know you’re as passionate about these issues as I am, which is no doubt why your initial question comes out more like a peremptory challenge than an inquiry. Nevertheless, there are important issues here, and I will take a stab at speaking to them.
Of course I believe in evaluating the quality of student learning, both what they’ve learned and the conditions we imagine and provide to foster that learning. But now we’ve got not one but at least three things to assess:
- the student’s orientation toward learning (attitudinal, cultural, cognitive). One big difference between a rat in a Skinner box and a student in a learning environment is that the student brings memory, affect, expectation, etc. to the moment. What the cog-psy people call “appraisal” becomes crucial. And as Donald Norman points out, human beings infer intent and indeed the nature of other minds from the design of what they see and use. Schooling often sends very dismal messages indeed about the other minds who have designed such a deadening experience.
- what the student has learned–and now we have to think about what we mean by “learning.” Memorization? Insight? Creativity? Cross-domain transfer? “Going beyond what is given” [Bruner]? Mastery? Life-long self-directed learning and re-learning? All of the above? I choose “all of the above,” which means that “assessing student learning” must be complex, multi-source, longitudinal, and constantly revised in terms of what we educators are learning about brain science, learning environments, social aspects of learning, etc. etc. Doesn’t mean the assessments can’t be done, but I’ve yet to see an analytics paradigm that’s answerable to that complexity, and I suspect the paradigm itself is simply too limiting, too behaviorist in its model of mind.
- finally (or at least “finally for now”), we have to consider the very structures of schooling itself. While certain human concerns persist, or appear to (I’m not sure school really wants the disruption of true insight to dominate the experience, but maybe I’m just cynical this morning), the conditions and organization of schooling have changed over time, and not always for the better. Clark Kerr’s book The Uses of the University is very interesting in this regard. I also recommend, very highly, Seymour Papert’s The Children’s Machine, one of the most sensitive and poignant examinations of the uses of computers in education that I’ve ever read. Maybe *the* most. Right now I’m reading Norbert Wiener’s posthumously published Invention: The Care and Feeding of Ideas and learning a great deal from his thoughts on the technologies of education (including technologies such as funding, degrees, environments, etc. etc.). I just finished a fascinating article in Scientific American on “cognitive disinhibition” that suggests we should think about the role of disorder and eccentricity in education. The book Falling For Science: Objects In Mind also examines the oblique paths to deep learning, and the sometimes counter-intuitive ways in which the design of learning environments can encourage the learner to discover and explore those paths. I think also, with great admiration, of Chris Dede’s work on learning-as-bonding, and of Diane Ravitch’s newly awakened opposition to so-called high-stakes testing. For me, books and articles like these ought to frame the conversation. When I hear a speaker at a national conference say that supermarkets know more about their customers than schools know more about learners, I think the conversation is on the wrong foot altogether, and dangerously so. What a supermarket knows about my buying habits is not at all an apt analogy for what I want to know about my students’ developing cognition.
If you got the impression from my blog post that I don’t think we should assess student learning, please read the post again. The problem I mull over is the one that occupied Brigham: premature standardization and a “testing industry” (or, mutatis mutandi, an “analytics industry”) in which financial stimuli interrupt the necessary and messy process of ongoing research. Blackboard, for example, got enormously wealthy by giving higher ed a way to avoid dealing with the World Wide Web in any serious or innovative way. I remember being regaled with tales of improvements in everything from menu design to customer service while also listening to scornful commentary on “frills” like wikis, avatars in discussion boards, ingestion of RSS feeds, and of course all competing products. I also heard a lot about adoption rates, as if the very fact of widespread use was a reliable and complete measure of worth. As a sales technique, such talk was undeniably effective. As evidence of better opportunities for learning? Not so much.
Unless and until we acquire the patience, humility, and appetite for complexity that it takes to think and talk about learning, all other questions–allocating resources, evaluating teaching/learning technologies, etc.–are secondary. To assert a final answer to the question of resource allocation before we have suitably rich and complex questions about learning, let alone about assessing that learning, is to “do more widely those things that are now being done badly,” in my view. The huge danger is that resources will be allocated in the direction of anti-learning, or thin and superficial learning (they really amount to the same thing for me). For example: can we safely assume that grades in a course tell us everything we need to know about student learning, so that if grades go up, there’s been an improvement in learning? The grades tell us something, but what? And do they always tell us the same thing, across or even within a course? I don’t advocate eliminating grades as a measure of successful learning. I do advocate that we not design an entire system of assessing learning technologies around that single measure of success, when that measure itself begs so many questions about the nature and purpose and quality of learning.
On the macro scale, the degree completion stats also need more complex and nuanced thought, in my view. Reverse engineer it: if we find a way to nudge more students over the “C” line in more courses so that they pass the courses faster and thus finish the degree, what have we accomplished? Even if we assume that a “C” means the same thing in every class, and that a “C” is an acceptable outcome–and I do not assume these things by any means–what will we have when we have a nation of “college-educated” people who have squeaked by in factory-like classes based on memorize-and-repeat models of “learning” and “assessment”? Not the nation I’d want to live in.
The Internet has been transformational. College can be, too, but not by using metaphors of “management” in the way it thinks about fostering cognitive development. The honeymoon of edtech’s potential is almost over? What honeymoon? I don’t think higher education has progressed much farther in its relationship with interactive networked computing than awkward conversation on opposite ends of the sofa while the parents look on with disapproval.
Gardner,
Wow, Sooooo much here…where to begin (with a not nearly so eloquent a response)…
I want to note that my novice comments come from watching/listening/observing in edtech for just 3 1/2 years since that has been my exclusive focus. I don’t have nearly the wealth of experience of many others (including you). But…my observation so far is precisely what you say….there is no honeymoon. To have that awkward conversation would assume that participants are at least in the same room together. I have not found that to be the case in many contexts.
Here is my (simplistic) take fwiw….the extent to which edtech can (or will ever) make a difference/be transformational is the extent to which it causes us to rethink the very essence of the relational nature of teaching and learning – in its very broadest sense. And I mean there relations with each other, relations with what is to be learned, relations with our own reflections (some would call that metacognition)…
The essence of the power of technology imho is exactly what you note – ‘interactive networked computing’ and all of the ‘relational’ richness that enables/facilitates/makes possible – that is where the transformation lies. Put another way, the difference edtech (mindfully used) can or will ever make is NOT about the technology. The essence of the difference-making lies in the relations. Maybe that’s ‘connectivism’ … I would generally resist any attempt to reduce something so incredibly complex though.
Anyway, I’ll stop rambling. What is at the heart? I am not sure…still pondering (and filing away the titles you mention for future reading). Right now, I am reading Parker Palmer’s The Heart of Higher Education: A Call to Renewal for a different perspective. All this said, I am pretty sure that technology is not it (at the heart I mean)….and no amount of ‘analytics’ will be able to capture or describe ‘it’….
Maybe it’s the cave – we will only ever see the shadows…
Gardner,
As we often say after our always stimulating but perhaps (to others) polarizing conversations, I love you, man! I only question because I care . . . certainly about the topic, but more so you, especially now in your completely deserved position of influence as Chair of NMC and as a recognized leader in higher education. Still, let me attempt a return of serve.
Could the perfect approach to assessment be the enemy of the attainable good one? At least a start–or at what you might call the painfully deliberate courtship stage between ed tech and learning (more on that analogy later)? Again, show me better. And by that I mean show me an alternate approach to assessment that sufficiently scales to address the societal imperative of higher education failing nearly half of those who attempt to pursue it. I’ve long been a fan of QualityMatters or the Student Assessment of Learning Gains (www.salgsite.org), but haven’t seen faculty interest scale up. Maybe I’m not doing a good enough job promoting them (if it even is my job), but there’s a certain entrenchment with the grading system that amazes me: faculty hate it but don’t replace it. Why? In my primary role as an instructional technology support staff member, I’m certainly not standing in the way. But I am dependent on the faculty to define what they do or don’t value, if only because that gives me my cue for how technology might (I repeat “might”) solve pedagogical problems or create new learning opportunities.
Let me ask you a question: I’ve always marvelled at the rich student engagement you describe (and showcase at conferences) in your own or other courses, particularly the articulate, thoughtful and prolific students who blog or create wikis on the open web. I like to think I’ve created a few of those teaching moments in my courses, too, albeit in the dark underbelly of a BlackBorg course site. 😉 But in the back of my mind, I wonder: does this “work” for the weaker students? Are you seeing D & F bloggers in your class? If so, do they perform better academically in succeeding classes as a result? If so, let’s shine some light on that, ok? I’d call that learning analytics, too. Wouldn’t you?
In my research methods course last year, we talked about “operationalizing” the research question or hypothesis one wants to study. My hypothesis is that activity in the LMS (or ANY instructional technology that leaves a usage log) can be a proxy for student engagement. I can’t prove this per se, it’s a hunch. But I see learning analytics (e.g., the correlation of usage, course design and final grades) as a START at operationalizing the research question: does instructional technology improve innovation in teaching or learning? If so, how? Under what conditions? For whom and by whom? Fact: since 2007, students earning a D or F use our LMS 40 percent less than students earning higher grades. What if anything should we do about this? Certainly we should investigate it further, but dismiss it? Entirely? Call it coincidence?
Interestingly, Cole Camplese at Penn State is reporting some similar findings regarding use of PSU’s Confluence Wikis and Movable Type blogs by student GPA–he’s one of the first I’ve seen who is harnessing more open, social media platforms through integration of a common single sign on authentication system. It’s brilliant. And I thought you also attended David Wiley’s fascinating talk at the 2011 ELI annual meeting in which he showed a daily “waterfall” of student activity in the LMS by course grade, as well as an “Academic DNA” showing students who performed well in courses were more active. Again, to be clear, I never say that the LMS creates good students, but I am interested in how good students use the LMS, and what impact (if any) sharing this information with all students has on their awareness, motivation, and academic performance.
I know you tend to characterize this as reducing learning to clicks or behaviorism, but to dismiss it altogether eliminates a reasonable methodological approach that can lead to deeper, qualitative insights. I know because I’ve done it. When we noticed an Econ lecturer had huge increases in his average hits per student in his course, I called him up to find out why. That’s qualitative research. Turns out he had started leveraging a tool we turned him onto during our hybrid course redesign workshop, and now his course is the most active campus wide for the past two years, and his students are performing 20 percent better on his department’s common final exam he does not administer, develop or see, and that is required to pass the course. This is compared to students in other sections of the same course. We’ve been promoting his example for the past two years, and the percentage of courses now using the grade book has risen from 46% to 54%. Sure, there’s more to it and we are exploring our hypothesis further, but learning analytics is A research methodology that complements (and balances) what has largely been an uncritical adoption of the latest new toy’s potential to transform teaching & learning.
Let me ask you this: Does student attendance matter in a traditional, face-to-face course (even one with no instructional technology)? Of course it does. While there certainly is much MORE to engagement than this, if you don’t attend class, it’s unlikely that you will perform well and succeed. The key is the word “attend” which is the root for attendance and attention. Why should this be any different online? In any system, be it an LMS, blog, wiki, twitter or some other social media? Nobody is saying clicks are the be all and end all, but dismiss them altogether? Cmon.
During a recent conversation about student success with Freeman Hrabowski, president of UMBC (where I work), I raised the critique I thought you might offer about learning, engagement and grades (sorry, you weren’t here, so I did my best–I DO listen. ;-). He defined student success as not only a passing grade in one course, but also the following one that assumed pre-requisite knowledge and understanding. Of course we then talked about retention, graduation, eventual placement in graduate school or employment, and the enduring value of a liberal arts education in a constantly changing world. It was great. But at the core, I thought his dependent (transferable?) nature of conceptual understanding and mastery in courses (presumably the curriculum) made a lot of sense. It seems (to me) to rule out the instructor effect, which could be be positive or negative, and focuses on the student eventually taking more responsibility for his or her own learning. Cue Socrates here: “I cannot teach anybody anything. I can only make them think.”
By focusing on what the students do (or attend to) as a proxy for what they might understand, it also allows us to look backward at the learning experiences faculty design to facilitate this. Basically, look at who is doing well (assuming we can define this) and then use their own activity to reverse engineer or backward design the process to replicate it for others, including evaluating student performance that follows. And I would argue, this also helps uncover effective practices in teaching.
But to even get to that point, one has to accept grades as at least an initial indicator of performance (notice I didn’t say learning), and perhaps which courses and instructors to research further. Of course, one has to dig deeper, to perhaps understand how students who earn certain grades do or don’t engage with concepts, demonstrate mastery or perhaps are affected by an instructor’s course design. But at some point, we have to act on what we think we know if only because the implications of not doing so are dire for students who aren’t making it under the status quo.
Let me give you another example from our own institution: introductory chemistry. The pass rate used to be barely 60 percent–and some faculty liked that because it was seen as rigorous. Using a modified version of problem-based learning to replace the poorly attended recitation sessions of our traditional lecture-based intro courses, our Chemistry Discovery Center offered “discovery learning” sessions for all students in the foundational chemistry classes (CHEM101 and 102). Since 2005, when the CDC was established, the “C or better” pass rate in CHEM 101 has risen from 61% to 85%, with a simultaneous increase in academic standards (e.g., what is required to get a C on the same exams in previous courses). Similar trends in results are being achieved for CHEM 102, where the average pass rates have risen from 73.1% to 79.4%. Overall retention rates have increased by 5.5% and 1.7% for CHEM 101 and CHEM 102, respectively.
I assure you, my point is not to brag or throw numbers at you. But the key (to me) was that our former chemistry chair, Bill LaCourse, now acting dean, started with learning analytics (investigating the data) to frame the problem, propose a solution, and then evaluate it. More importantly, he felt that the intro courses themselves were turning students off of science altogether. Nationally, the data on retention in STEM fields seems to bear this out, too. But since 2005, our number of chemistry majors increased from 100 to 170 in 2010 (a 70% increase) and the number of biochemistry majors during the same period increased from 260 to 368 (42%). Based on this success, we have developed and expanded active and collaborative learning models to foundational mathematics, physics, and biology courses as well. In short, we used learning analytics to first be aware that our students were having a problem, identify possible causes, hypothesize and implement a solution and evaluate it. What’s wrong with that?
I’ve gone on far more than a commenter should, and I should get my own blog rather than clog up your airspace. But let me clarify the honeymoon analogy, which seems to have backfired a bit. Ever read “Oversold and Underused: Computers in the Classroom” (2001)? What about Manuel Castell’s “The Network Society” or David Noble’s “Digital Diploma Mills” series? We might dismiss people who don’t embrace technology like we do as “luddites” or cranks, but since the PC revolution, we’ve had a generation of folks like you and me touting the potential of instructional technology to transform teaching and learning. Well, has it? If so, how? In what ways? And can it scale to address the retention and student success problem we face now?
I once had a CIO tell me “I jumped on the teaching & learning bandwagon like everyone else in the 80s and 90s, but the faculty didn’t. In fact, they resisted. I’m not getting beat up by faculty wanting to innovate more with technology, but I live a whack-a-mole existence putting out ERP finance and HR system fires, or dealing with the latest security, privacy or identity theft issues.”
Indeed, this CIO’s experience seems to have been reflected in the annual Educause or Campus Computing “Top Issues” surveys that (until recently) have had teaching & learning ranked very low for CIOs. I actually think the renewal of interest in teaching & learning coincides somewhat with the interest in learning analytics to possibly help assess it. You tell me: how does the budget for academic computing compare to that of administrative computing at most schools? We’re not banks or hospitals, but you’d never know our core mission was teaching and learning if you looked at most university IT budgets. And yet as instructional technologists, we bicker amongst ourselves about which technologies to support or abandon with our scraps. I go to ELI and hear people complain about the cost of Blackboard, but do you know what we’re all paying for ERP systems these days? Cmon.
Rather than accept this is the way things have to be, I want to push for more investment in instructional technology, but not just for pedagogical R & D to explore or innovate. Higher ed needs to get more students through successfully. I think instructional technology can help faculty and students, but to do so we need evidence, not anecdotes of instructional technology’s effectiveness to get a seat at the resource allocation table. I want us to partner with faculty to identify what does and doesn’t work, and I want them to publish scholarly articles about it and have these count toward their promotion & tenure decisions. Ideally, learning analytics should inform or lead to scholarship on teaching and learning–and yes, I do mean both qualitative, quantitative or mixed method approaches.
What concerns me slightly is your ever widening critique, not in response to a retention crisis, but perhaps coincidentally in the midst of it. First it was Blackboard, then it was all Learning Management Systems, and now it’s analytics. What’s next? We can agree to disagree about what is an instructional technology, or which is “better,” but I never imagined you critiquing learning analytics itself. Heck, institutional research offices have been doing learning analytics on traditional, face-to-face courses for years. It’s not really new at all, but why is applying it to courses that use technology (in class or online) such a problem? Again, what would you propose we do differently? I hate to sound like the impatient parent wanting grandchildren, but if the courtship takes any longer or gets more expensive without producing results — including a methodology to define effectiveness — then I’d argue instructional technology is in jeopardy of becoming irrelevant. Perhaps higher education is already for the 75% percent of American adults who do not attain it.
Okay, back at me. 😉
John
Pingback: The College Course as an Experience (or set of experiences) | Pedablogy: Musings on the Art & Craft of Teaching
I love this conversation! A couple of thoughts from my perspective:
1. I sense low-stakes trial-and-error (and iteration) is a good way to grow some assessments of learning that might pass the Gardner test. Imagine a simple pre- and post-course assessment to see how students develop in regards to a given learning outcome, using conversation as the form of the assessment.
2. The benefit of Learning Analytics at this early stage for me is that it makes us ask a) what kinds of data related to student learning activities do we have sitting around and what do they show, and b) what kinds of data do we wish we had and what would they show? The long-term goal for LA (for me) would be to let students see into their own learning (not create an automated grading machine).
Dave
@ Dave:
Apologies to Gardner for taking up more of his blogosphere, but your comments are sorta at the heart of where I’ve been trying to go with looking at alternate measures of student performance and how they do (or don’t) correlate with behaviors that any system could shine light on through usage logs.
Specifically, I adapted the College Academic Self-Efficacy Scale (CASES) developed by Owen & Froman (1988) for our survey to UMBC students who voluntarily agree to be contacted about their actual use of our Check My Activity (CMA) feedback tool on how their use of the LMS compares with peers:
UMBC Check My Activity Survey
http://tinyurl.com/umbc-bb-cmasurvey
Since January of this year, our CMA has had more than 40,000 hits by just over 3,200 distinct students, and like our LMS usage generally, students who earn higher final grades tend to use it more than students earning lower grades. More than 150 students have agreed to be contacted about their CMA usage, but only about 20 have completed this survey, which I suspect is too long (though I’ve had a number of students test it and say it takes less than 10 minutes).
Section 2 (which is the longest) includes the CASES scale that asks students to rate their own “self-efficacy” at a number of skills, attitudes and beliefs that I believe even Gardner would agree are quality indicators of the types of students we want to see more of. It also asks them to “self-report” their GPA, but my hunch is that students who rate higher on the CASES probably earn higher grades. If so, then a CASES scale rating (not just grades) could provide context to the types of activity in the LMS (and our CMA tool) that could be enlightening to students who aren’t performing as well. If students are intrigued enough by the comparison of their LMS activity with peers, then it might be enough of a “bread crumb” interest trail that gets them to look associated skills, attitudes and beliefs. Honest self-assessment might then persuade them to get help (if they need it), which can lead to a number of interventions the university can provide.
Additionally, better insight into the types of students and their behaviors in the LMS might influence faculty design of courses to be more interactive vs. one-way, which the literature suggests is highly predictive of student success Yes, I know, there are a LOT of “Ifs” here, and I don’t yet have what I would call a significant sample size of CMA survey respondents to infer or generalize further. But this is where I’m trying to go.
One concern I have is that when I looked at the gpa of students who have at least agreed to be contacted (even if not many have completed the CMA survey above), the average was well above 3.0 for the average. In other words, these are not the students who need help, and I need to find alternate ways to understand why weaker students do (in fact) use the CMA tool. I want to know why they do so, what it tells them about themselves, and what (if anything) we can do to improve its effectiveness or insights for them and future students. I have some ideas I’m reviewing with our IRB, but if you have any thoughts or suggestions, let me know.
Thx,
John
John and Gardner,
This is an exceptional good discussion, many thanks to both of you for so thoughtfully advocating for your ideas.
I had crafted this big long reply, but thought this article by Steve Denning in Forbes, which quotes Alfie Kohn makes the same point. Which is, if we can’t agree yet on what it means to be educated, how can we move on and discuss how to assess it? Measure it? Improve it?
http://onforb.es/oN1Je4
Seems to me like you two have slightly different ideas of what it means to be educated and that, in part, is why you don’t agree on what to do about technology and its role in education. I found myself agreeing and disagreeing with both of you as I read this, I think for this reason. But honestly, I’m not sure why.
In any case, keep it up. You may think no one cares or is listening, but we do care and we are listening.
Thanks Jim–and yes, Gardner, too. Always good to have a foil who’s not a foe.
John
John,
I LOVE your UMBC Check My Activity project. Seeing what system usage stats tell us about student activities and how they correlate to other metrics and measures is great stuff. I’m also interested in a couple of other “views” that might add up into a bigger comprehensive picture.
1: what can analysis of learning-related data tell us about the inside (if you will) of learning, the thinking part–this it seems would be hidden in the words people use when they’re learning and reflecting on learning. Words in papers and paper drafts and blogs and peer-feedback writing and in reflection essays and early and late tests with essay answers and so on.
2. what can analysis of the systematic structures around learning tell us about how we construct learning environments or sequences or pathways and how we define what learning “chunks” are –the answer here might be hidden in syllabi, course descriptions, reserves reading lists, major descriptions, prerequisite requirements, etc.
And you inspired me to share my own thoughts about Learning Analytics (essentially this comment in a longer form): http://wedaman.wordpress.com/2011/08/09/learning-analytics/
Hullo Gardner
Elinor Ostrom has an interesting take on participative governance and complexity which might be of interest.