Author Archives: malpaso

On Adaptive Learning: A Partial Response to Audrey Watters

In “The Algorithmic Future of Education”  Audrey Watters offers a sweeping critique of adaptive learning, arguing that “robot tutors” (her phrase) don’t benefit learners, they are not anything new under the sun, and that, worst of all, they represent a cunning ploy by industry (in league with administrators and managers) to “subjugate labor” and to create “austerity”.  According to Watters, adaptive learning and “algorithms” propel us towards a mechanized world which devalues learning, devalues labor, and devalues “caring”. Her latest piece on adaptive learning is part of her general skepticism and ongoing criticism of the educational technology industry.  

Watters is one of the few observers and critics struggling with some of the deeper questions about educational technology. What is it’s shape? Who benefits? Where is it going? Who controls it?  What does it mean for privacy and autonomy? Of late, she has also set her sights on some of the utterly nonsensical claims coming out of the adaptive learning world. (No researcher in their right mind would make such claims, nonetheless the claims are out there and unfortunately cast a shadow over the entire industry.)  Audrey Watters is a gadfly in the best sense of the term and her arguments need to be taken seriously.

If Watters is asking the right questions, her answers are at times questionable. In a future post I hope to provide a fuller response to “The Algorithmic Future of Education.” In this post I have a narrower aim, which is to point out that Watters characterization of adaptive systems suffers from a factual error and this leads to sloppy generalizations:

“What makes ed-tech programming “adaptive” is that the AI assesses a student’s answer (typically to a multiple choice question), then follows up with the “next best” question, aimed at the “right” level of difficulty. This doesn’t have to require a particularly complicated algorithm, and the idea actually based on “item response theory” which dates back to the 1950s and the rise of the psychometrician. Despite the intervening decades, quite honestly, these systems haven’t become terribly sophisticated, in no small part because they tend to rely on multiple choice tests.”

The assertion is simply incorrect. It’s also not the first time Watters has portrayed adaptive systems as based primarily on multiple-choice questions; powered algorithmically by item response theory (IRT); and, having seen no advances in the “intervening decades”.  She repeats it in her TedxNYED talk and the theme runs as a current through her other publications.

Why is this incorrect? For example, Aleks, which is one of McGraw-Hill Education’s adaptive learning platforms, has never used multiple choice questions. It has been around for nearly a decade. Second, the algorithmic theory (Knowledge Space Theory) behind Aleks is unrelated to Item Response Theory (IRT). Third, unlike Knewton, Aleks is not a black box and never has been. Its algorithmic basis was developed by mathematicians and cognitive scientists working at UC Irvine and University of Brussels. As a result, there is an extensively published and peer-reviewed research trail on how it works and how it aims to advance learning outcomes.

The same can be said of modern Intelligent Tutoring Systems (ITS). Even a cursory investigation of ITS reveals that a shift occurred in the 1970s and 1980s away from Computer Assisted Instruction (CAI) systems, which were based on behaviorist assumptions, to ITS which tried to incorporate advances in computer science, cognitive psychology, and artificial intelligence. Intelligent Tutoring Systems have never been about multiple-choice questions nor have they drawn primarily on IRT as their underlying pedagogical framework.

As a cultural critic and cultural anthropologist Watters will appreciate, I am sure, the importance of genealogy.  IRT emerged in response to the increasing emphasis on high-stakes testing in the US. IRT has also been associated with regimes for measuring “intelligence”.  Aleks, and systems like it, have an entirely different genealogy. The were designed to provide learners with feedback. The world of IRT is the world of summative assessments. The world of well-designed adaptive systems such as Aleks, on the other hand, is the world of formative assessments. There is overwhelming evidence in learning science that formative assessments are among the most important levers we have for improving learning outcomes. Testing is not Learning. Researchers working in this space know it and get it. They have also known it for a long time.

The best research also shows that our goal should never be to replace teachers. Our goal should be to empower teachers and support student-centered learning environments. Some of us believe that technology has a place in realizing this goal. Some of us also believe that which technologies are effective, and in which contexts, should be demonstrated by research and evidence, not anecdotes. 

[Note: The views stated in this blog are my own. I am not speaking on behalf of my employer, McGraw-Hill Education. This post has also not been reviewed or cleared by my employer.]

The R Project for Statistical Computing

R is a free software environment for statistical computing and graphics. It compiles and runs on a wide variety of UNIX platforms, Windows and MacOS. It is supported by the R Project.

R is an ideal language for statistical computing and graphics. It provides a wide variety of statistical (linear and nonlinear modeling, classical statistical tests, time-series analysis, classification, clustering, …) and graphical techniques, and is highly extensible.

One of R’s strengths is the ease with which well-designed publication-quality plots can be produced, including mathematical symbols and formulae where needed.

R is available as Free Software under the terms of the Free Software Foundation‘s GNU General Public License in source code form. It compiles and runs on a wide variety of UNIX platforms and similar systems (including FreeBSD and Linux), Windows and MacOS

Blackboard to Go Private (Summary of Reactions and Commentary)

(Reuters) – Education software company Blackboard Inc (BBBB.O) said it would be bought by private equity firm Providence Equity Partners, a well-known investor in the education field, for $1.64 billion in cash.


Summary of Reactions and Commentary


Joshua Kim, BlogU (June 26, 2011)

Joshua Kim, Director of Learning and Technology at Dartmouth College, suggests (“Providence Equity and 5 Steps for a Successful Blackboard Buyout“)  the following steps must be taken by Providence Equity to grow the value of Blackboard:

  • Integrate the Existing Pieces. “The acquisitions that Chasen has pushed through have all been smart…..The degree to which these products can add up to an integrated and flexible service offering,…will drive short-to-medium term profitability.”
  • Retain Talent. “No one person is ever indispensable, but I think Ray Henderson comes awfully close.”
  • Fill Out Some Missing Pieces. “The most obvious missing pieces to the Blackboard Educational Services company are media are…”
  • Invest in Mobile and Analytics Divisions. “Mobile and analytics are where I see the growth occurring.”
  • Focus on the Global Market. “I don’t think that Blackboard should get into the educational publishing business, or become a direct supplier of e-learning. Rather, I think the global strategy they should follow is that of providing integrated educational platforms that can rapidly scale.”


Rip Empson, TechCrunch (July 5, 2011)

Educated Buy?: “Although the announcement made no specific mention of cost or personnel cuts, these tend to be an inherent part of this kind of acquisition, so Blackboard may be on its way to a bit of a shakeup — or at least some streamlining and trimming — in the coming year.”

“While Blackboard’s products are certainly widely-used, with over 5,000 education institutions using the company’s software, Blackboard has not always had the best reputation among students, especially in terms of user experience. In fact, my college opted for its own customizable educational software for this very reason.”

Jim Farmer, e-Literate Blog (July 2nd, 2011)

Jim Farmer, chairman of Sigma Group Inc, provides a detailed analysis (“Investment Bankers and Blackboard’s Future, Part One“). Some excerpts:

  • “…the expectation of private equity firms for earnings exceeds Blackboard’s 2010 earnings. A combination of higher prices for annual software licenses and reduction of staff and lower services will, in the short run, be needed to achieve this higher profitability.”
  • “The average annual ROI of private equity firms for buyouts is 19.6%, though they may accept a less aggressive figure for one or two years. Assuming a market capitalization of US$ 1.5 billion, earnings would need to be $292 million. This is $271 million more than current projections for 2011. To meet their expectations software prices would increase 52.3%.”
  • “Likely the forthcoming changes at Blackboard will benefit a strong competitor—Desire2Learn. It too has a talented and dedicated management team. CEO John Baker has been an exceptional leader and responsive to the changing needs of higher education.”
  • “If the acquisition is completed, Providence can easily increase revenues, at least for a few quarters, by increasing annual license prices. But to sustain the company and its brand they will have to resolve several potential problems in a challenging environment: Retain the education community’s respect, restrain prices in the face of competing open-source projects and Desire2Learn, integrate the separate products into a suite, and continue software development and product support. This is not simple or certain.”


Josh Keller, The Chronicle of Higher Education (July 1, 2011)

Keller (Private-Equity Firm to Buy Blackboard for $1.64-Billion) reviews the history of the company, including its going public in 2004, acquisition of several major competitors, and expansion into new markets such as mobile, analytics, and emergency notification. The article has quotes from Blackboard and industry spokesmen:

  • Vicki Tambellini, president of the Tambellini Group, a higher-education consulting firm. “For a time, this is not going to give institutions confidence…I think this will create doubt in terms of what it means to the institutions going forward.”
  • Trace Urdan, an analyst at Signal Hill, “There’s no need for them to make a really big, significant change to it … It’s not broken, it’s probably just not a growth stock anymore.”
  • Ray Henderson, president of Blackboard Learn, “played down the possibility that Blackboard might scale back in some areas, although he said the company had not yet discussed those issues in detail with Providence.” In his blog on the acquisition, Henderson notes that “Private equity now provides us an alternative ownership model that’s more agile.”


Steve Kolowich, Inside Higher Education (July 5, 2011)

Kolowich (“Blackboard Gets Bought“) notes that more than half of nonprofit colleges and universities use Blackboard Learn; that learning management systems are now considered “mission critical”; and that “any major shakeup at Blackboard necessarily has a ripple effect across its thousands of higher ed clients.”

  • In a letter to clients CEO Michael Chasen states that “I and our management team will remain in place and we do not anticipate any changes to our day-to-day operations and practices.”
  • Sam Segran, CTO at Texas Tech University: “We have some concerns…Any time somebody goes into private equity, one of the concerns we have is profit motivation and less motivation in terms of meeting educational needs.”
  • Ray Henderson “countered the notion that Blackboard might divest from some of its more recent product offerings, such as Blackboard Mobile and Blackboard Analytics — the latter of which several analysts cited as a potential loss leader. “I think it’s fair to say there’s strong shared belief in the importance of mobile education and a growing institutional demand for analytics.”


Reprising The Joel Test: 12 Steps to Better Code

Joel Spolsky published this list in 2000 and it appeared again in the classic Joel on Software. It’s worth a reprise. If you are involved in software development I encourage you and your team to take The Joel Test.

According to Spolsky (and I agree), “A score of 12 is perfect, 11 is tolerable, but score 10 or lower and you’ve got serious problems.  The truth is that most software organizations are running with a score of 2 or 3, and they need serious help.”


The Joel Test

  1. Do you use source control?
  2. Can you make a build in one step?
  3. Do you make daily builds?
  4. Do you have a bug database?
  5. Do you fix bugs before writing new code?
  6. Do you have an up-to-date schedule?
  7. Do you have a spec?
  8. Do programmers have quiet working conditions?
  9. Do you use the best tools money can buy?
  10. Do you have testers?
  11. Do new candidates write code during their interview?
  12. Do you do hallway usability testing?

Learn to Fail Elegantly

In this week’s video Joichi Ito, CEO of Creative Commons and serial entrepreneur, reminds us that “learning to fail” is at the heart of innovation.

Building a culture of innovation means building an organizational culture which not only tolerates failure but encourages it. The art of innovation is how to learn from failure, which is just another name for play and experimentation.

Facebook Rolls out Full Session Encryption

Programs such as Firesheep have made it clear that Wi-Fi networks are highly insecure. It doesn’t take a sophisticated hacker to compromise your accounts over Wi-Fi networks.

An important step is for sites to enforce HTTPS (encryption) for the entire session instead of just for passwords and credit card information. Facebook is to be commended for offering full HTTPS support. Although compatibility with third-party applications is still being ironed out, I highly recommend that you enable the “Browse Facebook on a secure connection (https)” option.

Here’s how. Go to the “Account Security” section of the Account Settings page. Make sure you click the checkbox under “Secure Browsing”.  It should look like this:

Gates Foundation’s New Program

Inside Higher Ed

Gates Foundation’s New Program
October 11, 2010

The Bill and Melinda Gates Foundation, which has become a major force in pushing for improvements in college completion rates, is now linking that goal to technological innovation.

The foundation today announced a new grant program that will disburse about $20 million over two years, says Josh Jarrett, its senior program officer for postsecondary success. It will focus on projects aiming to scale new technologies to many students — particularly low-income students who are at the highest risk of failing or dropping out.

(Full Story)

New LMS from Instructure Goes Open Source

Campus Technology

New LMS from Instructure Goes Open Source
February 1, 2011

By Dian Schaffhauser

A Utah-based company has fired a warning shot across the bow of learning management system (LMS) companies, including market leader Blackboard, with the announcement that it’s turning its new LMS into open source.Instructure has publicly released the source code to its Canvas learning management system, which was launched in 2010. Currently, 26 institutions have signed contracts with Instructure, including 17 within the Utah Education Network. The company said that more than 100 other schools are currently evaluating its application.

(Read More)

IT Portfolio Management in 10 Slides

IT Savvy firms use portfolio management, in concert with governance and technology strategy, to maximize return on investment and to enable innovation.

Based on a framework developed at MIT’s Center for Information Systems Research (CISR), I have prepared an overview of “IT Portfolio Management in 10 Slides”:

Practical Portfolio Management

I am presenting a session on IT Portfolio Management at the 29th Annual Minnesota Government IT Symposium. The approach will be to view IT assets holistically using financial portfolio management techniques.

Here is the synopsis:

Effective prioritization of IT projects is key to optimizing IT investments: it can mean the difference between improved business performance or wasted resources. Using the IT portfolio framework developed at MIT CISR (Center for Information Systems Research), the presentation will present practical techniques and tools for jump starting or revitalizing your IT portfolio management initiative.

By attending this session, you will:

  • Learn practical techniques for classifying and tracking IT investment classes
  • Acquire a common vocabulary and framework for prioritization based on financial portfolio management
  • Gain familiarity with cost-effective tools and templates that won’t bust your budget