MOOC-y Lessons

What I Learned from Harvard…

cs50xIn order to expand my horizons in computers and to help me live up to the incorrect assumption that I’m some sort of programmer, and also to learn more about MOOCs and online learning, (and thus, to steal ideas and teaching tips) I enrolled in Harvard’s CS50X massively open online course, along with over 150,000 of my closest friends.  My plan was to keep this page updated with my latest thoughts and lessons on what this course teaches me about online learning.  6,852 words later (sorry about that), this is the wisdom that I have gleaned.  Enjoy…

What is CS50X?  It’s a popular introductory computer programming course at Harvard;  there are over 600 on-campus students enrolled this semester.  The on-campus lectures and small-group sessions are recorded and posted online for the MOOC students, who will use these to perform the exact same assignments through the same software, and the same learning platform as the “real” students.   While I get to learn programming for free and get a cool certificate in April, providing I have completed all 8 problem sets and quizzes by then, the on-campus suckers actually pay to learn the same information only getting a few credits toward a lousy Harvard degree in return.     *obligatory wink*

Why did I pick this MOOC?  For one thing, it was available.  I’ve been signed up for multiple Coursera  courses, but the earliest one I’m interested in starts in June.  The others are TBA.  They have a lot of courses, just not all at once, so it’s not very convenient for the academic impulse-buyer.  I looked at courses from the Canvas LMS, which lets instructors create free MOOCs, probably as an effort to get more people to use their LMS and help their company expand.  None grabbed my interest as the jumping off point for my MOOC research, although I’ll probably pick up at least one of those later.  I found CS50X in the MIT-developed edX MOOC site, which is 90% computer science content.  It seemed interesting, so I signed up.  Now here’s the rest of the story…

December 5 — Week 0 Take-aways:

  • I like having a Week 0.  It was challenging, but made great efforts to take baby steps in helping students prepare their heads for an online course.  Students were shown how to get around the learning platform, although this was only done after getting into the right place of the learning platform.  So make sure your students can access the material that tells them how to access material.
  • Proposed timeline — a MUST for a course that students complete at their own paces.  I need to complete the problem sets by mid-April.  That’s far from now, so I can put this week’s lesson off, right???  Oh, the timeline says “no.”
  • I was surprisingly excited about seeing the introduction videos from MOOC students around the world.  I’m not a bubbly socialite, but actually seeing and hearing the people with whom I would be learning made me feel more comfortable, especially since I started the course about a month late.  I liked seeing the other late-comers, so I knew that I wouldn’t be playing catch-up alone.
  • CS50 is a big deal to Harvard students.  It’s a mostly unnecessary, yet very challenging elective that they wear as a badge of honor.  That’s not an exaggeration.  They have t-shirts.  The first lecture featured this musical number performed on stage by former students.  The course you teach should be this cool and should unify your students this much.  The far-reaching influence of this low-level computer science course is pretty amazing.
  • The instructor tells his journey here.  He didn’t program until taking CS50 on a whim.  All of the other completely inexperienced students now feel like he’s one of them, and they have a chance too.
  • What about the students who have coded since middle school?  The course has “hacker” options, so everyone has something challenging to learn from the course.
  • Lecture videos are at least 70 minutes long.  It was tough to find chunks of time large enough to make it worth the effort to get in study mode.  After over a week, I still hadn’t found enough time to get around to watching the first full lecture.  Let alone the 70-minute follow-up.  Let alone the 45-minute recitation session.  But breaking this info into the 5-10 minute chunks that are recommended for online videos would be ridiculous.  Then I would have 37 videos to watch in the week instead of 3.  Way more daunting.  I would also be more likely to take lots of procrastination breaks between clips.
  • A “Hey, you haven’t completed anything yet.  Is there something you need help with?” email in the first couple of weeks would go a long way toward retaining student enrollment.  Instructors seem to rely on the discussion boards and personal initiative to take care of this.  There are tons of posts about procedural class issues, like where to go for specific information, and the questions are answered quite thoroughly by peers.
  • The lectures have a few pop-up quiz questions, often, slightly harder than the example just explained.  These aren’t graded; they’re just checks for students, but they should tell why ‘B’ is wrong and ‘C’ is right.  Otherwise, they’re not that effective for helping students learn.
  • The first program was done with MIT’s Scratch program.  It’s free to download and play around with, and is easy to learn.  We used it to learn overall concepts and terminology.  Good first activity, low entry-level, yet a challenge to each student, based on how ambitious he/she wanted to be.  (My project lives here.  Watch out for the monsters.)
  • I followed along with the other students, who posted their assignment by sharing it on the discussion board.  It didn’t show up as being turned in though.  ???   I had to search awhile to find the submission requirements hidden in a tiny hyperlink below a video.  The video was large enough that I needed to scroll down to find the hyperlink, but I hadn’t done this on the first viewing.  To use a metaphor that will not be understood by students in 5 years:  Don’t put the important information below the fold.
  • I also found the actual assignment requirements in that hidden hyperlink.  The requirements had been explained, I thought, in the video, but I found that there were many details left out.  Instructors need to be aware that students will assume that video information about an assignment will be exactly the same as text information, and will likely default to just watching the video if they think that viewing both will be redundant.  3 different strategies to design your way around this problem:  1) Make sure that any video contains all of the important details from the text;  2) Make sure that the videos are different enough from the text that students realize that these are two different sets of information, and that viewing both won’t waste time with unnecessary repetition; or 3) Tell students to stop whining and view both, just to make sure they’re not missing anything.  Whatever strategy you choose, one good bet is to post the assignment specifications along with the submission requirements, so that students can check them more easily.

December 13 — A numbers update:  In mid-September, enrollment was around 53,000. In mid-October when the course was launched, over 100,000 students were enrolled (no specific number found.) I joined at the end of November, and there are still new students posting their “Hello” introductions to the discussion boards daily.

According to an Admin’s discussion board post today, 5,666 students have submitted their first completed problem set, which is recommended to be turned in by December 23, just to keep a good pace. I’m really curious to see what those numbers will look like in April, as retention rate throughout a course is a big concern when looking at the success of MOOCs.

harvard logoDecember 20 — Week 1 Take-aways:

First, a point to be clarified:  The MOOC runs from October to April.  I’m not sure why it’s split into 8 “weeks,” but I’ll just go with it.  I did find out that having a Week 0, which I thought was a great idea for getting the feet wet, is actually just how programmers talk.  They count 0-7, rather than 1-8, for this 8-“week” course.  I could see potential problems when a student in “Week” 7 puts off a project, thinking he/she has more time to do it, but by that time, if you don’t understand how programmers communicate, you probably haven’t been paying enough attention to the course.  Now may the tips commence…

  • Make sure your students know the depth of understanding that is necessary to prepare themselves for the lesson.  They shouldn’t have to ask, “Do we need to know this?” because you’ve already outlined the learning outcomes and objectives.  As I started watching the first lecture video of the week, I felt like I didn’t know what I needed to learn.  I was paying attention to the information, and I understood the explanations of coding, but I felt like I was being given a bunch of stuff to hold, which may or may not be useful later.  “Should I be taking detailed notes over this, or will just watching the video and using the online transcript be enough to accomplish this week’s task?”
  • Help your students by always making the lesson’s problem – the reason for doing the lesson – the first item they see.  As I worked through the week’s problem set, I realized that there were many tiny things that I hadn’t paid enough attention to, because I didn’t realize their relevance.  It was VERY difficult to track down these pieces of information in order to make my program work.  This is probably a dropping off point for many students enrolled in a MOOC, who realize that they may have gotten in over their heads.  The setup of each week’s materials is:  Lecture 1, Lecture 2, Walk-Through (recitation), Problem Set, Shorts.  Being linear-minded, I worked in that order, and that was my problem.  I didn’t know what I was solving until I was on step 4 for the week, which is bad course design.
  • If you post class lecture videos for online students, edit out the unnecessary parts. A contributing factor for my difficulty was that the video included a lot of extraneous information that added to the appreciation of the topic (a longer-than-necessary video about a NASA rocket exploding because of some bad code), or that didn’t apply to me as an online student (announcements about on-campus activities).  My “Does it matter?” filter had to be switched on and off so many times that it ended up being stuck somewhere in the “Maybe” zone.  Online material is, perhaps unreasonably, expected to be streamlined and focused.  Once the students start tuning out part of a video, this becomes more and more likely for the rest of the information.
  • Make sure online students don’t feel like second class students.    Especially from the announcements about things like the lunch-time pizza programming sessions on campus, I felt a bit left out and forgotten.  I realized that throughout the 6 hours of lecture and recitation videos from Week 0 and Week 1, the 100,000+ online students had never been addressed directly.  Although it is completely accurate to say that we are piggy-backing onto this “real” course and are in no way entitled to any special favors from Harvard for gracing them with our unpaid enrollment, that’s not the case with most online courses.  So if you teach an online course that uses lecture video from a face-to-face class, make sure you are aware of your online viewers while you lecture, and let them know that you are making considerations for them and their learning.
  • Don’t skimp on the second helpings.  The “Walk-Through” recitation lecture seemed disjointed from the main lecture.  Or, rather, it’s like the recitation wasn’t aware of what information was covered in the lecture, so it was being covered again, rather identically.  It took me awhile to decide that this was good.  I was getting a double dose of the most important aspects, so it was probably better for retention in the long-run.  What I remembered from the lecture was reinforced.  What I didn’t remember from the lecture stood out as a gaping hole, which I needed to make sure I fixed this time around.  I don’t think this would be as effective if the same person gave the lecture and the recitation, though, because then the experience is likely to be 100% identical and would seem too redundant to students.  The second dose needs to be a slightly different flavor. 
  • Short videos for “just-in-time learning” – Another way that CS50 achieves this extra helping (that may be a pun), is to have “Shorts” videos, which are quicker explanations of specific topics.  They are basically the same as what was said in lecture, but by a different person.  They are probably accessed by students who realize that they just need help on this one idea and don’t want to find it in the big lecture video.  If, quite understandably, it is too much work to create separate videos for each part of a lecture, at least provide an outline of the lecture video with time-codes noted, so that students can more easily find what they need.
  • Giving students enough information to get started, then augmenting it as they need more help, makes the lecture video a much more active learning process.  The repetition of the material in recitation let me work slightly ahead of the walk-through lecture.  I already knew how to solve many of the programming issues posed, but when I got stuck, along came the recitation to give me a nudge, and then I was off and running again.  Students who want a challenge get to work ahead with a safety net.  Students who aren’t there yet can take smaller steps with the video.
  • Bad examples are good.  When writing programming code, the visual setup is important.  Instructors showed several examples of a “bad style,” which made the perfectly functional code difficult to read and understand.  Don’t assume that students will pick up on subtleties; show them the difference between good and bad, and help them prevent common mistakes.
  • Man your boards.  The big selling point of a MOOC is that students can learn from each other, which means that students are teaching each other, which leads to better learning.  With CS50, students are encouraged to post “pseudocode” to help their peers understand the coding logic, but to discourage any temptation of copying someone’s code outright.  As a result, if students have specific enough question, we need to post directly to an instructor.  I posted a question that I felt would only need a quick look, and a, “Oh, this is what’s wrong,” reply, but never got a response.  My quick question turned into hours of searching online and back through the video lectures.  This was pretty disheartening.  If you have 100,000+ students, you’d better have a way to support them all.
  • Ask for feedback.  Along with the submission form at the end of the week’s lesson, was a survey to get feedback about the course.  It asked for information about the types of students in the course, interest level, time commitment, enjoyment…  It also asked specifics about how the lectures and walk-through videos were conducted, about whether the discussion boards were helpful, etc.  Especially when trying something new, like Harvard is doing with this brand new MOOC, student feedback is important.  All instructors should work a few surveys into their semester plans.  Equally important, is showing the students how you use it.  Instructors should specifically address particular survey items with their students, identifying things that will be changed and explaining why some things are the way they are and will not be changed.  Knowing that they are heard, even when not agreed with, helps students feel like an integral part of the course.

code January 3– Week 2 Take-aways:

I sometimes feel like a big game hunter when beginning a new class.  I am hesitant to march in without knowing the lay of the land, so I tend to be a bit tentative and observant until I figure out the right strategy.  When I’m comfortable that I know how to be successful, I do the right prep work and take my shot.  I felt like that’s where I landed this week.  I finally figured out the right strategy to work through the CS50x course.  I found that I needed to view the course’s materials out of their given order, which involved trudging through information that I didn’t understand, just to familiarize myself with what I would be doing after I knew what I was doing.  It was like reading an uncompleted Mad Lib.  “You will use _______(name of a function) to complete a ________(Greek word), using a ________(mathematical term).”  Once I knew the blanks, I could look for what would be filling them, and I was amazed at how many tiny hints of code were sprinkled through the lectures.  Once I saw the big picture, I understood that they were going to be immensely helpful, and that made this week a cake walk compared to last week.

  • Help your students strategize.  I covered this a bit in the previous week’s discussion, but I think it’s worth a second mention.  Don’t leave students to fend for themselves.  Stating clear learning outcomes and objectives, providing weekly work load timelines and recommendations, and explicitly describing to students your course’s well-planned sequences of introduction, exploration, concept-development, practice, and execution will help your students know how to succeed.
  • Do your students know how to take notes?  Especially if you provide your PowerPoint slides or your lecture notes online, do they know how to use them correctly?  In the past weeks I had relied mainly on CS50’s provided notes, which just didn’t work for me when I was stuck on a problem, because I hadn’t engaged with them during the lecture.  This week I made a conscious effort to take better notes that would be helpful in completing the week’s challenges.  I added in my own interpretations, wrote down questions that I needed to ponder, and recorded coding ideas for solving the problem set.  If you’ve never shown students what good note-taking looks like, you should.  While your job might be to teach students the history of the bond market, not to teach them study skills, it’s worth the 10 minutes to cover this in class.  Ask a successful student to make a copy of his/her notes to show next semester’s students how they should not only record your words, but should be adding their own interpretations, questions, ideas, and extensions of the content.
  • Test their bug-finding ability.  This week’s lecture started with an activity in which faulty code was displayed, and students had to pinpoint the problems and fix.  It’s a great way to review the previous lesson to prep for the next lesson, and it is a great use of Higher Order Thinking Skills.  To complete the debugging process, students have to understand the purpose of their program, they have to understand any initial parameters that need to be set up, they have to understand the correct sequence of steps, and they have to understand how to express it in the proper language.  Whether it is analyzing the potential problems of an experiment, critiquing a persuasive essay, or spotting the problem in an accounting statement, this type of high-level check for understanding can and should be used in any field of study.

diagram of higher order thinking skills

  • Consider back-channel learning.  While I wasn’t able to participate in this due to my being an asynchronous student, I did think it was interesting that during the lectures this week, in-class students were given the option of using a live discussion board to post questions to grad teachers who were manning the board and could quickly address the issues without derailing the main lecture.  If you have the man-power, this might be worth a try.  Many students don’t want to monopolize the instructor with what they see as small questions, and this seems to be the perfect way to clear up their confusion and allow them to keep on learning.  As a lecturer, I would fear that this would lead to a distracting side-conversation, or that students would feel dissuaded from regular Q&A during a lecture.  Having a GTA-manned discussion rather than a student-regulated board would take care of one problem, and actively engaging the students with Q&A, rather than relying on them to pipe up when they had a question, would take care of the second.  If your course doesn’t have a built-in discussion board platform, Twitter is an easy way to conduct a back channel conversation, using your course’s name as a hashtag.  I just discovered Google Moderator, which seems to be a better tool for classroom use, because it allows students to submit questions and vote each other’s posts up or down the list, so an instructor can answer the most pressing questions first.  A few tutorial videos emphasizing different uses and aspects of Google Moderator can be viewed here:
  • Keep asking for feedback.  I was really disappointed that this week didn’t conclude like last week, with a student survey about the course.  After a good deal of frustration over the previous week’s challenges, relating as much to course design as content, I felt like the course was losing an opportunity to improve.  They still asked questions about the work load and how many hours were spent on programming, but it seems that they should also be tracking overall feelings about the course and how students experience the platform, the videos, the discussion boards, or anything else.  If this MOOC is a testbed for future courses, there is a very real need for continually tweaking its setup.  Also, students appreciate the chance to praise, to help, or to vent, whichever the case may be, and asking for their feedback is one more way to connect online students to the course and to the real human behind their instruction.


discussJanuary 28 — Week 3 Takeaways

This latest installment of my quest to find online best practice from inside Harvard’s CS50x MOOC took a slightly different direction than previous weeks.  Now that I’m in the groove of the course, I have been able to reflect less on working with the content and more on working with my peers in the class.  And I’ve realized that I’m not a very strong asset to them.

To be fair, I’m in what the course calls “the less comfortable section,” when it comes to programming knowledge, so I just don’t have much to provide in the way of advice when someone is asking for help finding a bug in their code.  I don’t know whether too feel bad about this, but I take answers from the board; I don’t give them.  This self-actualizing assessment of my discussion board parasitism led to a few questions that are important for designers of online courses to consider.

  •  What proportion of students needs to know what they’re doing in order to ensure that the group’s progress?  Too few helpful students, and the questions of the confused majority will not be answered quickly enough, and the faculty are too outnumbered by the 100,000 students to keep up.  With CS50, it is acknowledged that there are students in the “hacker section” that know what they are doing.  I don’t know if all courses could assume that such a knowledgeable group would exist to act as peer teachers.  Course designers would need to assess this, and possibly include more resources to help struggling students, if the peer teaching strategy isn’t keeping up with the demand.
  • How do you ensure that the knowledgeable peers will make the effort to help others?  In most MOOCs there is no compulsory discussion board requirement, and they rely on the good nature of the students to inspire the interaction.  How is that enough to keep the class going?  This problem is solved by what is advertised as one of the greatest advantages (and challenges) of a MOOC:  its “massive” enrollment.  In a class of 100, let’s say that a 5% rate of go-getters is leading the charge.  With a MOOC, the same rate equals 4,995 more potential teachers.  If even a fraction of them are plugged into the discussion boards on a regular basis, the course will have a great infusion of student-led assistance.  So maybe that first question of mine will never be a problem for a MOOC.  The strong sense of unity and mutual endeavor that CS50 promoted from day 1 (again, they have proudly-worn t-shirts) is probably the best way to tap into this potential.
  • How should Q&A discussion boards be set up in a course?  Where should they live?  I’m not talking about the usual “Respond to the chapter you just read,” discussion boards.  It’s the, “Hey, here’s something that I ran into.  Can anyone help me?” questions that don’t seem to have a good home in most courses.  If these exist in a course, they tend to be labeled something like, “General Q&A,” and students just don’t use them.  Questions sit, unanswered.  Busy students are focused on the current week’s board (for a grade), not the one that has been mostly empty since the first week of class.  With this in mind, instructors should provide a fresh thread with each week or unit, which invites students to ask their individual questions.  It stays current for both askers and answerers, and all of the students who are neither.
  •  Should the Q&A discussions be graded?  While it’s probably not appropriate to require this type of question that is often tangential to the lesson or about one student’s specific situation, these discussions often involve much deeper thinking than an assignment requires, and they can often break new ground for the students, as well as the instructor.  Because of this, it is a good idea to reward this interaction.  It adds a great deal to what Bill Pelz, professor at Herkimer County Community College, Sloan Consortium award winner, and someone with whom any online education specialist should become familiar, calls “Social Presence” and “Teaching Presence.”

He can explain it better than I can:

Developing the sense of collaboration in learning is something that this “extra” discussion thread can provide, and students should be rewarded for contributing to it.

 A couple of other take-aways along different lines:

  • Show students in your classroom videos –  I didn’t realize that it bothered me until it went away, but the previous weeks’ videos of small group meetings only showed the instructor.  Students asked questions off-camera, and he talked with them and answered the questions.  Pretty simple.  But this week, the camera moved around the room to actually show the other students in these interactions, and it made a big difference in how I perceived the group session.  It felt more like a real class, and I felt much more like a “real” student.
  • “Review, Review, Review” –  as stated by my Teaching Methods instructor in college.  I think that in online classes there is a tendency to complete modules and move on without looking back.  The group session this week looked at student code from previous problem sets, and students compared different strategies, and pinpointed better and worse practices.  Don’t forget to review content multiple times in later modules.
  • Show students the deep end of the pool – We often teach isolated skills with the mantra, “You’ll need to know how to use this later when you do __(blank)_.”  Students only believe that for so long.  When will “later” happen?  This week’s assignment showed us what “later” looks like by having the problem set come from a giant, scary program for a simple word scramble game.  We had three tasks to complete in order to make the game work correctly.  We didn’t need to completely understand the entire code of the program, but it was explained to us at a level appropriate for our current level of understanding and our current need.  It gave a much better idea of the scope of what we are working toward, in a non-threatening way, since we only needed to worry about a small sample of the big problem.

bubble sheetMarch 7 — Week 4&5 Takeaways:

As the MOOC completes what was the first third of the on-campus course, we get to experience our first test.  While there is a slight reprieve from problem sets, extending this round into another week, the focus of the lecture sessions and the small-group sections turned toward test prep.  So what does a MOOC test look like, at least for CS50?

  • List the Do’s and Don’ts – The test started with very explicit descriptions of what is and is not acceptable while taking the test.  No wiggle room was left for students to justify additional resources not condoned by the CS50 staff.  Sometimes instructors forget about the flip side, failing to tell students what is condoned, and only the more daring students might use a resource that greatly benefits them, while unsure students struggle without it and are hurt by their over-caution regarding the sanctity of The Test.  The CS50 test instructions did a good job of explaining both sides.  OK: anything from the CS50 course, other than the discussion board.  Not OK:  anything from anywhere else, or any human interaction.
  • Academic rigor? – The Test is likely the limiting factor when determining whether a MOOC has the academic rigor to match that of a “real” course.  In this case, there was a lot of factual information that, if I had to, I could have learned (or memorized).  But I didn’t have to, so I didn’t.  I knew that with the open-note test, I could easily search the posted lecture notes on the CS50 site and search my own notes files (taken with Evernote).  The on-campus students, though, had to actually stuff this information into their brains for rote recall later.  So the searchability of an open-note test somewhat cheapens the MOOC experience, compared to the “real” course.  But most of the questions were based on logic and the experiences gained through the first 4 problem sets, which is where the MOOC regains some of its lost rigor.  In the age of Google it is less important whether you can remember a set of facts, since it’s not that tough to look them up.  Assessments in MOOCS and other online courses should make a significant shift toward the higher-order thinking problems that prove mastery of a subject far beyond knowing the definition of a term.  The CS50 test was tough.
  • Location, Location, Location – It seems like an obvious tip, and one that I’ve stated before, but students need to be able to find things.  Especially in an asynchronous course in which students are left to make their own way through material.  If I were designing a course in which students took a quiz between Week 4 and Week 5, that is where I would put the link for the test.  I would not locate the link on the opening page of the course, next to the syllabus, which hasn’t been viewed in 4 weeks.
  • Re-design on the fly — Keeping with the previous situation, if you are monitoring discussion board posts and see that many students are having trouble finding a quiz link, after posting the hyperlink in their discussion threads, go ahead and scatter it a few more places in the course.  If the previous students had trouble finding it, the remaining students will have trouble finding it.  You probably didn’t design your course perfectly, so look at each wave of confusion as the opportunity to make it better for the future, now.
  • Estimated timing – It’s helpful to know how long the test is before we start it.  Help us plan by giving us a time estimate or at least the number of questions.  Online students may be working around tight schedules or on a time-limited library computer somewhere.
  • Encouraging retakes – The CS50 test can be retaken as many times as a student wants.  Even if a student passes, retaking a test is another intense exposure to the most difficult material.  To encourage students to try a retake, our answers were emailed to us after their test was submitted.  Not the questions, just the answers.  Now if we want to retest, we wouldn’t be shooting in the dark as much, and would feel like we are being given an extra tool for success.  Instructors should consider this as a good strategy to increase long-term learning in students who try to master information, not just leave it all on their bubble sheets.
  • Don’t forget the review – Especially in something like computer programming, where a lack of understanding now can lead to major problems down the road, it is important to remember that students need feedback on their feedback.  An overall grade and a check mark on a particular question don’t really do much to help a student learn.  The CS50 instructors took some time to review the most contentious questions in the small-group section following the test.  While the MOOC students aren’t getting one-on-one feedback from instructors, seeing them work with the small-group students is an extremely effective way for us to review our own understanding of the test material.  What if you don’t have an on-campus class to drive your post-test analysis?  How do you know what to discuss with the faceless masses and their innumerable differences of mastery?  If your test questions and decoy answers are crafted in an intentional manner, the incorrect answer that a student chooses should tell you something about their misunderstandings.  Your video review can address students in a way that explains what their assumptions probably were, and help anyone who answered ‘B’ know what they need to fix, which is different from anyone who answered ‘C,’ because they were making a different assumption to reach that answer.

denied certificate of completion May 20 — Final MOOC Experience Wrap-up

Through the final weeks of my Harvard MOOC experience, I learned a few things about setting rigorous expectation in a course, the value of competency-based grading (or “badges,” to fit the trending lingo), and how to plan for a final project that is valuable to students.  I also procrastinated enough on this last report that the nice folks at CS50 posted some very good data on the success of their course, which you can view here.

  • Motivation, and the difference between 60% on everything and 60% on each thing:  Around the fifth of the eight problem sets in the course, I had to make a choice that is at the heart of any challenging MOOC.  How much work am I willing to put in for 1) learning stuff, and 2) a shiny certificate, from this unrequired MOOC?  This point comes as the Difficulty and time requirement in the course surpasses the Perceived Benefit that the knowledge gained will provide to my life (as shown in the very professional graph below).  In short, if I don’t think the depth of the course will be useful, should I do it anyway?  If I was paying for college credit, then yes.  But for a free MOOC???

cs50 graph

    •  I calculated that if I skipped a couple of problem sets but continued with the course and the final project, I would be able to pass the course with well over the required 60% and get both the knowledge that I felt I needed, and the shiny certificate.  But upon a 2nd, 3rd, and 4th reading of the syllabus, I realized that I may have misinterpreted the course requirements, “Score at least 60% on this, this, and this…”).  That means on the whole bundle, right?  No.  I had to score 60% on each problem set.  That just wasn’t going to happen with the amount of effort that I was willing and able to put into the course.  From the first chart on the CS50 data page, it appears that this was the pivotal point for many of the MOOC participants, because following this area, there was a much lower rate of dropping out.  While the goal of the MOOC is to teach students, it doesn’t seem like the reward was motivating enough for roughly 90% of the 10,000 students who stated that their goal was to complete all course requirements.
  •  Shiny certificates do work though:  The ego-boosting idea of a Harvard certificate hanging on my office wall kept coming to mind as I decided whether to trudge on or to bail.  I’m sure it made a difference to some of the other students facing the same decision, and to many of us who are looking forward to the course’s next incarnation in the fall.
  •  The point of badges:  After I realized that I wasn’t going to be successful in this course, I did continue to work on the last problem set and the final project.  To me, these last assignments were back in Positive Benefit Land, so I would be wasting opportunity not to learn from them.  I am curious how many “dropouts” went back to the course to work on these.  Thinking again about motivation and the injustice of a linguistic mistake pulling the rug out from my plan of skating by with a minimum grade, I gathered some thoughts on how a badge grading system could affect courses like this.
    • If I was able to score 60% and get a certificate, mine would be as meaningful as a student who aced every piece of the course.  That is hardly fair or representative of what each of us had learned.  Badges signifying which programming skills each student has mastered would be tangible and understandable information for future employers to see, lending credibility to the learning outcomes of MOOCs and any other course.
    • If I knew half-way through that I wasn’t going to be successful with the entire course, why continue?  With a badge system, students can fail on one part but understand that there is still something to work toward, as they complete the remaining sections of the course.  CS50’s graph showing student completion of each problem set would look less like a continual drop-off, and could show a surge of activity on the later parts of the course that students wanted to continue.  Many more final projects would have been completed, showing what the other 9,000 active students got out of the course.
    • The goal could be to collect a percentage of the total programming skills, rather than scoring a 60% in each program assignment.  Even within one programming challenge, students might master four skills but fail one, and the program doesn’t run well enough to score the necessary 60%.  A skill-based badge system would reward a student who is able to obtain 60% of the entire course’s expected learning outcomes, even if part of the course was over his head.  The a la carte course would serve students better, according to their own needs and interests.  CS50 does this on the top end, by providing the “hacker edition” options on assignments.  Why not extend this to the rest of the course? (says the guy who doesn’t have to design a computer grading system capable of determining which skills within a program are mastered and which are not)
  • A valuable final project:  You can’t expect the 150,000 students who signed up for the course, or even the 1,388 who successfully completed the entire course, to care about the same computer programming applications and uses, so the culmination of a course like this should be something chosen by individuals.  CS50’s requirement for the final project is that you use skills from the course to make something useful.  That’s pretty much it, other than making a short video to show it off.  I put more effort into personal projects than in cookie-cutter assignments with questionable utility to me.  Self-determined projects do a much better job of letting an instructor see just how much each student has learned and is able to do on their own, which is kind of the point of any class.
  • Final thoughts on the data:   MOOC skeptics will look at the 0.9% rate of success among the 150,000 students who signed up as absurdly low, and proof that MOOCs just aren’t going to hack it when the novelty wears off.  But the data analysis page points out that 50,000 of the sign-ups didn’t even log in once.  Should these window shoppers be counted in the success/failure rate?  Overall, only around 11,000 students even intended to do all of the required work.  The vast majority of participants were just trying to learn something new or kill some free time on Friday nights, and weren’t interested in the certificate of completion.  In the end, 1,388 students completed a rigorous course in a potentially life-changing, career-starting set of skills, and without the MOOC they wouldn’t have the opportunity or motivation to do so.  I am curious about how many of the successful students were among the nearly 2,000 who reported taking the course “as a refresher,” compared to how many were newbies to programming.  That is some data that would really show that a MOOC like this is an effective tool for teaching a difficult subject like programming to complete beginners.

One thought on “MOOC-y Lessons

  1. Pingback: What I’m learning from Harvard… « Instructional Design @ ESU

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s