Ask HN: What makes an online course amazing?

[dewy]

For those of you who have taken online classes (MOOCs, at college, etc.), what have been the things that made the best classes stand out? (Or conversely, what made the bad ones bad?)

Why I'm asking: I teach a college data science 101 class with at least 1 section online every semester (and will now be switching to all online for at least the next month - thanks, coronavirus...). At the same time, I'm in the process of redesigning the course, and making new teaching materials and assignments.

I've taken completed several amazing MOOCs in the past (e.g. Princeton's Algorithms I & II, Stanford's Machine Learning), as well as abandoned many more that were simply terrible, so I am no stranger to the online student experience, and I know the things that made certain courses amazing for me personally (e.g. great videos, and well integrated assignments with automated feedback). However, every learner is different, and I would love to hear different perspectives so that I can better redesign my own course.

Was it something about the videos? The assignments? The structure? The instructor? Or something else? Thanks!

[ivansavz]

For me the biggest game-changer for online video lectures has been this Chrome plugin that allows for fine-grained control of video speed: https://chrome.google.com/webstore/detail/video-speed-contro...

If you have control over the courseware platform used for your course, make sure it uses a compatible video player based on html5 video and not some custom implementation (very rare).

In case video speed controls is not available on your courseware, you can pre-process videos to speed them up to 1.5x using this script (save as `fastervid.sh` and run on video lectures before uploading)

    #!/bin/bash
    if [ -z $1 ]; then
      echo "usage $0 input_video.mp4"
      exit -1
    fi

    echo "Converting $1 to 1.5x speed..."
    ffmpeg -i "$1"  -filter_complex "[0:v]setpts=0.6666666666666*PTS[v];[0:a]atempo=1.5[a]" -map "[v]" -map "[a]" "tmp-$1"

    echo "Delaying audio of $1 by 60ms"
    ffmpeg -i "tmp-$1" -itsoffset 0.06 -i "tmp-$1" -map "0:0" -map "1:1" -acodec copy -vcodec copy  "faster-$1"

    # cleanup temp file
    rm "tmp-$1"

[ivansavz]

Completely unrelated to the video speedup hacks, but also in the vein of getting a lot of information across faster, one thing you can do for your courses would be to create Concept Maps.

The CmapTools software is very good for this: https://cmap.ihmc.us/cmaptools/

Overall I find concept maps on of the best tools to show an overview of the course (macro maps) or topic (zoomed in maps). As a teacher with extra context about data science you can easily put together something like this for your course that will show learners what there is to learn and allow them to keep track of their progress (e.g. wow when we started all these words were unknown to me, but now I know about 70% of the words on this map already).

I have heard teachers use CmapTools in more active way (asking learners produce concept maps individually or collaboratively), but don't have experience with that myself.

A paper about concept maps: https://cmap.ihmc.us/docs/pdf/TheoryUnderlyingConceptMaps.pd...

And an example of the concept maps I use in my books: https://minireference.com/static/tutorials/conceptmap.pdf

[boneitis]

More in this vein, it genuinely floors me that UML-like diagrams (used very loosely and without strictness) aren't more of a thing.

There are so many walls of text out there going over a system's flow that could be expressed in just a couple paragraphs and a companion sequence diagram.

[dubya]

I've found that plug-in to work even where sites try to prevent speed control and skipping, for example our various campus mandatory online training courses.

This is also very satisfying:

document.querySelector('video').playbackRate = 100;

but the videos are often served from a different host and I forgot to record the setting that makes it work in that case.

[QuinnWilton]

I also enjoy sped up videos, especially for lectures, but if you preprocess videos to speed them up, make sure to also include the original videos. Students that don't speak your language natively, for example, may be handicapped by fast videos.

[ricklamers]

It might not be the most useful answer given the context of your question. But in my experience the thing that made the best courses stand out were the depth of understanding of the instructor and their basic didactic skills. Are they able to explain the concepts clearly to a novice student? Are they able to structure the course into a logical progression of topics?

To me, what I consider to be secondary aspects such as interactive quizzes or homework assignments never really made a difference. One thing I noticed that really did not work well for me was asking quiz questions in between that did not require understanding just memorization of what was just said 30 seconds ago. It's like you're checking if they're paying attention v.s. are grasping underlying concepts.

I would recommend you check out CS50 by David Malan (if you haven't already) as it's somewhat of a gold standard for MOOCs in my book.

Good luck with your MOOC! The world needs more of them, make sure you make them as publicly accessible as possible.

[madhadron]

> One thing I noticed that really did not work well for me was asking quiz questions in between that did not require understanding just memorization of what was just said 30 seconds ago.

Be careful of saying what works well versus what doesn't work well based on perception. I pretty much always use this kind of inline question in lectures, usually gauged to require the student to make a single deductive step from the material just presented. And they all look at me like I'm an idiot for expecting something that simple...except that then some of them get it wrong and have to reintegrate the knowledge, and the number of queries after the fact from misunderstandings of the material went down dramatically when I started doing it.

So I actually tell the students up front that this seems absurd, but play along, because it really does help the integration of knowledge.

[superasn]

My answer is very general and I only have experience taking a lot of online courses via sites like udemy but the key factor that makes any course stand out is practical real life examples. It makes everything super interesting and you learn a lot more quickly.

Also I've seen that a bit of humor does wonders for learning. Recently did an online course called StatQuest with Josh Starmer(1). I never knew stats / ml could be made this interesting.

(1) https://www.youtube.com/user/joshstarmer

[newsbinator]

Perhaps an unpopular opinion, but I feel humor detracts from a course most of the time. It's like breaking the 4th wall in cinema: you can do it, and some people do it to great effect, but as a rule you probably shouldn't.

Dan Ariely is unusually gifted, so his humor works.

For the vast majority of courses, I prefer when they stick closely to the material, so I can watch it at 1.7x or 2x without breaking stride.

[Dowwie]

Dan Ariely made extensive use of humor in his lectures and it made a great difference

[baron816]

You should keep in mind the key difference between your offering run of the mill MOOCs--you're students have to complete your course if they want to maintain their GPAs and graduate.

This should substantially change how you teach your class. A normal MOOC has to make learning "easy" so that students keep coming back. Unfortunately, when learning is "easy" it doesn't stick very well.

Effective learning needs to be challenging and force students to dig deep and think critically about what you're teaching them. If you give out hints or shortcuts, they'll probably like you more, but they're not going to build the deep understanding they need to succeed in further courses and in their career. If you're good at making them think critically, the whole learning process will feel hard for them, possibly as if they aren't learning anything, and they might resent you for that. So it's good that they're stuck with you. Do the right thing and make sure they're really understanding the material and not just memorizing formulas and patterns.

[ramesh1994]

I've taken Georgia Tech's OMSCS program. What really sets this apart is the active feedback on assignments / course interaction from Professors and TAs in Piazza.

Current online courses have really watered down content and zero/terrible ways of evaluating the material.

Things that I like too see in an online course:

- Keep assignments on a regular basis (on par with your campus schedule). Quizzes are great to keep them engaged but do not use them to replace regular assignments.

- Design them to be auto-graded as much as possible. Bonus points if there was some way the students can evaluate their solution locally.

- Something like Piazza/Forum setup to answer student questions and have an appropriate number of TAs.

- Post all the resources used in the videos to be accessible for later reference (slides and links).

If you're recording content for future offerings, ensure that the content is consistently updated to stay relevant.

[closeparen]

To the extent that students are writing code: a high quality "autograder" test suite. A tight feedback loop on the correctness of your work/understanding is the most powerful learning aid I've ever encountered.

Although it's a step on the right direction, I'm not a fan of Coursera's "upload a zip file and wait 10 minutes" approach: even that much delay means you've lost your mental "working set" by the time feedback comes. My college professors would give those to us as Make targets in the project scaffolds, which made it closer to "TDD."

In a similar vein, thoughtfully designed interactive environments with good difficulty progression, like Micro-corruption [0]. It's easy to imagine something like this for data science (here's an RStudio workbench and a dataset, go find the needle in the haystack, etc).

[0] https://microcorruption.com/login

[optimaton]

All of the following in one website or tool/sw:

1/ Interactive tutorials and quiz format like brilliant.org

2/ A Wikipedia like comprehensive wiki embeded with course website for all kinds of reference.

3/ An online community — could be a piazza/forum.

4/ A github like service integrated with course site specifically for that course where students can submit their projects and be evaluated by other forum members/mentors etc.

[Dowwie]

Dan Ariely's MOOC on Irrational Behavior (coursera) remains my favorite online course and one of the best overall.

The content was engaging. It included guest talks. The course didn't shy from hard work, including dozens of required and recommended papers. The curated papers were my favorite part. The course exposed me to an entire ecosystem that I was unaware of. Too many MOOCs are watered down but this one was just right.

My best advice is to create a course that management will resent open sourcing because it is so valuable. Demand autonomy and use it create a legacy.

[tokipin]

The biggest issue I've seen in online courses is production values. In fact I think crap production values and the lack of a quality bar is why platforms like Coursera have done poorly compared to their potential.

The most obvious issue is usually microphone quality. It's baffling that people spend so much effort developing a course and multiply that effort by a shit microphone. If you aren't willing to spend/get money for a good microphone, your philosophy on what is important for online teaching may have some blind spots IMO.

[maher_au]

I have developed a collaborative learning framework to support the development of learning experiences. The 4 key phases are

1. Resource curation 2. Guided active retrieval practice (questions, tasks, self assessment and feedback) 3. Collaborative experiences (professional examples, tadks and feedback) 4. Reflection and connection. (Connecting this learning to what has happened before or is likely to happen next

I will get the chapter (With evidence) up on a preprint server shortly -likely the end of the month. If you want the model - dm me on Twitter @maher_au

[aryzach]

Appropriate assignments. For me, that often makes or breaks a MOOC. I want to be stumped in an intellectual way, but not a cryptic way. I don't want to feel like I'm just trying to decode the professors cryptic problem. I want to feel like it's a genuine hurdle to learn the content well, yet still accessible, and definitely not busy work.

Good example: write an interpreter for the given (made up and simple) programming language. We had to figure out how to implement certain language features such as clojures.

Bad example: in networking course, we had to recall details about certain protocols, and do calculations. While this might be useful to know how to do in practice, it didn't contribute anything to my understanding, and it just always felt like tedious busy work, and lacked projects that contributed to the course goals.

I don't know data science, but here's a project I've heard of: Somebody's hobby was disc golf. Discs come with ratings on them (I'm assuming things like curve and distance). They created a controlled experiment to take various data points from many discs, and compare them to their stated ratings, and apply a bunch of data science-y things to the data.

Edit: I'd add some type of reference resources like a wiki, and ideally some auto-grader that gives feedback if possible (like what tests failed)

[aryzach]

I'd also add, why not just find an appropriate data science MOOC to use? I'm sure there are tons out there that have had more time and resources to pour into it. You could at least use one as a jumping point so you don't have to reinvent the wheel.

[jplayer01]

What MOOC did you take where you had to write an interpreter?

[aryzach]

https://www.coursera.org/learn/programming-languages-part-b

This is Part B, but there's also parts A and C. While I recommend the whole course, you don't need to take it all if you just want the interpreter project.

The networking course was discontinued, but these are the videos: https://www.youtube.com/playlist?list=PLEAYkSg4uSQ2dr0XO_Nwa...

The videos aren't terrible, but the homework was worthless.

[jplayer01]

Thank you.

[samvher]

I was going to say that all my favorite courses were kind of unique, and didn't share any particular characteristics, but writing this comment I found they actually all share excellent problem sets and an engaging instructor. Having taken ~25-30 online courses, the ones that stood out were:

- Discrete Optimization (Coursera/Melbourne university, Pascal van Hentenryck): had competitive optimization assignments and course leaderboards. The professor oozed enthousiasm and the nature of the assignments was great for the competitive setup (many different approaches possible, and there were always problems which you would not be able to solve to optimality).

- Microeconomics (EdX/MIT, Jonathan Gruber): very good assignments that made you discover things yourself. Professor presented with a lot of humor and this course was clearly refined over many iterations on campus. Lectures were actually recorded in a classroom setting so not tuned for MOOC delivery at all.

Honorable mentions for Computer Graphics (EdX/UC San Diego, Ravi Ramamoorthi, perfect pacing and balanced, broad set of assignments) and Underactuated Robotics (EdX/MIT, Russ Tedrake, fascinating material and challenging, elegant assignments).

[leggomylibro]

Hands-on, practical examples. If a class only includes video explanations and some quizzes on what was said in the lectures, I usually forget the subject matter quickly. The best MOOCs that I've taken are the ones that have periodic "lab" assignments which ask me to think and apply the subject matter in ways that the lectures may not have explained in precise detail.

Without naming names, here are two examples:

I once took a MOOC about "bare-metal" microcontroller programming which didn't include any microcontroller coding. Useless.

I once took a MOOC about digital circuit design which used an in-browser circuit simulator for problem sets and labs. Extremely helpful.

I feel sort of cheated when I work through an online course, only to find out that it would have worked just as well if it were a series of YouTube videos.

For college-level data structures, maybe you could ask students to implement their favorite DS from a set of tricky ones? You could write automated tests to verify them so grading is easier for you and development is easier for them.

[wespiser_2018]

I've taken almost 10 MOOCs and 5 courses in GT's OMSCS program, spanning from a member of Ng's first ML course, to a currently lackluster offering I'm taking for credit.

To take a step back, I think what makes a course great is just its ability to transfer knowledge to students and to spark curiosity (further learning after course finishes). Usually, when you take a course in person, the folks teaching the class have ample opportunity to ensure that transfer is happening, in online courses, the onus is really on the student.

Thus, in an online course, you need to provide clear learning goals, for instance, "dynamic programming", a set of lectures that teaches those concepts, then homework or project based work that directly refers to what has been said in the lectures. Decide what your resources are, say, a book, your videos, and a few homework assignments, then try to have the resources build off each other, and make sure information is available in multiple places. Sometimes with difficult concepts, you need things explained twice, from two different sources!

The next aspect of a great course is the fostering of a learning community. Make sure the students have a place where they can engage (slack, discord, et cetera) and ask questions. Having students make a social connection has been shown to increase their outcomes in online learning, and for me, the early Coursera courses were amazing for this.

Finally, there is the aspect of of running the course. Some students don't do well learning online, and I some of this can be explained by the loss of human to human contact and relationship building and mentoring that can help struggling students during in person office hours. A percentage of students truly rely on this, and there is no online alternative. If you don't grok them pythonz, data science can be a nightmare (R 4 lyfe!!!).

Someone much smarted than me figured out you need 3 things to learn: time invested, a consistent environment, and feedback. Make sure you have enough TAs to actually give student feedback, or figure out an autograder for problems of varying difficulty.

[xyg]

I would say these aspects are important:

An instructor's understanding of the topic he or she is teaching. It is hard to describe but the vision and the depth are just different. In some occasions, I find that it is my preparation prevents me from fully understanding what the instructors are seeing.

An instructor's passion for the topic, which brings moments for learners to feel "this is fascinating /amazing /cool /interesting. I have never thought about that". It seems that passion cannot be faked or manipulated, which means one cannot be prepared to demonstrate passion when teaching a course.

Teaching and presentation skills, can be trained.

Visualization and other technicals, can always be improved.

[uinerimak]

One of the things I like in courses is that fact I can know who's studying with me and in few cases I actually became friend with one or two people.

Also in online courses it's an important factor that not many e-learning sites fully understand the potential

[sriram_malhar]

Exercises and frequent quizzes. Passive learning is not learning.

[kylebenzle]

Agree with frequent tests. Also live video as much as possible.

[Eridrus]

This might not really translate to you since the context I have taken online courses in is that I have wanted to learn more so that I could be a more effective software engineer. So I'm looking for things that show applications quickly.

Courses I've gotten a lot out of: Andrew Ng's ML course and Yaser Abu-Mostafa's learning from data course.

In both courses I really felt like I was learning something meaningful. I very clearly remember how mind blowing it was to me that you could turn complex high level tasks into curve fitting. I excitedly told a colleague of this thing I had learned about called linear regression who turned out to be surprised I had not taken a statistics class. Same thing for generalization bounds & VC dimension in Yaser's class.

I don't know if this can really be generalized to all courses since it might just be matching students to courses that make sense to them, but the worst line in a MOOC is "Well, this is boring, but you're just going to have to grind through it so that we can solve problems later". Shout out to the Convex Optimization MOOC I dropped.

For Data Science 101 this should be easy. Find some problems that students could imagine wanting to work on and use those problems to drive what you talk about.

Also, a pet peeve of mine is courses trying to make me do math by hand. I am 100% going to ask a math package to do an integral for me every time this happens. Sure, somebody has to know all the math deeply, but it's not me.

So I guess the main takeaway here is that you need to understand your audience. What is appropriate for me is probably not appropriate for 1st semester college kids.

[quickthrower2]

I don’t want amazing. I want efficient! Teach me at a speed not to slow and not too fast. Cover the correct topics.

A course with missable sections would be good so that if you already know ES6 you don’t need to listen through it on every JS course. Cheat sheets might help with this too.

I don’t care much for a super interesting voice as long as it’s a reasonable voice (not monotonous or unclear) and I can learn the thing.

[jerome-jh]

Having taken a number of MOOCs, I would say, for the videos:

- the speaker must be standing

- the tone should be light, not necessarily funny, but definitely not always the same

- there should be illustrations

- anecdotes, personal notes, interviews: maybe not at every video but at least one per chapter.

- videos should last less than 15min.

The talk must not aim to be exhaustive. Students will master the material through provided readings, exercises or quizzes. Some may only watch the videos, others will do the full course.

You will probably not have the time to prepare quality videos before the COVID-19 outbreak, so you may limit yourself to an introductory video for every chapter. You may also provide the solution for a couple of exercises in a video on the model of Kahn academy.

You should watch the making-of at the end of "Mindshift" on Coursera, by Barbara Oakley.

Note that "Introduction to logic" on Coursera has no videos but provides great tools and both funny and serious exercises.

Bad videos may spoil the course by making your students bored before they even start learning. So insist on quality, not quantity.

[ldenoue]

Why does it feel important to you that the speaker be standing?

[toomuchtodo]

Standing uses body language to convey authority and confidence when presenting. I’d theorize this helps the consumer of educational content in being engaged, which is more difficult when watching videos versus being present during an in person lecture.

[ldenoue]

but if the teacher is fully shown, or perhaps the upper body (in Japanese culture, people like to see the upper body during webcam sessions, not just the head), then it would still convey authority and confidence while watching the video?

Authority seems to come in part from the actual height difference, so that could be lost if the student watches a video... But standing still gives other benefits, see https://www.genardmethod.com/blog/bid/161977/Sitting-or-Stan...

[jerome-jh]

In my experience, a standing speaker is usually less boring.

B. Oakley mentioned in my post is filmed full-body and then varies the framing, half-body, head, with digital zoom. That is explained in his making-of.

[jplayer01]

I'm curious about why people aren't naming any courses that exemplify the good or bad things they're mentioning.

[dominotw]

MOOC's are super boring for people who don't like watching videos.

I really hope somone builds a mooc on top of something like leetcode.com . Start with easy problems see how far you get, get a hint, if you can't then watch a video of concept you are missing and so on.

[diehunde]

I used to love watching video but since I started reading more books I just can't watch for more than 5 minutes. Recently I discover educative.io which is like reading articles. It's great if you don't like watching videos and want to learn software engineering stuff.

[elliekelly]

It’s not easy for me to explain (and I would imagine not easy to create in lesson plans) but I think it’s something Colt Steele consistently does well in his Web Dev Bootcamp on Udemy.

Here’s my best effort at articulating his teaching style: He walks students directly into the mistakes they’re going to make and then helps guide them out. Most instructors will teach the concept and leave it to students to avoid making mistakes. I learn a lot a better when I know what mistakes to look out for and what to do when I’ve encountered one. But more importantly, I feel like I’ve really mastered the material when I’m confident that I can get myself out of any mess I can get myself into.

[abdullahkhalids]

Can you give a specific example of this technique?

[ailun]

Check out David Joyner’s work at Georgia Tech:

http://lucylabs.gatech.edu/

One of the keys for me is how I get feedback. Multiple choice quizzes can be well-designed. Most are not. But they scale well.

Personally I like very challenging assignments, but with very lenient grading and partial credit, with lots of detailed feedback from the grader.

If you’re teaching a college course, maybe you have enough TAs to make that happen. You’ll see some of Dr. Joyner’s work is on scaling grading systems and feedback.

[decentralised]

In my formal education days, I was perma-frustrated that none of my teachers had achieved anything impressive with their work (apart from being published), and I often felt like I was being handed out second-hand knowledge to regurgitate during the exams... and I had to get a job to pay for that level of education!

To me what's really good about MOOCs is that I can have access to some of the best content and teachers from MIT, etc all from the comfort of my living room.

[mgozaydin]

Cost of a 3-credit course f2f ar Stanford is $ 5.000. But the real cost of an online course ( same course ) , is less than $ 10 . I tell them sell it at $ 100 . But there is a dilemna . How come the same thing is $ 5000 and $ 100 . Even the one for $ 100 is better . Solve the equation . I say make a consortium of online degree programs . Provide degrees from consortium . Any idea .

[hither2]

2 cents:

I think the internet is the best way to distribute information, but for me one of the worst places to learn.

If the learning is solely done via the computer I will fail to even get near to completing the course (self-study, no pressure). But by using pens, paper, books etc, my ability gets to a normal level.

So, encourage them to behave close to the way they do in your classes.

[j45]

I'm a technologist in Education. This frightens a lot of educators who are pretenders in Academia trying to lead technology development.

Supporting instructors, and instruction directly has been my main area of focus for the past year, in addition to being directly involved in developing online education platforms for the past 20 years in K-12, Academia and Industry.

I'm not sure if you have a tech background, but making sure you have competent technology advisement at all stages is quickly emerging as a key differentiator. If decision makers in academia haven't had technical literacy to make technical decisions, it shouldn't be a surprise if the experiences are not engaging, because they aren't leveraging digital interactions for what they're capable of, and instead only what a limited non-technical academic background can imagine.

Important:

- Not taking the lecture format classroom of the past 100 years and putting it online in video. MOOCs are handicapped before they even begin, quite often. Sure, great instructors who can explain well are good, but they are the exception, and not the norm. Another key problem remains.. god help you if you have to change the video. The key I focus on is how students are experiencing MOOCs as being the main lens. Academic institutions can have a hard time letting the frame go to be student centric instead of instruction-centric. Online education shouldn't be rooted in the practice of distance education (correspondence, textbooks, phone calls)

- The baseline of the student is that they are far more digitally literate and competent than the institutions they are walking to and most often more than their instructors.

- Much of the Academic online education world is designed and anchored in a past of a web browser on a desktop.. by educators who do not have competency in the possibilities or capabilities of tech.

- When it comes to making digitally engaging learning experiences, first you have to consider if academics rarely learn how to teach, let alone digitally.

- I would seriously pay attention to how students are interacting and studying digitally and focus far less on the existing taxonomies and nomenclature as it does not relate well to self-directed learning.

- The Copernican view of curriculum being the sacred cow of Academia is rooted in Math not changing much in a few hundred years, and generally the rate of change in curriculum being slow. In the real world, Academia is quickly reaching a point where they cannot update fast enough to keep pace with the change in the world, and the gap between students and entering a job is increasing.

- Again, see what students are doing to stay relevant - it's not always formal education.

- Learning how to learn is a course I reference early and often to all sides as the key skill being missed.

[kingludite]

In work and learning I keep getting back to an analogy with a trainer and an athlete. People have been training for a long time. The goal is to excel, constantly improve and ideally become the best. The job of the trainer is to monitor if the athlete is working hard enough to improve while not overdoing it. If recovery takes to long the gains will be smaller. One might even get injured. Every type of training has clear goals one of which is the duration of the challenge. A work day is usually 8 hours not a 30 min race to the end. It seems to me a huge challenge to do this remotely. It seems the camera shouldn't be pointed at the teacher but at the student.

[pezo1919]

To me it's giving the right intuition and perspectives. I don't really care about the "raw" material/math, it comes only after good visualization based intuition.

[mgozaydin]

admi,nistrators . Page design is real bad . Make it attractive .