Abstract

Meaningful learning requires that the learner engage in substantial cognitive processing during learning, but the learner's capacity for cognitive processing is severely limited - hence the situation called cognitive overload. This is the central assumption of Chandler and Sweller's (1991; Sweller, 1999) cognitive load theory and Baddeley's (1998) working memory theory.

Research Recommendation

Knowledge segmenting is a mechanism by which the information offered is broken down into bite-size segments. Learner is allowed to digest intellectually one segment before moving on to the next. With this approach, the learner has time and capacity to organize and integrate the information resulting in efficient knowledge construction.

Evidence of Impact

Mayer and Chandler (2001, Experiment 2) broke a narrated animation explaining lightning formation into 16 segments. Each segment contained one or two sentences of narration and approximately 8 to 10 sec of animation. Students who received the segmented presentation performed better on subsequent tests of problem-solving transfer than did students who received a continuous presentation. The effect size in this study was 1.36 - referred to as the segmentation effect.

Implementation on ScootPad

Prescribed learning objectives are broken down into discrete units of content. Each unit is further segmented into multiple bite-sized concepts which are interconnected and sequenced using the knowledge map. Students practice these fine grained concepts in progression with processing time between successive concepts - leading to mastery of the learning objective as a whole more effectively. Furthermore, teachers have the ability to design their own custom learning paths (by cherry picking the necessary segmented concepts) aligned to their instruction or curriculum maps.

The figure above is a representation of the ScootPad Knowledge Map which shows how fine-grained segments i.e. concepts (honeycombs) are interconnected to enable knowledge progression.

Abstract

The fundamental hypothesis underlying research on multimedia learning is that multimedia instructional messages that are designed in light of how the human mind works are more likely to lead to meaningful learning than those that are not. Thanks to recent technological advances, pictorial forms of instruction are becoming widely available, including videos and dazzling computer-based graphics. However, simply adding videos or visuals does not guarantee an improvement in learning - that is, all multimedia presentations are not equally effective.

Research Recommendation

Instruction design should be sensitive to how humans learn and based on the three cognitive science principles of learning namely, (1) dual channels - the idea that humans possess separate channels for processing visual and verbal material (2) limited capacity - the idea that each channel can process only a small amount of material at any one time; and (3) active processing - the idea that deep learning depends on the learner's cognitive processing during learning (e.g., selecting, organizing, and integrating).

Evidence of Impact

In 11 of 11 experiments conducted by Richard Mayer and his colleagues, involving paper-based lessons on brakes, pumps, generators, and lightning and computer-based lessons on brakes, pumps, lightning, and arithmetic, learners who received corresponding graphics with words performed better on transfer tests than learners who received words alone (Mayer, 1989, Experiments 1 and 2; Mayer & Anderson, 1991, Experiment 2a; Mayer & Anderson, 1992, Experiments 1 and 2; Mayer et al., 1996, Experiment 2; Mayer & Gallini, 1990, Experiments 1, 2, and 3; Moreno & Mayer, 1999, Experiment 1; Moreno & Mayer, 2002a, Experiment 1). The median effect size was 1.39, which is a large effect. In short, there is strong and consistent support for designing lessons with words and pictures rather than with words alone.

Implementation on ScootPad

ScootPad developed an innovative instruction delivery mechanism by painstakingly integrating the three recommended cognitive science principles of learning to maximize learner's processing capacity leading to much deeper learning. Specifically, 1) instruction is delivered across both visual and auditory channels simultaneously using high quality video lessons, 2) each bite sized lesson is designed to maximize the limited capacity available through each channel and 3) each video lesson is followed up with problem solving to enable active processing and seamless recollection of cognitive processing via visual and auditory channels.

To learn more, review how students leverage our Instructional Lesson Videos to learn a new concept or refresh their knowledge.

Abstract

Today's commonly-used study strategies involve massed practice methods where students review concepts immediately after they were introduced. This gives one the impression of mastery of the concepts because the knowledge is still fresh and familiar. In contrast, research indicates that this study method involves surface-level short-term memorization rather than deep learning. Such study methods are unproductive in the long term - we must consider alternative, research-based learning strategies to ensure long term learning.

Research Recommendation

A more beneficial study strategy that is supported by research is the retrieval practice method that is spaced and interleaved, which involves recalling information from memory over a longer period. Although this recommended approach seems unproductive to the student because it is slower and requires more effort, scientific evidence proves that it leads to higher levels of conceptual learning and successful application of knowledge and skills over the long term.

Evidence of Impact

Shea and Morgan (1979) contrasted random and blocked schedules of practice where blocked practice involved sequential trials at Task 1, Task 2, and Task 3, with all trials for a given task being completed before moving on to the next. Random practice, in contrast, involved the same number of trials at the three tasks, but the order was randomized so that a given task was never practiced on successive trials. After practice in an acquisition phase, retention tests were given after 10 days. The results show a strong advantage for retention for the subjects who practiced under the random conditions in acquisition. That is, even though the random conditions were less effective during the acquisition phase, they were better than the blocked conditions on the random retention test. These differences are especially impressive given the ecological validity of random tests; that is, most real-world behaviors are not produced in blocked contexts.

Implementation on ScootPad

Through painstaking instrumentation, ScootPad implemented an innovative learning process powered by mastery-learning methods coupled with proven spiral and spaced learning strategies. ScootPad's adaptive mastery engine delivers continuously personalized learning to each student through painstaking experimentation and implementation of research-backed learning strategies namely interleaving, scaffolding, retrieval and spacing. As a result, ScootPad enforces the process of spaced recollection of information from memory, thereby strengthening mental pathways that lead to higher levels of conceptual learning and successful application of knowledge over the long term. These research-backed techniques enable ScootPad to make learning stick!

The figure above illustrates how ScootPad implements the spiral and spaced learning techniques.

To learn more, review How ScootPad Works to understand the spiral and spaced learning implementation on ScootPad.

Abstract

Growth mindset is defined as a belief that construes intelligence as malleable and improvable. Students with growth mindset are likely to learn by a mastery approach, embrace challenges and put in effort to learn. For instance, growth-minded individuals perceive task setbacks as a necessary part of the learning process and they "bounce back" by increasing their motivational effort.

Research Recommendation

Incorporate growth mindset intervention techniques which teach students that intelligence is not a fixed quality. Intelligence can be nurtured through challenging tasks, as intelligence grows with hard work on challenging problems. Subtle feedback and messages related to growth mindset can have noticeable effects on students' attitudes and motivation that will lead to long-term outcomes.

Evidence of Impact

Empirical studies have revealed that growth mindset has positive effects on student motivation and academic performance. Recent research has also shown that mindset is related to student outcomes and behaviors including academic achievement, engagement, and willingness to attempt new challenges. Numerous studies have shown the effects of growth mindset interventions on students' achievement at all ages. According to Dweck, teaching growth mindset to junior high school students resulted in increased motivation and better academic achievement. Her findings revealed that students in the growth mindset intervention group outperformed those in the control group (who received excellent training in study skills), indicating improved learning and desire to work hard. A growth mindset intervention was especially impactful with student outcomes in particular subjects such as science and mathematics.

Implementation on ScootPad

ScootPad implemented a two pronged strategy to foster growth mindset in students. First, with the Growth Mindset messages, ScootPad pushes expertly-designed subtle yet effective and inspiring messages to students right on their dashboard which promote growth mindset in students continuously. Secondly, students receive constructive and encouraging feedback immediately yet seamlessly as they continue their learning journey, so they are reminded of their performance and encouraged to "embrace difficulties" - which lead to metacognition and helping students "Avoid Illusions of Knowing".

The figure above illustrates how ScootPad delivers growth mindset oriented feedback to students.

To learn more, review the Growth Mindset implementation on ScootPad.

Abstract

Interleaving is a well-researched study strategy with positive effects on learning. When practice is interleaved rather than blocked, the practice of different skills is intermixed rather than grouped (e.g. abcbcacab instead of aaabbbccc). In the traditional blocked method, each concept is practiced and mastered one at a time in a "sequential" or "streak" method. Such massing encourages quick and effortless processing of information leading to reduced attention and short-term learning. Interleaved practice, by contrast, involves practicing multiple concepts simultaneously in a "spiraled" method leading to more robust encoding of information and better long-term learning.

Research Recommendation

Each interleaved practice should involve a set of practice problems drawn from several concepts so that no two problems of the same kind appear consecutively, thereby requiring students to choose the appropriate concept knowledge on the basis of the problem itself. Such "shuffled review," also helps promote retrieval practice, a cognitive learning technique for recalling information from memory, thereby strengthening mental pathways that allow learning to happen.

Evidence of Impact

In studies conducted by K. Taylor and D. Rohrer, the interleaving of different kinds of practice problems more than doubled subsequent test scores (77% vs. 38%) even though the degree of spacing was held constant. This was the first study to assess the benefit of interleaving while controlling for the benefit of spacing, as previous studies have assessed the conjoint effects of interleaving and spacing. Thus, although the spacing effect is one of the most robust effects in learning, these studies demonstrate that interleaving per se can improve test scores by a large margin. Whereas the interleaving of practice boosted test performance, it impaired performance during the practice session. Hence, interleaving proved to be an instance of what Robert Bjork and his colleagues have termed a "desirable difficulty", because the difficulty incurred during the practice session proves to be ultimately worthwhile (e.g. Bjork, 1994; Christina & Bjork, 1991; Schmidt & Bjork, 1992). These authors have previously identified both interleaving and spacing as desirable difficulties.

Implementation on ScootPad

ScootPad developed an intelligent and highly scalable adaptive engine to deliver adaptive practices interleaved continuously with each practice problem drawn in real-time based on a number of dynamic variables and constraints (powered by student intelligence, concept intelligence and content intelligence) and convolutions of these variables with each other. Through painstaking instrumentation, our adaptive engine has been fine tuned over the years to deliver these real-time recommendations at scale using a complex discrete optimization algorithm requiring only a subsecond processing time to execute.

The figure above illustrates how ScootPad delivers continuously interleaved practice to students.

Abstract

Humans learn items in a list more effectively when they are studied in several sessions spread out over a long period of time, rather than studied repeatedly in a short period of time, a phenomenon called the spacing effect. The opposite is called massed practice and consists of fewer, longer practice sessions where learning is concentrated in continuous blocks. It is generally a less effective method of learning. For example, when studying for an exam studying more frequently over a larger period of time will result in more effective learning than intense study the night before.

Research Recommendation

The spacing effect - that is, the fact that spacing learning events apart results in more longterm learning than massing them together. This robust phenomenon has been demonstrated in hundreds of experiments dating back to Ebbinghaus (1885/ 1964). Learners would profit from taking advantage of the spacing effect, both in classrooms and during unsupervised learning - and doing so is feasible because spacing does not take more time than massing, it simply involves a different distribution of time.

Evidence of Impact

In three different experiments conducted by N. Kornell, The spacing strategy resulted in significantly more learning than the massing strategy in all three experiments. Combining the three experiments, 90% of participants learned more in the spaced conditions than the massed conditions, whereas only 6% of participants showed the reverse pattern. "Space learning over time" is the first research-based recommendation in a recent practice guide from the U. S. Department of Education's Institute of Educational Sciences (Pashler et al., 2007). In a recent review of the literature, Lisa Son and Dominic Simon write, "On the whole, both in the laboratory and the classroom, both in adults and in children, and in the cognitive and motor learning domains, spacing leads to better performance than massing" (2012).

Implementation on ScootPad

With ScootPad's innovative spiral curriculum and distributed learning, each concept is practiced over a longer period and revisited across multiple spaced practice sessions. Furthermore, inspired by Hermann Ebbinghaus's work on memory retention and learning curves, ScootPad integrated the forgetting curve into it's reinforcement algorithms leading to strategically spaced mastery checks depending on how and when a student practiced each new concept - helping the student make connections over time and creating more robust pathways for retaining and recalling information over a longer term.

The figure above illustrates how ScootPad delivers spaced practice and revisiting each concept over a longer period.

Abstract

The ultimate academic goal is for students to become independent lifetime learners, so that they can continue to learn on their own or with limited support. Scaffolded instruction optimizes student learning by providing a supportive environment while facilitating student independence. However, scaffolding depends on the ability to identify knowledge gaps - the difference between what a student has learned and what the student was expected to learn. One of the more consequential implications of knowledge gaps is their tendency, if left unaddressed, to compound over time and become more severe and pronounced, which can increase the chances that a student will struggle academically or socially, possibly even dropping out of school.

Research Recommendation

Learners do best when working within their Zone of Proximal Development (ZPD), this means building in supports and scaffolds, then slowly fading them as learners gain more knowledge and skills. It's a seamless and timely process in which students are given support until they can apply new skills and strategies independently. As they begin to demonstrate task mastery, the assistance or support is decreased gradually in order to shift the responsibility for learning to the students.

Evidence of Impact

To prepare students for the challenges of the 21st century, many research studies and their findings advocate a shift from a focus on declarative knowledge to a focus on critical thinking (e.g., Hurd, 1998; Kuhn, 1999; Spektor-Levy, Eylon, & Scherz, 2009). Scaffolding can play a key role in building students' critical thinking abilities by supporting students as they engage in complex processes rather than teaching them didactically needed skills before engaging in complex processes (Lee & Songer, 2000; Linn, 2000; Sinatra, 2010).

Implementation on ScootPad

Through painstaking implementation and refinement over the years, ScootPad developed an innovative remediation mechanism which will automatically identify knowledge gaps in real-time and deliver instructional scaffolds in the conepts they are learning and their prerequisite concepts, at or below their grade level. Scaffolding on ScootPad is dynamic and continuous driven by three types of contingencies: 1) Instructional contingency - how to support activity 2) Domain contingency - what to focus on next 3) Temporal contingency - if and when to intervene. Powered by the innovative knowledge map, ScootPad is able to scaffold to deeper concepts (across depth of knowledge (DOK) a.k.a. vertical scaffolds) and to prerequisite concepts (across grade levels a.k.a. horizontal scaffolds). In this way, ScootPad's intelligent adaptive learning platform provides each student with the right level of support every step along the way, effectively remediating all knowledge gaps and moving them towards mastery of the entire curriculum.

The figure above illustrates how ScootPad enables automatic remediation through dynamic and continuous scaffolding.

Abstract

When we think about learning, we typically focus on getting information into students' heads. What if, instead, we focus on getting information out of students' heads? "Retrieval practice" is a learning strategy where we focus on getting information out. Through the act of retrieval, or calling information to mind, our memory for that information is strengthened and forgetting is less likely to occur. Retrieval practice is a powerful strategy for improving academic performance without more technology, money, or class time.

Research Recommendation

Retrieval practice is a strategy in which calling information to mind subsequently enhances and boosts learning. Deliberately recalling information forces us to pull our knowledge "out" and examine what we know. For instance, I might have thought that I knew who the fourth U.S. President was, but I can't be sure unless I try to come up with the answer myself (it was James Madison). Often, we think we've learned some piece of information, but we come to realize we struggle when we try to recall the answer. It's precisely this "struggle" or challenge that improves our memory and learning - by trying to recall information, we exercise or strengthen our memory, and we can also identify gaps in our learning.

Evidence of Impact

In three studies from a variety of educational settings and content areas, instructors implemented retrieval practice in the form of brief short answer quizzes, followed by immediate feedback. For the review condition, researchers asked students to study a review sheet containing the same information as quiz questions but with answers written out. In the study, clicker questions were read aloud with answer statements, in contrast to students individually clicking in an answer during quizzes. In the two studies with "business as usual" control conditions, typical classroom activities continued in lieu of quizzing or reviewing content. Results: Regardless of final assessment delay, all three studies found a significant benefit of retrieval practice compared to reviewing or "business as usual" in authentic classroom settings, confirmed by large effect sizes (greater than d = .80).

Implementation on ScootPad

With ScootPad's innovative spiral curriculum and distributed learning, each concept is practiced with strategically delayed subsequent retrieval over a longer period of time. Furthermore, inspired by Hermann Ebbinghaus's work on memory retention and learning curves, ScootPad integrated the forgetting curve into it's reinforcement algorithms leading to strategically spaced retrieval a.k.a mastery checks - helping the student make connections over time and creating more robust pathways for retaining and recalling information over a longer term.

The figure above illustrates how ScootPad enables delayed retrieval and spaced assessment.

Abstract

Benjamin S. Bloom was an American educational psychologist who made significant contributions to the classification of educational objectives and to the theory of mastery learning. Dr. Bloom's research highlighted an educational phenomenon which was initially reported in 1984 in the journal Educational Researcher. Bloom found that the average student tutored one-to-one using mastery learning techniques performed two standard deviations better than students who learned via conventional instructional methods - that is, "the average tutored student was above 98% of the students in the control class". Additionally, the variation of the students' achievement changed: "about 90% of the tutored students ... attained the level of summative achievement reached by only the highest 20%" of the control class. Hence this phenomenon, Bloom's 2 Sigma Problem, is named after him.

Research Recommendation

By definition, mastery learning is a method of instruction where the focus is on the role of feedback in learning. Furthermore, mastery learning refers to a category of instructional methods which establishes a level of performance that all students must "master" before moving on to the next unit.

Evidence of Impact

In a series of studies conducted by Walberg (1984), Bloom classified alterable variables that may have, in combination, a 2 sigma effect as the following "objects of change process": learner, instructional material, home environment or peer group, and teacher. Bloom considered and tested various combinations of these variables, focusing only on those variables that individually had a 0.5 or higher effect size. These included: 1:1 tutorial instruction (effect size 2.0), reinforcement (effect size 1.2), feedback-corrective mastery learning (effect size 1.0), cues and explanations (effect size 1.0), student classroom participation (effect size 1.0), student time on task (effect size 1.0), improved reading/study skills (effect size 1.0), cooperative learning (effect size 0.8), homework (graded) (effect size 0.8), classroom morale (effect size 0.6), initial cognitive prerequisites (effect size 0.6) and home environment intervention (effect size 0.5). Though Bloom concluded that one-to-one tutoring is "too costly for most societies to bear on a large scale", Bloom conjectured that a combination of two or three altered variables may result in a similar performance improvement.

Implementation on ScootPad

Through painstaking implementation and refinement over 5 years, ScootPad incorporated the most impactful mastery learning techniques from Dr. Bloom's recommendations into the platform, including 1:1 tutorial instruction, reinforcement, feedback-corrective mastery learning, cues and explanations, initial cognitive prerequisites and home environment intervention. ScootPad's innovative spiral and adaptive learning process is a practical realization of Dr. Bloom's mastery learning techniques, one that is as effective as one-to-one tutoring and capable of delivering a 2 sigma effect on student achievement.

The figure above illustrates how ScootPad delivers continuously personalized mastery-based learning to every student.

Abstract

Personalized System of Instruction, also known as the Keller Plan, was developed in the mid-1960s by Fred Keller and colleagues. It was developed using mastery learning theory and is based on the idea of reinforcement in teaching processes. While traditional teaching is "same pace, different learning", a key distinguishing factor of PSI is that it instead advocates "different pace, same learning".

Research Recommendation

Keller recommended that effective instruction should incorporate five principles, which are the essential elements of the Keller Plan: 1. Written materials - The primary presentation of new content should be through written texts. Given the forms of media available at the time when the Keller Plan was developed (e.g., lectures, movies, audio records, television, radio, paper-based text, etc.), paper-based texts gave students the greatest freedom; books and texts are portable, can be read at one's own pace, can be started and stopped at any time, can be easily reviewed, and can be marked by the reader. As an application of behaviorism, the Keller Plan was meant to maximize the number of operant behaviors that could be reinforced; this could best be done with written materials rather than have the learner be a passive observer of other media. Digital media available today could provide the same kinds of learner control and presumably could be incorporated into a contemporary implementation of PSI. 2. Units of content - Subject matter material should be broken down into separable, meaningful units. These units could have various kinds of relationships; for example, one could provide prerequisites for understanding a second, or the second could provide deeper elaboration of a preceding unit. In any case, specific learning objectives should be definable for each discrete unit of content. 3. Self-paced instruction - Students should be allowed to advance through the course material at their own pace. While an instructor might specify the order in which learning units are completed, the learners should decide when and at what rate they learn. This means that learners can move through a course as quickly or as slowly as they choose. 4. Unit mastery - Students must satisfy a mastery requirement in one unit before proceeding to the next. Students must demonstrate mastery of a unit's objectives to a certain level of quality. If the student does not reach the threshold, they are redirected to unit materials (or supplements if provided) and then take an equivalent form of the unit assessment. From the point of view of behaviorism, demonstrating mastery and being allowed to continue to a subsequent unit was presumed to be reinforcing. 5. Proctors - Human proctors are an important part of the Keller Plan. The proctors could be "external" to the course (adults or peers brought to the course from external sources) or "internal" (advanced students in the course who are doing well, have completed all units to date, and have good interpersonal skills). Proctors are the arbiters of unit mastery; they "certify" mastery, discuss areas of weakness, and direct students to the next units. Behaviorists were always concerned about bringing conditioned behaviors under the control of "natural" reinforcers; interactions with the proctors were presumed to provide natural social reinforcers that encouraged learning behaviors and perseverance in the course.

The figure above illustrates the Personalized System of Instruction as discussed by The Keller Plan.

Evidence of Impact

Fred Keller and his colleagues conducted a series of studies and published a summary of their findings. Based on the evidence gathered from these studies, a traditional course might have all students follow the same weekly lectures, exercises, etc., followed by an end-of-course exam that all take at the same time - but possibly with huge variations in learning outcomes (e.g., 95% achievement for a strong student, but just 55% for a weak one). In a course run according to PSI, all students must pass a high threshold of achievement on each module within the course (for instance 90%). The difference between weak and strong students would then be that the stronger ones are able to finish the course quicker, while the weaker ones would need more time.

Implementation on ScootPad

Through meticulous design and painstaking implementation, ScootPad incorporated the five key principles of The Keller Plan namely 1) Written materials 2) Units of content 3) Self-paced instruction 4) Unit mastery and 5) Proctors - to enable reinforcement throughout the learning process.

The figure above illustrates how ScootPad drives reinforcement throughout the learning process.

Abstract

Imagine that you're teaching a math remediation course full of fourth graders. You've just administered a test with 10 questions. Of those 10 questions, two questions are very simple, two are incredibly hard, and the rest are of medium difficulty. Now imagine that two of your students take this test. Both answer nine of the 10 questions correctly. The first student answers an easy question incorrectly, while the second answers a hard question incorrectly. Which student has demonstrated greater mastery of the material? Under a traditional grading approach, you would assign both students a score of 90 out of 100, grant both of them an A, and move on to the next test. But this approach illustrates a key problem with measuring student ability via testing instruments: test questions do not have uniform characteristics. So how can we measure student ability while accounting for differences in questions?

Research Recommendation

Item response theory (IRT) attempts to model student ability using item level performance instead of aggregate test level performance. Instead of assuming all items contribute equally to our understanding of a student's abilities, IRT provides a more nuanced view of the information each item provides about a student. It is founded on the premise that the probability of a correct response to a test item is a mathematical function of parameters such as a person's latent traits or abilities, as well as item characteristics (such as difficulty, "guessability," and specificity to topic). While IRT is very popular and has a proven track record in psychometric testing, many common IRT models make assumptions that can be limiting in the context of adaptive learning. For example, a common assumption is that a student's proficiency is constant, or at least that it is being measured at a single instant. This does not match the expectations of an adaptive learning system, where assessments are continuous over a long period of time in which the student is expected to learn. Another common limiting assumption in IRT models is that the overall ability of a student can be condensed into a single parameter. While this assumption might be reasonable when estimating student abilities with a single assessment, it is much less likely to be true in an adaptive learning environment where students interact with content across multiple fine-grained concepts. Modeling these finely grained skills requires extending IRT to support multiple proficiencies and providing a way for these proficiencies to interact. This interaction may be compensatory (where a high ability in any relevant concept can be used) or non-compensatory (where sufficiently high proficiency in every relevant concept is necessary).

Implementation on ScootPad

ScootPad's innovative Knowledge Map allows the platform to measure student knowledge at a very fine-grained concept level. The map also maintains interdependencies between individual concepts such that it becomes possible to draw inferences about, for instance, how a student's proficiency in a concept could change based on changes to proficiency in its prerequisite concepts. ScootPad makes learning dynamic by allowing proficiencies to change over time. For instance, if a student fails to recall previously gained concept knowledge, proficiency in that concept goes down. Similarly, because skills and knowledge increase over time, giving more weight to more recent assessments can facilitate the learning process and encourage teaching practices that are focused on learning growth. These modifications to the IRT framework coupled with the knowledge map and real-time student proficiency computations, enable ScootPad's platform to more rapidly and accurately measure student knowledge in order to deliver personalized learning experiences in real-time and contunuously.

The figure above shows the Knowledge Map where proficiency in a concept is contingent on the performance across the interdependent web of concepts.

Abstract

Traditional assessments have typically consisted of selected-response (multiple-choice) questions which present several limitations. Selecting a response from a list of options (or even randomly guessing an answer) has proven to be a lot less reliable than responding by constructing an answer or by completing an interactive task to contribute to a response. Technology-enhanced items on the other hand have the potential to provide improved measures of student knowledge, create more engaging assessment environments, and reduce the effects of guessing and test-taking skills.

Research Recommendation

Assessments using technology-enhanced items offer several advantages, both from an educator and learner perspective. TEIs allow for the creation of assessment environments that can resemble real-world scenarios in which students might interact with information using technology. This makes TEIs inherently more authentic and engaging for students. In addition, students are already relatively familiar with many of the elements facing them in TEIs, such as the drop-down menus and text-highlighting features. In that sense, technology-enhanced items are already quite intuitive to the modern-day learner. In addition, the use of touchscreen devices is naturally appealing to the modern-day learner because it's human nature to want to touch and interact with one's environment. Numerous studies have demonstrated the substantial difference in students' motivation to complete tasks that involve interactive engagement.

Implementation on ScootPad

Built from the ground-up, ScootPad's innovative content authoring and delivery platform enables rapid development of tech-enhanced items. ScootPad's TEI item bank enables more accurate measurement of student knowledge while offering deeper student engagement using touch-and-click interactivity and promoting high-level thinking and problem-solving. ScootPad supports an extensive and ever-expanding list of 50+ technology-enhanced item types including classification, ordering, matching, highlighting, labeling, charting, graphing and much more. Powerful data-driven insights from millions of real-world student responses help us generate deep content insights and intelligence that help us continuously improve content difficulty, alignment, accuracy and quality.

To learn more, review our 50+ Tech Enhanced Item Types and take a test drive of the sample items.

Abstract

The notion of representing content relationships in a structured format is not new, as such our knowledge ontology builds on the graph theory and the mathematical structures used to model pairwise relations between objects. However, where we innovate is in our ability to build a flexible and scalable model using ontology-based information retrieval approaches and techniques. Such scalable knowledge model allows diverse content to be easily represented and connected with each other to enable adaptivity at scale across content domains.

Knowledge Map

ScootPad developed a novel instrument called the Knowledge Map to address scalability of knowledge modeling across content domains. It represents arbitrary content in a semantic graph structure consisting of concepts (nodes representing fine grained skills) and the relationships between them (edges representing the dependencies). We designed the knowledge map to facilitate true learning progression reflecting our core belief that one’s knowledge base is a scaffold that supports the construction of all future learning (Alexander, 1986). The flexibility and scalability of this structure provides a way to diagnose student understanding (and misunderstanding), remediate deep misconceptions, build on prerequisites, and power intelligent intervention strategies.

Knowledge DNA (K-DNA) Algorithm

Through painstaking experimentation and continuous improvement, ScootPad developed the Knowledge DNA (K-DNA) algorithm to compute performance characteristics of each individual concept by analyzing data from all student interactions and learning outcomes, excluding anomalies, from the entire network of students on the platform. Resulting concept-level performance insights include Concept Difficulty Index (CDI), Concept Scaffold Index (CSI), Concept Instruction Index (CII) etc. Ingested in real-time, these insights power our adaptive engine to deliver progression, remediation, scaffolding and learning path optimization. Designed for scale, this algorithm's real-time computation requires only subsecond processing time and is independent of the number of students and the number of concepts.

Predictable Knowledge Heatmaps

Knowledge Heatmap is a data visualization model based on continuous analysis of millions of real-world student experiences to predict which nodes (i.e. concepts or prerequisite skills) within a specific content domain (ex: 3rd grade math) are the most difficult for students to master. These heatmaps are designed to help teachers and administrators identify critical areas for instruction, supplemental resources, teacher support and professional development.

Real-Time Learning Path Optimization

To understand how real-time learning path optimization works on ScootPad, consider this example: Google Maps offers the static route and directions from point A to point B. Yet, when you start your journey at point A, the Google Traffic algorithm retrieves real-time updates relevant to your route and recommends an optimal path to reach your destination avoiding known bottlenecks and accidents. Similarly, on ScootPad, while the Knowledge Map offers a static path for learning progression, the adaptive engine retrieves concept-level performance insights in real-time (generated by the Knowledge DNA algorithm) of all concepts in the path to recommend an optimal learning path for the student to achieve desired mastery. As more students use the platform, more concept-level insights are available resulting in larger data sets and more accurate learning path optimization.

The figure above is a representation of the ScootPad Knowledge Map which allows diverse content to be easily represented and connected with each other. For example, concepts (honeycombs) are connected by prerequisite relationships (arrows).

Abstract

The key ingredient that enables continuously personalized learning for a student is that student's proficiency data and their learning journey towards mastery. Over the past 5+ years, ScootPad has painstakingly developed and refined the Student Learning DNA that contains information about what the student knows, how they achieved the knowledge, and their progression towards mastery. This profile is progressive, which means it keeps getting smarter the longer the student interacts with the platform.

Student Learning DNA (SL-DNA) Algorithm

ScootPad developed the Student Learning DNA (SL-DNA) algorithm to compute performance characteristics of each individual student in real-time by analyzing all achievement data points including diagnostic results, practice problems answered, lessons watched, remediation received, interventions triggeres and assessments taken, as well as non-achievement data and learning goals. Key student insights generated by this algoritm include Knowledge Retention Index (KRI), Concept Engagement Index (CEI), etc. These student insights coupled with concept and content insights enable the adaptive engine to deliver real-time optimization of the learning path for each student. Once enough data is collected, the platform will start uncovering patterns and applying smart predictions to enhance their learning experience and journey towards mastery. The more data in a student's learning DNA, the more effective the platform becomes at serving up targeted learning content for that student.

Designed to scale, this algorithm's real-time computation requires only sub-second processing time and is independent of the number of students and the number of concepts.

Learning Continuum

ScootPad's mastery-based learning is sensitive and conducive to learning continuum to guide learning over time so that student competence in the domain can be advanced coherently and continuously. For instance, if a student has already mastered a grade level curriculum on ScootPad and moves to the next grade level curriculum, they start off "warm" with their learning path (as opposed to starting "cold" with no data). The Adaptive Engine takes into account the entire Student Learning DNA which reflects the student's previously mastered concepts and their unique trajectory through the learning path and uses this intelligence to maximize student learning continuously from that point forward.

Learning Path Visualization

ScootPad puts a lot of emphasis on visualizing student data and progress, with a focus on making it clear, accurate, and user-friendly. This kind of visualization has been proven to directly affect student achievement. In Classroom Assessment and Grading that Work (2006), Robert Marzano reminds us that graphic representation of student progress and achievement is so motivating and helpful that it results in an increase of 26 percentile points in student achievement when used regularly. Nothing is more dissatisfying to students than feeling like the challenges they face are essentially arbitrary and culminate in nothing. The Learning Path Visualization feature answers the student need for continuity and meaning by affording students a sense of long-term investment in the learning process.

The screenshots above represent how ScootPad helps students visualize their learning path and mastery progress.

Abstract

Content intelligence means having the full context of an individual piece of content. It allows us at ScootPad to make better decisions about not only how we develop the content (questions, lessons etc) but also how we present the content to the students. Every piece of content is built with a singular goal of improving the learning outcome of the concept in question. Our content experts don’t guess how the students engage with the pieces of content or how effective the content is, they have data-driven actionable insights in front of them so the content can be improved continuously.

Content DNA (C-DNA) Algorithm

Through painstaking experimentation and continuous improvement, ScootPad developed the Content DNA (C-DNA) algorithm to compute performance characteristics of each individual content item by analyzing data from all student interactions and learning outcomes, excluding anomalies, from the entire network of students on the platform. Resulting item-level performance insights include Item Difficulty Index (IDI), Item Standard Deviation (ISD), Item Discrimination Index (IDSI) etc. These insights fuel the content improvement process empowering our content experts to continuously enhance the content quality and effectiveness.

Intelligent Item Analysis

ScootPad developed an intelligent item analysis based primarily on data-driven insights. This empowering process examines student responses to individual content items in order to assess the quality of those items within the concepts and of the concept as a whole. Knowledge of item difficulties, item discrimination, and distractors help our content experts make quicker and more accurate decisions about whether to retain items, revise them, or eliminate them from the pool. These insights are calculated with the assumption that overall quality of the concept derives from the quality of its content.

With these content insights at their fingertips, our content experts can easily answer questions such as:
- Was a particular item as difficult as you intended it to be?
- Did the item do a good job of separating students who knew the concept from those who did not?
- How effective were the item options (i.e., keys and distractors)?

Data-Driven Content Improvement

A continuous analysis of learners' performance data powered by AI can be incredibly valuable in identifying opportunities for content improvement. Such analysis can assist in modifying the level of difficulty, alignment to concept, accuracy, quality, and relationship to pedagogy. We not only enable real-time content revisions but we also help monitor the performance of the improved content to ensure the changes are working and effective.

Abstract

As Herman (2006, p. 122) observes, "whether and how children are able to engage in particular learning performances and the sequence in which they are able to do so are very much dependent on previous opportunities to learn." ScootPad's Adaptive Engine is built on the core belief that real-time progressions lay out a continuum to guide learning over time so that student competence can be advanced coherently and continuously.

Data-Powered Engine

ScootPad has been a data-driven, data-focused, data-hungry company right from its inception, having understood very early on that data was the key ingredient to personalize learning for every student, at scale. And yet, data alone doesn't enable the level of deep personalization needed to maximize learning outcomes. Over the past 5+ years, ScootPad has painstakingly honed its data gathering and interpretation capabilities and has used this data together with the most effective evidence-based strategies and techniques drawn from cognitive science, learning science and data science including interleaving, retrieval, scaffolding and spacing to develop the adaptive engine capable of delivering unparalleled, dynamic and continuously personalized mastery-based learning for every student at scale.

Tailor-made Learning Paths

ScootPad's Learning Path is an intuitive and flexible tool to define the scope of any desired curriculum (example: 2nd grade math, 3rd grade english etc.). It provides a logical structure to organize the scope of the content that students must master by breaking it down into separable, meaningful units (modeled on the five key principles of The Keller Plan). These units enable the sequence in which students are introduced to the content. Each learning path can have an unlimited number of units and each unit can have one or more concepts across any content domains. ScootPad's content library offers curriculum and content to support many US and International Curriculum Standards. Premade learning paths are available for each grade level and aligned to each supported curriculum (example: the Common Core Grade 5 Math Learning Path covers 100% of the Common Core standards a 5th grader must master). ScootPad also allows users to design their own custom learning paths using an intuitive drag-and-drop user interface. During this design process, ScootPad applies the Knowledge Map to recommend a logical structure and progression of the concepts - users can follow the recommendations or overwrite it and design their own sequence and grouping as they see fit with their own instruction methodologies.

Mastery-Based Progression

Students are assigned a learning path that reflects their learning goals/objectives (example: 2nd grade math, 3rd grade english etc.). Modeled after mastery-based learning, each student is expected to master all concepts from the learning path but allowed to do so at their own pace (traditional learning: "mastery is variable, pace is fixed" vs. mastery learning: "mastery is fixed, pace is variable"). Heritage (2011, p. 3) suggests that learning progressions provide descriptions of "how students' learning of important concepts and skills in a domain develops from its most rudimentary state through increasingly sophisticated states over a period of schooling." Inherent in these views of progressions is the idea of a coherent and continuous pathway along which students move incrementally through states of increasing competence in a domain. ScootPad's adaptive engine incorporates mastery-based progression when recommending what a student should learn next and only allowing students to move ahead when they mastered the dependent concepts. With mastery-based progression, every incremental state builds on and integrates the previous one as students accrue new levels of expertise with each successive step in the progression.

Continuous Adaptivity

Traditionally, products claiming personalized learning capabilities identify a student's level of proficiency in a subject matter only at a single point in time and only by using a fixed number of questions (with a diagnostic test, knowledge check, quiz, assessment etc.). Then, it will seek to develop a curriculum to fill the identified gaps of knowledge. In such single-point adaptive solutions, the curriculum remains static and ignores students' ongoing progress and struggles. ScootPad, however, enables continuous adaptivity to meet each student's unique needs in real-time. ScootPad analyzes each student response in real-time taking into account their latest proficiency and will adjust the rigor (number of questions to practice and their respective depth of knowledge) and adjust the learning path (the next concept to learn). In other words, ScootPad's adaptive learning process is dynamic and thereby continuously adapting to each student's unique needs and adjusting the learning path in real-time to achieve mastery in the desired concepts. With ScootPad's mastery-based progression model, students' work at their own pace and advance to the next unit only when they learned all concepts in the preceding unit. Once a student masters all concepts from the learning path across all units, they're deemed to have mastered the assigned curriculum and are ready for new and advanced curriculum.

The figure above illustrates how ScootPad deilvers a dynamic and continuously adapting learning experience to each student.

Abstract

Self-learning, often referred to as self-adapting, is a form of Artificial Intelligence. While it doesn't allow software programs to perfectly mimic the human experience, it does give programs an added layer of sophistication and the ability to recognize patterns, learn from data, and become more intelligent over time. This concept is also referred to as the Data Network Effect. Data network effects occur when a product, generally powered by artificial intelligence, becomes smarter as it gets more data from more users. There are plenty of examples of data network effects: for example, recommendation engines that are now everywhere from Amazon (products you’ll want to buy) to Netflix (movies you’ll want to watch) to LinkedIn (people you’ll want to connect with) that keep getting better with more users/data.

Self-Learning Smart Engine

At ScootPad, having understood very early on that data was the key ingredient to deliver personalization at scale, our goal has always been to build data feedback loops into our platform, first manually, then automatically so that the platform gets more intelligent over time autonomously. Today, as more students are using our platform, more data (structurally unique with a high degree of correlation) is available. Our data analysis algorithms, namely Knowledge DNA (K-DNA), Student Learning DNA (SL-DNA) and Content DNA (C-DNA), are able to recognize more patterns and offer much deeper insights. Our engine is continuously learning from these insights to recommend what a student should learn next and which content can best enable mastery.

The figure above illustrates how ScootPad's Adaptive Engine is self-learning and getting smarter over time as more students contribute more data to the platform.

Open Problems

ScootPad is committed to continued research and improvement of the platform AI capabilities to enable deeper and impactful personalized learning experiences for students.

Smarter Relationships between Knowledge Map Nodes: ScootPad's Knowledge Map is an interconnected web of thousands of nodes representing discrete concepts with hundreds of thousands of interconnections between them. Historically, these relationships have been manually assigned by content designers and validated/certified by the respective subject matter experts. However, given ScootPad's access to data from historical usage by students and by leveraging techniques from graph theory, it is possible to auto classify and make these relationships smarter. Such smart relationships can enable our engine to learn from these relationships and deliver an optimal learning progression for new students in future.

Identifying Student Learning Styles: Everyone learns and understands concepts in different ways. One student may prefer to directly launch into practice questions on a concept, another student may prefer to watch a video and then jump into the practice. At ScootPad, we have over 5 years of data of students interacting with content and questions on the platform and we are constantly mining this data to identify patterns of student behavior. Student Learning style identification is an area of active research at ScootPad and a logical next step to enhance our adaptive engine's abilities to learn from the data and automatically guide what a student should do next.