Ryan Lab Group Meetings

Spring 2014 (2142)

First Week of Fifteen, Mon. 01-20-14

| back to top | next |

Spring 2014 (2142)

Second Week of Fifteen, Mon. 01-27-14

| previous | back to top | next |

Spring 2014 (2142)

Third Week of Fifteen, Mon. 02-03-14

| previous | back to top | next |

Spring 2014 (2142)

Fourth Week of Fifteen, Mon. 02-10-14

| previous | back to top | next |

Spring 2014 (2142)

Fifth Week of Fifteen, Mon. 02-17-14

Wed., 02/19/14

Present: John Riter, and Derek Rickards. We are working on designing an experiment that applies the KLI framework to the Interleaving and Feedback experiment, with an eye to getting better retention. We noted that spacing - via repeated testing (practicing retrieval) may be what we need.

I emailed John the KLI paper and the Interleaving with Feedback paper. He will begin by reading the Interleaving paper first, then the KLI paper. Derek is already working on developing the materials for an experiment to run in the Fall of 2014 (to improve retention). He will try incorporating the spacing idea (perhaps expanding spacing).

| previous | back to top | next |

Spring 2014 (2142)

Sixth Week of Fifteen, Mon. 02-24-14

Wed., 02/26/14

Met with Erik Schwambach. On the training materials (blocked - with feedback) he added prompts to self-explain on the initial example (before the feedback).

One way to incorporate the S.E.'s would be to compare only two conditions between subjects. One condition would be the interleaved, with feedback condtion without the S.E., and the other would be the same, but with the S.E. That would give us the most subjects per condition, but would only test the addition of the S.E.'s. It would not provide any replication of the previous results.

Or, we could have three conditions, the worst condition from the previous study (blocked-no feedback), the other two would be the best (interleaved - with feedback), without and with the S.E.'s.

To get a complete replication and extension we would need all 4 previous conditions and either add one more (the best plus SE) or cross SE with the other two factors to give eight conditions. But that would require a huge number of subjects unless we could figure out how to do it within subjects. But doing all those conditions within subjects could be very difficult.

I sent Erik the interleaving paper that is in progress just in case he has not already seen it.

The plan going forward is to consider how best to make use of either spacing, SEing, or both.

| previous | back to top | next |

Spring 2014 (2142)

Seventh Week of Fifteen, Mon. 03-03-14

| previous | back to top | next |

Spring 2014 (2142)

Eighth Week of Fifteen, Mon. 03-10-14

| previous | back to top | next |

Spring 2014 (2142)

Ninth Week of Fifteen, Mon. 03-17-14

Wed., 3/19/14

Conversation with Dr. Baranczyk: A good way to do a manipulation within a class might be to use D2L. Randomly assign students to take different versions of a "Quiz". I put quiz in quotation marks because it might be possible to use the quiz as a way to manipulate some learning event, as well as collect data. It would also have the advantage of taking work off of the instructor, thus helping to get their buy in. Also, because the students would see it as a quiz, and not just an experimental activity, it would help with their task engagement.

| previous | back to top | next |

Spring 2014 (2142)

Tenth Week of Fifteen, Mon. 03-24-14

Wed., 3/26/14

My thoughts before the lab meeting:

For Fall 2014 - The idea would be to cross types of knowledge components (KC's) with types of instructional events (IE's) so that the four cells would comprise congruent and incongruent combinations of IE's with KC's. One way to accomplish the IE (suggested by Baranczyk) would be to use "quizzes" in D2L. We would need to identify several pairs of KC's (with each member of the pair theoretically amenable to a different kind of IE). Each pair could be from one module in Stats. The modules and KC's would have to be ones that any Stat instructor would teach. Then, divide each class randomly into Group A and B. For each module one of the groups would receive a congruent combination of KC and IE, and the other would receive an incongruent combination. The assignment of congruent or incongruent to A or B would be counterbalanced so that each group of students would receive the same number of congruent and incongruent IE's, but on different modules. One problem would be to equate the difficulty of the modules and we should plan in advance to examine students separately based on ability levels (Ryan, 2013).

References

Ryan, R. S. (2013, May). Practicing retrieval of facts in statistics benefits high ability but hurts low ability students. Poster presented at the 25th annual convention of the Association for Psychological Science, Washington, DC.

Erik and Derek's email of 3/9/14:

Hello Dr. Ryan, sorry for this delayed response. Me and Derek met up on Wednesday, March 5th. We talked on how to bring my idea (S.E. problems to help critical understanding) and Derek's (spacing tests to increase retention).

We thought of 3 possible conditions.

1) Spaced testing w/ interleaving, feedback, and S.E.

     -introduce spacing and S.E. to see how well they score. See if the average becomes the new highest/most efficient

2) Interleaved w/ feedback and S.E. but not spaced like #1

    - using the study's most successful condition (interleaved with feedback) but introducing S.E. to see if it increases the scores.

3) Blocked, no feedback, but w/ S.E., not spaced like #1

    -using the study's worst condition (blocked with no feedback) but introducing S.E. to see if it increasing the scores.

Results of meeting with Derek 4 - 5:22pm, Wed., 3/26/14

We have already shown that we can improve students' learning of the associations between descriptions with different designs (between - within - 2 grp - 3 or more grp) and the appropriate tests by using feature focusing and feedback to get them to induce the designs, and interleaving to make the associations. But we're still not getting retention. Maybe we need to focus on how to get retention. And maybe we should not worry about getting them to retain their ability to induce the designs from the descriptions. Instead, first we should focus on getting them to retain just the associations between the designs and the appropriate tests. So if we add spacing and retrieval practice we should be able to get that retention. (Although there may be a danger that if we give them the designs, rather than forcing them to induce them, then that might interfere with our ability to get retention - but let's hope that's not a problem).

Also, we should not worry about involving S.E.'s at this point. They may be helpful for higher level learning, but we aren't there yet. Or, they may be helpful for retention of the induction process, if, later, we try to get better retention of that.

Also, we should not worry about crossing types of KC's with types of IE's to get congruent and incongruent conditions. That might be useful for a more theoretical experiment that was designed to support the claims of the KLI paper. But it would not be of as much interest to teachers who simply want to improve learning.

So, t tests could be one module. Anova could be another module. Maybe we could add correlation or chi square or both. One group of students would get the module with spacing and retrieval practice, and the other group would serve as a control that would not get the spacing and retrieval practice. Then for another module, the groups could switch. The task, at this point, would not be to induce designs from descriptions and then make the correct associations. Right now, the task would just be to learn the associations, and then, more importantly, to retain them.

Now we have to figure out how to implement the conditions using D2L.

| previous | back to top | next |

Spring 2014 (2142)

Eleventh Week of Fifteen, Mon. 03-31-14

Wed., 04/02/14

John Ritter stopped in. We discussed the fact that the group working on this project is working a good bit just on their own and only stopping in to update me or get further instructions as needed.

As John figured from reading me previous email in which I sent out the notes from the 3/26/14 meeting with Derek, we now need to be working on how to implement the study described there on D2L. They do not have to worry about the nuts and bolts of how to get it on D2L. Rather, they can work on two things. First, what exact materials could we use. Second, how would we present them so that one group would get retrieval practice and spacing, whereas the other group would get exactly the same thing, but without retrieval practice and spacing. And yet, both groups have to get exactly the same thing other than that manipulation, and both groups have to see the activities as graded quizzes in order to get good task involvement.

I thought about whether their task should be only to make an association between a set of features and the name of a procedure. If so, we might be able to get better retention of just such facts. But they could possibly learn and retain just bare bones facts such as "If there are two groups, not three or more, then use a t test, not an anova" without much understanding of what that means. On the other hand, they might actually be able to acquire greater understanding, but we would not know if we do not give them the opportunity to do so, and then test for it.

So, here's what we could do. The materials could be descriptions of research situations, but always presented along with an explicit statement of what the features were, and what procedure goes with those features. Then, when we test them, we could first test for just the association between the features and the appropriate procedure. Then we could also give them test items in which we present just the description (without the explicit statement about the features) and ask what would be the appropriate procedure. That way, if they DID have the ability to induce the features (and maybe even retain the ability to do that), then we would have given them the opportunity to do it, and to show us that they did it.

I pointed out to John that maybe they could find appropriate materials from among the stuff they already have from the interleaving experiments. Also, the feedback manipulation that we used might be able to be adapted into a practicing retrieval procedure. That might be able to be done by making the feedback into delayed feedback, rather than the immediate feedback that we used.

It probably will not matter whether the experiment spreads into the time that they receive the formal instruction on the task materials, as long as it was the same for both conditions, because the formal instruction, unfortunately, did not have any impact according to the results of the interleaving studies.

The challenge, however, will be to design a procedure to implement the manipulation while keeping all other things the same for both groups, and making sure it looks like actual graded assignments to the students (there's no reason they couldn't actually be actual graded assignments).

Then implementing it as quizzes on D2L may not be too hard (Ha, ha, famous last words).

| previous | back to top | next |

Spring 2014 (2142)

Twelfth Week of Fifteen, Mon. 04-07-14

Wed., 04/09/14

Met with Derek Rickards. We figured out that in order to manipulate retrieval practice, we need to take the interleaving with feedback materials, and adapt them. We need to have not just a session of training with immediate feedback, but rather, a training session first, then a session of practicing retrieval with delayed feedback.

Fri., 04/11/14

While creating example materials for the crew to use to make the sets of materials I realized something. My training examples are worked examples and my study examples are problems to solve. There is research showing that learning from worked examples is improved if the worked examples include both correct and incorrect examples. That is, one group compares two worked examples, both of which are correct, whereas another group compares two worked examples, one of which is correct and the other is incorrect. We need to keep this idea in mind for perhaps future studies.

Also, a rep from Cengage came by to say she is going to be doing a demo of their web enrichment site. I asked about using the site to implement an experiment and she seemed interested.

| previous | back to top | next |

Spring 2014 (2142)

Thirteenth Week of Fifteen, Mon. 04-14-14

| previous | back to top | next |

Spring 2014 (2142)

Fourteenth Week of Fifteen, Mon. 04-21-14

| previous | back to top |

Spring 2014 (2142)

Fifteenth Week of Fifteen, Mon. 04-28-14

Wed., 04/30/14

Met with Derek briefly. He emailed me the materials. I'll check them when I can and get back to him. He will be available during the summer.

Spring 2014 (2142)

After the semester ended

Wed., 08/27/14

Over the summer I developed the experiment for the Stats classes in which we manipulate practicing retrieval and amount of spacing between two practice sessions. I developed the materials to be presented on D2L as quizzes. All the Stats instructors agreed to use the quizzes as items that count towards the students' grades (how much will be up to the instructors). The students will be asked for their informed consent to use their performance on the quizzes as data for our study. The IRB ap was approved over the summer. The grad assistants will visit all the stats classes to obtain the informed consents.

I met with Derek Rickards today.

Derek Rickards and John Riter timed the EBL activities enough to give me an idea of how much time to tell the students the activities will take. They will continue to time them at the beginning of the semester.

Also over the summer I received word that my sabbatical request was denied. The application basically asked for a one year sabbatical to do a cognitive task analysis of the content of our Statistics courses in order to develop a series of experiments based on the KLI framework that could be run in the Stats classes. The committee denied the application because they felt that I was asking for time to develop a project rather than developing the project first and asking for time to run it. In discussions with the chair of the committee we determined that if I already have developed an experiment, and especially if it is the kind in which I need to run hundreds of subjects, they need to be run individually, and it takes, say, a couple hours to run each one, then the committee would think that it would be appropriate to take a sabbatical to run the experiment.

So, I will ask Derek and John to work with me on doing the cognitive task analysis over these next two semesters with a goal of developing an experiment in which we manipulate self-explanation. According to the KLI framework it will probably be S.E. ing that would be the likely candidate for a type of instructional event to apply to the higher order concepts and principles in Statistics. But in order to conduct an experiment we will have to identify specific concepts, operationalize them, and develop all of the specifics for a method for training and prompting S. E.'s specific to those concepts. Such an experiment would be appropriate to test the ecological validity of using S. E.ing on statistics concepts, and would be the kind of labor intensive experiment that the sabbatical committee might be more likely to approve.

Derek and John can work on that and check in with me as needed for guidance and feedback from me.