Ryan Lab Group Meetings

Spring 2010 (20106)

Prior to start of semester

Nothing

Spring 2010 (20106)

First Week of Fifteen, Tues. 01-19-10

No meeting yet.

| back to top | next |

Spring 2010 (20106)

Second Week of Fifteen, Tues. 01-26-10

No meeting yet.

| previous | back to top | next |

Spring 2010 (20106)

Third Week of Fifteen, Tues. 02-02-10

No meeting yet.

| previous | back to top | next |

Spring 2010 (20106)

Fourth Week of Fifteen, Tues. 2-09-10

No meeting yet.

| previous | back to top | next |

Spring 2010 (20106)

Fifth Week of Fifteen, Tues. 2-16-10

No meeting yet.

| previous | back to top | next |

Spring 2010 (20106)

Sixth Week of Fifteen, Tues. 2-23-10

Tues. 2-23-10:

Present: Samantha Crist and Mara

We analyzed PR2 pub web to get graphs of the two way interactions. Then we analyzed PR2 web lab exactly the same way (no exper factor) and got exactly the same graphs.

| previous | back to top | next |

Spring 2010 (20106)

Seventh Week of Fifteen, Tues. 3-2-10

Tues. 3-2-10:

Present: Mara Wilde, Chelsea Muehsam, Dale Kappus

Chelsea and Dale will be on interleaving. They will meet with me at 1:00 tomorrow

Checked on all info for all of them in the members file.
Asked Mara to check accessing the Ryan Lab group folder from home
Explained the PR2 paper to Chelsea and Dale

Checked where Mara and Samantha are. They have scribus on Mara's laptop. They can use it together.

I read through the drafts they have so far of the method and results. I gave them a few notes, but there was not much to comment on. Mara will meet with me at 12noon on Friday and have more for us to go over. The graphs that she put in the results section did not show. Mara will try downloading OpenOffice to see if that corrects the problem.

Wed. 3-3-10:

Present: Heather Shaw, Chelsea Muehsam, Dale Kappus

Got Scribus running on Heather's laptop

Explained the older and most current Interleaving experiments

Gave Heather the submission for APS 2010, and gave her instructions for beginning to write the paper. She will write a draft of just method and results. She will write the Fall 2008 study first. Then, for the Spring 2009 and Fall 2009 studies, she can just explain the changes. She will meet with me to move forward on the draft Mon, 3/15/10 at 1pm.

Then I showed them the data file for entering the Spring 2010 data. I'll email all of them copies of the file. For now, we will not enter data until we are sure the computer in OM 381 is not infected. But, once we are sure of that, Heather will get Chelsea and Dale going entering data.

| previous | back to top | next |

Spring 2010 (20106)

Eighth Week of Fifteen, Tues. 03-09-10

Tues. 03-09-10:

Spring Break

| previous | back to top | next |

Spring 2010 (20106)

Ninth Week of Fifteen, Tues. 03-16-10

Mon. 03-15-10:

Meet with Heather Shaw regarding writing up the previous interleaving studies for APS?

Tues. 03-16-10

Met with Mara regarding the PR 2 paper. She had a re-write of the method and results. I gave them a few more notes. When they follow those notes, they should be done with method and results. Mara will meet with me on Friday 3-19-10 at noon to give me summaries of articles for the intro.

Fre. 03-19-10

Met with Steve Craig - we finished the intro to the CP3 APS 2010 paper

Met with Mare Wilde re: the PR2 paper - we did another re-write of the method and discussion. She summarized the literature for me. They will now work on the intro and disc. and bring it to me on Tues.

| previous | back to top | next |

Spring 2010 (20106)

Tenth Week of Fifteen, Tues. 03-23-10

Tues. 03-23-10: 2:00pm

Met with Heather Shaw at 2pm. She is writing the paper for the APS poster presentation on interleaving in May. She will write the method and results first. For the method, she will write the method for Fall 2008 as Experiment 1. That is the one in which, although performance was low, we at least got a significant interleaving advantage on the early retention test. Then she will write a results section for that experiment. Then she will write up the method and results of Experiments 2 & 3 (Spring 09 and Fall 09 respectively). For those, she will just explain the changes, and report that there was little improvement in performance (.40 ish at best) and no more signs of any effect of the manipulation.

In the discussion she can suggest the changes that we made for Spring 2010 (which we now know did improve performance, but did not even include a manipulation).

She is now working on just writing up the method for Experiment 1 in the lab, based on my notes, and she will bring it to me when she is done.

She came to me with one re-write. I gave her more notes. She did not come back by 4pm. We need to set up another meeting.

Tues. 03-23-10: 4:00pm

Met with Mara regarding the PR2 paper. She had a nice re-write of the method and results. We found that Table 1 had come from the wrong output, but we found which output it should come from and she can easily fix that. She had a very good first draft of an intro from Samantha. We edited it. She will write up the edits and give them to Samantha and then come to me next time with a first draft of the discussion. Also, they will email me an e-copy of the intro to run through Turn-it-in to make sure it does not pull too much from their cited sources. We will meet next on Friday 3/25/10 at noon.

Wed. 03-24-10: 2:00pm

Met with Steve Craig. We did a little more re-writing on the intro. It should now be done unless we decide to improve it a little more later. We also found the source material from Ryan 2009 that can be used to write up the method. We figured out that what we need to do with that source material is to delete the choice task, but add the aptitude measures. We will need to explicitly tell the reader that we are reporting some of the same data as reported in Ryan 2009, but that more data was added, that we are not reporting the choice task because that was reported in Ryan 2009, and that we are reporting the aptitude measures because they were not reported in Ryan 2009. I wrote in a note to Steve to that effect in the draft of the paper. I made sure all the relevant files were copied to his flash drive. He will meet with me on Friday with his first draft of the method.

Fri. 03-26-10: 12:00noon

Met with Steve Craig. He had done a good draft of the method. We edited it some. We are up to Draft 6. Next he will have to explain the 9 test problems. I gave him the EPA 2005 paper that was done as a talk at EPA. It explains the 9 test problems.

Importantly, we are thinking about doing a new experiment next year. We will keep it simple. The goal is to just replicate the effect of Matching Features on the find amount problems.

Fri. 03-26-10: 2:00pm

Mara came in with a draft of the discussion as the final part of the PR2 paper for Scranton. I didn't need to do any editing other than I thought they should remove the last sentence. It just referred to the need for more research without specifying what it needed to address. Mara will bring me a draft on Tuesday with the title page, etc.

| previous | back to top | next |

Spring 2010 (20106)

Eleventh Week of Fifteen, Tues. 03-30-10

Tues. 03-30-10

Met with Mara regarding proofread the final draft of the PR2 paper. Also, Sean Snoddy attended his first meeting.

I did a quick proofread. There are a few things, such as adding page numbers, and formatting less like a ms, that Mara will do. We set up our next meetings for Friday 4/2/10, 12pm to about 4 with Mara, and Sat. 4/3/10, at 8:30am to about 11 for Mara and Samantha to work on any final edits on the paper, and then move on to the poster.

Sean is doing the log. He knows about the subject protection training and will work on it. I gave him (on his network drive) the whole ACME literature folder and advised him to start with the seminal articles. We will meet next at the regular Tuesday lab time.

Wed. 03-31-10 - 1:00 pm

Met with Heather. She needs to look into funding from the department and will meet with Dr. Meehan later this afternoon to look into that. I sent an email to all department faculty asking if there are any other female students who will be rooming at the convention with whom she could share.

We worked on the method section for Experiment 1 - Fall 2008. We got through the materials. We did the procedure for the training. She will continue with the procedure for the immediate test using the script as a guide. We will meet again on Friday 4/2/10 at 9am.

Wed. 03-31-10 - 2:00 pm

Met with Steve. We edited the materials for the CP3 paper for APS. Next will be the procedure. We will meet again on next Wed. same time, and on Friday 10am to 12.

Fri. 04-02-10 - 12:00 pm

Met with Mara to finalize the paper and begin the poster. We finished a draft of the paper (in Openoffice) and converted it to .pdf. We started working on the poster. Mara will check to see if there is a conference program, what time we really need to be there, what time we set up to present, etc. She will double check on the size of the poster boards.

| previous | back to top | next |

Spring 2010 (20106)

Twelfth Week of Fifteen, Tues. 04-06-10

Tues. 04-06-10

No meeting.

Fri. 04-09-10

I worked on the poster for PR2 for Scranton. Got all the graphics done. Began putting them in.

Met with Steve. We re-worked the method section. We may leave it as is, although I may try to make one improvement involving the problem of what order to present the materials. This was draft 8.

Next, Steve will work on the results. We will meet next on Wed. 4/14/10 at 2pm to 4pm.

| previous | back to top | next |

Spring 2010 (20106)

Thirteenth Week of Fifteen, Tues. 04-13-10

Tues. 04-13-10

Met with Sean Snoddy. Did a little brainstorming on where to go with the ACME project

Wed. 04-14-10 - 1:00 - 2:30

Met with Heather to continue working on the interleaving paper. Next is the procedure for the immediate test using the script as a guide. We also wrote up the results section for Experiment 1. We decided that she will work on writing up Exps. 2, 3, & 4. She will write up the changes for each in a general way as an introduction. Then in each method section she will detail the changes. We will meet again at 1pm on Friday.

Wed. 04-14-10 - 2:30 - 4:00

Worked with Steve on results. I did more correlations. The output is only in the Lab Group folder. The correlations are for (a) training conditions with each set of three posttest problems, (b) cover1-6 with each set of three posttest problems, and (c) cover 7-9 with each set of three posttest problems. I did them with and without controlling for pretest.

Fri. 04-16-10 - 1:00 -

Met with Heather to continue working on the interleaving paper. She used the output summaries, which had notes about the changes in the versions, to write up the methods and intro's to Exp 2 and 3. Fall 2008 is Exp 1, Spring 2009 is Exp 2, Fall 2009 is Exp 3, and Spring 2010 (now) is the next experiment, which is only referred to in the discussion. Next we will use the submission notes to develop the intro and discussion.

She is working on the paper on her own and will email me the draft when she is finished (Draft 5). We will meet again on Wed., 4/21/10 at 1:00 pm.

| previous | back to top | next |

Spring 2010 (20106)

Fourteenth Week of Fifteen, Tues. 04-20-10

Tues. 04-20-10

Met with Sean Snoddy. We are looking for good research questions for the ACME project.

One idea would be to manipulate whether we teach a mathematical concept with just the principles involved in the concept or with real life examples designed to illustrate the principle. We would ask whether the difference between these two teaching methods would affect learning the math concept. So we would need a reliable and valid measure of learning of the concept.

The above is based on Jaffe 2008 saying that generic examples are better than concrete examples.

Testing with short anwer questions is superior to multiple choice testing to produce better performance on a later multiple choice exam (McDaniel, M.A., Anderson, J.L., Derbish, M. H., & Morrisette, N. (2006).

When the testing effect occurs, does it only occur for the information they got correct.

Sean will next read Rittle-Johnson and Star (2007), Rittle-Johnson and Star 2009, and Ng and Lee (2009). He will look for what we could manipulate in our statistics classes to improve learning of which statistical test goes with which research situation. He will also think about other concepts in statistics that we could target to improve.

I'll see about getting feedback from someone in Education about the above ideas.

We may also get Sean doing a lab study this coming Fall manipulating the same types of things we are manipulating in the classroom, or something else if he comes up with a good idea of his own.

Sean will be around in the summer, so he could even start working on developing materials for the next study coming up in the Fall.

Wed. 04-21-10 - 1:00pm

We worked on introductions for Experiment 2 and 3. We finished up through all the method and results sections for all the expeirments (except for the tables). We also have introductions for Experiments 2 and 3. Next, Heather will work on the tables. I will work on the initial intro and the discussion, etc. Our next meeting will be this coming Friday, 4/23/10 at 1pm to about 3pm.

Fri. 04-23-10 - 9:30am

Met with Dr. Terry Stahler, the Chair of Secondary Education in the School of Education. We discussed the fact that cognitive psychologists need to take the findings of cognitive psychology from lab studies and partner with education professionals to do research in how such findings can be applied in actual classroom instructional practices.

First, I found that we agreed on some of the important principles of what enhances learning, but call them by different names. For example, what cognitive psychologists call massing (as opposed to distributing) practice, educators call chunking. I explained that I had found from the cognitive psychology literature that there isn't much evidence to support the claim that different learners have different learning styles that benefit from specific instructional methods. That, of course, does not mean that the claim wouldn't be supported if the right kinds of studies were done, but rather that, to date, they haven't been done. Dr. Stahler pointed out that it would be difficult to do the necessary studies because it would be difficult to define learning styles in a way that enough educators would agree on. Therefore, she said that because she is confident that different learners learn in different ways (regardless of what those might be), she believes that what teachers need to do is to simply use multiple methods rather than just one, in order to increase the probability that each learner will be reached by one of the methods. My response was that I agreed that there was such a difficulty, but that I thought it would be worth the effort to try to overcome it.

We also talked about which authors and which journals each of us read. It turned out that there was very little overlap. Dr. Stahler gave me three articles to read as examples of the kinds of research that educators rely on. I'll send her some examples of the kinds of articles I read. She also pointed out that she keeps up with The Journal of Teacher Education as a way to motivate herself to keep current with the literature. I said that I tend to read by searching on topics I'm interested in. But I can see how keeping up with a specific journal might be a good approach.

Our discussion ranged over several other topics, not the least of which was our common connection to Shaler Township in Pittsburgh. We talked a good bit about how I need to get input from educators in order to do research that be helpful in the effort to apply cognitive principles to education, but specifically, in ways that educators would agree are important and helpful to their field, not just in ways that cognitive psychologists feel improves their understanding of learning principles as uncovered in the lab.

Dr. Stahler said that Dr. Patricia Walsh-Coates, a member of the Secondary Ed. faculty, might be a good person to approach about collaborating. Dr. Stahler suggested that we get a small discussion group going as a beginning step. She said that a good time to begin meeting might be after the semester is over.

I'll send some articles to Dr. Stahler, and I'll contact Dr. Walsh-Coates. I'll also look for some good times to meet and see when Drs. Stahler and Walsh-Coates could meet so that we can plan to get started.

Fri. 04-23-10 - 1:00pm

Met with Heather on the Interleaving paper. We finished the general introduction, although we may refine it later. Heather will work on the discussion and send it to me. At that point, I'll probably add the title page and references and finalize the editing of the entire paper.

| previous | back to top |

Spring 2010 (20106)

Fifteenth Week of Fifteen, Tues. 04-27-10

Tues. 04-27-10

Met with Sean. He has a good idea about manipulating whether students are first given generic examples, followed by concrete example versus concrete first, then generic. He will do further searching on this more specific idea. He will also contact his high school teachers from Emmaus to see if they would be interested in getting involved in this research.

For this Fall we should start piloting one of the ideas.

Wed. 04-28-10

Met with Dr. Patricia Walsh Coates regarding ideas for ACME. She said that they are going to start teaching a new couse in Spring 2011 called Principles of Learning. Dr. Walsh Coates will teach it. This will be a course for undergrads in education. It will be a 4 credit course that will be part lecture and part practicum in the local secondary schools. She said that the practicum part could be a venue for us to do the kinds of research that we talked about. However, she said that it would probably not be practical to implement the research component in the first semester that it was being taught because they would be still working out the bugs from the course. However, she saw Fall 2011 as a possible time that we could begin.

I will keep in contact with Drs. Stahler and Walsh Coates regarding when we could start meeting on a regular basis once the Spring semester is over.

Summer 2010 (20111)

Mon. 05-10-10

Met with Sean. He brought in Koedinger, Alibali, and Nathan 2008 "Trade-Offs Between Grounded and Abstract Representations: Evidence From Algebra Problem Solving".

For less complex problems (single reference - meaning X appears only once in the equation), there is a "verbal advantage", that is, students' performance is better for a grounded representation (a word problem), than for an abstract rep. (just the equation).

For more complex problems (double reference - X appears in two places), there is a symbolic advantage - they do better with the equation than the word problem.

So a possible question for us could be - do desirable difficulties have more of an impact on grounded or abstract representations.

Another question is what the long term retention and transfer would be for using these different representations.

Students can do more abstract problems in the same time. So try different time intervals, and see if one type of rep is better than the other for a specific interval.

What are the effects of MIXING generic and grounded.

Sean will contact the high teachers he knows when he narrows down an idea. He will explain that we need their input on what would be valuable to teachers.

==============

Regarding the idea of how preference for abstract vs. concrete examples is related to intelligence -- Sean was trying to think of how we could do that as a web survey. We might be able to use a vocabulary test, plus some web based quick IQ test as part of a survey.

I gave Sean a .pdf of Lefevre & Dixon 1986.
================

POSSIBLE IDEA FOR STEVE CRAIG FOR NEXT YEAR

Return to DS3 but strengthen the manipulation somehow. Maybe more training. Maybe include training on the equation to see if they can recognize how to apply it to the find amount problems. That would be more realistic. In a classroom they would probably not teach by the steps and with hold the equation.
===============

Mon. 05-17-10

Met with Sean again. Our best approach from this point would be to try contacting the two high school teachers to see what they think of doing research on one of the ideas that Sean has come up with.

How we should approach the teachers:
Email the teachers - Applying cognitive principles to actual instructional practice.
We know what works in the lab. But, is what works in the lab even of any interest to teachers?
If so, what questions would you want to see investigated?
We have some ideas, but what do you think of them, and what ideas might you have?
The ideas - For example - the Koedinger article and your ideas from it.

Mon. 05-24-10

For Summer 2010 and Fall 2010 there is the continuation of the interleaving study and the plan that Sean Snoddy is working on to do a study in a high school class room. I also will still have Steve Craig, Mara Wilde, and Dale Kappus available. Steve is willing to do a followup to CP3.

It might be good if everyone could work on followups to the interleaving study, so as to not spread myself too thin, but I want Sean to do his own original study. The guiding principle, as I've been thinking all along, should be that the studies should be both doable and of interest to the field. I also want to make sure that I'm moving in the direction of the ACME idea. Therefore, I want to make sure that the studies are of interest to teachers. Sean is working on that by contacting algebra teachers from his high school in Emmaus. Another way we are working on that is by discussing research ideas with Drs. Terry Stahler and Patricia Walsh-Coates from our Secondary Ed. department. I just emailed them to ask when we could start meeting.

Another concern I have is that I would like to get something out of all the work that I have put into the comparing problems studies. I know there has been a lot of theoretical work on that principle. But I just don't have the resources to do studies like that.

Except for the next interleaving study, I can't really pin down what I'll have everyone doing until I get feedback from teachers.

It seems like I'm working with multiple constraints:

Continue the interleaving study.
Get something out of the CP studies (have Steve to a followup since he is already familiar?).
Have Sean come up with an original idea.
Avoid difficult theoretical experiments.
Do studies in actual classrooms.
Do studies that are about applications that teachers believe are useful.
Do studies that are like the kinds of studies that are published in recent issues of specific journals so as to be able to target those journals for publication.
Do studies that could attract funding from IES.

So we need information from high school teachers, from Drs. Stahler and Walsh-Coates, from IES, and from some journals.

I just sent an email to Drs. Stahler and Walsh-Coates.
Sean is working on getting info from high school teachers.
Looks like the next thing I should do is go to the IES website and the websites of some journals.

Sean has emailed two of the teachers, Thursday last week. Two emails at least went through. Another is waiting until the inbox is not full. He will keep checking that one. One is a math teacher. Another is a Chem teacher, and that one is a friend of Sean's father so he expects to be able to reach that one. The one with the full inbox is on maternity leave and is a physics teacher.

We discussed in more detail the experiment that Sean wants to do.
Students would be trained in how to do both grounded and abstract versions of some kind of algebra problem.
They would later take the standard midterm algebra test that Emmaus uses.
What we want to know is whether practicing all abstract versions, all grounded versions, or mixtures of various proportions of grounded and abstract versions, is best for producing the highest performance on the test.

However, this amounts to a more complicated version of a simpler experiment. The simpler experiment would test whether matching vs mis-matching the type of test with the type of practice produces the same effect for abstract practice as for grounded practice. The left graph shows no interaction between type of practice and match of test. The graph on the right shows that, for grounded practice, you do better on a grounded test than an abstract test, but if your practice is on abstract, then even though you do better on the matched test, you don't do that much worse on a mismatched test.

In other words, abstract practice is better because there is more benefit on the grounded problems if your practice is abstract, than there is benefit for abstract problems if your practice is grounded.

It looks like I could end up doing the Interleaving followup, Sean's original study, and one more ACME type study (CP3 related or not).

I just heard back from Dr. Stahler and Walsh-Coates
Dr. Stahler says she is overwhelmed right now and asks me to let her get her bearings first. So I will have to try again, but I don't know how long to wait.
Dr. Walsh-Coates says she wouldn't be able to work on this until the 2011-2012 academic year, when she has had a chance to teach the lecture/practicum course, and so wants to put it on hold for now. So I should get back to her probably at the end of Fall 2010 to see how the new course is going.

Mon. 06-07-10

One thing I need to do is contact Drs. Stahler and Walsh-Coates again to see if I can get them to tell me how soon I should contact them again. That will help to prevent dropping the ball on those contacts.

Then I should look into a study that Sean could do over the summer to get his feet wet. To get subjects we would have to try to recruit from the summer classes. Unfortunately, some are on-line, one is at Reading Hospital, and one is in England. But the ones that are not are:

Summer I:
Stat - Stoffey
Personality - Rains
Abnormal - Rauenzahn

Summer II:
General - Baranczyk

Also, I might be able to use students from a summer course in MAT 017 or MAT 105

HERE'S AN IDEA FOR A QUICK AND FAIRLY EASY PROJECT THAT WE COULD PROBABLY DO OVER THE SUMMER, AND IT MEETS SEVERAL OF MY CRITERIA FOR WHAT I NEED TO BE DOING.

It is not a difficult to do experiment on a theoretical claim.
It could be done in some math classrooms (maybe with some non-math students as well in summer to boost the sample size)
It would be about an application that teachers might find useful
It is directly linked to the Koedinger, Alibali, and Nathan (2008) study in Cognitive Science
It might even be a lead in to a proposal for a grant from IES

I am skeptical of Koedinger's claim that there is a verbal advantage for single reference problems. In his Experiment 1, he only gave an example of one problem where this occurred.

The word problem version was:

Mom won some money in a lottery. She kept $64 for herself and gave each of her three sons an equal portion of the rest of it. If each son got $20.50, how much did Mom win?

The equation version was:

(X - 64) � 3 = 20.50

Koedinger found that 0.83 of participants were able to solve the word version compared to only 0.23 for the equation version.

In talking to Dr. Gebhard of our math department, she said that no way could 0.83 of her students solve the word problem without help. And she was skeptical that only 0.23 could solve the equation. I pointed out that there are also other ways to set up the equation than the way that Koedinger did it. For example, my first inclination would be to set it up as 64 + 3(20.50) = X. I think a lot more than 0.23 of participants could correctly solve that equation.

So my idea for an experiment is this:

				Factor B: Type of presentation (within subjects)
				B1:Story Problem Presentation	B2:Equation Presentation
Factor A: Type of equation (between subjects)	A1: X embedded on left	Factor C: Counter- balancing (between subjects)	C1: First set of values	Mom won some money in a lottery. She kept $64 for herself and gave each of her 3 sons an equal portion of the rest of it. If each son got $20.50, how much did Mom win?	(X - 50) � 5 = 30.25
	A1: X embedded on left	Factor C: Counter- balancing (between subjects)	C2: Second set of values	Mom won some money in a lottery. She kept $50 for herself and gave each of her 5 sons an equal portion of the rest of it. If each son got $30.25, how much did Mom win?	(X - 64) � 3 = 20.50
	A2: X isolated on right	Counter- balancing (between subjects)	C1: First set of values	Mom won some money in a lottery. She kept $64 for herself and gave each of her 3 sons an equal portion of the rest of it. If each son got $20.50, how much did Mom win?	50 + 5 (30.25) = X
	A2: X isolated on right	Counter- balancing (between subjects)	C2: Second set of values	Mom won some money in a lottery. She kept $50 for herself and gave each of her 5 sons an equal portion of the rest of it. If each son got $30.25, how much did Mom win?	64 + 3 (20.50) = X

Mon. 06-14-10

Meeting with Sean Snoddy. He gave me the materials. We edited them a bit. We did the IRB application and a script.

Wed. 06-16-10

We have IRB approval to run the Grounded vs. 2 Abstract Representations study.

The classes in which we can run in Summer I are:

Dr. Stoffey                   Stats                              MTWH      10:15 - 12:35 - OM 297
Dr. Rains                        Personality                  MTWH    10:15 - 12:20 - OM 280
Dr. Rauenzahn             Abnormal                     MTWH      8:00 - 10:05 - OM 276
Dr. Glenna Gebhard      Intro Math 017             MTWH      8:00 - 10:05 - LY 114
Dr. Amadou Guisse       Colleg Alg. 105          MTWH      10:15 - 12:20 - LY 214

Then we can check on the Summer II courses.

				Factor B: Type of presentation (within subjects)
				B1:Story Problem Presentation	B2:Equation Presentation
Factor A: Inputs (requirement)	A1: Inputs Unknown (Unwinding required)	Factor C: Counter- balancing (between subjects)	C1: First set of values	Mom won some money in a lottery. She kept $64 for herself and gave each of her 3 sons an equal portion of the rest of it. If each son got $20.50, how much did Mom win?	(X - 50) � 5 = 30.25
	A1: Inputs Unknown (Unwinding required)	Factor C: Counter- balancing (between subjects)	C2: Second set of values	Mom won some money in a lottery. She kept $50 for herself and gave each of her 5 sons an equal portion of the rest of it. If each son got $30.25, how much did Mom win?	(X - 64) � 3 = 20.50
	A2: Inputs known (No unwinding required)	Counter- balancing (between subjects)	C1: First set of values	Mom had $64. Each of her 3 sons had $20.50. How much money did they have altogether?	50 + 5 (30.25) = X
	A2: Inputs known (No unwinding required)	Counter- balancing (between subjects)	C2: Second set of values	Mom had $50. Each of her 5 sons had $30.25. How much money did they have altogether?	64 + 3 (20.50) = X

Koedinger argues that for the double reference problems, it is not reasonably possible to unwind the word problem. And, because people have a great deal of difficulty producing (as Heffernan would say) the equation for such problems, they have a great deal of difficulty with the word problem. By comparison, if the correct equation is set up for them, then they have less trouble solving the equation than they have figuring out how to set it up. That is not to say that they don't make errors on the equation, such as not following the order of operations. But such errors are not prevalent enough so that they do as poorly on solving the equation as they do on trying to set it up from the word problem.

On the single reference problems Koedinger argues that people are able to accomplish the unwinding when attempting the word problem, and that by unwinding, they do not have to solve an equation. If they are given the equation, they have trouble solving it because they make errors, just as they do on the equation for the double reference problems. Thus, Koedinger is arguing that for single reference problems it is easier to do the word problem than the equation because unwinding a word problem to avoid having to solve an equation is easier than solving the equation.

However, perhaps the reason Koedinger et al. found that people could do the single reference word problem more easily than the equation is not that unwinding is easier than solving the equation, but that people don't unwind the word problem. Instead, they may set up an "all-inputs-known" equation and then solve that. The all-inputs-unknown equation may be easier to solve than an input-unknown equation. So an experiment is needed to examine what kinds of equations people create for a single reference problem, and to examine whether all-inputs-known equations are easier to solve than input-unknown equations.

When people are presented with a single reference word problem, they might set up 64 + 3(20.50) = X. Or they might not set that up, but may mentally represent the problem that way and then solve it by just following the order of operations to evaluate the expression on the left. Even though it may appear that setting up that equation involves the process of unwinding, that may not actually be the case. If you interpret "Mom won some money in the lottery" as a phrase that should be represented symbolically as the unknown, and then interpreted the phrase "and gave" as a phrase that should be represented as subtraction, then you would begin either writing X - 64, or you would at least mentally represent the word problem that way. According to this account, you would then proceed to set up the input-unknown equation (X - 64) � 3 = 20.50. Koedinger assumes that such an equation is the equivalent equation for the word problem.

But if you interpret the first sentence as just setting the context for the problem to come, you might not think you had to translate it into a symbolic representation. Then you might start by writing (or mentally representing) 64 when you came to it in the next sentence. The next thing you come to is "and gave each of her 3 sons an equal portion of it." If your mental model started with $64 as an amount of money that Mom has, and the money she gave to her sons as more money, then you might write (or represent) 64 + "the total of the son's money". Because the next sentence tells you how much each son got, and you already know there are 3 sons, you might then represent "the total of the son's money" as 3($20.50). The last phrase is the first one that asks a question. People might interpret that question as asking them how much money is there altogether, including both Mom's and the son's money. People might interpret a question as being the part of the problem that tells them what to consider to be the unknown. Therefore, people would interpret that question as asking them for the unknown total amount of money, which they would represent as 64 + 3(20.50) = X. Therefore, they would represent the word problem as a problem in which all the inputs to the needed operations are known. This would result in no need for any unwinding (reversing operations) when solving the equation. According to this account, people are not "unwinding" in the sense of reversing operations when they interpret the word problem,

On the other hand, if you are given the equation (X - 64) � 3 = 20.50, there are two operations, division and subtraction. For the division, the output is known (20.50), and one input is known (3), but the other input is unknown. It can found by unwinding, that is, reversing the division and multiplying 3*20.50. Now, the output of the subtraction is known (61.50) and one of the inputs is known (64), but the other input is the unknown for the problem. So you can perform another step of unwinding, reversing the subtraction by adding 64 and 61.50, to get that unknown 121.50. So solving the input unknown equation does involve unwinding, which is actually harder than not unwinding at all.

Is it better to present the word version first because students will be likely to use the unwinding strategy and succeed, whereas if you present the equation first, they may make more errors? But if for single reference problems people don't unwind, they just set up an input known equation, then they are not learning anything about unwinding, which is actually part of learning to solve equations. Maybe it would be better to explicitly teach converting a single reference word problem to an input unknown equation (which people may not spontaneously do, but may be missing out on some important learning). Then you could explicitly teach that solving the equation is like unwinding the word problem.

Tues. 06-22-10

The above does not address an important issue, at least not explicitly enough. I am assuming that the all-inputs-known version of an equation is easier than the input unknown version. That is an empirical question.

Now that some of the data from the "Grounded Versus 2 Abstract" experiment is in, I have found that the all-inputs-known equation does not appear to be easier than the input unknown equation, at least for the 2 specifice equations I'm using.

First, the experiment described above was simplified to:

Type of Problem (between subjects)
Word	Input Unknown	Output Unknown
Mom won some money in a lottery. She kept $64 for herself and gave each of her 3 sons an equal portion of the rest of it. If each son got $20.50, how much did Mom win?	(X - 64) � 3 = 20.50	64 + 3 (20.50) = X

So far (I'll delay reporting actual percentages and significance tests until more data is in), performance on the word problem is numerically superior to performance on either equation. When participants in the word problem condition do set up an equation (they are all instructed to try), they all set up the all-inputs-known version shown above. No one set up the input unknown version. And, after setting up the equation, they all correctly solve it. At the same time, the participants who are given the all-inputs-know equation, but not the word problem, do not solve the equation any better than participants solve the inuput unknown equation.

The error that is almost always made on the all-inputs-known equation is to add 64 and 3 first to get 67, and then multiply 20.50 by the 67 (one subject just multiplied 20.50 times 7).

This shows that Koedinger, Alibali, and Nathan (2008) is right, and my intuitions, and those of the math professors I have talked to, are wrong. Specifically, we were skeptical that participants would perform as well on the word problem as they were reported to have done in Koedinger, Alibali, and Nathan.(2008). Also, I assumed that participants would do better on the all-inputs-known equation than on the input unknown equation.

The only place I was right was that, for those participants in the word problem condition who did set up an equation, they all set up the all-inputs-known version.

So what's going on here? One thing to note is that when the participants set up the all-inputs-known equation, they always set it up in this order:
64 + 3(20.50) = X. That is the same order in which it is presented to the participants in the all-inputs-known equation condition. Those participants then go on to almost always make the error of adding the 64 and the 3 before multiplying by the 20.50. None of the participants in the word problem condition, who set up the equation for themselves, made that error. Therefore, the semantic content of three sons receiving $20.50 each enabled the word problem participants to multiply 20.50 time just the 3, before adding just the 64.

It is notable that the error described above violates the order of operations, but follows a "left to right" strategy. Perhaps if the all-inputs-known equation had been presented as 3(20.50) + 64 = X, then the performance on the equation without the word problem would have been much better. That is what I should investigate next.

It is also possibly the case that if the word problem had been re-worded so that the 3 sons receiving $20.50 each were mentioned before mentioning that the mother also kept $64 for herself, then the participants may have set up the equation as 3(20.50) + 64 = X. However, as noted above, the word problem participants do perfectly well with the equation set up as 64 + 3(20.50) = X, when they set it up themselves from the word problem.

If it turns out that the ordering of the equation affects performance, then the practical problem is how to best remediate that. A question that arises is why would students make such an error? Do they not realize that the order of operations applies to the all-inputs-known equation? After all, the order of operations applies to simplifying expression specifically, not to solving equations generally. I need to find out from the math professors what students are usually taught about ordering their steps when they are solving equations. I think that when you are trying to isolate an unknown in an equation, you are supposed to follow the reverse of the order of operations. But, it's been so long since I was taught that, and I feel like I solve equations rather intuitively, that is, without thinking about the ordering of the steps when you are isolating the unknown, that I'm not sure if I'm right. Maybe it's not as simple as reversing the order of operations for simplifying an expression.

So, one way students could make the error of adding 64 and 3 first, is that they don't realize that the order of operations does apply once the unknown is isolated and all you have left to do is simplify the expression on the other side of the equals sign. But another way is that they may know that they should use the order of operation, but they don't remember it correctly. On the one hand, how you remediate the problem might be best seen as dependent on which is the actual reason for it. But on the other hand, math teachers might say that the reason for the error is no doubt different for different students. Therefore, they might think that it would be better just to try to remediate both.

Here is where I need input from math teachers to determine what would be a good research question. Do they think it's important to know specifically which of two possible reasons lie behind a particular kind of error? Do they think it would add efficiency to teaching if, for example, you could diagnose which misunderstanding caused that error for a given student, and then have a way to remediate it that has been shown to work? Or would they rather just have a known way to remediate both possible sources of the error and use both as part of the initial teaching of how to solve equations.

As a cognitive psychologist, I think it would be important to know whether presenting an equation with the elements arranged so that the order of operations follows a left to right sequence makes it easier for students to solve an equation. But as a math teacher, you might say that for more complex equations, it might not be possible to set it up that way. Or they might say that students simply need to be able to solve an equation whether or not it is set up that way. If so, does it help us to know whether or not the error I have been seeing is really the result of an incorrect left to right strategy?

Nervertheless, as a cognitive psychologist I might like to know how students develop their concepts of "equation" versus "expression". However, that might be the kind of question that would be hard to delve into at a teaching, rather than a research, institution.

Perhaps a better question would be something like this. Would it help students to learn, retain, and know when and how to apply the order of operations if we gave students "in order" equations first, and then showed them explicitly the correct order of steps is only left to right because the left to right order just happens to follow the order of operations? Would that be useful for helping them form the correct concept of order of operations?

Also, what if we taught the concept of higher order and lower order operations explicitly. Then we could point out that part of the rational for the order of operations is to do the higher order operations before the lower order ones. You would have to also point out that grouping symbols have their own significance. Would it be better to say that they create, in effect, an even higher order operation? Or would it be better to say that they are used when the nature of the problem requires that you do a lower order operation before a higher order operation?
Is it better to try to present such a rationale for why the order of operations is what it is? Does that make the knowledge more interconnected with other knowledge, which is supposed to enhance learning? Or would such a rationale be distracting and hurt learning because students wouldn't understand it's relevance? Perhaps there would be a U shaped learning effect here. Perhaps presenting the rationale would initially be unconnected to anything else the student knows and would actually interfere with learning the order of operations itself. But later, the idea of higher and lower order relations might apply to other things the student was learning in similar ways that it applies to the order of operations, resulting in that other understanding of the importance of higher and lower order relations reinforcing the student's understanding of the order of operations.

To try to boil all this down, I'd like to first do a simple comparison of the "in order" (3(20.50) + 64 = X) and "out of order" 64 + 3(20.50)) versions of an equation. Then, assuming I find that the "in order" is easier, I'd like to use that knowledge to develop a method to remediate the misunderstanding that equations should always be solved left to right and do the research to test it. I'm afraid that the math teachers might say, just teach them the acronym PEMDAS, but then they have to remember it and remember when to apply it. Is that a problem? Or is that a source for more good research questions?

I guess I have to talk to the math teachers to find out. I guess I should also run this by Koedinger to get his advice as well.

Another, larger, question that might lead to some studies is "Why do single reference problems provide enough semantic support to afford unwinding, whereas double reference problems do not provide enough semantic support to afford untangling?"

This quote is from Anderson, J.R., Reder, L.M., & Simon, H.A. (non-published) Applications and Misapplications of Cognitive Psychology to Mathematics Education. Retrieved from http://act-r.psy.cmu.edu/papers/misapplied.html, 06/23/10. It is among the white papers about the Carnegie Learning Tutor.
"The amount of transfer appeared to depend in large part on where the attention of subjects was directed during the experiment, which suggests that instruction and training on the cues that signal the relevance of an available skill might well deserve more emphasis than they now typically receive--a promising topic for cognitive research with very important educational implications."

Wed. 06-23-10

Here are the results for 58 participants in the Grounded vs. 2 abstract representations experiment.

Results

In the word problem condition, 19 of the 20 participants (95%) were correct, compared to only 7 of the 19 input unknown participants (37%) and 9 of the 19 output unknown participants (47%), X² (2, n=58) = 15.76, p = .0004. The word problem performance was superior to both the input unknown performance, X² (1, n=39) = 14.83, p = .0001 and the output unknown performance X² (1, n=39) = 10.92, p = .001. There was no difference in performance between the input unknown and output unknown conditions, Fisher exact test (two tailed) p = 1.0. In the word problem condition, exactly half of the 20 participants produced no equation, but 9 out of 10 of them correctly solved the problem. Among the other half, who did produce an equation, all of them produced the output unknown equation, and all of them correctly solved it.

Wed. 06-28-10

Going back to Sean's idea for an experiment. He would do something like the following:

Training (Type of examples)		Test
Conditions	All grounded	Emmaus's standard midterm algebra test Some grounded Some abstract Total score
	75% grounded 25 % abstract
	50% grounded 50% abstract
	25% grounded 75% abstract
	All abstract

Sean's hypothesis is that training on a mixture of grounded and abstract would lead to the best total score. Specifically, he thinks that the 25% grounded, 75% abstract would produce the best total score. His reasoning is that abstract training is generally better because it trains student in the principles, but that they need some concrete examples as illustrations of how the principles work.

So here are the questions we need to answer:

What do math teachers think of this idea? Is it a question worth pursuing?
We need to get the materials. We would need the algebra test, and textbook items for training examples.
What about the proposed mechanism? What would be alternative possible mechanisms? What different predictions do they make that we could test?

However, in talking to Sean, he now is not as sure that training with the grounded representation would be that helpful on the abstract (because, in our Grounded vs 2 abs study, our subjects did so surprisingly well on the word problem, but so surprisingly poorly on the equation).

So he said that it comes down to more to a question of the effect of training with a word problem on doing another word problem, vs. an equation, and the effect of training with an equation on doing another equation, vs. a word problem. In other words, it goes back to the simpler experiment that I illustrated above.

However, he also is interested in people's ability to make connections between the equation and the word problem. That reminded me that making connections between one problem and another was somewhat the focus of my dissertation work. However, in my dissertation, the connections were between two members of a pair of word problems, and their effect on an new word problem, as well as on the ability to generate an equation.

So, I'll give Sean some of my work from my disseration to read. I gave him the Cognitive Technology paper, and the version of the Mechanical Feature Matching paper that was rejected by JMRE. I also gave him a copy of the huge ms for the 3 experiments that was abandoned by Jonathan and me.

I told him to read the Cog Tech paper first, because that presents my first dissertation experiment, and thne the JMRE paper, because that presents the second one. I told him he can read the 3 experiment paper later if he wants to.

The idea will be for him to use those experiments to come up with an idea for an experiment. One possibility would be a version of the simple experiment --- the effects of training with one kind of representation (a word problem or an equation ) on performance on the other. Another possibility would be to examine the effects of different methods of training people to make the connections between the word problems and the equations on performance on both types of representations.

=====================

Now, going back to the results of the grounded vs. 2 abstract experiment: I have a proposed explanation for many of the errors - the left to right strategy. Let's see if I can think of a simple test of whether that explanation holds.

		Placement of Unknown
		Input Unknown	Output Unknown
Congruency of Left to Right Strategy With Order of Operators	Left to Right Incongruent with Order of Operations	(X - 64) � 3 = 20.50	64 + 3 (20.50) = X
	Left to Right Congruent with Order of Operations	(X - 64) = 20.50 3	3 (20.50) + 64 = X

Mon. 07-12-10

The above experiment will be done as a compete within subjects design with multiple instances of each of the four types of equation. The instances will be created by simply using different values. The values are illustrated below with the Output unknown, congruent equation:

3 (20.50) + 64 = X
4 (31.10) + 42 = X
2 (15.20) + 33 = X
5 (10.30) + 25 = X

The 16 resulting equations will be presented in two blocks. One block will be the 8 incongruent equations, randomly ordered. The other block will be the 8 congruent equations, randomly ordered. Which block occurs first will be counterbalanced accross subjects.

The reason for the counterbalancing is that I suspect that doing the congruent equations may facilitate doing the incongruent ones, but not the reverse. I don't expect to see any effect of placement of the unknown. However, I do expect to see an advantage for the congruent condition over the incongruent condition, although this may occur mostly for the incongruent first subjects.

THE PROBLEM HERE IS THAT THE SUBJECT WOULD BE COMING UP WITH THE SAME ANSWER 4 TIMES. THAT WOULD INTRODUCE THE PROBLEM OF PRACTICE EFFECTS. THE ONLY SOLUTION WOULD BE TO USE 16 DIFFERENT SETS OF VALUES AND RANDOMLY ASSIGN VALUES TO EQUATION TYPES FOR EACH SUBJECT.

BASICALLY EVERY SUBJECT'S MATERIALS WOULD BE CREATED SEPARATELY. THAT COULD BE DONE, BUT IT WOULD BE TREMENDOUSLY LABOR CONSUMING. WE COULD GET JUST AS MUCH INFORMATION BY JUST USING A TWO GROUP DESIGN AGAIN. JUST GIVE ONE GROUP 64 + 3 (20.50) = X, AND THE OTHER GROUP 3 (20.50) + 64 = X.

THAT WOULD BE A MUCH CLEANER DESIGN AND WOULD BE MUCH LESS LABOR INTENSIVE.

Using the more simplified design, we could run the subjects more quickly, which will probably give us more subjects. We could try recruiting from the following classes.

Dr. Stoffey I/O MTWH 10:15 - 12:20 - OM 283
Dr. Baranczyk Intro MTWH 8:00 - 10:05 - OM 278

I'll also contact the Math professors

Sean, starting the 27th, will be working 9pm to 6am.
Some time will have a week of 4am to 12:30pm.
Normally he would have to leave here at 1pm to get to work. Wednesdays he is normally off.

Regarding Sean's idea for an experiment: He read the Cog Tech similarity judgment paper. He said that in his study he is interested in how people learn the abstract structure (what was accomplished by my similarity judgment task), but that he would be interested in the effects of training with different types of materials (grounded/word problems vs abstract/equations) rather than the effects of training with different tasks.

I asked him if he has devised any materials yet. But he is still stuck waiting for his teacher to get back to him to enable him to get the realistic classroom materials.

I suggested that we might want to also consider the effects of training with BOTH types of materials, but manipulating something like whether the teacher explained the relationship between the word problem and the equations, or whether the subjects were trained to self-explain and had them self-explain the relationships. I also mentioned that we would have to control for amount of training and time on task by having the non-self-explaining subjects receive more materials than the self-explaining subjects.

It occurs to me that there may be some literature out there on effects of self-explanation on math learning.

Mon. 07-19-10

Met with Sean. Showed him the Koedinger email. We talked about the last idea - what effect does doing the word problem have on doing the equation. Sean also thought that we could look at what effect doing the equation has on doing the word problem.

In order to have a proper control, I suggested that if we give one group of subjects the word problem followed by the equation, we should have the other group given some task similar to doing a word problem, before doing the equation. If we have the other group do the equation first, followed by the word problem, then we have a problem of a ceiling effect on the word problem. In other words, I would expect that people would always do well on the word problem. Therefore, doing the word problem after the equation would be expected to still lead to high performance on the word problem, but that would not mean that doing the equation improved performance on the word problem.

Of course, we could probably pretty easily find word problems on which people performed poorly. Then, we could investigate whether doing the equation first would improve performance on the word problem. I would think that it would not. The reason the lottery word problem might improve performance on an equation is because of the semantic support provided by the word problem. But I would not expect doing an equation in advance - even if it was one that subjects could do well on - to have any benefits that would transfer to doing a difficult word problem. Remember, the difficulty of difficult word problems stems from that "grammar of algebra" problem as well as from the problem of "comprehension", that is - translating the words into the CORRECT algebraic expression.

So, alleviating the difficulty of difficult word problems might benefit more from a more direct instruction approach. BUT, direct instruction including something like self-explanation, spacing, interleaving, and the like.

We also talked about the effect of doing the lottery word problem on something like Hewitt's verification task.

Sean took with him a copy of the schedule for running subjects this week. He is working that graveyard shift this week. So he just needs to look at the subject running schedule and figure out when he would work in sleeping in order to decide when he could help with running the subjects. He will email me back.

Mon. 08-02-10

Sean tried to do a forward search on Koedinger, Alibali, and Nathan, 2008 (Trade offs ... the one with the materials for my Summer 2010 studies). He says he found one article that might be relevant. But the article was Kirshner (1989) "The Visual Syntax of Algebra".

Mon. 08-16-10

Sean got an email from the teacher from his high school district, Margaret Hoffert. She has Algebra I and Algebra III/Trig. Usually 20 - 25 students per class. Sean would try to use her classes for his study.

The design would be to give training with homework problems - different ratios of abstract (equation examples form the text they use) and grounded (word problems from the text they use) - and use the usual classroom materials or our own test as the dependent measure. We would see if she would teach a unit on something that would not be in the curriculum. Or at least not at that time. That way we could use the test as our dependent measure without having the test affect the students' course grade. Or we could even select some topic that she would be teaching later, and that she knows is an especially hard topic. That way, our experiment wouldn't affect their grade at the time, but would help as prior exposure to some material that they would have later.

Sean would wait until the teacher is back - Early September - Sean will contact her to get the materials. He is done at 11:00am on Fridays. He could be there at 11:30 - 12:00 to see her. They get done at about 2:30.