Showing posts with label quasi-experimental research. Show all posts
Showing posts with label quasi-experimental research. Show all posts

Oct 11, 2011

How can researchers' measure students' motivation, volition and attitude toward technology?

Or we can re-phase that sentence to: how can researchers measure students' motivation, volition and attitude of a technology-integrated course?

The article Sarah provided is very interesting. To me, it is obviously a very "learning science" article. It followed a very traditional quantitative research design. It is quasi-experimental, which means the participants are not randomly sampled, but researchers randomly assigned the participants to two treatment groups and also randomly assigned the teachers to different treatment groups. Independent variable is whether the group receive MVEM or not. There are 4 dependent variables to be measured: students' motivation, students' volition, students' attitude toward technology, and students' performance score. One-way ANOVA is used to determine if variance of homogeneity is fulfilled during the pre-test and if there is are significant results of difference two treatment groups in regards of four different dependent variables.  A abbreviated survey was designed to apply in the middle of the courses to measure the motivation and volition levels of individual student in order to provide appropriate instructional message via email.

Unfortunately the results of students' motivation and performance didn't meet the researchers' expectation that had significant different after the MVEM, but it has the positive effect on students' volition and attitude.   The researchers explained that could be attribute to students already had high motivation from the beginning of the class. The authors also  estimated that students will spend 30 mins. to read the message they send,  optimistically assumed that the reading time will be increased by 2.5 hrs if it is a semester-long study. How did they estimate the time? Will students' motivation, volition and attitude be increased linearly regarding time increases?

Another interesting point that brought by the researcher in the implication section is they way they encourage practitioners (online instructor)  to apply MVEM to enhance students' motivation, volition and attitude toward technology integration. They stated:
"An outside observer might note that MVEM seem promising but simply are not feasible due to the time it would take to create customized MVEM for each student. However, it would not necessarily require a great time commitment on the part of instructors. Audience analysis is something that all good instructors do anyway" (p.107).
They even proposed to develop an automated system to recognize the volitional and motivational patterns of students. I strongly disagree the way they want to interact with students based on my personal teaching philosophy. I don't think students' reflection can be easily measured and analyzed by the computer. To provide appropriate feedbacks and formative assessment has always been a job of instructors and it is always time-consuming. Nonetheless, the research was framed by positivism paradigm. To me, it is cognitivist even behaviorist, which means, everything can be measured and controlled. Students' learning can be stimulated by instructional messages that researchers provided. You give the message, they received then they should react with proper behavior. It's a little pity they didn't collect any qualitative data, such as student' essay, which may become evidence to help triangulate their results.

At the end of the reflection, I just want to mention ARCS model, which I believe I've read it in Reiser's book. However, the book is in my boxes on the way to Houston so I don't have a chance to check it out. Here I attach a diagram from the Internet, which I think will explain the model more explicitly by dividing it into several steps:

Sep 27, 2011

A Bad Example that Quantitative Research Could Give...

http://www.classroom.umn.edu/projects/ALCOverview.html


I recently read an article talks about the impact of formal learning space on student learning. The author conducted a quasi-experimental research and seek to understand the relationship between formal learning spaces and student learning outcomes.

Here's how the author did his research. He wanted to compare two datasets of students' leaning outcome in two different learning spaces--one is a traditional classroom setting: blackboard in the front, students sitting in rows...etc; another is so-called Active Learning Classroom (ALC), which offers big round tables, switchable laptop technology for presentation, and wall-round marker board...etc. Except for different learning spaces, all other variables were controlled. However, the samples were from students who voluntarily enrolled the course and is impossible being random, this research cannot be categorized as experimental design.

The author found out one of the variable cannot be controlled because of the convenient sample set: students' academic achievement. In order to gain more credible results, the researcher collected students' ACT score, and claimed the score "have been demonstrated to be reliable and valid predictors of grades, especially among first-year college students." The ACT scores showed that students in traditional classroom had higher average of ACT than the students in ALC before they enter the college, statistical method was used by the author to standardized the score, and predicted that under all controlling variables, students in traditional classrooms should will gain higher average score in the end of the semester than students in the ALC classrooms.

The results shows that students in traditional classroom did gain higher score, but not as high as predicted, and the difference is statistic significant. Thus the author concluded that the impact in different learning space did exist and influence student learning.

That's it! Probably the weakest research I've ever read in recent years. The author only collected two sets of data, used t-test and multiple regression to analyze the data, then got the statistic significant result. Regardless the weak connection of the literature review session and the research design, what can this result actually tell us? We are always talking about statistical significant and practical significant in our stats classes. Although there is nothing wrong with the statistics in the study, as a reader, I cannot be persuaded there is a real difference because students' average score in the traditional classroom is only "3 points short" than the research estimate. Ironically, the students in ALC classroom actually earned less average score than the traditional one. The researcher manipulate the simple data and sought to present it in multiple fancy way to show the how much the "difference" the ALC score increases.

What do I expect to see in the topic related to the different learning space? I want to see the how innovate facilitation differ from traditional classroom can impact students' learning. Their learning achievement, can only be shown in their final scores, but different assignments or classroom activities they were engaged in. I want to see the patterns of students interaction in different learning spaces. I want to see how students and teachers percept in a different learning environments. From neither the results nor research design in this study, I am not informed any characteristic that is unique in an innovative learning space. The results are meaningless to me.

It could become more meaningful if this research only a pilot study for other research. Moreover, if the research design was duplicated and conducted at least three time and got the similar results, I will be more convinced.