Using ChemSkill Builder to Improve Student Performance in General Chemistry

Name: J. J. Breen & P. T. Czech
Dept/Program: Chem/Biochem

Start Date: July 1, 2003
End Date: July 1, 2004

Project Type: Collaborative

Collaborators: C. Baer, K. Overly, and J. Pike

Original Proposal

Briefly summarize the scope of your project. What were the results? Accomplishments? Did you accomplish as much or more than anticipated?

During the 2003-2004 academic year all students (approximately 140) in General/Intro Chemistry sections were enrolled in ChemSkill Builder (CSB), an online teaching tool for General Chemistry supported by the McGraw Hill Companies. During the fall semester the student’s recorded scores on CSB exercises were a component contributing to final grade. In the spring '04 semester, for roughly half the students, CSB points contributed to the final grade; for the other half, CSB was purely optional.

In the following table the percentage of students receiving less than satisfactory grades are listed for the past five years. While the percentages for the past year are near the 5-year lows, they are well within the range displayed by these numbers. Data from three or four additional years will be required to more accurately assess the effectiveness of including the CSB in these courses.

 

 

Overall we feel that the project should be termed a success. When viewing the survey results found later in this report, students were quite positive about using the CSB despite a significant number of “bugs” in the program and the troubles with the internet experienced by all of us on the Providence College campus.

Additionally, the faculty involved with this project and teaching General Chemistry have agreed to continue using the CSB next year.

 

Please describe any unanticipated issues, challenges or barriers that you encountered during your project. How were these resolved?

One collaborator chose to not use the student results from the CSB as a part of their grade for the second semester. His rationale was that he did not traditionally include homework results as a component of the class grade during the second semester and was not motivated to deviate from this policy. From the responses on the final survey in this instructor’s sections, it appears that only 12 students out of the 34 students finishing the course used CSB regularly. The responses of these 12 students are tallied separately in the survey results.

 

Please highlight how you assessed the success of your project, including its impact on student learning?

As we described in our original proposal, our main assessment instruments were student satisfaction surveys conducted at the end of each semester. The results of our first survey were submitted with our December report and the results of our second survey are included with this report. Overall the results from both surveys are rather similar with the most significant differences coming from the students for whom participation was optional in the second semester. In terms of impact on student learning we remain very focused on reducing the fraction of students in these courses receiving less than satisfactory grades (WD, F, D-, D, and D+)

 

To what extent was your formal assessment process useful in evaluating the benefits of their project?

We were especially interested in the results of Questions #6 and #7. To the question “I learned a great deal for CSB assignments?”, over 70% of the students agreed or strongly agreed with the statement. To the question “I prefer CSB assignments to graded homework assignments”, over 80% of the students agreed or strongly agreed with the statement. Going into this project we suspected that our students would likely prefer a web-based approach to homework assignments, however, it was uncertain whether or not they would actually value the experience. It appears from our results that a clear majority of this particular group of students felt that CSB was an effective tool for learning general chemistry material.

 

In retrospect is there anything you would have done differently in designing or implementing your project?

In response to the numerous complaints concerning “bugs” in the program and trouble with the campus internet, we are considering an alternative strategy for scoring the skill builder. Since the student frustrations are mainly associated with trying to achieve the maximum possible points on each assignment the following scale should alleviate many of their worries.

 

Have your professional skills advanced as a result of this project?

No

 

What, if any, future plans do you have for the use of instructional technology in your courses?

We will continue to require the students to enroll in the CSB and for the most part continue include their CSB scores as a component of their final course grades.

 

Please indicate any on or off-campus presentations or publications you have made or plan to make as a result of your project.

We will continue to survey the students and monitor the effectiveness of using the CSB in General Chemistry. We expect that after a few years of data gathering we will be in a position to submit a short paper to the Journal of Chemical Education detailing the effectiveness of the CSB in reducing the fraction of students receiving a less than satisfactory grade in General Chemistry.

 

Is there anything else we should know about your project?

We would like to thank both the Instructional Technology Development Program and the Faculty Technology Project Review Committee for their support of this project.

 

---

 

Results for the Spring 2004 Chemistry SkillBuilder Student Survey

Based on 45 responses from the two classes where CSB counted towards the student’s grade and 12 responses for the two classes (34 students total) where CSB was not counted towards the student’s grade

  [SA = 5, A = 4, N = 3, D = 2, SD = 1]

The following are the student responses to a question about any problems they consistently had with the ChemSkill Builder.

1) Sometimes when a right answer was entered, ChemSkill says it was wrong.

2) Sometimes it did not save, but otherwise it was okay.

3) Spelling errors mark it wrong, getting logged out of the site after a certain time.

4) Sometimes we would be doing sections that we hadn’t covered in lectures, yet which would present a problem.

5) Sometimes it marked correct answers as incorrect and notations were often a pain.

6) Subscript and superscript; it sometimes marks answers wrong when they are right.

7) Problems saving scores, many times the page would say please wait but never load and my scores for that day would be lost. Also, sometimes when the right answer was put in, it was declared wrong.

8) Sometimes things weren’t mentioned in lecture.

9) The website often froze in the middle of exercises and you would have to do the whole section over again.

10) I had trouble with some of the notation. I often had correct answers that were wrong because of how they were typed in.

11) Sigfigs, and the ^ or \ (to make the things subscript or superscript) sometimes they weren’t right.

12) It doesn’t always work properly.

13) Not saving scores (often) resulting in me having to do the section again.

14) The issue of exponents was a constant distraction and annoyance.

15) - if sign for charge was on the wrong side of the #, the whole prob was wrong; - would sign off and not save score of completed assignment

16) In certain sections I would put in the right answer and it would keep telling me that it was incorrect.

17) ChemSkill Builder occasionally had wrong answers that it thought were right and vice versa. Also, some sections seemed to make it impossible to get a 100 or because of some internal error with the program.

18) Some assignments would not save correctly and had to be done several times before a grade was reported. Occasionally, the significant figure rules used by the program were inconsistent.

19) There were many times when it did note record the score or it marked questions wrong that were right. That was extremely frustrating.

20) There were a couple of times when I lost points even though I put in the right answer.

21) I had problems in a few areas, I had one section that would not load correctly. For instance 5-3 would load the problems for 5-1 and give me 5-3 credit for redoing the section. Also, a lot of times if I typed the same answer but in a different way then the program it would call it wrong.

22) Sometimes an answer would be put in correctly and the computer would not read it.

23) It would sometimes call a correct answer wrong and the superscript/subscript they were a pain.

24) Computer glitches - whole problem wrong for minor error.

25) Some software problems; they were not always consistent.

26) It would mark my answer wrong when it’s right sometimes.

27) They didn’t always do sig.figs. Sometimes right answers were marked wrong. Some provided answers were vague.

28) Just some technical difficulties - not saving scores.

29) Sometimes it was too picky with the answers: e.g. rounding off and so.

30) Sig figs seem very unclear sometimes.

31) ChemSkill Builder was very sensitive to carats (^) and slash keys (/) that it was a nuisance. You would get the answer right, but you wouldn’t because of the program’s structure.

32) The scores wouldn’t save sometimes. I had to do a section 2 or three times before it would save my score.

33) Basic programming problems: answers would be accepted, but several times it would not store my grades. I don’t know if it was my computer though.

34) Many times I would enter the correct answer and it would say it was wrong, and then display the same answer I wrote. The exercises were very helpful, however they were extremely frustrating. I would get to the point when I wasn’t learning anything, just memorizing. I would prefer not working on a computer.

35) No problems.

36) Significant figures and putting in the right answer and then still having it say it was wrong.

37) I wish we had more time to do them. We were given usually 2 or 3 days and sometimes with other work it was frustrating to spend so long on Skill Builder.

38) Sometimes they used the incorrect number of significant figures and it impacted the # of times I had to attempt the problem.

39) Sometimes even if you have the right answer, it says you’re wrong because you didn’t type it exactly the way it wanted.

The following are the student responses to a call for any other comments they wished to share regarding ChemSkill Builder.

1) I really like it as a complement to lecture and the book.

2) I enjoy the fact that the Skill Builder will show you how to do the problem if you get an answer wrong.

3) Overall I think it is helpful. It helped me to prepare for the tests.

4) Good tool but it needs some work and analysis from the professors.

5) There were some sections we did that were not covered in class.

6) Sunday at 5 seemed like a hard time to remember to complete the work by.

7) I liked that you could repeat a section until you got 100, but it got very frustrating because you had to do EVERY SINGLE Problem over again in order to raise your score. If you mess up a written homework assignment, you can just fix that one. I found I wasted a lot of time doing the same problems many times... problems I had gotten right and understood the first time.

8) My only comment would be in regards to the problems I had (notation). Maybe a trouble shooting sheet of some kind would help if that is at all possible. Otherwise I found it to be very effective.

9) I like Skill Builder. The program itself needs a little polishing, but it’s a good tool. I prefer it over graded assignments.

10) Some assignments were very easy while some were very hard, not many were in the middle.

11) They were better at teaching you the material than book work because it showed you how to do the problem if you got it wrong.

12) Besides the few glitches I felt it was a great help.

13) I think it’s a good idea to use ChemSkill Builder for the class because it allows the student to receive instant feedback for his/her work. Overall, I would say it was a very useful tool.

14) I feel that it is a beneficial exercise. It gives good feedback about how to answer problems that were done incorrectly and is helpful in general.

15) Frustrating; can’t go back and change.

16) I like it.

17) I like it. I think it helps to comprehend the material and the repetition of repeated problems helps to solidify the material.

18) They really helped because they were good for practice problems.

19) They were helpful in the way to help one understand. But it took a long time plus frustrate people at times.

20) I think it is a good idea, but it is aggravating to do them because you have to keep doing them over and over. Also, sometimes you could do them well and not really learn anything because you would just memorize how the problems were done.

21) Good for the first half of the year - not for the second.

22) CSB was very helpful. It made the text info much more clear and explained where you were wrong in a problem.

23) It helps to understand the concepts but is sometimes confusing when comparing it to problems on quizzes and tests.

24) I would have preferred graded homework assignments from the book.

25) Great for studying - helped me a lot because when you get it wrong it shows you what you should have done instead of just giving the answer.