Information technology (IT) tools in education have primarily been intended to serve two distinct goals, first expanding access to educational opportunities for learners who would not otherwise have them, and second, improving the quality and effectiveness of teaching and learning for all. The Digital Planet 2006 report from the World Information Technology and Services Alliance (http://www.witsa.org/) states as follows: Indeed, despite the expected peaks and valleys, countries around the globe are finding ICT the indispensable technology for increasing productivity, raising the standard of living, delivering greater educational opportunities, improving healthcare and human services and eliminating barriers to greater participation in world markets. [italics added by author]
Unfortunately, the evidence that IT tools are 'delivering greater educational opportunities” around the globe extends primarily to the meaning of 'greater” as 'extended” or 'more widespread.” Obviously, 'greater” can also mean 'better” or 'more effective,” but the evidence for the impact of IT tools in attaining this second goal is sorely lacking. The bottom line is that the vast bulk of the research and evaluation studies conducted to date indicate that IT tools can be used successfully to extend educational opportunities to otherwise disenfranchised learners, but the dream of enhancing the quality or effectiveness for all with these same IT tools remains illusive. One plausible reason for the lack of impact with respect to enhancing the quality of instruction and improving learning outcomes is that there has not been a sufficient investment in sound evaluation strategies when IT tools are being developed in educational contexts. The purpose of this chapter is to describe the state of the art of evaluation strategies for IT tools in education, with a special focus on mixed methods used in service of the design and development of educational software.