ABSTRACT

During 2010, 2011 and 2012, approximately 20 colleges took part in developing and improving a nationally normed standard test for the core elements of the IS Model Curriculum published in 2010.  This case study looks at one of those colleges and presents how they have used their performance data for ongoing accreditation purposes.  The Information Systems Department at California State University,  Los Angeles (CSULA), submitted a standalone University Program Review Self Study and an AACSB 5-Year Maintenance Report integrated into the college’s AACSB report. Both groups commented on the quality of the department’s reports and assessments primarily because of their five years of IS Model Curriculum (ISA) examination assessment data.  This paper presents the accreditation and assessment methodologies utilized and the complexity arising from the different goals and objectives of the business college and the information systems department.

Key Words: assessment, accreditation, IS Model Curriculum, ICCP, program review, normed testing

1. OVERVIEW

The following assessment requirements are quoted from the three accreditation agencies which are impacting Information Systems (IS) programs.

INSTITUTION LEVEL: Regional Accreditation Organizations

 -   WASC

§          “SLOs and expectations for student learning at all levels; reflected in policies, advising, information resources, etc.

§          Systematic program review includes SLOs, data on retention and graduation; includes reference to external evidence and evaluators

§          Quality assurance processes; assessment and tracking; comparative data; use of results to revise/improve”

PROGRAM LEVEL: National Accreditation Organizations

–    AACSB

1. What will our students learn in our program? What are our expectations?

2. How will they learn it?

3. How will we know they have learned it or not?

4. What will we do if they have not learned it? “

–    ABET

§      ”The program must regularly use appropriate, documented processes for assessing and evaluating the extent to which both the program educational objectives and the student outcomes are being attained.

§      The results of these evaluations must be systematically utilized as input for the continuous improvement of the program.”

All Business Schools and the IS Programs must meet or exceed relevant requirements if they hope to maintain a quality accredited program and attract students in a competitive environment.

The IS department at CSULA offers a BS CIS degree and a MSIS degree that closely follows the IS Model Curriculum from AIS and ACM. The Center for Computing Education Research (CCER) offers nationally normed examinations that are direct measures of the IS Model curriculum for 2002, 2010 and MSIS 2006 curriculums. The CSULA department for IS has approximately 300 majors and twelve faculty members and is a small department compared to the college’s 4,000 business students and 70 business faculties. 

The complexity of ACCSB accreditation for an IS department in a business school rises from several factors including

·        differing AACSB and ABET IS accreditation goals and objectives,

·        differing AACSB and ABET IS curriculum and course structures.

·        differing AACSB, ABET IS, and university program reviews and required outlines, and

·        differing college and IS department outcome assessment goals and objectives.

·        changing faculty over assessment periods and thus varied understanding of the needs and processes.

The expense and difficulty of consistent reporting takes staff away from important research and teaching assignments to focus on institutional reporting and administration.  A consistent model that reduces effort and provides for changing faculty should be desired by all IS programs. This paper presents a well tested and successful model. 

The IS assessment complexity is substantially reduced since the CCER sponsored IS Model Curriculum exam results are reported by both IS 2010 courses and student learning outcomes and ABET IS learning outcome objective structures.  Research evidence by Snyder (2010) also supports the use of the IS Model Curriculum exam. His research “compares ISA exam scores earned by students in one particular Computer Information Systems program with scores earned by the same students on the ETS Major Field Test (MFT) and shows that the IS Model Curriculum Exam appears to measure knowledge retention in the same manner as the MFT. Since the MFT is recommended as a valid program assessment tool, the ISA Exam should be similarly recommended.”

The AACSB provides a detailed outline for the Maintenance Report (AACSB, 2012).  Responding to the requirements in the standards and the maintenance report outline generally involves two levels of committees.

·      College maintenance committee (a member  from each department/program)

·      Department/program maintenance committees (2-3 members each)

o     Responsible for annual reports and 5^th year maintenance report

·      College outcomes assessment committee (a member  from each department/program)

o     Department/program outcomes assessment committees (2-3 members each)

o     Responsible for annual reports and 5^th year maintenance report

·      College coordinator of accreditation activities (an almost full-time position)

Specialized data base software is essential to maintain the required accreditation information and improve the efficiency of the committees.   CSULA uses Digital Measures software.

The college committees are primarily responsible for the following sections of the outline,

1. Situational Analysis

2. Progress Update on Concerns from Previous Review

3. Strategic Management - College Policy Level Only

4.  Participants

5. Assurance of Learning- College Policy Level Only

6. Other Material

The department/program level committees are primarily responsible for the following sections of the outline.

3a. Department/Program level mission, goals, objectives and strategic plan

5b. Department/Program level outcomes assessment methods, measurements and feedback

Since the Information Systems degrees or programs involve primarily items 3a and 5b, the scope of this paper will discuss those topics.

2. IS DEPARTMENT MISSION

The department’s mission is based on AACSB and WASC requirements and must be a compatible subset of the university and college mission statements.  The CSULA statements follow.

CSULA Mission

“Cal State L.A., a member of the California State University (CSU) system, offers excellent and innovative educational opportunities to an urban student population that reflects the diversity of the Los Angeles basin.”

College Mission

“The College of Business and Economics central mission is to be the educational gateway through which students of diverse backgrounds emerge as graduates having the abilities and sensitivities needed to be highly competitive in local, regional, and global markets.”

Department Mission

“The mission of the Department of Information Systems is to educate and train a diverse population of students for leadership roles in Information Technology (IT) by focusing on technical competency, managerial and interpersonal skills, effective and ethical decision making, and long-term professional growth in a dynamic field that is globally oriented.”

AACSB requires that the entire self-study be derived from and compatible with the mission statements.  Extreme care must be used to assure this compatibility. The department mission should be revised following each five year regional accreditation cycle discussion, and approved by the entire faculty.

3. IS DEPARTMENT GOALS AND OBJECTIVES

Goals and objectives implement the IS/IT mission statement.  There are two sets of industry standard goals and objectives available, from the IS Model Curriculums (AIS & ACM) for 2002 and 2010 and from ABET the IS requirement for program educational objectives (PEOs) and student outcomes (SOs) – student learning outcomes and post-graduation employment outcomes.

ABET Student Outcomes (called objectives by AACSB)

(a) An ability to apply knowledge of computing and mathematics appropriate to the discipline

(b) An ability to analyze a problem, and identify and define the computing requirements appropriate to its solution

(c) An ability to design, implement, and evaluate a computer-based system, process, component, or program to meet desired needs

(d) An ability to function effectively on teams to accomplish a common goal

(e) An understanding of professional, ethical, legal, security and social issues and responsibilities

(f) An ability to communicate effectively with a range of audiences

(g) An ability to analyze the local and global impact of computing on individuals, organizations, and society

(h) An understanding of processes that support the delivery and management of information systems within business application environments

  These Objectives/Student outcomes are measured by the IS Model Curriculum Examinations for 2002 and 2010.  The CULA IS department has been giving this test as a final examination to all of their IS majors for the last six years.

IS Model Curriculum Goals

1.                   IS professionals must have a broad business and real world perspective.

- covers ABET objectives a, b, and c

2.                   IS professionals must have strong analytical and critical thinking skills.

- covers ABET Objectives d, e, and f

3.                   IS professionals must have interpersonal communication and team skills and have strong ethical principles.

- covers ABET Objectives g, and h

4.                   IS professionals must design and implement information technology solutions that enhance organizational performance.

- covers ABET Objective i

The following chart extracted from CSULA’s 2011 Annual report illustrates the type of information available for both Objective and Course assessment from the IS 2010 ISA exam.                               

 Objective c

An ability to design, implement, and evaluate a computer-based system, process, component, or program to meet desired needs

                                                      r = .9 (17 question points)

                                     a = .7

                        Both national and CSULA exam means rounded to 45%.  

4. IS DEPARTMENT STRATEGIC PLAN

AACSB expects refinements to the department strategic plan at least every five years based on feedback from assessment and on external technical and environment changes.  Many strategic plans follow the maintenance report outline.  A typical outline using that approach follows.  It is modeled on several published AACSB related strategic Plans.

Typical Scope of IS Strategic Plan

Situational Analysis (with SWAT Analysis)

Finance, Technology, Curriculum, Student, Faculty and Assessment related

Mission, Goals and Objectives

Planning Processes and Outcomes

Strategic Management Processes

Outcomes Assessment processes

Financial Strategy Process

Feedback and Continuous improvement Processes

Strategic Initiatives and related Action Plans

Finance, Technology, Curriculum, Student, Faculty and Outcomes Assessment related

CSULA’s strategic plan also includes introductory material on Why Information Systems, and separate sections on the BS and MS degree programs.

5. OUTCOMES ASSESSMENT

Outcomes assessment/assurance (AOL) of learning as used in AACSB accreditation can be defined as 

“The systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development.”

(Palomba and Banta, 1999).

Its scope is defined by AACSB in a white paper as

“The outcomes assessment process should include

 1. Definition of student learning goals and objectives

2. Alignment of curricula with the adopted goals

3. Identification of instruments and measures to assess learning

4. Collection, analyzing, and dissemination of assessment information

5. Using assessment information for continuous improvement including documentation that the assessment process is being carried out in a systematic, ongoing basis.”

(AACSB Assurance of Learning Standards: An Interpretation, 20007)

These AOL principals can be operationalized as Accountability and Continuous Improvement.

This section illustrates how CSULA implemented Accountability through use of a standard curriculum and the use of the IS Model Curriculum Examination, and how it implemented Continuous Improvement through feedback to all faculty annually of the examination’s detailed course level results.

The remainder of this section is organized as shown in the AACSB AOL Standard.

5.1 Demonstrate a high degree of maturity in terms of delineation of clear learning goals

The College set up four high level goals for each of the programs

·      Master Business Fundamentals

·          Master Analytical and Critical Thinking

·          Master Ethical, Interpersonal, Communication, and Team Skills

·          Master a Functional Area

Therefore the IS department was required to use them as is high level goals.

The college then produced a standard set of subsidiary goals and objectives for tailoring by the business-oriented functional areas.  After an on-going discussion, the IS department moved ahead by using the ABET standard goals and objectives pattern set forth in Section 3.  We argued that this pattern of goals and objectives allowed us to do true qualitative and normed quantitative assessment, the core of accreditation’s purpose.

5.2 Demonstrate the implementation of outcome assessment processes

Through the use of their committee structure and total faculty approval of their assessment actions, they believed they were fully meeting AACSB requirements.  The key is that every faculty member annual receives a detailed tabulation of the performance of their students by question area for each of their courses. Sample ABET and IS 2010 course reports follows.

Extract from an ABET Course Level Exam Report

Extract from an IS 2010 Course Level Exam Report

At the department level CSULA have available the objectives level reports that give them a quantitative evaluation of their overall program’s status. A sample follows.

Extract from an ABET Goal/Objective Exam Report

These reports are primarily for internal use.  CSULA’s actual external assessment document includes the following charts.  They serve as the bases of their evaluation of current annual status.

5.3 Demonstrate use of assessment information to improve curricula

The following trend chart shows the overall performance of the IS program since we began using of the ISA Exit Exam.

CSULA also included an overall explanation of why their scores were below average

“UCLA and USC skim the top 15% of Los Angeles basin high school students (a California educational policy) leaving CSULA with the B and C students. “

CSULA therefore claim their performance is very good considering the pool of available (remaining) students.

CSULA then explains each of bends in the national and CSULA curves.  For example the national 2005-2006 drop was due to the addition of numerous programs following the 2004 beta tests. The 2006-2007 gains represent the first set of feedbacks.  Also, it appears that the IS Model Curriculum Examination for 2010 is more difficult than the IS 2002 test. More faculty involvement in review of test item level performance is required.  Additionally, evaluation and use by more colleges and universities broadens this nationally normed (and even internationally normed) instrument increasing its value and thus performance comparatives.

CSULA’s rise in 2009-2010 was interpreted as due to a curriculum change motivated by their downward sloping scores – which could only have emerged from consistent and continuous testing.

5.4 Closing the Loop

The key to maintaining quality is an emphasis on feedback of quantitative performance to each faculty member.  The preceding chart indicates two significant teaching and curricular feedbacks that improved CSULA scores.  This type of feedback can be achieved only by the continuous use of a national test such as the IS Model Curriculum examinations.

5.5 Continuous improvement

A major AACSB evaluation measure examines continuous improvement.  It is the opinion of the authors, that the use of the feedback available from this universal IS Model Curriculum exam by colleges and universities can be a major factor that will lead to their success and positive assessment by accreditation agencies.

6. FINAL THOUGHTS

IS programs should consider themselves fortunate by the continuing support of the standardized curriculums and certification examination programs in our field.  Few other fields have this advantage.  All of the IS oriented degree programs need to support these efforts by periodically using the IS Model Curriculum Examination.

Approximately half of IS programs are outside business schools and they may find the ABET criteria more consistent with their engineering or science faculties’ approaches to accreditation.  Since the IS Model Curriculum examination provides direct ABET measures those ABET accredited programs or those considering ABET accreditation should consider using the IS Model Curriculum exam as their primary continuous assessment and improvement tool.

7. REFERENCES

AACSB Accreditation Coordinating Committee (2007).  AACSB Assurance of Learning Standards: An Interpretation.

http://www.aacsb.edu/publications/whitepapers/AACSB_Assurance_of_Learning.pdf

AACSB International (2012).  Maintenance of Accreditation Handbook.

http://www.aacsb.edu/accreditation/maintenance_of_accreditation_handbook.pdf

AACSB International (2012).  Fifth-Year Maintenance Report Outline.

http://www.aacsb.edu/accreditation/business/maintenance_docs/maintenance_fifth_year_report_outline.pdf

ABET Computing Accreditation Commission (2012). 

Criteria for Accrediting Computing Programs, 2012 - 2013.

http://www.abet.org/DisplayTemplates/DocsHandbook.aspx?id=1805

Catherine A. Palomba and Trudy W. Banta (1999).  Assessment Essentials: Planning, Implementing, and Improving Assessment in Higher Education.  Jossey-Bass Publishers, San Francisco.

             Information Systems Analyst (2012).  Institute for the Certification of Computing Professionals, Des Plaines, IL 60018.

Contact the Author:  ~  Click Here