Back-to-School October 2021

University of Iowa:  Ross Miller

At the University of Iowa, the Elements of Success application was developed utilizing data from the Unizin data platform to show students how they are using Canvas compared with their peers.  The latest release of Elements of Success allows students to compare the following with other students in their class:

• the number of files they have accessed

• the total of graded assignments

• the number of days that they have accessed their Canvas site

This set of metrics enables students to determine if they might need to focus on an area of study.  This type of comparison is only made possible with the UDP Event store.

Elements of Success (EoS) has been in production for seven years. Most faculty and instructors using EoS with their classes are ‘return’ customers.  Currently, 18 sections are using EoS with over 5,500 unique students.  Each semester the students whose instructors make EoS available report that they like knowing exactly how they are doing in their class and have a better sense of control over their outcome.

Penn State University: Bart Pursel

The Unizin Data Platform plays a critical role in helping Penn State deliver Canvas activity data to advisers through our Elevate application. Penn State has an early progress reporting campaign university-wide in weeks 3 and 4 of the semester. Depending on how a course is designed, some courses may not yet have any graded assignments, providing very little data to consider when anticipating a student struggling in a course. Elevate provides a new stream of data, helping to inform if, when, and how an adviser might reach out to a student early in a semester. This helps Penn State move towards a more proactive model of advising.

Through the first six weeks of our fall 2021 semester, 243 advisers (representing both faculty and staff advisers) used Elevate to explore 2,006 students’ Canvas activity to help inform if, when, and how these students are contacted. This includes advisers from University Park, as well as advisers from 17 of our Commonwealth Campuses. 31.7% of students enrolled across Penn State are being advised by an adviser that uses Elevate.

Advisers that use Elevate participate in unit-level training, including specific training on the tool itself and training that leverages case studies that include data from Elevate combined with data from other platforms, designed to surface ethical and privacy questions that need to be considered when using Elevate. Asynchronous training also exists within the application itself, intended to be accessed just in time.

The UPD and Elevate also help University administrators identify students who may have enrolled at PSU but are not attending any of their courses within the first weeks. By identifying and engaging these students who do not plan on attending PSU early in the semester, we can ensure they don’t incur tuition costs, and at the same time make sure these students aren’t included in our official reporting data, that can go on to impact enrollment and graduation rate calculations that are reported to various outlets.

University of Minnesota:  Jeff Weber

It never hurts to be reminded, Unizin partnership initiatives have led to favorable pricing, vendor accountability, and valuable data additions to the UDP.  Qualtrics is a vendor that dominates the survey application market and is not currently under a Unizin agreement.  With few competitive alternatives, contract renewals with Qualtrics are challenging and can lead to double-digit percentage increases in price.  At the September LTSO meeting, nine Unizin institutions indicated that Qualtrics is available to all or a portion of their user base.  Discussion at that same LTSO meeting revealed responsibility for Qualtrics varies at each institution and often resides outside of central OIT.

The University of Minnesota is seeking a long-term strategy to increase competition in the survey application market to benefit institutions and leverage during negotiations. To do this:

• We are exploring pilot opportunities with promising vendors to gain working experience and provide feedback on making their products more competitive.

• We hope that other Unizin institutions are interested in joining this pilot potentially.

• Ultimately this may end up with broad-based consortium interest in both a Qualtric Unizin-based relationship and interest in Qualtrics data in the UCDM. That has yet to be determined and will take time.

If you or your institution is interested in exploring a pilot opportunity with a non-Qualtrics survey vendor or to provide input on a potential Unizin partnership with Qualtrics, email Jeff Weber weber101@umn.edu.

Ohio State University: Marcia Ham, PhD

with Maureen Guarcello, Ph.D., Linda Feng

Many of us have enjoyed the opportunities provided through various Unizin community and advisory groups to engage in discussions around the ethical use of data analytics. It is an important topic to discuss moving forward in leveraging data at our institutions to advance learning systems—technical systems through teaching and learning strategy systems through assessment systems. It is also an enormous topic, and perhaps that is one reason ethics may seem to take a back seat during the work of innovation. We can get excited by the great potential promised through innovations and easily forget to slow down and consider all ramifications of the development—the good, the bad, and the ugly. By identifying potential unintended consequences of our innovations, we can build in strategies for mitigating and hopefully even eliminating them.

The August 2021 issue of Educause Review highlighted the article, “Discrimination in a Sea of Data: Ethical Implications of Student Success Analytics,” which discussed the threat of unintended consequences of using student data for analytics without proper vetting, especially for bias. As we work together as a consortium developing data systems and learning technologies—some specifically focused on student success through learning analytics—it becomes crucial for us to integrate the consideration of potential bias during our technology development and use processes. The data literacy efforts should reach campus boundaries, not just the academic centers where most of the focus resides. Powerful analytics are generated within business and financial affairs, residential life, bookstores, human resources, and campus planning.

A commitment to the continued inclusion of ethical considerations was recently demonstrated with the final approval of a set of guiding principles for the application of learning analytics that was shared among the Unizin community last spring. Those suggested principles are meant to be guidelines member institutions can reference as they work through innovations that leverage student data for learning analytics. Still, they can be more broadly scoped by institutions as well. For instance, the fifth principle states that “analyses should recognize and mitigate bias and strive to minimize adverse impact on individual students or groups of students.”

As we develop increasingly advanced learning systems—such as those using artificial intelligence (AI) and machine learning (ML)—the opportunity for bias to occur compounds. The Educause Review article mentioned earlier dives into the risks for bias in these technologies and the different types of bias that we should be aware of as we look to create mitigation strategies. Although the article focuses on AI and ML, we can easily see how the various types of bias could creep into less advanced uses of data and why now is the time for thoughtful consideration during technology development and use. So, how could the fifth guiding principle play out in developing or purchasing systems and processes for learning technologies?

As institutions grapple with procurement decisions where AI or ML—whether simple or advanced—is involved, it is worth considering whether they are making a purchasing decision or more or a hiring decision. In many ways, buying a product that uses AI or ML has a long-term impact akin to bringing in a new hire. Let’s think of it more like a hiring decision. We are more likely to carefully evaluate the values of the company that makes the product, for example, assessing whether the product’s algorithms contain bias. Ultimately, as we think collectively about the ethical implications of advanced learning technologies, we need more transparency regarding the purpose for which data will be used, under what conditions, and what protections will exist for an individual’s identity. We must influence vendors to help their customers understand the extent to which AI or ML may be used in their products, much like ingredients are listed on nutrition labels. (Responsible food labeling practices have evolved in response to consumer (and some regulatory) pressures.) This is especially important when the potential harms resulting from under-examined bias can have high-stakes consequences.

University of Iowa:  Jane Russell

Using the Unizin Data Platform data, the University of Iowa designed a new dashboard to support student learning. The Instructional Analytics Dashboard (IAD) is a comprehensive analytics tool that helps instructors make data-informed decisions about their teaching and create a supportive learning environment by providing insight into student behavior and performance. The IAD’s audience is exclusively instructors, while Elements of Success (mentioned in the previous article) is primarily designed for students.

Using the IAD, instructors can see class trends of engagement and identify patterns of behavior and potential concerns that might prompt them to communicate with specific students or potentially change their class design or set of activities. The IAD combines learner data from the UDP with other student data elements, such as demographic information, courses students take, and student majors, to generate near real-time reports. Because the IAD utilizes PowerBI, it is easily customizable. It is recommended that instructors consult the IAD a few times each week. Many have chosen to review it daily.

IAD was first developed for one large-lecture chemistry class at the beginning of the pandemic as all instruction had transitioned online rapidly. The instructor for this course was concerned about student engagement in the online sessions. Zoom lectures were divided into small groups, with a student peer learning assistant leading the discussions. Due to the reliance on Zoom rather than the traditional LMS, Canvas, student attendance and engagement were difficult to track. Some students attended only video recordings of the lectures, while others may have participated in some synchronous lectures and watched the same recording. The IAD provided the instructor with how many students joined each discussion group, how many accessed the recorded lecture, and how many did both or did nothing.

In fall 2020, IAD began with a single chemistry course. The word spread rapidly as the chemistry instructor was delighted and shared his enthusiasm with other instructors. In spring 2021, the IAD pilot added five new courses. In fall 2021, the dashboard expanded further to include five more courses. With each expansion, the IAD designer, Jane Russell, Director of OTLT Research & Analytics (https://unizin.org//teach.its.uiowa.edu/people/jae-eun-jane-russell), wanted to ensure that feedback from involved instructors was incorporated to improve the dashboard. Even though the current pilot focuses on gateway courses, there is a need to customize the dashboard to different disciplines and instructional styles.

The Instructional Analytics Dashboard encourages instructors to reflect on their course design and when and how to best communicate with their students to promote learning and student success. The current pilot focuses on capturing the correct data in a way that is meaningful for each course.  The next challenge will be to scale it more broadly to impact more courses and their students.

University of Michigan:  Sean DeMonner

This fall semester, the University of Michigan has sourced a few additional services for the coming academic year via Unizin.  These new services include GradeScope, TurnItIn Feedback Studio, and NameCoach.   All have been newly contracted for the fall term through Unizin.

Each service has an outreach and communication campaign that includes stories in relevant university publications, digital signage, mass emails, social media placements, and Canvas touts (banner announcements in our Canvas instance). Face-to-face announcements are essential opportunities to encourage awareness and build relationships.  These new services have been announced at the Unit Reps meeting, the Canvas Summit event,  and the Strategic Technology Advisory Committee. Additionally, the University of Michigan typically provides training programs for faculty as they evaluate and begin to use a new service.

In particular, NameCoach is getting the most attention and buzz. It aligns well with the presidential DEI initiative. NameCoach is a great way to help build an inclusive community quickly at the beginning of the new academic year.

Information Technology Services (ITS) supports a broad range of activities that systemically increase awareness of existing and new services across campus through a11y focus, internal training, HR Processes, workshops, etc.   Future plans include Salesforce expansion, MCommunity integration, the Michigan IT Symposium for technical staff, and even commencement and events.

University of Iowa: Valerie Henessee

When the University of Iowa (UI) sought a method to provide Top Hat, a student engagement tool, to instructors, it leveraged its Unizin membership and partnership with Top Hat to rapidly expand access and lower student costs.

By offering Top Hat through Unizin’s Order Tool, the University of Iowa reduced the Top Hat student license cost by over 50 % off the retail price while taking advantage of efficiencies created by Unizin’s automatic billing and opt-out processes. Unizin’s development team also created a historical entitlements function that allowed students to use licenses they had already purchased without being charged again. And instructors could order Top Hat through Unizin’s Order Tool like other inclusive access materials.

This marks the first time that Unizin has made Top Hat licenses available to students in this way.

To effectively promote the service, the UI’s Office of Teaching, Learning & Technology (OTLT) focused on previous Top Hat users and instructors of large enrollment courses. OTLT also developed a support system for instructors, including regular training sessions centered on teaching best practices for Top Hat and a Top Hat Users group.

In fall 2021, instructors in 55 courses with nearly 5,000 students opted to use Top Hat. The projection is that 10,000 UI students may use Top Hat during the 2021-2022 academic year to engage in their classes.

ICON Direct, the UI’s inclusive access program of which Top Hat is a part, also continues its rapid growth, with a 13% increase in orders in fall 2021 over fall 2020. This semester, approximately 14,800 students have at least one ICON Direct digital material in their classes.

Valerie Henessee, who shared this story, was hired to support the steady rise in orders through the inclusive access program. “I’ve found students and instructors enjoy the ease of access that eTexts offer and students also appreciate the cost savings over physical texts,” says Valerie Henessee, ICON Direct eText coordinator.

You can learn more about the University of Iowa’s ICON Direct program by visiting the Office of Teaching, Learning & Technology’s website.

The Unizin Data Platform added support for multiple SIS ingests per day and simplified manifest file names.  By consuming SIS data more often, institutions can ensure that their context data is up-to-date when performing research and analysis.  Also, the simplification of manifest files names improves readability for institutions and the Unizin team.

Order Tool delivered improvements to the ordering experience and resolved minor bugs as reported by our member institutions.

Derek Gleim, interim CTO and Director of Engineering co-presented “Jump-Starting Learning Analytics with Data Ready Apps” at the IMS Global Learning Impact online conference.  Unizin has long been involved with the data-ready app initiative within IMS Global.  The IMS Data Ready Apps program was first proposed to look at the problem of adopting learning analytics strategies holistically rather than through the lens of a data standard or an integration technology standard.  Its goal is to provide an opportunity for a provider to demonstrate an ability to take in data in a specific agreed-upon format (e.g., Caliper) and show that they can use that data to answer an agreed-upon set of questions around a particular, narrow topic? If a provider can do that, any institution could potentially “point their data” at the provider and have access to at least a baseline set of analytics.

Ross Miller, Sr. Application Developer at the University of IA, Kyle Unruh, Unizin Data Solutions Architect,  and Sara Bolf, Unizin Data Analyst, presented “Modeling Student Success Based on Digital Activity”. The University of Iowa realized that they had a deficiency in data-driven tools to identify students who are struggling academically. To help identify these students, the University proposed using a network of models that give a broader and more reliable student assessment during a semester. Previous modeling efforts relied heavily on historical measures of academic performance (cumulative GPA, ACT scores, etc.). They omitted relevant behavioral metrics that capture student engagement and effort early in a semester. In this session, a model that relies primarily on digital student behavior was presented. Used in conjunction with its interpretable insights, the proposed model offers student success staff a tool to help identify at-risk students and enables more targeted conversations with students.