Last week was week 5, which means I am grading the hardest assignment in my Improvement Science course, the problem of practice paper. For this assignment, my EdD students must identify their organization’s goals and the data that demonstrates their progress in relation to these goals. They then have to identify a regular, routine practice that is getting in the way of those goals, and explain how and why this problem of practice is where they should focus their improvement efforts. Essentially, I teach data-driven decision-making for continuous improvement in educational organizations.
I spend a lot of time in Week 5 writing, “showing me the evidence.” Don’t just tell me what you think. You have to explain to me why you believe that, break it down, and show me the evidence.
I’ve been thinking a lot about goals and metrics myself lately as a member of one of the departments slated for elimination. And as the AAUP chapter president, I am anxious for all of our faculty who are having the same experience.
Now two weeks into this fiasco, my department still has not been given the data on which we are being judged. We have only been given our department’s set of Z-scores. Without knowing what went into a Z-score, it really doesn’t mean that much. And if the data itself is flawed, well, then garbage in, garbage out.
Some real garbage
Just as a small example, administrators pulled faculty productivity from the Academic Analytics (AA) database. I didn’t know it existed until about 9 pm last Saturday. My entry included three (3!!?) conference presentations. Total in nine years. That represents only 7% of my total national presentations. AA was somewhat better at capturing my peer review publications; it managed 80% of those, but only 17% of my citations.
I can’t imagine anyone feels comfortable being judged on less than 100% of their full research or creative activity productivity. And it appears that through their Z-score calculation to compare departments to one another, the admin is dismissing how AA weights different research outputs based on academic standards in those fields. For example, books that are more in History and English than in my field, in which the peer-reviewed journal articles carry more weight.
We were only able to address our productivity when we dug out the source data ourselves. If the data sources and processes used are not shared, we’re just taking stabs in the dark about why programs are being eliminated. And for our colleagues in TMDF and architecture, AA does not seem to capture all of the creative activity produced in those fields.
We’re also taking stabs at how they calculated instructional efficiency and wondering how they used contextual data. It seems clear that they didn’t consider that, as a graduate department, a doctoral methods course with 16 students is actually pretty large. Or that faculty in my department advise anywhere from 12-16 doctoral students at a time and may sit on as many as 20 committees. I currently have 16 doctoral students, four of whom are working on dissertations and another five who are working on proposals. They and I would much rather that I be reading the mounting pile of dissertation drafts than writing this post.
Show us the receipts
EDAD is not alone in wondering where the receipts are. Even before the proposed cuts were announced, the Faculty Senate passed a resolution asking that the administration share the data and metrics with the wider university community. The Faculty Senate president had no update for us when our Senator asked recently. The resolution is below. Everyone should be asking their Faculty Senator to follow up on this resolution.
Stonewalling is not transparency
The EDAD department has also asked Vice Chancellor Volkmer to meet with us to answer our questions about the data. He has refused, and so here are my receipts for the day, Volkmer’s email:
“We are committed to supporting a clear understanding of the metrics, which we hope is reflected in the comprehensive documentation and ongoing campus engagement efforts. If you have any questions, please feel free to submit them via email. We’ve found this to be an efficient and effective method for providing timely and thorough responses. There are many people involved in the sourcing of metric data, and getting them all together would be problematic. Through this process, we anticipate being able to address your inquiries prior to Wednesday. Thank you for your continued collaboration.
Best, James”
I’ll let you all parse that and determine if you think it meets the moment in terms of transparency and shared governance, because I’m too tired from trying to teach my students, keep at least my funded research project going, and collect accurate data on behalf of my department.
P.S., As of Wednesday, September 24th, Volkmer stonewalled us again and did not answer the questions submitted via email.
Sarah Zuckerman is an Associate Professor of Educational Administration and the UNL AAUP Chapter President
