July 2024 CIO Update
Apologies for the delay in posting this entry. July was an eventful month! Below I discuss a major IT disruption that affected VCU, and I also feature an update on a cost saving and sustainability innovation in our classroom technology area.
Global IT Disruption
In the early hours of a Friday morning this month, a seemingly commonplace configuration file update wreaked havoc on VCU’s IT infrastructure. CrowdStrike, the endpoint protection service provider, inadvertently pushed out a faulty configuration file to approximately 8.5 million Windows-based computers and servers worldwide. Unfortunately, VCU was not spared from the chaos.
VCU utilizes this service to secure computers and servers across campus. There was no way to repair the affected computers remotely so each and every one had to be visited in-person by our teams to effect a repair. All this started a mobilization of the VCU TS team beginning well before sunrise and deep into the afternoon and into the following week to resolve for our customers.
One thing that helped VCU was that Network Services, while finishing up work on a planned change, noticed increased traffic on the VCU network and took steps to stop the traffic until they could figure out what was happening. This proactive measure likely minimized the impact on VCU servers, desktops, and laptops. After determining the source of the problem, the Information Security Office quickly developed a process to aid the teams across campus who would be tasked with repairing the affected systems. The Endpoint team continued to iterate on this process, streamlining the repair process considerably. By midmorning, the University Computer Center team restored services, and most hosted online services remained accessible.
Once VCU online and hosted services were available, the incident became a more localized support issue, with desktop and classroom technicians working to restore affected computers. From centrally supported classrooms to individual endpoints, more than 3,000 computers have been repaired, onsite at customer locations or in the RamTech/fixIT location at the VCU Bookstore. There are currently about 200 computers across campus remaining to be repaired as we are able to get access to them.
None of this could have been accomplished without extensive collaboration, communication, and willingness to support each other during this time. This unified response to the incident underscores what we strive demonstrate every day – that when we work together, we can be incredibly responsive and effective. Thank you for all your hard work. It is greatly appreciated by me and, more importantly, those we serve.
Eco Savings for classrooms
What happens when classrooms are empty, but AV systems are left on? Years ago, the Media Support Services (MSS) team from Academic Technologies implemented standard system designs and procedures, and diligently maintained them through the years. Expanding upon the existing environment, a Crestron “Fusion” Roomview server, hosted at VCU, is now used for monitoring, controlling and reporting AV systems in 450 classrooms. Mariusz Ostasz, an audio visual programmer in the MSS Engineering and Design team, developed “ECO”, a programming function that monitors and shuts down classroom AV systems to reduce cost, reduce energy use, and increase the lifecycle of the equipment.
When AV systems are left on and there is no occupancy detected in the rooms, and no one interacting with the UI, the control processor is set to turn off the systems at a designated time (“ECO time”). The processor counts the number of hours elapsed until the next time someone interacts with the Classroom UI, and reports these numbers to the Fusion server. This total number of hours is called “ECO savings”, and the countdown continues, even when there is a preset shutdown time.
Fusion provides an annual report of the total ECO savings, which we use to calculate a breakdown of electricity savings and display runtime usage. Display usage is rated by hours, so each hour they are not on results in cost savings. Through Mariusz’s efforts, we can show real ECO savings.
| 9.07 c/kWh average commercial electricity rate in Richmond previous year: 6.86 c/kWh | 4.40 c/hr average electrical cost to run one of our displays previous year: 3.3 c/kWh | 23 c/hr average hourly cost to run one of our displays |
| ECO savings: 181,548 previous year: 156,977 | Display hours saved: 248,437 calculated using a multiplier for the number of displays in a classroomprevious year: 186,490 | Cost savings for hardware operation: $57,338.34 previous year: $43,041.20 | Electricity savings: $10,836.44 previous year: $6,152.37 |
| Total SAVINGS: $68,174.78 previous year: $49,493.57 | |||
The increased cost of electricity, larger number of systems (STEM was added) and larger number of displays per room in some cases (STEM 112, 202, 216), are major contributing factors to increased savings this fiscal year.
Fun Facts
STEM Building: Room 112 (4 displays), Room 202 (10 displays), Room 216 (10 displays)
Combined, the total saved is 38,830 display/hours
- Total number of times that ECO has shut the systems down: 8,570 (This is equivalent to 1 room shutting down somewhere on campus every hour!)
The top 3 centrally managed spaces that reported the highest “ECO hours saved”:
- Egyptian Auditorium, with 3,539
- Lyons 443, with 3,085
- Hibbs 430, with 3,028
Want to learn more? Call VCU Media Support Services at 804-828-1098.
I wish everyone a great start to the new academic year and am excited to get started!