Please see the slide decks for the agenda and meeting content. Content not covered in the slide deck is noted below.
Workday Outage Debrief
Workday Outage 11/2/23 - Workday has provided us a very detailed Root Cause Analysis (RCA) Report regarding the incident. We have provided the summary statement below for everyone. As we shared the day of the incident, the initial cause of the incident was confirmed to be a catastrophic power outage in the WD5 Data Center which disabled primary, secondary, and backup power systems.
Their RCA includes 12 preventative and follow-up actions to help prevent an incident such as this in the future. The total down time for this incident was less than 12 hours. The Disaster Recovery process was initiated, but they did not cut over to the alternate server on the east coast. If an outage is over 12 hours, our Service Agreement requires them to restore us at our secondary server on the east coast.
This was the first unanticipated Workday outage we have had in our nearly 5 years as Workday customers. We would like to extend a huge thank you to all of you for your patience and understanding during the outage.
We are refining our Workday Issue Outages and Internal Communications Plan, as well as collecting any lessons learned. We do want feedback on how that day was handled, so if you have any suggestions on what we could have done better or differently, please do not hesitate to email Twyla and she will include them in our debrief information. We really value your feedback.
Note: The statement mentions minimizing data loss during the outage. There were some integrations that didn’t run for a certain period; however, our team had already investigated it, and we were already structured in a way that all of our integrations were complete by the time the outage started. We felt good about how it was handled.
Note: We have already modified and are working on finalizing standard messaging should this happen again.
Workday’s Official Summary Statement:
“Resolution Date Thursday Nov. 11, 2023, 9:16 p.m. America/Los Angeles (GMT-0700)
As of November 2, 2023, 9:16 p.m. PST, this issue is resolved. All WD5 Production and Non-Production tenants are available as expected. A Root Cause Analysis document will be available by Nov. 10, 2023, and will be attached to this alert.
In summary, at 4:49 a.m. PST on Nov. 2, 2023, Workday identified a large-scale service disruption impacting both WD5 Production and Non-Production environments comprising approximately 10% of Workday customers. Immediate response was initiated by our internal teams, our co-location vendor, and the power utility. The cause was confirmed to be a catastrophic power outage in the WD5 data center disabling primary, secondary, and backup power systems.
Even after power was restored, an initially unstable power environment created additional challenges, and service restoration was more difficult and took longer than expected.
At the beginning of the incident, Workday immediately initiated parallel recovery efforts including Disaster Recovery (DR) pre-checks as well as continuing with protocol to restore in place. A fix-in-place restoration was determined to be the quickest approach for tenant availability and to minimize data loss. Greater than 996% of Production tenants were restored by 2:22 p.m. PST. The remainder of Production tenants were restored by 4:09 p.m. PST and all Non-Production tenants by 9:16 p.m. PST. No case of customer data loss was identified.
We understand the significance the Workday service has to your operations and your business. And for this, we sincerely apologize for the disruption. Customer service is a core value at Workday, and we remain committed to supporting our customers’ needs with the highest level of urgency.”
Workday Release Updates
2023 Equal Pay Profile Update:
- Launched the 2023 Equal Pay Talent Review for Executive Branch, excluding the Secretary of State.
- Updated language for a current process in Career and Development Planning functional area.
- Comms: CHRO Class Comp provided communication regarding this item.
An inbox task to complete the Equal Pay analysis process was sent out. If anyone needs to update their career information, they can do so through the inbox task or use the knowledge article Career Information for a step-by-step process. The link to the knowledge article is provided
here.
If you are submitting a case regarding the equal pay analysis, please specify this so that our team doesn’t mistakenly believe the case is in reference to making a salary correction.
RPT: NRPPT (No Retro Processing Prior To) Report Update:
- Replaced the PAY | NRPPT report to be the HCM | Employee NRPPT Dates report.
- We added the Supervisory Organization and the PDC field.
- Comms: Communication happened directly with agencies that made the request & Workday Wednesday.
November Release Items:
- 2 CPERS Integration Updates
- PEBB Open Enrollment Updates (processing, validating and updating rates)
- December COLA preparation/testing
- Various Differentials, Time Entry code, and other CBA updates
Allowances not paying temporary hourly employees in pay results:
Currently allowances are not paying correctly to temporary hourly employees. After the change, the impacted population will be paid their allowances correctly. This item will only address allowances currently in use, any allowance plan that is not currently in use will be addressed later when it is appropriate/necessary. Comms: Workday Wednesday.
Establish a PTO Leave Bank:
New bargaining Agreement for IAFF and KFFA. Both CBAs have agreed to this change. Per the KFFA CBA, this does not apply to the 40 hour per week employees. Currently impacted employees have separate leave balances for the leave times identified in the description, after the change they will have one PTO Leave Bank. Comms: Directly with impacted Labor Units.
Time-off Validations:
Currently employees can request time off in increments that are not supported by the state time tracking. Once this update happens, time off requests and balances will be maintained the same as Time Tracking and reported correctly. Need to add a validation to all time off plans that will only allow the use of up to 2 digits after the decimal. An error message will be used to notify employees when trying to take an action that is no longer supported. Comms: Workday Wednesday.
Payroll & Time Tracking
Federal Tax Withholding Update:
The federal income tax update will not change the amount of federal tax deducted from an employee’s mid-month pay. Federal taxable wages are now based on the new first of the month combined federal taxable wages earned in a single pay period (calendar month) based on employees W-4 in Workday. For example, employees that receive a mid-month pay, the federal tax that is deducted from the md-month pay will be reduced from the total federal tax on the first of the month pay. The Federal Tax Visual resource will be distributed via E-News and on Workday Drive for Payroll Partners to share with their agency employees.
State Tax Withholding Update:
The state income tax update will not change the amount of state tax deducted from an employee’s mid-month pay. This update will calculate an 8% flat rate state tax amount for earnings paid on the mid-month pay date. If the 8% flat rate state tax is more than the amount that was deducted on the mid-month pay, then that difference will be added to the state tax deducted on the employee’s first-of-the-month pay. As a result, the employee may see an increase in state tax deductions for the first-of-the-month pay. The State Tax Visual resource will be distributed via E-News and on Workday Drive for Payroll Partners to share with their agency employees.
October off-cycle (run 2) payroll will process on Nov. 8, 2023
- October off-cycle pay date is Wednesday, Nov. 15, 2023
November on-cycle (run 1) payroll will process on Nov. 27, 2023
- November on-cycle time entry cutoff is 5 p.m. on Nov. 22, 2023
- Cutoff for 24/7 agencies is Nov. 26, 2023, at 5 p.m.
- No time entry in Workday from 5 p.m. on Nov. 26 until 8 a.m. on Nov. 28, 2023
- No daily check (off cycle) processing on Nov. 27 and Nov. 28, 2023
Note: Payroll Processing calendar for 2023 and 2024 are located as follows:
Workday Payroll Year End Forum was held on Nov. 2, 2023:
Year End Forum Resources sent out via E-News Nov. 8, 2023, and an updated E-News went out Nov. 14, 2023, with updated Resources including Year End Overpayments Process Updated.
Year End Resources can be accessed on Workday Drive: Drive>Payroll Information and Resources>2023 Year End Forum.
Complete 2023 Timeline:
- Nov. 20, 2023, Federal & State tax adjustment help cases due.
- Nov. 22, 2023 (Run 1) Non-cash taxable reimbursements entered in November pay period. Overpayment Gross-ups.
- Dec. 8, 2023 (Nov. Run 2) Underpayments deadline. Separated Employees with Net Pay Positive Adjustment deadline. Cash taxable reimbursements entered in November pay period.
- Dec. 21, 2023 (Dec. Run 1) December terminations cutoff date for all final payment requests for tax year 2023. All information must be entered on the spreadsheet.
- Dec. 26 to 29, 2023 (Blackout dates) No off-cycle payments will be issued.
- End of December/Early January Review Deceased Report.
- Jan. 10, 2024 - Ensure all employees update their addresses in Workday. After Jan. 10, 2024, the W2 file will be created, sent off to a print plant. Distribution is required by Jan 31. 2024.
Create help cases for Federal & State tax adjustments as soon as possible. Last day to process any-and-all payments is Dec. 22, 2023. All final checks have to be requested via the off cycle check process by the deadline. If it’s requested after the deadline and they’re terming 12/31, payday will be in January.
PEBB Arrears:
Charges to PEBB arrears is caused when there is no money to take out premium deductions for PEBB. This goes on a report called the Arrears Report. DAS has undertaken a large effort in cleaning up and reconciling the process over the past several months. The initial bill invoice went out to agencies this week so there should be a detailed list of the month and the actual deductions it was referring to for those charges. The invoices went out to the accounting managers. The accounting manager is correcting the invoices, and all future invoices will be going to the appropriate payroll person first. That way, payroll can work with accounting to get the invoices paid.
Compensation
Equal Pay Calculator Implementation - The CMP Equal Pay Tool and CMP Equal Pay Tool with Details report were removed. As of Mar. 2022, agencies should use the equal pay calculator to determine salary placement. Prior to the development of this calculator, agencies were tasked with looking at different points of data/information to do an analysis for salary placement.
Although technically accurate, the CMP Equal Pay Tool (which was removed), did not tell the entire story. For example, the average length of service in the position reflected the time an employee was on the position number. Some agencies frequently move employees on and off position numbers as a part of position management, which made this data point problematic. Additionally, the average time in job profile reflected the time an employee was on a specific job profile. As you are aware, we have classification studies that impact job profiles – and all of the classifications that were created as a part of The Oregon Management Project (TOMP) are new. As these data fields were not reliable (and agencies should use the equal pay calculator to determine salary placement), the CMP Equal Pay Tool was removed.
However, we also know that some agencies used the CMP Equal Pay Tool for position management. We worked with HCM Workday to build a new report titled “CMP Executive Branch Positions by Agency and Job Profile.” This report was tested with a few agencies who use it for position management, and the feedback received was positive.
If your agency feels an employee needs to be placed higher in the salary range due to a specific skill set you are looking for in a broadly defined classification, the Position Management Detail report and the Current Worker Detail report both can be used as reference points. However, as a reminder, agencies should use the equal pay calculator to determine salary placement. If going above the expected step generated from the equal pay calculator, the agency needs to use the notes section to explain hiring above the expected step, including references to state policy and CBA, and the agency needs to be prepared to defend that decision if audited or challenged through other legal venues.
Please send us an email if you have additional questions:
December 2023 COLA - The December 2023 COLA includes the majority of representations. However, there a few exceptions such as bargaining groups that either have not settled, groups that have different dates or various other reasons.
We were working on testing and validation, which has since been completed. A Workday announcement has been posted in the Workday announcements section, it features timelines, information and details about agencies responsibilities, and important information about actions you may need review/complete.
As a reminder, in-progress actions such as job changes, hires, and other compensation changes will block the COLA from being processed. If you can review in-progress actions and complete as many as possible as we get closer to the implementation timeline. This helps reduce the need for follow-up and manual work for those who did not get the COLA applied automatically through our mass processing.
Another group to review are those who have in-flight actions. This includes hires that are being processed. For example, maybe they started the hire process prior to the COLA and their date of hire is after the COLA. Take a good look at the compensation of those being processed to make sure they do include the COLA. If they do not, they can be updated.
Also, review future dated actions such as completed future dated hires. We are processing the COLA for Dec. 1 so if somebody has a completed future dated hire on Dec. 15, for example, the COLA does not automatically apply to them.
Reports available for use to find in-progress and future dated events for employees are: “CMP | In Progress Compensation Changes”, “HCM | Employee Events in Progress by Selected Initiated and Effective Dates”, and “HCM Employee Events by Selection Report” (these are available in the HR Reports Worklet under the Compensation Reports and Worker Reports Tabs).
General Reminders/Information
The Time Off | Compensatory Time | Above 0 Hours report was updated. We have added the Time Off Balance (greater than) prompt. Reminder, you can search for representation / union in the Organization prompt. The Organization field will only show organizations. Supervisory Organization is an entirely different prompt.
Oregon1 is refreshing this Saturday. For more information, please see System News on your Workday homepage.
For those who can’t attend, please make sure your teams (HR, payroll, budget, etc.) know where to find the Workday Wednesday meeting notes.
Please submit a help case in Workday if you have any questions about the information provided today.
In partnership,
The Workday Oregon Team