US Cloud Services Down

Incident Report for IT Portal

Postmortem

Incident Summary: SQL Services affecting central login experienced downtime following routine updates. Issues were encountered when reconnecting encrypted volumes after system reboot.

Root Cause: A recent Windows update modified the default behavior of PowerShell's Invoke-WebRequest cmdlet. Our automated scripts that handle volume reconnection were executing without the -UseBasicParsing parameter, causing them to pause and wait for user confirmation instead of executing non-interactively.

Resolution: Encrypted volumes were manually reconnected to restore service availability. Our automation startup scripts have been updated to include the -UseBasicParsing parameter in all Invoke-WebRequest calls, ensuring scripts execute without requiring user interaction.

Preventive Measures:

  • All PowerShell automation scripts have been audited and updated with the -UseBasicParsing parameter
  • Testing procedures will be enhanced to validate script execution in non-interactive environments following system updates
  • Additional monitoring has been implemented to detect script execution failures more rapidly

Timeline:

  • 4:53 PM EST - Issue detected, services down
  • Manual intervention initiated
  • 5:46 PM EST - Services restored, incident resolved
Posted Dec 22, 2025 - 08:10 EST

Resolved

This incident has been resolved.
Posted Dec 17, 2025 - 05:46 EST

Monitoring

SQL Services affecting central login was down after updates. Issues reconnecting encrypted volumes after reboot. Volumes were manually reconnected and services are operational.
Posted Dec 17, 2025 - 04:53 EST
This incident affected: US (US API, US Cloud).