Service announcement 7 December

There was a DNSSEC signer cut-over failure on 7 December, which was resolved.

There was a temporary DNSSEC signer cut-over failure on Monday, 7 December 2012, which was resolved.


7 December 2012


Up to 24 hours

Services affected

APNIC reverse zones


DNSSEC signer cut-over failure resulted in cache mis-match of signed reverse DNS information

Sequence of events:

  1. 07/12/2012 19:00 UTC+10 – The switch where the active DNSSEC signer is connected failed.
  2. 07/12/2012 21:00 UTC+10 – Our DNS distribution servers were configured to use standby DNSSEC signer.
  3. 08/12/2012 04:00 UTC+10 – The standby signer re-signed all zones according to its daily schedule, with all zones verified to be valid as published.
  4. 08/12/2012 04:30 UTC+10 – APNIC authoritative DNS servers received updated copy of all signed zones with different set of ZSK.
  5. 09/12/2012 00:27 UTC+10 ­ External validation failure report was first posted by Sebastian Wiesinger to caused by cached data having different ZSKs than fresh data.
  6. 10/12/2012 13:00 UTC+10 – Announcement was posted on APNIC website about DNSSEC validation failure.
  7. 11/12/2012 18:30 UTC+10 – Announcement was updated with timeline and more details.


Any DNS resolvers that had cached records prior to the ZSK change will have a failure in validation of records fetched after the update.  It will have taken up to 24 hours after the ZSK update on 08/12/2012 04:00 UTC+10 for all DNS records to be consistent and valid.

Mitigation and Improvement:

  • Update the disaster recovery process to make sure keys are synced before switching to standby signer.
  • Improve DNSSEC policy auditing to verify that keys do not change unexpectedly.
  • Request DNSSEC vendor assistance in automation of key synchronisation between active and standby signers.

Affected zones:

We apologize if there is any loss of facilities or inconvenience caused.


Contact Us Should you require assistance in dealing with any problems arising from this outage, please contact the APNIC Helpdesk.