Service announcement 7 December

There was a DNSSEC signer cut-over failure on 7 December, which was resolved.

There was a temporary DNSSEC signer cut-over failure on Monday, 7 December 2012, which was resolved.

Date:

7 December 2012

Duration

Up to 24 hours

Services affected

APNIC reverse zones

Description:

DNSSEC signer cut-over failure resulted in cache mis-match of signed reverse DNS information

Sequence of events:

  1. 07/12/2012 19:00 UTC+10 – The switch where the active DNSSEC signer is connected failed.
  2. 07/12/2012 21:00 UTC+10 – Our DNS distribution servers were configured to use standby DNSSEC signer.
  3. 08/12/2012 04:00 UTC+10 – The standby signer re-signed all zones according to its daily schedule, with all zones verified to be valid as published.
  4. 08/12/2012 04:30 UTC+10 – APNIC authoritative DNS servers received updated copy of all signed zones with different set of ZSK.
  5. 09/12/2012 00:27 UTC+10 ­ External validation failure report was first posted by Sebastian Wiesinger to dns-operations@lists.dns-oarc.net caused by cached data having different ZSKs than fresh data.
  6. 10/12/2012 13:00 UTC+10 – Announcement was posted on APNIC website about DNSSEC validation failure.
  7. 11/12/2012 18:30 UTC+10 – Announcement was updated with timeline and more details.

Impact:

Any DNS resolvers that had cached records prior to the ZSK change will have a failure in validation of records fetched after the update.  It will have taken up to 24 hours after the ZSK update on 08/12/2012 04:00 UTC+10 for all DNS records to be consistent and valid.

Mitigation and Improvement:

  • Update the disaster recovery process to make sure keys are synced before switching to standby signer.
  • Improve DNSSEC policy auditing to verify that keys do not change unexpectedly.
  • Request DNSSEC vendor assistance in automation of key synchronisation between active and standby signers.

Affected zones:

101.in-addr.arpa
103.in-addr.arpa
106.in-addr.arpa
110.in-addr.arpa
111.in-addr.arpa
112.in-addr.arpa
113.in-addr.arpa
114.in-addr.arpa
115.in-addr.arpa
116.in-addr.arpa
117.in-addr.arpa
118.in-addr.arpa
119.in-addr.arpa
120.in-addr.arpa
121.in-addr.arpa
122.in-addr.arpa
123.in-addr.arpa
124.in-addr.arpa
125.in-addr.arpa
126.in-addr.arpa
14.in-addr.arpa
150.in-addr.arpa
153.in-addr.arpa
163.in-addr.arpa
171.in-addr.arpa
175.in-addr.arpa
180.in-addr.arpa
182.in-addr.arpa
183.in-addr.arpa
1.in-addr.arpa
202.in-addr.arpa
203.in-addr.arpa
210.in-addr.arpa
211.in-addr.arpa
218.in-addr.arpa
219.in-addr.arpa
220.in-addr.arpa
221.in-addr.arpa
222.in-addr.arpa
223.in-addr.arpa
27.in-addr.arpa
36.in-addr.arpa
39.in-addr.arpa
42.in-addr.arpa
43.in-addr.arpa
49.in-addr.arpa
58.in-addr.arpa
59.in-addr.arpa
60.in-addr.arpa
61.in-addr.arpa
0.4.2.ip6.arpa
2.0.1.0.0.2.ip6.arpa
3.0.1.0.0.2.ip6.arpa
4.4.1.0.0.2.ip6.arpa
5.4.1.0.0.2.ip6.arpa
8.1.0.0.2.ip6.arpa
9.1.0.0.2.ip6.arpa
a.1.0.0.2.ip6.arpa
b.1.0.0.2.ip6.arpa
c.0.1.0.0.2.ip6.arpa
d.0.1.0.0.2.ip6.arpa
e.0.1.0.0.2.ip6.arpa
f.0.1.0.0.2.ip6.arpa

We apologize if there is any loss of facilities or inconvenience caused.

 

Contact Us Should you require assistance in dealing with any problems arising from this outage, please contact the APNIC Helpdesk.

Contact:

Phone: