Skip Maine state header navigation

Agencies | Online Services | Help

State of Maine Seal 

Maine State Government

Dept. of Administrative & Financial Services

Office of Information Technology (OIT)

Major Incident Procedure

 

I. Statement

This Procedure sets OIT’s Major Incident command, control, and communication protocol.

 

II. Purpose

Utilizing a pre-defined Procedure, Agencies and OIT will collectively ensure the best possible response to Major Incidents.

 

III. Applicability

This procedure applies to:

 

1.      Executive Branch Agencies, irrespective of where their applications are hosted.

 

2.      Other State government branch applications hosted by OIT and/or utilizing the State WAN.

 

IV. Responsibilities

A.    Chief Information Officer (CIO): Communicates catastrophic Major Incidents to Commissioners.

B.     Chief Technology Officer (CTO): Declares Major Incident, owns, executes, and enforces this Procedure.

C.     Duty Manager: Facilitates remediation of Major Incidents during off-business hours. 

D.    Enterprise Security Officer: Advises Incident Commander in case of security breaches. 

E.     Incident Commander:  Section Director closest to the Situation (business hours) or Duty Manager (off-business hours).  Owns, manages, and leads Major Incident response, remediation, and reporting.  Forms and manages Incident Command Team, serves as or designates Incident Communicator.

F.      Incident Command Team: Responds to the incident. Either a virtual team or an on-site team (preferred).  May include Vendors and/or Key Agency Personnel.

G.    Incident Communicator: Leads all internal and external written communications and coordination efforts in a timely manner (timely communication is imperative).   Works closely with the Technology Business Consultants to keep all parties informed and updated.

H.    Manager, Customer Support: Identifies potential Major Incidents, notifies other appropriate parties, and updates the Customer Support Status (CSS) page.     

I.       Operational Manager: Identifies potential Major Incidents, notifies other appropriate parties, provides timely updates to the Incident Communicator, and participates in remediation.  This is typically the line manager closest to the Situation.

J.       Root Cause Resolution Team (Optional): Performs post-remediation root-cause investigation.

K.    Section Director (e.g. Core Section Manager, Application Director, etc.): Facilitates remediation of Major Incidents during business hours.

L.     Technology Business Consultants (TBCs): Communicates/liaison to Key Agency Personnel.

 

V. Directives

A. Awareness & Initiation

1.            The Manager, Customer Support or Operational Manager becomes aware of a potential Major Incident, immediately notifies the other party, and the Incident Commander.

2.            The Incident Commander reports the potential Major Incident to the CTO.

3.            The CTO determines whether this event is a Major Incident.

 

B. Response

1.            The Incident Commander designates the Incident Communicator and forms the Incident Command Team, comprised of OIT staff (optionally including, Key Agency Personnel and/or Vendors), and activates the Incident Command Center (51 Commerce Drive, Room 414) if necessary to help facilitate remediation.

2.            The Incident Commander consults with the Incident Command Team to identify quantitative remediation metrics and remediation strategy.

3.            The Incident Communicator provides all verbiage and timely updates to Customer Support.  Customer Support posts verbiage to the CSS page in a timely manner.

Updating the CSS page automatically updates the OIT Core Status and News (CSN) page (http://csn.state.me.us).  The current Duty Manager is also listed on the CSN Page. 
To the extent known, the update covers:

(1)         The nature of the Incident in plain language,

(2)         The projected impact on Agency operations and/or citizens,

(3)         Quantitative metric(s) of what constitutes remediation,

(4)         The remediation steps being undertaken,

(5)         Estimated time for remediation, and

(6)         Estimated next update time.

 

4.            The CIO/CTO, at their discretion, notifies the DAFS Commissioner, DAFS Communications Director, and affected Agency Commissioners.

5.            The Incident Communicator notifies affected TBCs and other OIT personnel as necessary, including, but not limited to OIT Extended Managers and affected OIT staff.
 

6.            The Incident Commander initiates remediation steps, including reaching out to OIT resources (while briefing their command chains), initiating communication with vendors, suppliers, partners, Key Agency Personnel, et al.

7.             The TBCs ensure that the affected Agencies are adequately briefed.

8.            The Incident Communicator ensures the OIT CSS Page is updated at planned intervals, until remediation. To the extent known, updates covers the six items identified above.

9.            For security breaches, the Incident Commander contacts the Enterprise Security Officer for advice. State Law[1] mandates notifications under certain kinds of security breaches.

 

C. Diagnosis & Remediation:

1.            Affected Operational Managers and their teams (likely already part of the Incident Command Team) diagnose the cause, and estimate remediation time.

2.            Each Operational Manager ensures that the Incident Commander remains fully briefed.

3.            The Incident Commander creates a Footprints ticket for the Major Incident.

4.            Operational Manager(s) and their teams perform necessary remediation. All changes must follow pre-established emergency change control procedures.

5.            The Incident Commander determines if/when quantitative metric(s) are met.

6.            Restoration Priority Order (Subject to Governor’s Office approval):

(1)         Core Information Infrastructure, network, email, etc.

(2)         Citizen Health & Safety

(3)         Revenue

(4)         Citizen Financial Services

(5)         Regulation

(6)         Provider/Vendor Financial Services

(7)         All Other Services

 

D. Post-Remediation:

1.            Operational Managers and their teams document the Footprints ticket(s).

2.            Incident Commander ensures Footprints tickets are created for any follow-up activities and that all Footprints tickets are linked.

3.            If root cause is not identified, the Incident Commander may create a Root-Cause Resolution Team to investigate root cause(s) and recommend permanent solutions.
 

4.            Upon resolution, the Incident Commander creates a preliminary report, distributed to impacted customers within two business days (48 hours). An OIT Major Incident Report is also created, distributed to all concerned parties, within five business days. Both reports must receive CIO and/or CTO approval before distribution. Once approved, they are distributed and attached to the Footprints Major Incident ticket.

 

VI. Definitions

A. Major Incident: Event that the CTO judges to have a significant impact on governmental information operations. Examples include:

·         Network, email, or other app outage, for two hours or longer, significantly affecting governmental productivity and/or public service.

·         Security breach, significantly compromising either the credibility or operational capability of the government.

Catastrophic Major Incident examples include:

·         OIT data center fire, disabling the majority of information operations. 

·         Cyber-attack that shuts down the entire State network.

VII. References

VIII. Document Information

Initial Issue Date: February 26, 2014

Latest Revision Date: July 11, 2014

 

Point of Contact: Henry Quintal, Architecture-Policy Administrator, OIT, (207) 624-8836.

 

Approved By: James Smith, Chief Information Officer, OIT, (207) 624-7568.

 

Position Title(s) or Agency Responsible for Enforcement: Greg McNeal, Chief Technology Officer, OIT,  (207) 624-7568.

 

Legal Citation:  5 M.R.S.A. Chapter 163 Section 1973 paragraphs (1)B and (1)D, which reads in part, “The Chief Information Officer shall: “Set policies and standards for the implementation and use of information and telecommunications technologies…” and “Identify and implement information technology best business practices and project management.”

 

Waiver Process: See the Waiver Policy[2].



[1] Notice of Risk to Personal Data, Title 10, Chapter 210-B, http://www.mainelegislature.org/legis/statutes/10/title10sec1347.html

[2] http://maine.gov/oit/policies/waiver.htm