Maine State Government
Dept. of Administrative & Financial
Services
Office of Information Technology
Deployment Certification Policy for
Major Application Projects
I. Statement
All major computer application projects must undergo a
collection of certification tests to determine their readiness to be deployed.
The Chief Information Officer (CIO) makes the final determination whether or
not an application is ready to be placed into production.
II. Purpose
As Maine
State computer
applications have become more complex, more expensive, more interconnected, and
more exposed to the external world, it has become even more important to evaluate
them properly before they are deployed into production. The price of failure
includes, at a minimum,
1) Customer hardship,
2) The added technical challenge and
expense of fixing an application when it is already in production, and
3) Should there be a breach of
confidential data due to an intrusion attack on an improperly secured
application, there may also be additional legal and statutory ramifications.
Addressing these vulnerabilities, this policy will provide
the CIO with a standard and objective process to support the final go live
decision.
III. Applicability
This policy applies to both new applications and upgrades of
existing applications, should they qualify as major application projects that
are owned by all:
1) Executive
Branch and semi-autonomous State agencies irrespective of where the
applications are hosted and
2) Applications
from other Maine State government branches that are
hosted on computer devices operated by the Office of Information Technology or
that traverse the State’s wide area network.
It consists of a collection of tests designed to provide a
high degree of assurance that the application will indeed meet its
expectations, post deployment. Prior to their deployment, all major application
projects must undergo this self-certification without exception. Unless granted
an explicit waiver by the CIO, the bona fides or the past track record of an
organization cannot be substituted in lieu of this self-certification.
IV. Responsibilities
A.
Application Owners: The Application Owners are
responsible for the execution of this test collection and the submission of the
results thereof to the CIO. This submission consists of:
1.
The names and signatures of the individuals designated as the
project/product manager, the executive sponsor and the technical leader, and
2.
A summary result (Passed/Failed/Not Applicable), and a short
explanatory paragraph clarifying and elaborating on that summary result,
corresponding to each of the tests identified below.
Any part of the actual testing may be outsourced to
one or more third parties without prejudicing the ultimate responsibility of
the Application Owners. Irrespective of who actually executes the test
collection, the Application Owners still remain responsible for their
execution, and the results thereof.
Regarding the effectiveness of self-certification in
the absence of external verification, it is noted that the Application Owners
are effectively vouching to the CIO that their application has successfully
satisfied this collection of criteria. Post deployment, should the application
fail on any of these criteria, then the Application Owners must explain to the
CIO how their application passed the earlier self-certification.
B.
Chief Information Officer (CIO): The CIO makes the
final go-live decision and conveys that decision back to the application owners
within five working days from the submission of the results. No major
application shall be installed in a production environment without this
explicit go-live decision from the CIO.
C.
Application Owners and Project Management Office (PMO):
The Application Owners and the PMO must consider this Certification process as
an integral part of the overall project plan. More specifically, this policy
should be an essential component of the product quality assurance lifecycle, as
well as the procurement and contracting processes. Thus, any product or service
contract related to a major application should include a payment holdback
provision, subject to passing this Deployment Certification.
The Application owners and the PMO must jointly
determine the viability of the testing plan as part of negotiating the Project
Definition Document, the detailed Work Plan, the risk analysis, and contingency
planning.
D.
Project Management Office (PMO): This Policy will be
executed and enforced by the PMO. The PMO will advise the application owners of
any potential risks with regard to satisfying this policy, and relevant mitigation
strategies, as part of their oversight.
V. Guidelines &
Procedures
A.
All major application projects will be subjected to the
following collection of tests:
1.
Operating Platform Test: Ensures proper functioning across all
combinations of relevant hardware and software components.
2.
Individual Use Cases Test: Ensures proper functioning of the
unitary features of the application.
3.
Combined Use Case Interactions Test: Ensures proper
functioning of the unitary features in concert with one another.
4.
End-to-end User Acceptance Test: Ensures the satisfaction of
end-user expectations.
5.
Accessibility Test: Ensures compliance with the Maine IT
accessibility policy.
6.
Data Conversion Test: Ensures the accurate migration of
appropriate legacy data.
7.
Input Boundary Values Test: Ensures well-behaved results
against illegitimate input data.
8.
External Interfaces Test: Ensures proper functioning with all
companion applications.
9.
Security Test: Ensures against unauthorized access.
10.
Performance Test: Ensures responsiveness against projected
average and peak processing loads.
11.
Limited Rollout Test: Ensures the viability of the final
rollout.
12.
Backup and Recovery Tests: Ensures disaster recovery and
planned rollback.
13.
Regression Test: Applies exclusively to upgrades of existing
applications. Ensures that a new version does not compromise existing
functionality.
B.
Brief descriptions of the tests are provided below.
These are generic enough to allow specific customization within the context of
an individual application.
1.
Operating Platform Test: The application must be actually
loaded, or configured, on all combinations of hardware, operating systems,
network configurations, terminal emulators, browsers, etc, that are planned for
production deployment, and verified that it works as expected, end-to-end, and
across the board. This includes all relevant client devices, network
configurations, as well as the full complement of web, application and database
servers. It will not suffice to accept the product vendors’ compliance
statements in lieu of actual testing. It is often the case that the development
environment is different from the deployment environment. The extent of such
variance could be subtle, e.g., the development environment could be Oracle on
Windows, whereas, the deployment environment could be Oracle on AIX.
Nonetheless, the application must be actually tested under Oracle on AIX prior
to its deployment. And the same holds for alternative network configurations,
terminal emulators, browsers, etc.
2.
Individual Use Cases Test: An application must have a
complete, stable, and up-to-date documentation of the full set of its Use
Cases. Beyond their expected outcomes, Use Cases must anticipate errors, and
therefore, incorporate robust error-handling and error-logging capabilities.
Each of the Use Cases must be executed individually, and verified that it
indeed delivers as expected.
3.
Combined Use Case Interactions Test: Application owners will
know in advance which Use Cases are likely to be invoked simultaneously with
one another. It is reasonable to presume, for example, that nightly batch jobs
will not execute at the same time as interactive transactions. By the same
token, it is reasonable to presume that within a sales application, order
entry, inventory control and shipping will indeed take place simultaneously.
Therefore, while testing this sales application, it is not enough to simply
ensure that the individual Use Cases work as expected. It is equally important
that the Use Cases for order entry, inventory control and shipping, all deliver
as expected when exercised simultaneously. Interactions among Use Cases depend
upon synchronization of data and processes, and therefore, such interactions
must be tested simultaneously. Potential flaws uncovered as a result of such
testing include unexpected file and record locks, process time-outs, sharing
violations, error propagation, etc. All likely combinations of Use Case
interactions must be tested, and verified that they indeed deliver as expected.
4.
End-to-end User Acceptance Test: While individual Use Cases
and Combined Use Case Interactions exercise component parts of an application,
the End-to-end User Acceptance Test involves a representative sample of actual
end-users performing their daily jobs holistically, using the entirety of an
application. Returning back to the example of the sales application above, an
element of the End-to-end User Acceptance Test may involve entering a new sales
order, initiating the fulfillment of that order, verifying the decrement of the
inventory, generating the shipping manifest, mailing out the customer invoice,
and printing out the internal management control reports. At the completion of
the End-to-end User Acceptance Test, the end-users must be satisfied that the
application does indeed meet all their expectations, or alternatively, that
they are willing to accept any deficiencies thereof.
5.
Accessibility Test: The application must be tested in order to
ensure that it meets State IT accessibility policy as follows: Applications
acquired or developed must be compliant with the Computer Application Program
Accessibility Standard - www.maine.gov/oit/accessiblesoftware. All application and content delivered through
web browsers must comply with the Maine State Web Standards – www.maine.gov/oit/webstandard and
the Web Accessibility Policy of the State of Maine – www.maine.gov/oit/accessibleweb.
6.
Data Conversion Test: It is likely that record structures and
formats of the legacy application were modified as a result of the migration
into the new application. It must be ensured via actual testing that all
business-critical data have indeed survived the migration. It is left to the
good judgment of the application owners to determine as to what constitutes
‘business-critical data.’ But once so determined, it must be ensured that such
data are indeed accessible from the new application. Should the new application
precipitate modifications to the existing work flows, then this step must also
include testing and validating the new work flows.
7.
Input Boundary Values Test: The application must be thoroughly
tested against input data that are outside the domain of the expected values.
At the least, this ensures graceful outcomes when users inadvertently enter
wrong values. But more importantly, this guards against the so-called Injection
Attacks, where illegitimate values are deliberately fed in order to compromise application
security. Input data beyond the expected domain may include violations of upper
and/or lower limits, invalid characters, missing or incomplete data, mutually
inconsistent data, etc.
8.
External Interfaces Test: An application must have a complete and
up-to-date documentation of all the data and workflow dependencies between
itself and all other applications that it interacts with. All such interactions
must be tested, and verified that they indeed deliver as expected. Interfaces
must anticipate errors, and therefore, incorporate robust error-handling and
error-logging capabilities. While it is desirable to exclusively utilize the
Test environments of the various applications when testing the various
interfaces, it may be necessary under certain circumstances to pair the Test
environment of this application with other environments of the companion
applications, as long as such other applications participate in the interface
on a read-only basis. The Enterprise Applications Office will provide further guidance
on this item, as needed.
9.
Security Test: The application must not allow unauthorized
access, or compromise any of its data or workflow. All personal, medical and
financial data, in motion, must be encrypted end-to-end, both inside and
outside the State of Maine
firewall. Data hosted on servers are not subject to encryption, but data
resident in portable computing devices must be encrypted. A full vulnerability
assessment and penetration test must be performed on the application, and it
must cover all relevant client devices, as well as the full complement of web,
application, and database servers. At a minimum, such a test should include
hardware and software configurations, virus/worm detection, war dialing,
password cracking, log review, forensic auditing, integrity checks, injection
tests, buffer overflow tests, cross-site scripting tests, and denial of service
tests. All applications should enforce standard best practices in terms of
complex passwords, access controls, incident reporting, IP routing, hardened
OS, redundancy, encryption, backups, firewall, intrusion detection, etc. Should
the Test and Development environments of the application require a less secure
configuration than the Production environment, then the Test and Development
environments must be completely separate from the Production environment in
terms of the subnet and the firewall. Beyond these generic requirements, the
application may need to satisfy additional specific, statutory requirements, as
set forth by HIPAA, FISMA/FIPS, SOX, GLBA, CROMERR, USA
Patriot Act, etc. The Enterprise Security Office will provide further guidance
on this item, as needed.
10.
Performance Test: Performance issues determine the
responsiveness of the application to its users, and therefore, its acceptance
and adoption. The application must be responsive under the projected average
load, as well as the peak processing load. Performance has three broad
components: the client, the network and the server.
For a fat-client application, the client must be configured
properly to obtain optimal performance. For thin-client applications, relevant
images and scripts must be cached at the client device on first login. All
applications, irrespective of client size, must safeguard against user
impatience, eg, repeatedly pushing an action button in the hope of speeding up
a transaction, etc.
Network throughput is a function of many different
factors, viz: the loading on the network, settings on switches, routers and
hubs, size of the transactions, payload to overhead ratio, etc. For the purpose
of this Policy, network throughput will be measured by combining the
payload-to-overhead ratio with transaction size, under very low server load
conditions. The application should tune the following two factors in its
control to affect this measurement: the screen buffer size and the transaction
size.
In order to safeguard against adverse user
perception, the application must establish a two-tiered response time
specification, one for data inquiry/lookup, and another for data modification
transactions. In both cases, the times must be derived from network captures,
starting from when a command is issued by the client, until the response
arrives at the client device, excluding network latency and network
transmission times.
For server-side tests, the application must generate
load using automated tools that simulate user behavior, including simultaneous
and staggered loading. Beyond response times, other aberrations that must be
investigated include non-linear performance, eg, response time increasing
disproportionately with loading, and response time varying during periods of
constant load. The server must be monitored for CPU loading, memory
consumption, disk I/O, etc, under average and peak loads. Either CPU or memory
loading approaching 90% should trigger further investigation, resulting in
further tuning, or revised capacity planning. Regarding disk I/O, a trigger for
additional investigation includes excessive swapping. It is left to the good
judgment of experienced OS administrators to determine as to what constitutes
‘excessive swapping.’ Should the server(s) prove inadequate to the demands of
hosting the application in spite of exercising all available tuning options,
then a revised round of capacity planning must be undertaken. The Enterprise
Operations Office and the Enterprise Network Services will provide further
guidance on this item, as needed.
11.
Limited Rollout Test: The deployment scripts must be validated
on a restricted test bed that includes all the components, including relevant
client devices and the full complement of web, application and database
servers. This is to ensure that there does not arise any unexpected eventuality
during the production rollout. It is reasonable to minimally edit the
deployment scripts in order to constrain their scope to the restricted test
bed, refine them as part of the limited rollout, and then revert back to the
full scope of deployment.
12.
Backup and Recovery Tests: Two distinct tests must be
performed as part of backup and recovery. The first is to restore the current
state, or as close to it as possible, from the backup media in order to
simulate recovery from a disaster. The second is to rollback the system to a
previous state from archived media in order to simulate recovery from a disastrous
upgrade, a series of flawed transactions, etc.
13.
Regression Test: This applies exclusively to upgrades of
existing applications, in order to verify that a new version has not adversely
affected previously working functionality, and that known issues previously
addressed have not re-surfaced yet again. A two-pronged regression test
strategy must be undertaken. Based upon the release notes and the known module
dependencies, a focused test suite must be administered for those Use Cases
that are actually affected as part of this upgrade. At the same time, a core
suite of essential functions must also be tested, irrespective of whether or
not they underwent any modification as part of this upgrade. It is left to the
good judgment of the application owners to determine as to what constitutes a
‘core suite of essential functions.’ Such a two-pronged strategy provides
failover protection against the fallibility of release notes, and incomplete
knowledge of module dependencies.
VI. Definitions
1. Application
Owners: For the purposes of this policy, the individuals designated as the
Project/Product Manager, the Executive Sponsor and the Technical Leader are jointly
and collectively identified as the Application Owners.
2. Major
Application Project: An Application Project is deemed to be major if it meets
the threshold of Enterprise Portfolio submission. At the discretion of the CIO,
the PMO, or the Application Owners, even an Application Project that does not
meet the threshold of Enterprise Portfolio submission may be deemed to be major
due to its complexity, operational impact, security impact, business
criticality, media or political exposure, etc. Should it appear that a large application
project has been decomposed into smaller projects where each of them fall below
the threshold of Enterprise Portfolio submission, then the Enterprise Portfolio
Management Office, the PMO, or the CIO may designate any or all of the smaller
projects as major.
3. Semi-autonomous
State Agency: An agency created by an act of the Legislative Branch that is not
a part of the Executive Branch. This
term does not include the Legislative Branch, Judicial Branch, Office of the
Attorney General, Office of the Secretary of State, Office of the State
Treasurer and Audit Department.
4. Use
Case: A Use Case is a well defined sequence of actions undertaken jointly by
the user and the application, which produces a predictable result of value to
the user. Thus, a Use Case captures a discrete functionality of an application
completely independent of the underlying system development methodology. The
full set of all the Use Cases for an application constitutes the complete value
added by that application.
VII. References
1.
Policy to Govern Information Security Risk
Assessments and to Ensure the Prompt Remediation of Deficiencies
2.
Policy to Safeguard Electronic
Information Residing on Portable Computing Devices
3. Computer
Application Program Accessibility Standard
4. Maine State
Web Standards
5. Web
Accessibility Policy of the State of Maine
VIII. Document
Information
1. Document Reference
Number: 7
2. Category: Applications
3. Adoption Date: October 30, 2006
4. Effective Date: October 30, 2006
5. Review Date: October 30, 2008
6. Point of Contact: Kathy
Record, Associate Chief Information Officer, Office of Information Technology,
State House Station #138, Augusta,
ME 04333,
(207)624-9502.
7. Approved By: Richard B Thompson, Chief Information
Officer, State House Station #138, Augusta,
ME 04333,
(207)624-7568.
8. Position Title(s)
or Agency Responsible for Enforcement: Mary Silva, Director, Project Management
Office, Office of Information Technology, State House Station #138, Augusta, ME
04333, (207)624-7574.
9. Legal
Citation: 5 M.R.S.A. Chapter 163 Section
1973 paragraphs B and D, which read in part
The Chief Information Officer shall: “Set policies and
standards for the implementation and use of information and telecommunications
technologies…” and “Identify and implement information technology best business
practices and project management.”
10. Waiver Process:
Submit a written request for a waiver to the Associate CIO.
