Testing

Digital Accessibility Testing Guidelines

Digital accessibility testing is the assessment of digital information and services, specifically electronic content (email, electronic documents, software, web applications, and websites), against established accessibility standards. The State of Maine requires digital information and services, developed, procured, or provided by the State of Maine to be accessible to individuals with disabilities, in accordance with our Digital Accessibility Policy (PDF), and any other relevant federal and state regulations, policies, and standards. This applies to both internal and external use.

Digital information and services procured through a vendor, have the option of state testing, vendor testing, or third-party testing. If testing is not conducted by the state, comparable test documentation must be submitted for assessment.

State testing, and assessments of vendor, or third-party, testing results are performed by the MaineIT Accessibility Team. While the Accessibility Team conducts testing, provides guidance, and provides assessments, the business owner remains responsible for ensuring the accessibility of their information and services.

Request Process

To request a test or an assessment, a state agency representative (applications manager or designee, etc.) submits an Assyst service request for an accessibility testing. A standard process is used to manage the request queue. The amount of time required can be impacted by the number of requests already in the queue, so please insure you provide as much lead time as possible when making a request. Credentials and appropriate permissions need to be securely provided for state testing to be performed.

State Testing

Electronic documents: The MaineIT Accessibility Team tests electronic documents following the AED COP guidance for WCAG 2.1 Level AA. A key to creating accessible electronic documents, is to use products that support accessibility, including the features/components of those products that are geared towards accessibility. One example being the Check Accessibility feature in Microsoft Word.

State Agency websites: The MaineIT Accessibility Team uses an automated web crawler for testing. Reports are provided, upon request, to agency webmasters, for manual inspection.

Web and desktop applications: The MaineIT Accessibility Team has adopted the Department of Homeland Security's Trusted Tester process for the testing of web and desktop applications. For web applications, review consists of a manual assessment of a sampling of sections (web pages, documents, screens) to confirm conformance with the Americans with Disabilities Act (ADA) through WCAG 2.1 Level AA and Section 508.

An executive summary and detailed report are provided when testing is completed. The detailed report identifies global, pattern, and individual issues for the pages tested. Global issues represent those found through the manual assessment that are assumed to span most or all pages. Pattern issues represent those found through a manual assessment that are assumed to span two or more pages.

Vendor Testing

Vendor submitted tests must show a process that is at least comparable to state testing. Submitted documentation must include a statement of conformance (or non-conformance) along with a list of tools used and a summary of the testing process to verify WCAG 2.1 Level AA compliance, and/or other applicable standards. The statement must include language that manual checks were performed. It needs to be clear that criteria that can only be checked manually were checked, and manual verification was done on any questionable tool results.

Requirements for Vendor (or Vendor Third-Party) Submitted Tests

1. The date of the test and person or party who performed the test.
2. A statement of conformance (including language stating that manual checks were performed).
3. A list of the tools used.
4. A list of the sections (pages, windows, etc.) that were checked.
5. A high-level summary of the testing process.
6. A completed checklist.
   • The following checklists are recommended, but other comparable documentation may be used.  Achieving Level AAA conformance is recommended but not required.
     1. WebAIM's WCAG 2 Checklist
     2. Section 508 ICT Testing Baseline
7. A list of any compliance errors and any remaining warnings. These may not be thoroughly reviewed but documentation of nonconformant results is particularly important as it allows the purchasing agency to prepare any necessary accommodations.
8. Make supporting test result documents available upon request.

Please note, if the vendor testing provided does not meet all of the requirements listed above, then supplemental materials and/or state testing may be required to provide sufficient information for an assessment to be made. Additionally, upon review of the content, questions may need to be answered and/or further clarity may be needed, in order to perform an assessment.

Sample Statements

The following samples are incomplete and are only meant to demonstrate acceptable submission language and materials for each requirement.

• Sample #1

  Company X with Product Y asserts compliance with WCAG 2.1 Level AA. We had a JAWS user, a Dragon Naturally Speaking user, and a switch user navigate the site and complete forms without assistance. None of them reported significant trouble. Tables were navigated successfully, and form error messages were understood, and the forms completed without users reporting issues. Color contrast was verified manually during development. The required checklist is attached.

  The following tools were used:

  • JAWS 18
  • Dragon Professional v15
  • WebAIM Color Contrast Checker
  • etc.

  The following pages and documents were checked:

  • Home page
  • Record ABC page
  • Inventory Details PDF
  • etc.

  • Sample #2:

  Company R with Product S asserts compliance with WCAG 2.1 Level AA. We followed the DHS Trusted Tester process on the attached list of pages and all results not listed as DNA were compliant. See the attached report for specifics.

  • Sample #3:

  Company A with Product B asserts partial compliance with WCAG 2.1 Level AA. We followed the WebAIM WCAG 2.0 checklist using the WAVE toolbar and the aXe browser extension with manual inspection when necessary. Manual inspection included color contrast, multimedia caption and audio description, audio transcript, and equivalent keyboard access. See the attached document for a list of non-compliant issues.

Other Accessibility Assessment Considerations

One commonly used measure of assessment for proposed and procured products are vendor-completed Voluntary Product Accessibility Templates (VPATs). The VPAT explains how digital products such as software, hardware, electronic content, and support documentation meet (conform to) the Revised 508 Standards for digital accessibility.

Specific to testing, if a vendor already has a completed VPAT 2.x (508 version) for the product that has or will be implemented, that covers some or all of the vendor testing requirements, it can be submitted to the Accessibility Team for review following the Request Process identified above. However, when that occurs, the request must clearly state which of the requirements are met by the VPAT and must include any additional supplemental materials (test results, etc.) for any requirements not covered by the VPAT.

Since VPATs are typically associated with proposed or procured products, if a product has already been acquired and assessed (through state testing, etc.) it is not necessary to also request a VPAT review, unless there is a belief that the VPAT contains important information that has not already been assessed.