Complaint Investigation Report
Interested Party v. Mid-Coast CDS

December 7, 2010

Complaint #11.028CS
Complaint Investigator: Jonathan Braff, Esq.

I. Identifying Information

Complainant: Interested Party
Address
City, Zip

Respondent: 	Kathy Alley, Board Chair
Greenlaw, First Fl.
11 State House Station
Augusta, ME 04333

Site Director: Jude Thomas

II.	Summary of Complaint Investigation Activities

The Department of Education received this complaint on October 19, 2010.  The Complaint Investigator was appointed on October 19, 2010 and issued a draft allegations report on October 21, 2010.  The Complaint Investigator conducted a complaint investigation meeting on November 3, 2010, resulting in a stipulation.  On November 3, 2010, the Complaint Investigator received 5 pages of documents from the Complainant, and received a 3-page memorandum and 47 pages of documents from Mid-Coast CDS (the “Site”) on November 9,
2010. Interviews were conducted with the following: Jude Thomas, site director for the Site;
Linda Reynolds, coordinator of Part C services, early childhood special educator and case manager for the Site; Katherine Tranzillo, speech therapist for the Site; Nora Armstrong, occupational therapist for the Site; Tera Kennedy, coordinator of Part B services and special education teacher for the Site; Pam Morse, special education teacher for the Site; Daphne Hinchcliffe, physical therapist; Darby French, speech therapist; and the Complainant.

III.	Preliminary Statement

This systemic complaint was filed by Interested Party, alleging violations of the Maine
Unified Special Education Regulations (MUSER), Chapter 101, as set forth below.

IV.	Allegations

1.   Failure to administer the Battelle Developmental Inventory in accordance with instructions provided by the producer of that assessment in violation of MUSER
§V.2.C(1)(e)

V.	Stipulations

1.	Evaluation report forms used by the Site include subdomain standard scores and corresponding standard deviations.

VI.	Summary of Findings

1.  The Complainant was contracted to conduct evaluations for the Site. Those evaluations included use of an assessment titled the Battelle Developmental Inventory (BDI-2).

2.  The Site provided the Complainant with a scoring summary sheet for use with the BDI-2 that included reporting of subdomain “standard scores” and corresponding standard deviations.

3.  According to the reference manual “Administration of the BDI-2 and Use and Scoring of the Record Forms” issued by the Riverside Publishing Company (the “Publisher”), the “scaled scores” for the assessment subdomains are “normalized standard scores with a mean of 10 and a standard deviation of 3, similar to the subtest profile scales used on many cognitive batteries.”

4.  After the complaint was filed, the Site revised the scoring summary sheet to replace the term “standard score” with “scaled score” in reference to subdomain scores, and notified the evaluators of this change.

5.  According to the Publisher’s web site, the minimum qualifications required for an evaluator administering the BDI-2 (L Level) are as follows:

Certification in occupational therapy, physical therapy, or other medical field. Other medical fields include: pediatrics, nurse practitioners, office nurses, visiting nurses, home health care workers for infants & young children, and head start specialists.1

AND

Specific undergraduate level training in one or more of the following:
- Intelligence/Cognitive Testing
- Basic tests and measurements
- Speech, hearing, language assessments
- Education diagnostics

1 Higher level operations require degrees in education or psychology.

- Developmental milestone assessment

6.  At the point that the Site took over responsibility for the Blue Hill peninsula region, there were 25 children receiving services through CDS.  At present there are 40 children in that region receiving services.

7.  During an interview conducted by the Complaint Investigator with the Interested Party, M. Ed., (the Complainant), the Interested Party stated the following: She is a special education consultant and has been in the field of special education for 30 years, first as a teacher and then a consultant. For the past 12 years, she has done assessments and provided early intervention services to children for CDS, first for CDS-Waldo County and then, after consolidation, for the Site.

She is glad that the Site changed the wording on the scoring summary sheets and made them accurate, but that doesn’t resolve the issue for her. She believes that the Site has people who are improperly administering the BDI-2.  They don’t understand the scores they are reporting. She called one of the case managers to discuss the problem with the score sheets. The case manager didn’t understand what she was saying about the scoring, and suggested that she talk with Nora Armstrong, who has used the BDI-2 a lot. She called Ms. Armstrong, but she didn’t understand the scoring issue either. She suggested that Ms. Armstrong should look in the manual, but Ms. Armstrong told her that she didn’t have a manual. She also spoke with Linda Reynolds about the issue, but Ms. Reynolds told her that she didn’t understand the
issue and didn’t have time to research it.

Ms. Armstrong told her that when she scored tests, she only had some tables from the back of the examiner’s manual. Ms. Armstrong also said she didn’t necessarily have the administrator’s manual when she administered the assessment, only a photocopy of the testing protocol with some written notes. She believes that one cannot properly administer the BDI-2 without the administrator’s manual. When doing an interview, one needs to follow the scripts contained in the manual. She attended a three-day training from the publishers of the BDI-2
in 2005.  She learned that one had to have the administrator’s manual when administering the
assessment so that it would be administered in a standardized way. It’s cumbersome to use the manual, so people create a short cut by writing out notes on the protocol page.

Both Daphne Hinchcliffe and Darby French told her that they have reviewed BDI-2 reports that contained scores that were impossible. She believes that items are being improperly scored based on a parent’s report that the child is able to do a certain task, even though the child doesn’t perform the task during the assessment. These practices produce unreliable scores with a tendency to over-score, so that children are being found ineligible who are actually eligible. The manual refers to subdomain scores being “invalidated by poor administration procedures.”

The BDI-2 is an L Level test on the producer’s qualification chart. Even if those administering the test have the required qualifications, however, they still have to know the proper procedures and follow them. They also have to know how to score and interpret the test, or give the test along with someone who does know how to do this. She believes that the Site is having the BDI-2 administered by former case managers who decide not to be case

managers anymore and become providers instead. They do testing without having received sufficient training. They miss the cognitive element to the evaluation because they lack training in this area.

She believes that, in her geographical area, fewer children are currently receiving early intervention services, particularly developmental therapy, than should be receiving them, and that fewer children are currently receiving early intervention services than were receiving them before the Site took over responsibility for her area.

8.  During an interview conducted by the Complaint Investigator with Daphne Hinchcliffe, LPT, Ms. Hinchcliffe stated the following: She is a licensed physical therapist specializing in pediatrics contracted with CDS.  She hasn’t done any developmental assessments with evaluators from the Site, and is unable to comment on whether the Site staff members are following proper testing procedure. The only instance she encountered of an evaluator who appeared not to have had adequate training, and which she told Interested Party about, involved an evaluator from another CDS office, not the Site.

9.  During an interview conducted by the Complaint Investigator with Darby French, SLP- CCC, Ms. French stated the following: She is a speech/language therapist specializing in pediatrics contracted with CDS.  She had been working with CDS for about 8 years, first through the Ellsworth CDS site and now through the Site. Part of her work involves administration of the BDI-2, to both Part C and Part B children. When she administers the test, she always has the manuals with her and someone, either she or her co-evaluator, reads from them to make sure the script is being followed exactly as much as possible. She has memorized much of some of the other tests she uses, especially the speech/language assessments, but she’s not as familiar with the BDI-2.

She has done four evaluations with the Site, all Part C, with the Interested Party, Ms. Hinchcliffe and, on one occasion, Ms. Armstrong, as her co-evaluators. On the latter occasion, she felt that Ms. Armstrong was not that comfortable with the test. Ms. Armstrong didn’t use the manuals, but showed her photocopied pages with written notes that she was using in place of the manuals. After the evaluation was completed, she spent a good deal of time reviewing the evaluation with Ms. Armstrong, and discovered that Ms. Armstrong had left out certain of the test items. Ms. Armstrong seemed to think it was acceptable to assume that the child could perform a task rather than give that part of the test. She had the feeling that Ms. Armstrong didn’t really understand the cognitive element of the assessment, and that there was a lot of guesswork involved. They wrote the report with a statement that the test had been invalidated, but the child was obviously in need of services and was determined to be eligible.
One of the people at the Site giving the test is Betty Wood, who had been a case manager at Ellsworth before the reorganization. Ms. Wood told her that she had never given the BDI-2 before she took her new position with the Site. So far, Ms. Wood has been acting as a co- evaluator with someone else who is experienced with the test, but she is concerned that if Ms. Wood is asked to administer the test to a Part B child, she will be doing it on her own.  To her knowledge, this has not yet happened.

Both she and Ms. Wood have been assigned to provide services to a child and are expected to follow the coaching model, rather than the clinical model they were familiar with previously. They told the case manager that they were uncomfortable with the fact that neither of them was familiar with this model. She also believes that the Site should be using local providers more often rather than sending people from Rockland.

10.  During an interview conducted by the Complaint Investigator with Jude Thomas, Ms. Thomas stated the following: She is the site director for the Site. There are five individuals at the Site that administer the BDI-2: Ms. Armstrong, Ms. Reynolds, and Katie Tranzillo for Part C children; and Pam Morse and Tara Kennedy for Part B children. For Part C children, a
team of two conducts the evaluation, and one of these evaluators is always part of the team. Most have master’s degrees, and they all have many years of experience. None of them were formerly case managers. They fully understand the cognitive aspects of the evaluation. They all have received training on the BDI-2 from trainers certified by the producers of the assessment. They are all experienced with giving evaluations. These evaluators know about standardization of assessment tools, and they do follow the scripts. They meet once a week to discuss issues concerning testing.

All the evaluators have a BDI-2 kit which contains the administration manual, but they don’t necessarily have the administration manual by their side during the evaluation. It’s distracting to the children for the evaluator to keep looking in the manual. The evaluators often copy pages from the protocol and write notes to themselves on the pages for reference. These are babies and toddlers who are being evaluated, not children sitting across a desk to whom you give instructions. The evaluator may put a toy in front of the child and say “Look at the toy.” If the child doesn’t respond, the evaluator may change the wording slightly. They might also ask the child’s parents if the child normally responds. The evaluator doesn’t want to spend
too much time on a single task, because it may make the child nervous.

The BDI-2 is not the only source of information used by the evaluators. Parent interviews are required as one of the criteria. Where a child doesn’t perform a task during the assessment, the evaluators might ask the parents whether the child typically does so.  The response may not influence the BDI-2 scoring, but it’s included in the report. The evaluators may also decide not to score the item, or to return and see the child again. They may decide to find the child eligible, provide services and repeat the assessment in 3 months. The evaluators are experienced professionals, and can exercise their professional judgment. The Site’s
evaluators err on the side of including children in the system.

She doesn’t know how the Interested Party would have information about the number of children being served by the Site. The Site is very busy in the Interested Party’s area, and there are two other individuals providing developmental therapy in that area in addition to the Interested Party. The Interested Party has been unwilling to provide reports in the format the Site has requested. The Interested Party’s reports also do not provide the amount of information the Site prefers. The Interested Party has refused to return a copy of the BDI-2 that she borrowed to the Ellsworth CDS office. When the Interested Party called Ms. Armstrong, it was at night and Ms. Armstrong was at home. Ms. Armstrong told her she was surprised by the call, the conversation made her uncomfortable, and she was not prepared to

discuss the matter with the Interested Party. Ms. Armstrong is experienced with administering the BDI-2 and has been trained in its use.

When the Site took over the region in which the Interested Party works, the Site began working with various providers who had previously worked with a different CDS site for years. These providers have not always been following the regulations with regard to timeliness and the content of evaluation reports. The providers also have been telling parents things that are not in their reports, making parents distrust what the Site is doing and saying. In August 2010, the Site began holding periodic meetings with the providers and providing training so they understand the Site’s regulatory responsibilities. The providers are also used to serving children in a clinical stetting, and the Site is trying to change their culture so that they understand the need to deliver services in the natural setting (Part C) or least restrictive
environment (Part B).  She believes that the providers had previously been left on their own to do things the way they wanted to do them. She believes that the Interested Party’s real
agenda is to have the CDS consolidation rescinded so that she and the other providers will no longer be under the Site’s direction.

11.  During an interview conducted by the Complaint Investigator with Linda Reynolds, M.Ed., Ms. Reynolds stated the following: She is an early childhood special educator and coordinator for Part C services for the Site. Up until July 2010, she had primarily been acting as a case manager, and she still does some case management. She administers the BDI-2 for Part C children as part of a team. Her graduate program included courses on intelligence and cognitive testing, basic tests and measurement. More recently, she took graduate level
courses on assessments focused on younger children. One of those assessments was the BDI-
2, on which she wrote a paper. She also attended a day-long training on the BDI-2 presented by the publisher.

She has her own BDI-2 kit, including the administrator’s manual. She is familiar with the scripts, and uses them consistently, following the protocol in the manual. The manuals are large and not always easy to pull out in the midst of an evaluation, so she copies some of the script questions and makes notes on the pages that she refers to. She has memorized many of the scripted questions. She is familiar with scoring of the BDI-2 – it was part of her course work and it’s also covered in the manual.

She has never observed anyone from the Site administering the BDI-2 improperly, and she is not aware of any instance when the test was invalid for any reason. She feels very comfortable with the people with whom she does the evaluations.

The Interested Party sent her an e-mail about her concerns with the BDI-2, and she forwarded it to Ms. Thomas and Ms. Kennedy. She wasn’t sure what the Interested Party was asking
her. The next time she saw the Interested Party, the Interested Party asked about the e-mail and she told her it had been forwarded. The Interested Party had a lot of things she was
unhappy about, and didn’t always raise them at appropriate times. She found the Interested
Party’s questions and comments surprising. The Interested Party had done evaluations in the
past without any problem. She knows that the changes that accompanied the CDS

consolidation have been hard for the providers because they were being asked to do things differently.

12.  During an interview conducted by the Complaint Investigator with Katherine Tranzillo, M.S., CCC-SLP, Ms. Tranzillo stated the following: She has worked professionally as a speech pathologist for 26 years, and is currently employed by the Site. Over the past 9 years, she has administered the BDI-2 hundreds of times. She was trained by the experienced evaluators with whom she first started administering the assessment; she hasn’t attended any formal training. She took courses at both the undergraduate and graduate level in assessments and statistics, including courses in speech and language assessments.

The administrator’s manual consists of one main manual and five other manuals, one for each domain. The manuals are always available when she administers the assessment – she carries one copy with her and there is also one available in the Site’s office. She sometimes shares her manuals with other providers depending on who needs them when. The evaluator doesn’t have to have the main manual when administering the assessment. The five individual manuals contain the scripts, and those she does have with her when administering the assessment.

Some elements of the scripts are very specific, but some expressly allow for flexibility. When required, she follows the script precisely. For some items, however, the manual says that the evaluator can change the wording as long as the evaluator understands and remains true to the intent of the task. For example, the script might have the evaluator ask the child to “draw” an object. If the child doesn’t understand the request, the evaluator could ask the child to “color” the object. The task is not measuring vocabulary, so it would be permissible to alter the language.

There are many items that can be scored either by direct observation or else by parental report.  If the child doesn’t perform the task, the evaluator might still want to know whether the child is usually able to do it. The child might be distracted by the evaluation context, and this may be useful information. In a situation like this, the evaluators would discuss how to score the item with reference to the criteria in the manual. She doesn’t believe that eligibility determinations are being made improperly.

13.  During an interview conducted by the Complaint Investigator with Nora Armstrong, M.S., OTR/L, Ms. Armstrong stated the following: She is employed by the Site as an occupational therapist, and in that capacity has administered the BDI-2 at least 50 times over the last 2 ½ years, both to Part C and Part B children. Before using the assessment, she attended full-day training on the test led by a representative from the Publisher. She then went out as part of an evaluation team with evaluators who were experienced with the BDI-2. She had course work in assessment tools related to occupational therapy. She received
instruction in the use of assessments, observed demonstrations and then engaged in supervised
administration of assessments. As part of her supervised field work in graduate school, she did observations of evaluations conducted in different settings.

Before beginning to use the BDI-2, she became very familiar with the administrator’s manual and proper scoring procedure. There is a copy of the manual and the five subdomain manuals in the Site office which are available to her. She also has her own BDI-2 kit, with copies of the five subdomain manuals. When administering the test, she will have with her whichever of the five manuals are relevant to the evaluation; if it is a complete evaluation, she would
have all of them with her. Some of the test items have specific scripts, especially for the older
children. These scripts have to be followed precisely, and she either reads them from the manual or she copies them out on a piece of paper and reads from that.

She has never been present during an evaluation where she felt that the BDI-2 was being used in a manner contrary to the training she received. She has never been made aware of an instance where in her opinion the assessment was being used improperly by someone connected with the Site. She doesn’t believe that there are fewer children receiving services
as a result of the manner in which the Site is using the BDI-2.  In fact, every instance in which
she administered the assessment to a Part B child has resulted in a determination of eligibility.

One of the case managers at the site had been talking to the Interested Party about the fact that a report that she did was in a different format than is typically used at the Site. There was a question about how the scores were reported, and the case manager referred the Interested Party to her. The Interested Party called her at home to discuss this issue, but she didn’t have the manual with her to enable her to have an in-depth conversation about it. She later looked at the scoring materials and the Interested Party’s report and saw what the Interested Party
was talking about to an extent, but it still wasn’t completely clear to her what the Interested Party was saying. The Interested Party was referring to something in the manual that was not relevant to how the Site uses the test and that she had never used.

14.  During an interview conducted by the Complaint Investigator with Tera Kennedy, M.Ed., Ms. Kennedy stated the following: She is a certified special education teacher for birth to 5 year olds, and is employed by the Site as a teacher and Part B coordinator. She has been
using the BDI-2 since it first came out in 2005.  She initially received informal training with
one of the standardization phase examiners for the BDI-2 at that time. Later, she took a graduate level course in assessments where she focused on the BDI-2, and then attended day- long training with a presenter certified by the Publisher in 2007.

She purchased her own BDI-2 kit before she began working for the Site. She always has the manuals with her during an evaluation, because she knows that there are certain things she needs to say exactly as scripted, and her memory skills are not that strong. Also, she tests children of various ages, and the scripts differ based on their age. She knows other evaluators who have memorized portions of the scripts, or who use notes on paper as cues to help them remember. Having to refer to the manuals is challenging when working with toddlers, but
less so with the older children.

She has never observed the BDI-2 being used improperly, and is not aware of an instance where the test was not properly administered or scored by someone connected with the Site. She believes that the staff responsible for administering the test are experienced and trained so that they understand and know how to administer the test.

15.  During an interview conducted by the Complaint Investigator with Pam Morse, M.Ed., Ms. Morse stated the following: She is employed by the Site as an early childhood special educator, and administers the BDI-2 to Part B children. She has been using this test since
2005, and has given it over 100 times. She has been familiar with the 1st edition of the test through her graduate work, which included coursework where she observed and recorded
various developmental assessments. When she was asked to work for the Belfast CDS site, she received training from the site director and another provider, both of whom were very
familiar with the assessment. She has not participated in training from the Publisher, and she is not certain whether the site director or provider was certified as a trainer by the Publisher.

There is a BDI-2 kit at the Site office that she shares with other employees. She always has the manuals with her when she administers the test. Before she sees a child, she will go through the protocol and flag the pages she needs to be looking at, and will do the same thing with the manuals so she can be sure she is following the appropriate protocol without disrupting the process and losing the child’s attention.

She reads the scripts from the manuals, but she is aware that others use excerpts and written notes. With Part C children, there is greater likelihood that one will stay within a given range so that excerpts should not be a problem, whereas she may have to go above or below the anticipated range. The important thing is that each child is being assessed in the same manner.  She has complete confidence in her team members; they all have different ways of ensuring that they follow proper procedure. She has not become aware of any instance where the assessment was not being administered properly.

VII.  Conclusions

Allegation #1: Failure to administer the Battelle Developmental Inventory in accordance with instructions provided by the producer of that assessment in violation of MUSER
§V.2.C(1)(e)
NO VIOLATION FOUND

It initially appeared that the focus of this complaint was on the distinction between “scaled scores” and “standard scores” for the various subdomains. Given that the Publisher itself describes the subdomain scores as “normalized standard scores,” the significance of this difference in nomenclature is unclear, and the investigation did not uncover evidence that the mislabeling of the scores on the summary sheet resulted in invalid results. Regardless, the Site acknowledged the error and corrected its forms to reference “scaled scores.”

The Complainant’s other concerns were in the areas of the availability and use of the administrator’s manuals, and of the qualifications of those administering the test. No evidence was uncovered that the manuals were not available to any evaluator who needed them. While some of the evaluators acknowledged that they did not always read from the manuals while administering the assessment, they explained how doing so could interfere
with the administration process. Instead, they utilized a combination of their memory, written

cues and selected excerpts of the scripts to ensure faithful adherence to the scripts when required. Even Ms. French acknowledged that she relies on her memory when administering tests with which she is sufficiently familiar.

As to the evaluators’ qualifications, each of the Site’s evaluators met the qualifications for administration of the test according to the Publisher’s own qualification criteria. No evidence was uncovered that any of the evaluators lacked sufficient training, experience or understanding so as to suggest that the Site’s entrusting the administration of the assessment
to her violated regulatory requirements. There was only one specific instance where the administration of the test was called into question. Even if Ms. French’s assessment of Ms. Armstrong’s testing acumen is credited, this does not amount to a systemic policy, practice or procedure at the Site, but is at most a personnel issue.

Furthermore, there was no evidence uncovered that any child has been denied services as a result of any activity of Ms. Armstrong or anyone else at the Site; to the contrary, there are more children receiving services under the Site’s supervision than there had been previously, including the child whose assessment was reportedly deemed invalid.

VIII. Corrective Action Plan

As no violation was found, none is required.