SUMMARY OF THE PERFORMANCE PARTNERSHIP GRANT (PPG) PERFORMANCE MEASUREMENT DEVELOPMENT PROCESS TO DATE

(This material provides some significant context related to the development of the strategies proposed by the current PPG work group)

Overview and Vision of the Plan

The primary goals of any performance measurement strategy include generating data that can be used for improving the quality of the services offered, results oriented management; and providing external accountability. The current conceptualization of performance measurement under a SAPT Performance Partnership Grant (PPG) would allow States to measure variables using a variety of methodologies and instrumentation; however, States would be required to collect data in "core" indicator areas and optional State-selected indices and assess their performance against negotiated objectives or targets using a continuous quality improvement framework. The PPGs would focus on State systems accountability by requiring States to measure current performance, set targets and adjust State system activities and priorities based on State's performance relative to these targets.

On November 7^th and 8^th, 2000, NASADAD"s State Data Advisory Workgroup on Treatment Performance Measures was last convened to flesh out some of the issues dealing with the PPG and web-based MIS systems. That group began the process in 1997 in a collaborative effort with CSAT.

The workgroup recommended that the PPG Implementation and Measurement Process should be built on Continuous Quality Improvement models in which structural, efficiency and effectiveness indicators are interrelated. It should also include how the collection and processing of the proposed variables will be implemented across States. At a December 4^th, 2001 SAMHSA/NASADAD meeting on Performance Partnership Implementation, it was suggested that a performance incentive program be developed. NASADAD has requested that this proposal be discussed further. It was also noted that PPG plan (SAPT Block Grant Application) development for the States should include means for ensuring public participation through public comment on plans and reports that the States produce. According to the legislation designating the SAPT Block Grant a Performance Partnership Grant (P.L. 106-310), a plan to be submitted by SAMHSA to Congress by October of 2002 must include how the States will receive greater flexibility, what performance measures will be used in holding States accountable for performance, definitions for the data elements that will be collected, the funds needed to implement the system, where these funds would come from, and needed legislative changes.

It is NASADAD’s position that the States should be allowed to demonstrate their performance only against relevant criteria within each State. Any appearance that States are directly "competitive" on any performance measures should be avoided. The developmental nature of outcome and performance measures, the lack of infrastructure support, consensus on uniform clinical instrumentation, and consensus on cost-efficient and effective methods for tracking outcomes, require that the Performance Partnership be flexible enough to allow States to begin to implement performance measurement within the context of their current operational capabilities. The reporting of identical, common performance measures across all States is simply not feasible at present. NASADAD believes that the use of obtainable, State-relevant data for any potential performance measure is vital.

In the past, there has been significant level of mutual State/Federal discussion surrounding treatment performance measurement. NASADAD’s State Data Advisory has met several times with CSAT and other Federal representatives to refine a core set of measures defined by the States’ participation in a Delphi process. The core effectiveness measures include changes in employment status, living arrangements, criminal justice involvement, and AOD use. Other measures dependent on linkages to related systems were proposed for inclusion in a "menu" of State option effectiveness measures.

The States were particularly concerned with practical implementation issues and operationalizing the defined variables. Of particular interest was the model of admission/post-admission follow-up versus a facility-based admission/discharge client change tracking system. SAMHSA expressed a need to demonstrate long-term client change and the States present seemed to buy in to an admission/six-months post-admission follow-up model. There seemed to be some concern that the AOD measure examining frequency of AOD use, a model measuring use 30 days prior to admission and use 30 days prior to discharge would be more practical for that measure. No consensus was reached on the AOD use variable however, the use of this 6-month post-admission protocol for most of the data variables measuring effectiveness was deemed acceptable by the workgroup.

Within the efficiency domain, it was agreed that measures of access and penetration would be critical. Defining a specific measure to assess treatment penetration rates was problematic and was deferred. Access and penetration indicators may need population-based figures to serve as denominators. Those figures, especially for subpopulations, are not consistently available except through separately funded needs assessment studies.

After the November 2000 meeting, discussions identified other conceptual issues such as resourcing/funding needs and infrastructure and staff requirements necessary for implementing a discharge/follow-up data system to collect treatment performance measures.

The States are concerned about the costs associated with implementing a performance measurement system. Those issues could be addressed, in part, by:

1. a formal statement issued by SAMHSA to the single State agencies (SSAs) which clarifies that block grant funds may be used for implementing PPG related data collection efforts: such clarification could emphasize the practicality of utilizing a portion of any block grant increases for that purpose;
2. for some variables, secondary analysis or data mining would be allowed to reduce measurement costs;
3. clarifying that measures requiring client follow-up would be sample-based; and
4. requiring that SAPT PPG applications - with a complete performance reporting component – be submitted every three years in concert with a needs assessment element. Expenditure data, budget projections and progress updates on performance based on administrative data readily available to the individual States might be done on an annual basis.

NASADAD will continue to use both face-to-face meetings and the Delphi process to examine and define performance measures and related issues with CSAT under its Cooperative Agreement with that agency. Those meetings will also explore data collection and methodological issues and to further develop the performance measurement process in general.

The use of social indicators or variables to inform needs assessment, State prioritization of needs, and State planning efforts is proposed to support and further quantify periodic, State-specific needs assessments. Data sources such as the National Household Survey of Drug Abuse and Healthy People 2010 may serve as the basis for some portion of this set of indicators.

State systems’ unique characteristics will need to be documented. This would be accomplished through a set of contextual variables displayed in a State profile somewhat akin to those formally published in NASADAD’s State Alcohol and Drug Abuse Profiles (SADAP). Information on proportions of clients served through specified modalities (TEDS definitions), demographic profile of the system’s clients, expenditures, average length of stay, and other variables could inform SAMHSA and the general public about the States’ distinct treatment systems. An anticipated benefit is that such information identifying the dissimilar aspect of State treatment systems would deter State-to-State comparisons of performance and outcome data.

Although the Children’s Health Act calls for waiver related criteria development, SAMHSA has indicated that it may seek legislative authority to add "core" measures specific to the targeted populations and remove specific process requirements. These targeted population measures would be utilized to assess State’s efforts relative to these populations rather than continuing with a process compliance model which includes a waiver authority. Inclusion of target-population specific performance and outcome measures were discussed at the November 7^th and 8^th, 2000, NASADAD State Data Advisory Workgroup meeting. For these target population measures, work will be undertaken with the States to develop consensus around data requirements. Target-population measures, as outlined in the Performance Partnership Block Grant (P.L. 106-310) would address the following populations and processes:

• services for pregnant women and women with dependent children (Section 1922 ( c));
• intravenous substance abuse services (Section 1923);
• requirements regarding tuberculosis (TB) and human immunodeficiency virus (HIV) (Section 1924); and
• improvement of the process for appropriate referrals for treatment, continuing education, and coordination of various activities and services (Section 1928).

States would be expected to report year-to-year on certain target-population related measures and their progress towards State-specified, time-framed performance objectives or targets. States not setting useful targets or States not performing acceptably in relation to their objectives or targets would have their State plans scrutinized to ensure the needs of these populations are prioritized in a manner consistent with good-faith efforts to improve services and outcomes for these target populations.

The next step is to convene the first in a new series of Performance Measurement meetings That meeting will be held in Washington, D.C. February 14 and 15, 2002. The purpose of the meeting is to obtain State agreement on the evolved model, to facilitate State input in the development, refinement and operationalization of the performance partnership process. Workgroups will be formed to review and assess possible social indicator data items to be incorporated in the States PPG application annual report, and to examine and develop proposed target-population measures and contextual data elements for the same. Three additional Performance Measurement meetings will be convened later this year to reach consensus on: (A) additional measures (e.g. penetration, social support, and target-population specific); (B) social indicator data to be included in the annual report; (C) contextual variables to be uniformly reported; (D) methodological issues; and (E) infrastructure needs.

The use of a web-based reporting system can help States collect necessary data elements in a more efficient and effective manner. Concerns regarding web-based reporting are currently being discussed and reviewed by the National Association of State Alcohol and Other Drug Abuse Directors (NASADAD) Single State Agency Web Workgroup (SSAWW) under CSAT’s NEDS II contract. The SSAWW is comprised of State AOD Directors and technical personnel and has been established to advise the development of individual State plans for the transition to web-based management information systems or to enhance existing systems.

 Return to Performance Partnership Grant page