Document alpinecountyca_gov_doc_88f6a4cf15

Behavioral Health Concepts, Inc. 5901 Christie Avenue, Suite 502 Emeryville, CA 94608 [EMAIL REDACTED] www.caleqro.com FY 17–18 MEDI-CAL SPECIALTY MENTAL HEALTH EXTERNAL QUALITY REVIEW ALPINE MHP FINAL REPORT Prepared for: California Department of Health Care Services (DHCS) Review Dates: August 02, 2017 ---PAGE BREAK--- - 1 - Alpine MHP CalEQRO Report Fiscal Year 2017–2018 TABLE OF CONTENTS ALPINE MHP SUMMARY OF FINDINGS 1 Introduction 1 Access 1 Timeliness 2 Quality 2 Outcomes 2 INTRODUCTION 4 Validation of Performance Measures 4 Performance Improvement Projects 5 MHP Health Information System Capabilities 5 Validation of State and County Consumer Satisfaction Surveys 5 Review of Recommendations and Assessment of MHP and Opportunities 6 PRIOR YEAR REVIEW FINDINGS, FY17-18 7 Status of FY16–17 Review of Recommendations 7 Changes in the MHP Environment and Within the MHP—Impact and Implications 11 PERFORMANCE MEASUREMENT 14 Total Beneficiaries Served 15 Penetration Rates and Approved Claim Dollars per Beneficiary 15 High-Cost Beneficiaries 19 Timely Follow-up After Inpatient Discharge 20 Diagnostic 21 Performance Measures Findings—Impact and Implications 22 PERFORMANCE IMPROVEMENT PROJECT VALIDATION 23 Alpine MHP PIPs Identified for Validation 23 Clinical PIP—Seeking Safety Treatment to Reduce Co-Occurring 25 Non-clinical PIP—Crisis Intervention Team 26 PIP Findings—Impact and Implications 27 PERFORMANCE AND QUALITY MANAGEMENT KEY COMPONENTS 29 Access to Care 29 Timeliness of Services 31 Quality of Care 32 Key Components Findings—Impact and Implications 36 CONSUMER AND FAMILY MEMBER FOCUS GROUP 39 Consumer/Family Member Focus Group 1 39 Consumer/Family Member Focus Group Findings—Implications 40 ---PAGE BREAK--- - 2 - Alpine MHP CalEQRO Report Fiscal Year 2017–2018 INFORMATION SYSTEMS REVIEW 42 Key Information Systems Capabilities Assessment (ISCA) Information Provided by the MHP 42 Summary of Technology and Data Analytical Staffing 43 Current Operations 44 Priorities for the Coming Year 45 Major Changes Since Prior Year 45 Other Significant Issues 45 Plans for Information Systems Change 46 Current Electronic Health Record Status 46 Personal Health 47 Medi-Cal Claims 47 Information Systems Review 48 SITE REVIEW PROCESS BARRIERS 50 CONCLUSIONS 51 and Opportunities 51 Recommendations 54 ATTACHMENTS 55 Attachment A—On-site Review Agenda 56 Attachment B—Review Participants 57 Attachment C—Approved Claims Source Data 60 Attachment D—PIP Validation Tools 61 ---PAGE BREAK--- - 1 - Alpine MHP CalEQRO Report Fiscal Year 2017–2018 LIST OF TABLES AND FIGURES Table 1: MHP Medi-Cal Enrollees and Beneficiaries Served in CY16, by Race/Ethnicity Table 2: High-Cost Beneficiaries Table 3: PIPs Submitted by MHP Table 4: PIP Validation Review Table 5: PIP Validation Review Summary Table 6: Access to Care Components Table 7: Timeliness of Services Components Table 8: Quality of Care Components Table 9: Distribution of Services, by Type of Provider Table 10: Summary of Technology Staff Changes Table 11: Summary of Data Analytical Staff Changes Table 12: Primary EHR Systems/Applications Table 13: Current EHR Functionality Table 14: MHP Summary of Short Doyle/Medi-Cal Claims Table 15: Summary of Top Three Reasons for Claim Denial Figure 1A: Overall Average Approved Claims per Beneficiary, CY14-16 Figure 1B: Overall Penetration Rates, CY14-16 Figure 2A: Foster Care Average Approved Claims per Beneficiary Figure 2B: Foster Care Penetration Rates, CY14-16 Figure 3A: Latino/Hispanic Average Approved Claims per Beneficiary, CY14-16 Figure 3B: Latino/Hispanic Penetration Rates, CY14-16 Figure 4A: 7-day Outpatient Follow-up and Rehospitalization Rates Figure 4B: 30-day Outpatient Follow-up and Rehospitalization Rates Figure 5A: Beneficiaries Served, by Diagnostic Categories, CY16 Figure 5B: Total Approved Claims by Diagnostic Categories, CY16 ---PAGE BREAK--- - 1 - Alpine MHP CalEQRO Report Fiscal Year 2017–2018 ALPINE MHP SUMMARY OF FINDINGS Beneficiaries served in CY16  36 MHP Threshold Language(s)  None MHP Size  Small Rural MHP Region  Central MHP Location  Markleeville MHP County Seat  Markleeville Introduction Alpine County is a small rural frontier county in the Central Sierra Nevada mountain range. The county is geographically isolated and is the least populated in California. Alpine County has no incorporated cities, but has five communities that are separated in winter by closed snow-covered mountain passes. The county has no stoplight, grocery store, bank, hospital, pharmacy, or jail. The largest employer is the county government and the primary industry is tourism. The Hung-A-Lel-Ti Band of the Washoe Tribe makes up about one-third of the population of this county. Alpine’s small population allows for specialty mental health services to be provided to every eligible individual. Alpine’s Mental Health Plan (MHP) engages in county-wide collaboration and timely identification and resolution of system and consumer-specific challenges. During the Fiscal Year 2017-18 (FY17-18) review, California External Quality Review Organization (CalEQRO) reviewers found the following overall significant changes, efforts, and opportunities related to access, timeliness, quality, and outcomes of the MHP and its contract provider services. Further details and findings from EQRO-mandated activities are provided in this report. Access The MHP relies exclusively on for medication services. The MHP has expanded its capacity over the past year, and currently has one vacant clinical position in Bear Valley. ---PAGE BREAK--- - 2 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Stakeholders reported that the county has no public transportation and the MHP’s transportation services are inadequate for consumers to access mental health services as needed. There is a high demand to transport consumers to/from clinical services and to pharmacies out of county for prescriptions. Crisis response remains disjointed due to the lack of in-county facilities and the reliance on the Sheriff’s Department to authorize all 5150s and transport consumers to emergency rooms and hospitals, which are out-of-county. The MHP does not have the authority to authorize 5150s. To improve crisis response, the MHP is engaged in a non-clinical PIP that focuses on the implementation of a Crisis Intervention Team (CIT) and Workgroup tasked with developing triage crisis response in Alpine County. Stakeholders reported that there is a cultural divide in Alpine between the two communities: Hung-A-Lel-Ti Tribal members and the Markleeville population. Cultural and mental health stigma was experienced by Tribal members and the MHP is working with local agencies to ameliorate this issue. Timeliness The MHP regularly met all of their timeliness standards over the past year and adeptly used their electronic Access and Information Log (in Microsoft Excel) to track timeliness measures. Staff received training on their information systems from Kings View. Quality The MHP established a Leadership Team, an Integrated Health Care Team (with Public Health), and a CIT (with law enforcement agencies), all of which were active in the past year. While the MHP collects data on programs and operations, they lack a truly data-driven system for program planning and quality improvement. The MHP submitted two Performance Improvement Projects (PIPs), both of which were determined to be active and ongoing. Stakeholders report that while formal mechanisms exist for communication from the MHP, word-of-mouth is the most common means of communication to learn about services, events, and issues. Outcomes The MHP has embedded outcome measures in their Electronic Health Record (EHR) and, in collaboration with Kings View, they are developing reports for routine analyses of individual and aggregate consumer outcomes. ---PAGE BREAK--- - 3 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 The MHP lacks a formal peer program and there are currently no job classifications for peers. In the spirit of wellness and recovery and to expand the MHP’s own capacity, the MHP and the county Board of Supervisors should consider developing a formal peer program with designated positions for peers. ---PAGE BREAK--- - 4 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 INTRODUCTION The United States Department of Health and Human Services (HHS), Centers for Medicare and Medicaid Services (CMS) requires an annual, independent external evaluation of State Medicaid Managed Care programs by an External Quality Review Organization (EQRO). External Quality Review (EQR) is the analysis and evaluation by an approved EQRO of aggregate information on quality, timeliness, and access to health care services furnished by Prepaid Inpatient Health Plans (PIHPs) and their contractors to recipients of State Medicaid managed care services. The CMS (42 CFR §438; Medicaid Program, External Quality Review of Medicaid Managed Care Organizations) rules specify the requirements for evaluation of Medicaid managed care programs. These rules require an on-site review or a desk review of each Medi-Cal Mental Health Plan (MHP). The State of California Department of Health Care Services (DHCS) contracts with 56 county Medi-Cal MHPs to provide Medi-Cal covered specialty mental health services (SMHS) to Medi-Cal beneficiaries under the provisions of Title XIX of the federal Social Security Act. This report presents the fiscal year 2017-2018 (FY17-18) findings of an EQR of the Alpine MHP by the California External Quality Review Organization (CalEQRO), Behavioral Health Concepts, Inc. (BHC). The EQR technical report analyzes and aggregates data from the EQR activities as described below: Validation of Performance Measures1 Both a statewide annual report and this MHP-specific report present the results of CalEQRO’s validation of eight mandatory performance measures (PMs) as defined by DHCS. The eight PMs include: • Total beneficiaries served by each county MHP; • Total costs per beneficiary served by each county MHP; • Penetration rates in each county MHP; 1 Department of Health and Human Services. Centers for Medicare and Medicaid Services (2012). Validation of Performance Measures Reported by the MCO: A Mandatory Protocol for External Quality Review (EQR), Protocol 2, Version 2.0, September, 2012. Washington, DC: Author. ---PAGE BREAK--- - 5 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 • Count of Therapeutic Behavioral Services (TBS) beneficiaries served compared to the 4% Emily Q. Benchmark2; • Total inpatient hospital episodes, costs, and average length of stay (LOS); • inpatient hospital 7-day and 30-day rehospitalization rates; • inpatient hospital 7-day and 30-day Specialty Mental Health Services (SMHS) follow-up service rates; and • High-Cost Beneficiaries (HCBs), incurring approved claims of $30,000 or higher during a calendar year. Performance Improvement Projects3 Each MHP is required to conduct two performance improvement projects (PIPs)—one clinical and one non-clinical—during the 12 months preceding the review. The PIPs are discussed in detail later in this report. MHP Health Information System Capabilities4 Using the Information Systems Capabilities Assessment (ISCA) protocol, CalEQRO reviewed and analyzed the extent to which the MHP meets federal data integrity requirement for Health Information Systems (HIS), as identified in 42 CFR §438.242. This evaluation included a review of the MHP’s reporting systems and methodologies for calculating PMs. Validation of State and County Consumer Satisfaction Surveys CalEQRO examined available consumer satisfaction surveys conducted by DHCS, the MHP, or its subcontractors. 2 The Emily Q. lawsuit settlement in 2008 mandated that the MHPs provide TBS to foster care children meeting certain at-risk criteria. These counts are included in the annual statewide report submitted to DHCS, but not in the individual county-level MHP reports. 3 Department of Health and Human Services. Centers for Medicare and Medicaid Services (2012). Validating Performance Improvement Projects: Mandatory Protocol for External Quality Review (EQR), Protocol 3, Version 2.0, September 2012. Washington, DC: Author. 4 Department of Health and Human Services. Centers for Medicare and Medicaid Services (2012). EQR Protocol 1: Assessment of Compliance with Medicaid Managed Care Regulations: A Mandatory Protocol for External Quality Review (EQR), Protocol 1, Version 2.0, September 1, 2012. Washington, DC: Author. ---PAGE BREAK--- - 6 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 CalEQRO also conducted 90-minute focus groups with beneficiaries and family members to obtain direct qualitative evidence from beneficiaries. Review of Recommendations and Assessment of MHP and Opportunities The CalEQRO review draws upon prior years’ findings, including sustained opportunities for improvement, and actions in response to recommendations. Other findings in this report include: • Changes, progress, or milestones in the MHP’s approach to performance management — emphasizing utilization of data, specific reports, and activities designed to manage and improve quality. • Ratings for key components associated with the following three domains: access, timeliness, and quality. Submitted documentation as well as interviews with a variety of key staff, contracted providers, advisory groups, beneficiaries, and other stakeholders inform the evaluation of the MHP’s performance within these domains. Detailed definitions for each of the review criteria can be found on the CalEQRO Website, www.caleqro.com. ---PAGE BREAK--- - 7 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 PRIOR YEAR REVIEW FINDINGS, FY17-18 In this section, the status of last year’s (FY16-17) recommendations are presented, as well as changes within the MHP’s environment since its last review. Status of FY16–17 Review of Recommendations In the FY16-17 site review report, the CalEQRO made a number of recommendations for improvements in the MHP’s programmatic and/or operational areas. During the FY17-18 site visit, CalEQRO and MHP staff discussed the status of those FY16-17 recommendations, which are summarized below. Assignment of Ratings Met is assigned when the identified issue has been resolved. Partially Met is assigned when the MHP has either: • Made clear plans and is in the early stages of initiating activities to address the recommendation; or • Addressed some but not all aspects of the recommendation or related issues. Not Met is assigned when the MHP performed no meaningful activities to address the recommendation or associated issues. Key Recommendations from FY16-17 Recommendation Evaluate the effectiveness of maintaining a 60-day baseline standard to measure timeliness from initial contact to first appointment. Status: Met • The MHP is not tracking initial contact to first appointment. However, they are tracking timeliness from the referral to a provider (from a clinician) to both the first offered and the first kept appointment. • While the MHP is maintaining 60 days as their standard for time to first appointment, actual timeliness for FY16-17 was 13 days. • The MHP contracts four hours per week of services through Kings View for both adult and children’s services. The lives out of state and has visited Alpine County once in the past year. Consumers and staff ---PAGE BREAK--- - 8 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 reported it would be helpful to have more frequent in-person contact. A clinician is always present during sessions. The MHP maintains that there is no waitlist for and that when no-shows occur, the time is used for case consultations with clinicians. • The MHP stated that due to a small staff and large workload, changing the standard on how timeliness to first appointment is measured is not a priority at this time. In addition, the MHP prefers to do additional talk-therapy sessions before recommending services. • As all appointments are through the MHP facilitates transportation from Kirkwood, Bear Valley, and other remote communities to the only site which is located at Hung-A-Lel-Ti in Woodfords. Recommendation Identify and increase staff hours/percentage of (Full Time Employees) FTE staff devoted to working with Kings View to upgrade the EHR, develop and analyze reports based on EHR capabilities and provide staff training. Status: Partially Met • An Administrative Assistant I has been trained in the past year to identify chart errors in coding and billing. • Through training by Kings View, as Administrative Assistant III has increased their knowledge and production of reports to analyze billable time, timeliness of documentation, and review clinical data entry. Additionally, many MHP staff participated in other Kings View trainings during the year. • The MHP receives a Alpine County Penetration Report produced by Kings View, but appears to do minimal analysis of the data for program planning and quality improvement purposes. • No new functionality or Cerner modules were added to the EHR during the year. Recommendation Build on the recent incorporation of client functioning outcome measures into the EHR, along with clinical staff training: o Institute regular and consistent application of these tools for baseline and follow-up measures of all consumers; and o Develop individual and aggregate level reports to monitor consumer outcomes and guide clinical care. (A variation of this recommendation is repeated from FY14-15 and FY15-16). Status: Met ---PAGE BREAK--- - 9 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 • The MHP included the following outcome tools in their EHR: Adult Needs and Assessment (ANSA)/Child and Adolescent Needs and (CANS); Patient Health Questionnaire (PHQ-9); Generalized Anxiety Disorder (GAD-7); and Milestones Of Recovery Scale (MORS). All active clients have received CANS/ANSA, and other tools as clinically indicated. • The MHP has begun to produce pre- and post-test results reports for internal review. They are working with Kings View to configure the reports in a way that will be helpful, as well as learning how to do aggregate analyses. • The MHP uses hardcopies of the Adverse Childhood Experience (ACE) Questionnaire; Pediatric Checklist; and the Commercially and Sexually Exploited Children (CSEC) Checklist, as appropriate for consumers. Recommendation Develop two active PIPs, one Clinical and the other non-clinical that are data driven with consumer input. Use sufficient data to establish the prevalence of an issue impacting Alpine’s consumers. Focus both PIPs on consumer outcomes, with measurable client focused indicators and interventions with repeated measures to assess implications over time. (This recommendation is repeated from FY15-16) Status: Met • The goal of the clinical PIP is to determine whether a brief intervention to reduce substance use among Behavioral Health Services (BHS) clients will reduce emotional and behavioral impairments and improve daily functioning. The MHP demonstrated solid use of their data to select this PIP topic. Preliminary results, although not statistically significant, suggest that using the Seeking Safety intervention for consumers with co-occurring disorders reduces consumer-reported This PIP is considered active and ongoing. • The goal of the non-clinical PIP is to decrease mental health crises that require a response by law enforcement, with a special focus on consumers who have experienced multiple crisis episodes that required a response by law enforcement. The MHP has implemented a Crisis Intervention Team (CIT) with law enforcement agencies to de-escalate situations and provide of appropriate safe and timely) crisis services. This PIP is considered active and ongoing. Recommendation As there are no clinical services provided in Bear Valley, investigate the level of need among local residents and determine how best to address them re-open the Bear Valley clinic; expand services). Status: Met • The Bear Valley clinic has recently reopened after severe flooding closed the facilities a year ago. The Bear Valley Mental Health Services Act (MHSA) ---PAGE BREAK--- - 10 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 program increased the staff time from 19 to 32 hours per week. Recruitment is in process for a part-time Clinician position. • Bear Valley has a large and growing MHSA program, which includes outreach activities and Skype services for individual and group therapy which are provided in the Public Health Clinic in Bear Valley. services are not offered in Bear Valley. • The Bear Valley clinic provides year-round, regularly scheduled clinical and appointments, as well as groups, which are conducted via Skype. During the summer months, the Clinical Coordinator drives to Bear Valley at least twice for face-to-face appointments and accommodates Bear Valley clients more frequently as needed. During the winter months, when inclement weather prohibits frequent travel to the area, the Clinical Coordinator drives out • The Bear Valley site is not Medi-Cal certified, therefore no Medi-Cal services can be billed. The MHP has no current plans to certify this site. ---PAGE BREAK--- - 11 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Changes in the MHP Environment and Within the MHP— Impact and Implications Discussed below are any changes since the last CalEQRO review that were identified as having a significant effect on service provision or management of those services. This section emphasizes systemic changes that affect access, timeliness, and quality, including any changes that provide context to areas discussed later in this report. Access to Care • The MHP has established a Leadership Team, an Integrated Health Care Team with Public Health, and a CIT with law enforcement agencies, all of which meet • The MHP increased their capacity by: hiring an MHSA Coordinator; filling MHSA Specialist and Native Wellness Advocate positions; and increasing the hours of the MHSA Specialist in Bear Valley. • The MHP continues to expand the depth and breadth of consumer groups that support outreach, engagement, and wellness and recovery for the population, primarily using MHSA funds. • Through a PIP, the MHP implemented a CIT to initiate an effective and consistent response to behavioral health crises by law enforcement, mental health and addiction professionals, and peers. If resources allow, the MHP, the Sheriff’s Office, and other first responders may implement the CIT similar to the Mobile Outreach Safety Team in neighboring Douglas County in the State of Nevada. Douglas County staff are providing training and support to Alpine County’s CIT efforts. • There is no hospital in Alpine County. Depending on ambulance service, consumers who require 72-hour holds are transported to either Barton Hospital in South Lake Tahoe (in El Dorado County) or Carson Tahoe Behavioral Health (in Carson City, Nevada). • The county has prioritized obtaining Medicare certification which they are hoping to secure over the next year. • In the Spring of 2018, the MHP plans to break ground on a new/larger 5,000 square foot building that will be located across the street from the existing Markleeville clinic. In addition to MHP clinical services, this new space will be used for other community activities. ---PAGE BREAK--- - 12 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Timeliness of Services • The MHP’s timeliness improved for FY16-17 services, as reflected in the CalEQRO FY17-18 Timeliness Self-Assessment (TSA) tool. • Eight MHP staff, in addition to one primary user, were trained to use the MHP’s electronic (Excel) Access and Information Log. Quality of Care • The MHP screens and assesses all children, but the MHP asserts that there continue to be no youth who meet criteria for Katie A. The MHP Director attended the California Department of Social Services Continuum of Care Reform (CCR) training. One Medi-Cal clinician attended a two-day Child and Family Teaming (CFT) training provided in Inyo County. • The MHP continues to maintain a number of dual Medi-Cal/MHSA funded programs including: Field Capable Clinical Services, which extends services to homes and community settings; Full Service Partnerships (FSPs) for all ages; and, school-based services including a school-based mental health clinician, Primary Intervention Program, playgroups, and Positive Behavior Intervention Support at the county’s one school, Diamond Valley School. • The Alpine County Superior Court has begun to more consistently refer defendants with co-occurring mental health and substance use disorder (SUD) diagnoses to the MHP for assessment and treatment. • The MHP maintained robust trainings in FY16-17 including the mainstay evidence-based practices (EBPs) of Mental Health First Aid (MHFA); Trauma Informed Care/Trauma-Focused Cognitive Behavioral Therapy (TF-CBT); Applied Suicide Intervention Skills Training; First Aid; and Dialectical Behavioral Treatment (DBT). MHP staff participated in additional trainings including: Drug Suspicion; Sex Trafficking (by Awaken); and Inside the Manipulator’s Mind. • Through MHSA funding, the MHP continued to operate many community activities. A few examples include a Talking Circle for Alcohol and Other Drug (AOD); Family, Teen and Community Movie nights; and yoga in Bear Valley, Kirkwood, Markleeville, Hung-a-Lel-Ti, and Woodfords. ---PAGE BREAK--- - 13 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Consumer Outcomes • The MHP implemented the 2016 Spring and Fall Performance Outcome and Quality Improvement (POQI) consumer satisfaction surveys. The MHP distributed the results system-wide, but did not analyze them. ---PAGE BREAK--- - 14 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 PERFORMANCE MEASUREMENT As noted above, CalEQRO is required to validate the following PMs as defined by DHCS: • Total beneficiaries served by each county MHP; • Total costs per beneficiary served by each county MHP; • Penetration rates in each county MHP; • Count of TBS Beneficiaries Served Compared to the 4% Emily Q. Benchmark (not included in MHP reports; this information is included in the Annual Statewide Report submitted to DHCS); • Total inpatient hospital episodes, costs, and average LOS; • inpatient hospital 7-day and 30-day rehospitalization rates; • inpatient hospital 7-day and 30-day SMHS follow-up service rates; and • HCBs incurring $30,000 or higher in approved claims during a calendar year. HIPAA Suppression Disclosure: Values are suppressed to protect confidentiality of the individuals summarized in the data sets where beneficiary count is less than or equal to eleven Additionally, suppression may be required to prevent calculation of initially suppressed data, corresponding penetration rate percentages and cells containing zero, missing data or dollar amounts ---PAGE BREAK--- - 15 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Total Beneficiaries Served Table 1 provides detail on beneficiaries served by race/ethnicity. Starting with CY16 performance measures, CalEQRO has incorporated the ACA Expansion data in the total Medi-Cal enrollees and beneficiaries served. See Attachment C, Table C1 for the penetration rate and approved claims per beneficiary for just the CY16 ACA Penetration Rate and Approved Claims per Beneficiary. Penetration Rates and Approved Claim Dollars per Beneficiary The penetration rate is calculated by dividing the number of unduplicated beneficiaries served by the average enrollee count. The average approved claims per beneficiary served per year is calculated by dividing the total annual dollar amount of Medi-Cal approved claims by the unduplicated number of Medi-Cal beneficiaries served per year. Regarding calculation of penetration rates, the Alpine MHP uses the same method used by CalEQRO. Race/Ethnicity Average Unduplicated Medi-Cal Enrollees % Enrollees Unduplicated Annual Count of Beneficiaries Served % Served White 144 40.3% 13 36.1% Latino/Hispanic 25 7.0% * n/a African-American * n/a * n/a Asian/Pacific Islander * n/a * n/a Native American 160 44.8% 21 58.3% Other 27 7.6% * n/a Total 357 100% 36 100% Table 1: Alpine MHP Medi-Cal Enrollees and Beneficiaries Served in CY16, by Race/Ethnicity The total for Average Unduplicated Medi-Cal Enrollees is not a direct sum of the averages above it. The averages are calculated independently. ---PAGE BREAK--- - 16 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Figures 1A and 1B show 3-year (CY14-16) trends of the MHP’s overall approved claims per beneficiary and penetration rates, compared to both the statewide average and the average for small-rural MHPs. $0 $1,000 $2,000 $3,000 $4,000 $5,000 $6,000 $7,000 $8,000 $9,000 CY14 CY15 CY16 Figure 1A. Overall Average Approved Claims per Beneficiary Alpine Small-Rural State 0.00% 2.00% 4.00% 6.00% 8.00% 10.00% 12.00% 14.00% CY14 CY15 CY16 Figure 1B. Overall Penetration Rates Alpine Small-Rural State ---PAGE BREAK--- - 17 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Figures 2A and 2B show 3-year (CY14-16) trends of the MHP’s foster care (FC) approved claims per beneficiary and penetration rates, compared to both the statewide average and the average for small-rural MHPs. $0 $2,000 $4,000 $6,000 $8,000 $10,000 $12,000 CY14 CY15 CY16 Figure 2A. FC Average Approved Claims per Beneficiary Alpine Small-Rural State 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% CY14 CY15 CY16 Figure 2B. FC Penetration Rates Alpine Small-Rural State ---PAGE BREAK--- - 18 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Figures 3A and 3B show 3-year (CY14-16) trends of the MHP’s Latino/Hispanic approved claims per beneficiary and penetration rates, compared to both the statewide average and the average for small-rural MHPs. $0 $1,000 $2,000 $3,000 $4,000 $5,000 $6,000 CY14 CY15 CY16 Alpine Small-Rural State Figure 3A. Latino/Hispanic Average Approved Claims per Beneficiary 0.00% 1.00% 2.00% 3.00% 4.00% 5.00% 6.00% 7.00% CY14 CY15 CY16 Figure 3B. Latino/Hispanic Penetration Rates Alpine Small-Rural State ---PAGE BREAK--- - 19 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 High-Cost Beneficiaries Table 2 compares the statewide data for High-Cost Beneficiaries (HCBs) for CY16 with the MHP’s data for CY16, as well as the prior two years. HCBs in this table are identified as those with approved claims of more than $30,000 in a year. See Attachment C, Table C2 for the distribution of the MHP beneficiaries served by approved claims per beneficiary (ACB) range for three cost categories: under $20,000; $20,000 to $30,000; and those above $30,000. MHP Year HCB Count Total Beneficiary Count HCB % by Count Average Approved Claims per HCB HCB Total Claims HCB % by Approved Claims Statewide CY16 18,909 598,296 3.16% $53,219 $1,006,318,438 29.02% CY16 * 36 n/a $0 $0 n/a CY15 * 30 n/a $0 $0 n/a CY14 * 26 n/a $0 $0 n/a Table 2: Alpine MHP High-Cost Beneficiaries Alpine ---PAGE BREAK--- - 20 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Timely Follow-up After Inpatient Discharge Figures 4A and 4B show the statewide and MHP 7-day and 30-day outpatient follow-up and rehospitalization rates for CY15 and CY16. 0% 10% 20% 30% 40% 50% 60% 70% Outpatient MHP Outpatient State Rehospitalization MHP Rehospitalization State Figure 4A. 7-Day Outpatient Follow-up and Rehospitalization Rates, Alpine and State CY15 CY16 0% 10% 20% 30% 40% 50% 60% 70% Outpatient MHP Outpatient State Rehospitalization MHP Rehospitalization State Figure 4B. 30-Day Outpatient Follow-up and Rehospitalization Rates, Alpine and State CY15 CY16 ---PAGE BREAK--- - 21 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Diagnostic Categories Figures 5A and 5B compare the breakdown by diagnostic category of the statewide and MHP number of beneficiaries served and total approved claims amount, respectively, for CY16. MHP self-reported percent of consumers served with co-occurring (substance abuse and mental health) diagnoses: 17.2%. 0% 5% 10% 15% 20% 25% 30% 35% Depression Disruptive Bipolar Anxiety Adjustment Other Deferred Figure 5A. Diagnostic Categories, Beneficiaries Served Alpine CY16 State CY16 0% 5% 10% 15% 20% 25% 30% 35% Depression Disruptive Bipolar Anxiety Adjustment Other Deferred Figure 5B. Diagnostic Categories, Total Approved Alpine CY16 State CY16 ---PAGE BREAK--- - 22 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Performance Measures Findings—Impact and Implications Access to Care • The MHP’s CY16 overall penetration rate decreased to 10.08% from the CY15 rate of 12.10%. This remains well above the small-rural MHP and statewide overall rates. • The MHP’s CY16 FC penetration rate decreased from CY15 and is below the small-rural MHP and statewide rates. • The MHP served no Latino/Hispanic enrollees in CY16, despite an increase in Hispanic enrollees in Alpine County. Timeliness of Services • The MHP had no hospitalizations in the CalEQRO data for CY16. Quality of Care • The MHP continues to have no HCBs. • The MHP’s overall average ACBs have decreased since CY14. The MHP’s average remains well above small-rural MHP’s, but below the statewide averages, which have both remained stable since CY14. • The MHP had no claims for Hispanic enrollees in CY16. • The percentage of MHP consumers with primary diagnoses of depression and increased in CY16 similar to statewide. The percentage of consumers in CY16 with “Other” diagnoses was greatly than the statewide trend. Consumer Outcomes • None noted. ---PAGE BREAK--- - 23 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 PERFORMANCE IMPROVEMENT PROJECT VALIDATION A Performance Improvement Project (PIP) is defined by CMS as “a project designed to assess and improve processes and outcomes of care that is designed, conducted, and reported in a methodologically sound manner.” The Validating Performance Improvement Projects Protocol specifies that the EQRO validate two PIPs at each MHP that have been initiated, are underway, were completed during the reporting year, or some combination of these three stages. DHCS elected to examine projects that were underway during the preceding calendar year. Alpine MHP PIPs Identified for Validation Each MHP is required to conduct two PIPs during the 12 months preceding the review. CalEQRO reviewed and validated two MHP-submitted PIPs, as shown below. Table 3 lists the findings for each section of the evaluation of the PIPs, as required by the PIP Protocols: Validation of Performance Improvement Projects.5 Table 3: PIPs Submitted by Alpine MHP PIPs for Validation # of PIPs PIP Titles Clinical PIP 1 Seeking Safety Treatment to Reduce Co-Occurring Non-clinical PIP 1 Crisis Intervention Team Table 4, on the following page, provides the overall rating for each PIP, based on the ratings given to the validation items: Met Partially Met (PM), Not Met (NM), Not Applicable (NA), Unable to Determine (UTD), or Not Rated (NR). 5 2012 Department of Health and Human Services, Centers for Medicare and Medicaid Service Protocol 3 Version 2.0, September 2012. EQR Protocol 3: Validating Performance Improvement Projects. ---PAGE BREAK--- - 24 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Table 4: PIP Validation Review Step PIP Section Validation Item Item Rating Non- Clinical clinical 1 Selected Study Topics 1.1 Stakeholder input/multi-functional team M M 1.2 Analysis of comprehensive aspects of enrollee needs, care, and services M M 1.3 Broad spectrum of key aspects of enrollee care and services M M 1.4 All enrolled populations M PM 2 Study Question 2.1 Clearly stated PM PM 3 Study Population 3.1 Clear definition of study population M M 3.2 Inclusion of the entire study population M M 4 Study Indicators 4.1 Objective, clearly defined, measurable indicators M PM 4.2 Changes in health status, functional status, enrollee satisfaction, or processes of care M PM 5 Sampling Methods 5.1 Sampling technique specified true frequency, confidence interval and margin of error NA NA 5.2 Valid sampling techniques that protected against bias were employed NA NA 5.3 Sample contained sufficient number of enrollees NA NA 6 Data Collection Procedures 6.1 Clear specification of data M PM 6.2 Clear specification of sources of data M PM 6.3 Systematic collection of reliable and valid data for the study population M PM 6.4 Plan for consistent and accurate data collection M PM 6.5 Prospective data analysis plan including contingencies M PM 6.6 Qualified data collection personnel M PM 7 Assess Improvement Strategies 7.1 Reasonable interventions were undertaken to address causes/barriers M PM 8 Review Data Analysis and Interpretation of Study Results 8.1 Analysis of findings performed according to data analysis plan PM NA 8.2 PIP results and findings presented clearly and accurately M NA 8.3 Threats to comparability, internal and external validity M NA 8.4 Interpretation of results indicating the success of the PIP and follow-up PM NA 9 Validity of Improvement 9.1 Consistent methodology throughout the study M NA 9.2 Documented, quantitative improvement in processes or outcomes of care UTD NA 9.3 Improvement in performance linked to the PIP UTD NA 9.4 Statistical evidence of true improvement UTD NA 9.5 Sustained improvement demonstrated through repeated measures. UTD NA ---PAGE BREAK--- - 25 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Table 5 provides a summary of the PIP validation review. Table 5: PIP Validation Review Summary Summary Totals for PIP Validation Clinical PIP Non-clinical PIP Number Met 18 5 Number Partially Met 3 11 Number Not Met 0 0 Number Applicable (AP) (Maximum = 28 with Sampling; 25 without Sampling) 25 16 Overall PIP Rating ((#Met*2)+(#Partially Met))/(AP*2) 78.00% 65.63% Clinical PIP—Seeking Safety Treatment to Reduce Co- Occurring The MHP presented its study question for the clinical PIP as follows: “Will the implementation of the 8-week Seeking Safety curriculum with clients having a moderate or high number of of a substance use disorder reduce emotional and behavioral impairments as measured in the GAIN-SS domains of externalizing disorders, internalizing disorders, substance use, or criminal behavior?” Date PIP began: October 2016 Status of PIP: Active and Ongoing The goal of this PIP is to determine whether a brief intervention to reduce substance use among Alpine County BHS clients will also reduce emotional and behavioral impairments and improve daily function. After training staff on the Seeking Safety intervention model and screening all current adult outpatient clients using the Global Appraisal of Individual Needs-Short Screener (GAIN-SS), the MHP implemented an 8-week Seeking Safety group counseling program for consumers with co-occurring disorders. The group counseling program included an incentive for attendance and retention, a $25 gift card at the completion of each group session. After completing the 8-week group counseling, participants were re-evaluated using the GAIN-SS. The total number of reported at the completion of the group was compared to the number of reported during the initial screening, representing “the past month”. Seeking Safety Clients who continue to meet study criteria are referred to continue ---PAGE BREAK--- - 26 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 with the Seeking Safety group intervention. New clients who meet the criteria for Seeking Safety will also be referred and may join the program at the start of each 8-week treatment interval. In the interim/preliminary analysis, the MHP reported that clients who participated in the Seeking Safety intervention did not report an increase in while those who declined to participate did report a higher number of occurring “during the past month” when pre-treatment and post-treatment scores were compared. The MHP plans to continue analyzing other treatment intervals over the next year. The MHP reported that the use of incentives to increase attendance and engagement in the Seeking Safety intervention appeared to be at least moderately successful. Relevant details of these issues and recommendations are included within the comments found in the PIP validation tool. The technical assistance provided to the MHP by CalEQRO consisted of detailed discussion regarding the sections of the PIP and how to strengthen them to ensure greater reliability of PIP results. For the MHP to continue the PIP, they should restate the study question to be more measurable by adding “as increased by x% or x score using the screening tool”. The MHP needs to add another measure that is more sensitive, and able to determine if the intervention is leading to real changes in outcomes for consumers. Several tools were discussed and the MHP will determine which one they will incorporate moving forward. Additionally, some changes in the data collection process are needed. While this is an initial short intervention, if the same clients are continuing the treatment modality, the MHP should consider changes to the treatment interval expanding to 12, 16 or longer weeks). For the second year of the PIP, the MHP will need to incorporate additional interventions for the PIP to be considered active and continuing. The MHP is encouraged to continue this PIP for a second year, using the recommendations stated above to improve the PIP. Non-clinical PIP—Crisis Intervention Team The MHP presented its study question for the non-clinical PIP as follows: “Will establishing a Crisis Intervention Team (CIT) Program in Alpine County decrease the number of behavioral health crises for individuals who require response and intervention by law enforcement during these crises by increasing their engagement in local long-term behavioral health treatment after the crisis?” Date PIP began: June 2017 Status of PIP: Active and Ongoing The goal of the non-clinical PIP is to decrease mental health crises that require a response by law enforcement, with a special focus on those consumers who experience multiple crisis ---PAGE BREAK--- - 27 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 episodes that required a law enforcement response. The MHP is addressing de-escalation and provision of safe and timely crisis services through the development of an Alpine County CIT Team and through increased consumer engagement in ongoing mental health services. To date, Alpine County has developed an interagency CIT and held several meetings. The team is in the process of discussing shared policies and protocols for crisis response including information and data sharing. Two Sheriff’s Deputies participated in a 40-hour CIT training and additional county staff will be trained in the coming year. The MHP was initially challenged in obtaining baseline data from the Sheriff’s department, particularly for crisis calls. The MHP was eventually able to obtain baseline data for 5150 calls and so are beginning with this indicator and will phase in other indicators as they become available. The PIP focuses on consumers who experience multiple, repeated crises that involve a response by a law enforcement agency. While law enforcement data include both residents and non-residents of Alpine County, the MHP is only able to focus on following up with consumers who remain in county. The study question needs to be strengthened and made measurable. The data analysis plan requires much more detail, as does the plan for data collection, data sharing, improved coding, and these could be added to the interventions. The interventions as written will only measure part of what is envisioned for PIP and need to be expanded (see examples provided in the PIP Validation Tool). This PIP is still in the early stages and the MHP is working on data collection, analysis, improvement of interventions, and indicators to strengthen their measurements. Relevant details of these issues and recommendations are included within the comments found in the PIP validation tool. The technical assistance provided to the MHP by CalEQRO consisted of detailed discussion regarding the sections of the PIP and how to strengthen them to ensure greater reliability of PIP results. The MHP is encouraged to continue this PIP for a second year, using the recommendations mentioned above to improve it. PIP Findings—Impact and Implications Access to Care • The clinical PIP screens all adult consumers, both existing and new, for inclusion. • The non-clinical PIP focuses on consumers who experience multiple, repeated crises that involve a response by law enforcement. While the data is for both ---PAGE BREAK--- - 28 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 residents and non-residents, the MHP is only able to focus on following up with county residents who remain in county. Timeliness of Services • The implementation of a CIT in Alpine County is designed to improve timeliness of crisis services for consumers, and ongoing consumer engagement with mental health services. Quality of Care • The clinical PIP treats consumers with co-occurring disorders for both mental health and substance use. • The focus of the non-clinical PIP is to decrease mental health crises that require a response by law enforcement, with a special focus on those who experience multiple episodes. Consumer Outcomes • Preliminary results suggest that using the Seeking Safety intervention for consumers with co-occurring disorders reduces consumer-reported • Through the implementation of a CIT and improved engagement with behavioral health services, Alpine County is working towards improving consumer outcomes for crises. ---PAGE BREAK--- - 29 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 PERFORMANCE AND QUALITY MANAGEMENT KEY COMPONENTS CalEQRO emphasizes the MHP’s use of data to promote quality and improve performance. Components widely recognized as critical to successful performance management include an organizational culture with focused leadership and strong stakeholder involvement, effective use of data to drive quality management, a comprehensive service delivery system, and workforce development strategies that support system needs. These are discussed below, along with their quality rating of Met Partially Met (PM), or Not Met (NM). Access to Care Table 6 lists the components that CalEQRO considers representative of a broad service delivery system that provides access to consumers and family members. An examination of capacity, penetration rates, cultural competency, integration, and collaboration of services with other providers forms the foundation of access to and delivery of quality services. Table 6: Access to Care Components Component Quality Rating 1A Service accessibility and availability are reflective of cultural competence principles and practices M The MHP has an active Cultural Competency Committee (CCC) which met four times in the last year with a standing agenda and regular meeting minutes. The CCC lacks designated peer positions. Though not a threshold language, the MHP provides all materials in Spanish. Alpine County has a high percentage of Native American residents, Native American Medi-Cal enrollees, and very high Medi-Cal penetration rate. Hispanic Medi-Cal enrollment is increasing in the county, as is the “Other” Medi-Cal race/ethnicity category. The MHP has no male clinical staff which some consumers reported was problematic. The MHP has one male staff person (the Driver) and one Native American employee. The TAY population remains the most difficult to serve due to the very small size of the community which results in a lack of privacy. Alpine County has no high schools or colleges. There are also cultural challenges for Tribal Youth due to stigma. Staff and consumers report that there is a cultural split in Alpine with virtually two communities, Hung-A-Lel-Ti Tribal members and the Markleeville population. Using MHSA funds, the MHP is working with other county agencies and community-based organizations to increase integration and collaboration in order to reduce stigma and improve overall ---PAGE BREAK--- - 30 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 relationships. The Bear Valley clinic has recently reopened after severe flooding closed the facility a year ago. The MHSA Program Specialist hours increased from 19 to 32 per week and recruitment is in process for a part-time Clinician position. With a growing MHSA program, provision of outreach activities and Skype clinical services have expanded with year-round, regularly scheduled clinical and appointments and groups. 1B Manages and adapts its capacity to meet consumer service needs M The MHP’s CY16 retention in services rate percent of consumers who received five or more Medi-Cal services in CY16) decreased greatly and is well below the statewide rate. The MHP provides considerable flexibility to engage and meet the clinical needs of Alpine County residents as evidenced by having a full-time driver who facilitates transportation for consumers to/from therapeutic services; allowing staff to work from various offices and field-based sites; and maintaining walk-in hours weekly. The MHP also accommodates consumers who arrive outside of those hours. The MHP contracts four hours per week of services through Kings View for both adult and children’s services. As all appointments are through in the Markleeville clinic, the MHP facilitates transportation from Kirkwood, Bear Valley, and other remote communities to the The MHP uses Crisis Support Services of Alameda County as their crisis line. The MHP increased growth by: hiring an MHSA Coordinator; filling MHSA Specialist and Native Wellness Advocate positions; and increasing the hours of the MHSA Specialist in Bear Valley. The MHP continues to provide extensive training opportunities for staff on a regular basis. The county does not have a mental health crisis stabilization unit, emergency room, hospital, or residential facility, making de-escalation of crises in the county difficult. However, the MHP has an ongoing CIT PIP to try to achieve better coordination and triaging of mental health crises. Two county sheriff deputies have completed the 40-hour CIT certification course. The county lacks any Federally Qualified Health Center, Rural Health Clinics, or other primary care clinics. Medi-Cal services are not provided in Kirkwood or Bear Valley, which are the county’s two smaller communities of approximately 150 residents each. 1C Integration and/or collaboration with community-based services to improve access M The MHP partners with programs and organizations throughout the county including the one school, Sheriff’s Department, Public Health and other community based organizations as evidenced by meeting minutes and group interviews. The MHP continues to contract with Tahoe Youth and Family Services, which sends clinicians to Alpine to provide individual and family counseling, crisis line, texting line, emergency shelter for youth, and prevention services. The MHP continues their contract with Crisis Support Services of Alameda County to answer ---PAGE BREAK--- - 31 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 their 24/7 Access and Crisis Line after hours and on weekends/holidays. The MHP collaborates with County Public Health on the Primary Care Project. The County does not have a Drug Medi-Cal (DMC) program, but provides alcohol and other drug services through their federal Substance Abuse and Prevention Treatment Block Grant. The Alpine County Superior Court has begun to more consistently refer defendants with co- occurring mental health and SUD diagnoses to the MHP for assessment and treatment. While Alpine County currently has no foster youth, there is one group home with 12 beds that provides services for youth from ten other counties. The group home has submitted their Short-term Residential Therapeutic Program application, but has yet to be certified. To date, Alpine county has not received any presumptive transfer requests, but they are prepared to contract with the if needed. Timeliness of Services As shown in Table 7, CalEQRO identifies the following components as necessary to support a full service delivery system that provides timely access to mental health services. This ensures successful engagement with consumers and family members and can improve overall outcomes, while moving beneficiaries throughout the system of care to full recovery. Table 7: Timeliness of Services Components Component Quality Rating 2A Tracks and trends access data from initial contact to first appointment M The MHP’s standard is 10 business days, which they met 100% of the time. Average wait time was five days for adults and children. The MHP calculates timeliness to first appointment from their Access and Information Log using the dates of initial phone or “walk-in” contact and first scheduled clinical appointment. 2B Tracks and trends access data from initial contact to first appointment M The MHP’s standard is 60 days, which they met 100% of the time. Average wait time was 13 days for both adults and children, which was an improvement from 17 days reported in last year’s Timeliness Self-Assessment (TSA). The MHP calculates timeliness to first appointment using the dates of the clinician’s referral for first appointment. The MHP is ---PAGE BREAK--- - 32 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 tracking timeliness data from referral to to both the first offered and first kept appointment. 2C Tracks and trends access data for timely appointments for urgent conditions M The MHP’s standard is one day, which they met 100% of the time for all consumers. The MHP calculates this metric from the Access Log by calculating the time from phone calls or “walk-ins” denoted as urgent to the date of first appointment. 2D Tracks and trends timely access to follow-up appointments after hospitalization M The MHP’s standard is seven days. The MHP met this 100% of the time for FY16-17 with an average wait time of five days, which has been the average for the last two years. CalEQRO data showed no hospitalizations for CY16. The MHP uses their Inpatient Log for hospital discharge date and compares this to the first post-discharge appointment on the Access Log. 2E Tracks and trends data on rehospitalizations M The MHP’s standard is 0% rehospitalizations. The MHP reported no rehospitalizations for FY16-17 and CY16 CalEQRO data show no hospitalizations. 2F Tracks and trends no-shows M The MHP raised their no show standards from 10% to 20% for clinicians/non- and from 2% to 20% for The MHP changed the no show rate standards because the MHP changed methodologies for calculating this metric. They now use Cerner Scheduler widget/functionality instead of the Access Log. The MHP met these new standards 100% of the time. However, the No Show rates increased from 3.1% for to 13% and from 0.07% to 14%, for and respectively. Quality of Care In Table 8, CalEQRO identifies the components of an organization that is dedicated to the overall quality of care. Effective quality improvement activities and data-driven decision making require strong collaboration among staff (including consumer/family member staff), working in information systems, data analysis, clinical care, executive management, and program leadership. Technology infrastructure, effective business processes, and staff skills in extracting and utilizing data for analysis must be present in order to demonstrate that analytic findings are used to ensure overall quality of the service delivery system and organizational operations. ---PAGE BREAK--- - 33 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Table 8: Quality of Care Components Component Quality Rating 3A Quality management and performance improvement are organizational priorities M The MHP established a Leadership Team, an Integrated Health Care Team with Public Health, and a CIT with law enforcement, all of which meet The Quality Improvement Committee (QIC) and CCC meet alternating months and have standing agendas with meeting minutes. There is an updated QI work plan for FY17-18 and an evaluation of the previous year’s work plan. The MHP’s clinical staff, MHP Director, and some MHSA staff have been trained in many EBPs and services including: Mental Health First Aid; Trauma-Informed Care; Motivational Interviewing; Inside the Manipulator’s Mind; First Aid; CIT; and, DBT among others. Staff have also been trained on newer topics included Sex Trafficking Training through Awaken; Drug Suspicion Training; and, CFT. These clinical trainings were conducted by external entities and MHP staff. Six staff (administrative and clinical) received one or more trainings from Kings View so the MHP can now generate their own reports to do data analysis, and to use data for management and clinical improvement activities. The MHP submitted both a clinical and non-clinical PIP, both of which are active and ongoing. 3B Data are used to inform management and guide decisions M The Administrative Assistant I for Contracted Providers maintains the MHP’s Access and Information Log, which includes routine access, crisis and test calls, and has trained eight MHP staff. The MHP’s penetration report contains demographic and service information on Medi-Cal enrollees, number of Medi-Cal enrollees served by the MHP, and prevalence rates by age, ethnicity, gender and language. However, the MHP appears to do minimal analysis of the data for program planning and quality improvement. The MHP’s FY 2015-16 Cultural and Linguistic Competence Plan includes: analysis of Alpine County residents by gender, age and race/ethnicity; penetration rates for Medi-Cal and MHSA services combined; and, consumer utilization by Medi-Cal service type. Staff report that internet connectivity and computer downtime unreliable and slow Wi-Fi and system crashes) interfere with workflow and service delivery, particularly for Cell phone coverage is also difficult in many areas of the county. Staff ---PAGE BREAK--- - 34 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Table 8: Quality of Care Components Component Quality Rating document technical problems in a log and work closely with IT staff to resolve. 3C Evidence of effective communication from MHP administration, and stakeholder input and involvement on system planning and implementation M The MHP’s “Guide to Mental Health Services” brochure (in English and Spanish) highlights available services, including culturally-specific services. The “Provider List” is updated and contains provider names, population specialty, services provided, language capability, and whether the provider is accepting new clients. The MHP has a website and a Facebook page. The MHP communicates to consumers through newsletters, billboards, emails, and a calendar that is delivered door-to-door in the community. The QIC and CCC generally meet in alternating months and have many members in common. Staff reported that they are not involved with these committees, but some staff participated in the PIPs. Consumers and family members reported that two-way communication is effective in Alpine County. 3D Evidence of a systematic clinical continuum of care M The MHP provides individual and group therapy as service modalities. As there are no designated case managers in the system of care, case management functions are performed by clinicians. While consumers report that therapists see them weekly or biweekly and work closely with them on their treatment plans, consumers did not feel they were receiving case management services. The MHP maintains the use of EBPs including Mental Health First Aid, Trauma Informed Care/Trauma-Focused Cognitive Behavioral Therapy, Applied Suicide Intervention Skills Training, First Aid and DBT. The MHP engages in collaborative care with probation and Child Protective Services. Kings View provides medication monitoring services for the MHP. Consumers who are prescribed medication are provided with written medication information. Frequent turnover of Tribal health providers makes it challenging to maintain a close working relationship between MHP and Tribal practitioners. Referrals for specialty healthcare cardiologists, dentists, and optometry) are challenging due to a lack of providers in county. ---PAGE BREAK--- - 35 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Table 8: Quality of Care Components Component Quality Rating 3E Evidence of consumer and family member employment in key roles throughout the system NM The MHP’s practice is to fill certain positions with consumer staff, but there are no designated positions where lived-experience is a requirement. Staff report that clients would like jobs within the MHP as few job opportunities exist in the county. Leadership maintains that designated peer positions, either employee or volunteer, have not been a priority. The county has a Job Counselor position, whose function is to connect peers with county employment, which is currently vacant. 3F Consumer run and/or consumer driven programs exist to enhance wellness and recovery M Alpine has two MHSA funded wellness centers, one in Bear Valley and the other in Markleeville), both of which are open to the public. Consumers reported that they learned about the wellness center through word-of-mouth and flyers; no packet was provided when initiating services. Staff report that the wellness center has regularly scheduled programs that are mostly attended by the Tribal community. No Wellness and Recovery Action Plan (WRAP) classes are available, and consumers often request herbal remedies. Staff and consumers report that transportation continues to be a challenge with only one driver available. Engagement with the teen population remains challenging as there is no high school in county and all youth are transported daily to a neighboring county in the State of Nevada. 3G Measures clinical and/or functional outcomes of consumers served M The MHP has included ANSA, CANS, PHQ-9, GAD-7 and MORS in the Anasazi/Cerner EHR. The MHP is implementing ANSA or CANS with all new consumers and conducting annual reviews with existing consumers. The MHP routinely uses the GAIN-SS to screen for co- occurring disorders in adults. The MHP uses hardcopies of the ACE Questionnaire; Pediatric Checklist; and, the Commercially and Sexually Exploited Children Checklist, as appropriate for consumers. The MHP is in the process of developing processes for routine analyses at the individual ---PAGE BREAK--- - 36 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Table 8: Quality of Care Components Component Quality Rating and aggregate level. The MHP is working with Kings View to configure the reports and learning how to perform aggregate analyses. 3H Utilizes information from Consumer Satisfaction Surveys PM The MHP provided CalEQRO with its Spring and Fall 2016 DHCS POQI survey results for children, caregivers, adults, and older adults. The MHP distributed the results system-wide, but did not analyze them. Key Components Findings—Impact and Implications Access to Care • The TAY population remains the most difficult to serve due to size of the community. • The MHP contracts services for four hours per week through Kings View for both adult and children’s services. • Staff and consumers report that transportation continues to be a challenge given that there is only one driver available to facilitate transportation for consumers to/from therapeutic services and to/from pharmacies out-of-county. Other staff do facilitate some consumer transportation needs. • Due to the size of the county, many agencies work closely together, but informally. Communication between agencies and with the public is often through word-of-mouth. The MHP also partners formally with other programs and organizations throughout the county and in neighboring counties. • Consumers and family members report that two-way communication is effective. Timeliness of Services ---PAGE BREAK--- - 37 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 • The MHP is meeting all of their timeliness standards. While the MHP is maintaining a 60-day standard for time to first appointment, actual timeliness for FY16-17 was 13 days. • The MHP is tracking timeliness data from referral to to both the first offered and first kept appointment. • From their Access and Information Log, the MHP can calculate all timeliness elements described in the FY17-18 CalEQRO TSA. The MHP receives approximately 400 calls per year and eight staff besides the primary user were trained on using the Access and Information Log in FY16-17. Quality of Care • The MHP established a Leadership Team, an Integrated Health Care Team with Public Health, and a CIT with law enforcement, all of which meet • The QIC and CCC meet alternating months and have standing agendas, with subsequent meeting minutes. There is an updated QI work plan for FY17-18 and an evaluation of the previous year’s work plan. • The MHP submitted both a clinical and non-clinical PIP, both of which are active and ongoing. • Kings View produces a Penetration Report by Fiscal Year which the MHP reviews, but minimal analysis preclude program planning and quality improvement. Consumer Outcomes • Case Management is done by all clinicians as the MHP has no case manager positions. While consumers report that therapists see them weekly or every two weeks and work closely with them on their treatment plans, consumers do not feel they are receiving case management services. • The MHP lacks an organized peer program with designated consumer positions that leverage lived experience to support clinical care, and enhance wellness and recovery. The county Job Counselor position that is supposed to connect peers in the county is currently vacant, which limits consumer employment options. • The MHP maintains the use of several EBPs and is in the process of developing protocols for routine analysis of outcome measures at the individual and aggregate levels. ---PAGE BREAK--- - 38 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 • The MHP conducts two surveys annually and while results are distributed system-wide, they are not analyzed or used to inform system change. ---PAGE BREAK--- - 39 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 CONSUMER AND FAMILY MEMBER FOCUS GROUP CalEQRO conducted one 90-minute focus group with consumers and family members during the site review of the MHP. As part of the pre-site planning process, CalEQRO requested one focus group with 8 to 10 participants each, the details of which can be found in each section below. The consumer/family member focus group is an important component of the CalEQRO site review process. Obtaining feedback from those who are receiving services provides significant information regarding quality, access, timeliness, and outcomes. The focus group questions are specific to the MHP being reviewed and emphasize the availability of timely access to care, recovery, peer support, cultural competence, improved outcomes, and consumer and family member involvement. CalEQRO provides gift certificates to thank the consumers and family members for their participation. Consumer/Family Member Focus Group 1 CalEQRO requested a culturally diverse group of adult beneficiaries and parents/caregivers of child/youth beneficiaries who are mostly new clients who have initiated/utilized services within the past 12 months. This focus group was culturally diverse and held at the Hung-A- Lel-Ti Firehouse in Markleeville, CA. Number of participants: 7 (Only one consumer initiated services within the past year, and therefore all comments are combined to protect the identity of the participants). General comments regarding service delivery that were mentioned included the following: • There are two sides: one, Native American and the other, residents of Markleeville. Native American participants stated that there is a difference in how staff treat them. Participants provided an example of being disrespected. Consumers in Markleeville did not believe that there were problems, but stated that they did not attend the Hung-A-Lel-Ti wellness center due to lack of transportation. • Participants reported that they completed the annual survey but never turned it in because they had no way to return it to the MHP. Recommendations for improving care included the following: • Add more energy medicine holistic) to clinical therapy instead of using only the medical model, as this would combine and honor both. Examples of holistic ---PAGE BREAK--- - 40 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 approaches include Eye Movement Desensitization and Reprocessing therapy, Emotional Freedom Technique or Tapping, Reiki, and other body work. • There are more opportunities in Markleeville and there needs to be parity. • There is a need for more transportation not only for clinical care, but also for groups and events. Existing transportation services need to be provided in a more respectful manner. • Staff need to be friendlier and more welcoming on the phone. • Lunches should be provided. Interpreter used for focus group 1: No Language: N/A Consumer/Family Member Focus Group Findings— Implications Access to Care • Stakeholders reported that there is a cultural divide in Alpine between the two communities: Hung-A-Lel-Ti Tribal members and the Markleeville population. Cultural and mental health stigma was experienced by Tribal members, and the MHP is working with local agencies to ameliorate this issue. Consumers reported primarily hearing information informally, through others, as opposed to official means of communication from the MHP such as flyers, posters, and meetings. • Transportation was the barrier that most identified for access to care. Timeliness of Services • Timely access to initial and ongoing services was reportedly not a problem for new and existing consumers. Quality of Care • WRAP is not currently being used as a clinical tool. Consumers requested an opportunity to learn about and utilize this or a similar evidence-based model to fully engage in and facilitate their recovery. ---PAGE BREAK--- - 41 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 • Concerns were voiced regarding consumers having to call the MHP to get permission to speak with a member of the Mental Health Board. • Consumers shared their desire to receive alternative therapies in addition to talk therapy and medication and consumers wished the doctor and clinicians were open to their suggestions. Consumer Outcomes • The MHP should consider the addition of paid (or stipend) peer support positions by hiring consumers and family members. Peer positions not only empower consumers, they help consumers to realize their potential for helping others and contributing to their own recovery. • There was a perception among the focus group participants of not being liked by staff. Consumers did not feel that they had a voice or were being heard. ---PAGE BREAK--- - 42 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 INFORMATION SYSTEMS REVIEW Understanding an MHP’s information system’s capabilities is essential to evaluating its capacity to manage the health care of its beneficiaries. CalEQRO used the written response to standard questions posed in the California-specific ISCA, additional documents submitted by the MHP, and information gathered in interviews to complete the information systems evaluation. Key Information Systems Capabilities Assessment (ISCA) Information Provided by the MHP The following information is self-reported by the MHP through the ISCA and/or the site review. Table 9 shows the percentage of services provided by type of service provider. Table 9: Distribution of Services, by Type of Provider Type of Provider Distribution County-operated/staffed clinics 75% Contract providers 25% Network providers 0% Total 100% Percentage of total annual MHP budget dedicated to supporting information technology operations (includes hardware, network, software license, IT staff): The budget determination process for information system operations is: MHP currently provides services to consumers using a application: ☒ Yes ☐ No ☐ In pilot phase ☒ Under MHP control ☐ Allocated to or managed by another County department ☐ Combination of MHP control and another County department or Agency ---PAGE BREAK--- - 43 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Number of remote sites currently operational: One Identify primary reason(s) for using as a service extender (check all that apply): ☒ Hiring healthcare professional staff locally is difficult ☐ For linguistic capacity or expansion ☐ To serve outlying areas within the county ☐ To serve consumers temporarily residing outside the county ☐ Reduce travel time for healthcare professional staff ☐ Reduce travel time for consumers services are available with English speaking practitioners (not including the use of interpreters or language line). Summary of Technology and Data Analytical Staffing MHP self-reported technology staff changes (FTE) since the previous CalEQRO review are shown in Table 10. MHP self-reported data analytical staff changes (in FTEs) that occurred since the previous CalEQRO review are shown in Table 11. Table 10: Technology Staff IS FTEs (Include Employees and Contractors) # of New FTEs # Employees / Contractors Retired, Transferred, Terminated Current # Unfilled Positions 1 1 0 0 ---PAGE BREAK--- - 44 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 The following should be noted with regard to the above information: • The MHP has one main site in the Hung-A-Lel-Ti clinic, and no remote sites. The MHP contracts for four hours of a week. • The MHP has no threshold languages and, for several years, has had no consumers who speak a language other than English. • Though the MHP has not hired—and does not have—a full or part-time staff person solely devoted to data analyst activities, two of the three administrative staff and three of five Medi-Cal clinical staff received training from Kings View. EHR and Information Technology (IT) training activities included Anasazi/Cerner documentation, group notes, progress notes, third party billing, report writing, scheduling, and Kings View billing. The Medi-Cal Administrative Assistant I has the capability to develop and run reports. • The one county IT Department employee stated that he devotes approximately 15 percent of his time to the MHP. Current Operations • Kings View continues to provide support for both technology needs and data analytics. • The MHP utilizes Citrix to connect to the application service provider (ASP) hosted application via the county internet connection. The ASP host (Kings View) utilizes virtualized terminal service connected to physical database servers. • Though not hired as data the MHP’s Clinical Coordinator, Medi-Cal Administrative Assistant I, and MHSA Administrative Assistant III have received fairly extensive Kings View training and are considered Subject Matter Experts/”super users” of the EHR and/or the Access and Information Log. • The MHP continues to use Excel for their Access and Information Log Scheduler) and eight staff were trained in its use during FY16-17. Table 11: Data Analytical Staff IS FTEs (Include Employees and Contractors) # of New FTEs # Employees / Contractors Retired, Transferred, Terminated Current # Unfilled Positions 0 0 0 0 ---PAGE BREAK--- - 45 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 • The MHP continues to use Microsoft Access for data extraction and report generation. • Kings View continues to produce penetration reports with updated data during each Fiscal Year. • The MHP has rapidly increased the use of through Kings View. The number of consumers served increased from 20 in FY15-16 to 86 in FY16- 17. • The fragility of the internet infrastructure in the county, extreme weather events, and outages caused by Cerner’s system upgrades were largely responsible for the downtime for the EHR (8.57%) in FY16-17 and the MHP is tracking this downtime. Table 12 lists the primary systems and applications the MHP uses to conduct business and manage operations. These systems support data collection and storage, provide electronic health record (EHR) functionality, produce Short-Doyle/Medi-Cal (SD/MC) and other third party claims, track revenue, perform managed care activities, and provide information for analyses and reporting. Table 12: Primary EHR Systems/Applications System/Application Function Vendor/ Supplier Years Used Operated By Community Behavioral Health (CBH) EHR (Client Data, Assessments, Treatment Plans, Doctor’s Homepage, MCO) Cerner 4 Kings View Priorities for the Coming Year • The MHP would like to expand the number of available clinical forms in the EHR. Major Changes Since Prior Year • The MHP implemented the new Progress Note functionality in March 2017. • The MHP converted to the non-axial diagnosis review and implemented DSM-5 upgrades in April 2017. Other Significant Issues ---PAGE BREAK--- - 46 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 • The MHP does not provide Medi-Cal services in their Bear Valley site and does not intend to Medi-Cal certify this site. This is due to the low consumer volume. • The MHP does not measure/track staff productivity. Plans for Information Systems Change • Implementation of Cerner Community Behavioral Health with Kings View as ASP remains in progress in Year Current Electronic Health Record Status Table 13 summarizes the ratings given to the MHP for EHR functionality. Table 13: EHR Functionality Function System/ Application Rating Present Partially Present Not Present Not Rated Alerts Cerner CBH X Assessments Cerner CBH X Care Coordination Cerner CBH X Document imaging/storage Cerner CBH X Electronic signature— consumer Cerner CBH X Laboratory results (eLab) Cerner CBH X Level of Care/Level of Service Cerner CBH X Outcomes Cerner CBH X Prescriptions (eRx) Cerner CBH X Progress notes Cerner CBH X Referral Management Cerner CBH X Treatment plans Cerner CBH X Summary Totals for EHR Functionality 6 0 6 0 Progress and issues associated with implementing an electronic health record over the past year are discussed below: • The MHP has not implemented electronic consumer signatures and this is not a current objective for this year. Limited internet capacity in the county does not ---PAGE BREAK--- - 47 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 make it feasible to implement consumer signatures in the field, such as in the Diamond Valley School. • Prescribing is done by Kings View via The MHP enters prescriptions into the EHR manually, which are transmitted to them by Kings View. • Document imaging can be done from two computers/work stations at the Markleeville clinic. Consumer’s Chart of Record for county-operated programs (self-reported by MHP): ☐ Paper ☐ Electronic ☒ Combination Personal Health Record Do consumers have online access to their health records either through a Personal Health Record (PHR) feature provided within the EHR, consumer portal, or third-party PHR? ☐ Yes ☒ No If no, provide the expected implementation timeline. ☐ Within 6 months ☐ Within the next year ☐ Within the next two years ☒ Longer than 2 years Medi-Cal Claims Processing MHP performs end-to-end (837/835) claim transaction reconciliations: If yes, product or application: Microsoft Excel. The data file is created by the ASP. Method used to submit Medicare Part B claims: ☐ Paper ☒ Electronic ☐ Clearinghouse Table 14 summarizes the MHP’s SDMC claims. ☒ Yes ☐ No ---PAGE BREAK--- - 48 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Table 15 summarizes the most frequently cited reasons for claim denial. Information Systems Review Findings—Implications Access to Care • The MHP provides through Kings View at the Hung-A-Lel-Ti clinic near Markleeville. The number of consumers served increased from 20 in FY15- 16 to 89 in FY16-17. • The MHP does not have capability in Bear Valley or Kirkwood. The MHP currently uses Skype when necessary in winter months in Bear Valley in the Alpine Public Health Clinic). • The MHP may start working towards their CIT PIP and Committee. The MHP may consider implementing a Mobile Outreach Safety Team similar to Douglas County, Nevada, which includes behavioral health, law enforcement, and paramedic responders. Timeliness of Services • The MHP calculates timeliness measures through their Access and Information Log, which is separate from the EHR. Eight MHP staff were trained in use of this log in FY16-17. Quality of Care Number Submitted Gross Dollars Billed Number Denied Dollars Denied Percent Denied Gross Dollars Adjudicated Claim Adjustments Gross Dollars Approved 626 $192,249 1 $478 0.25% $191,771 $0 $191,771 Table 14: Alpine MHP Summary of CY16 Short Doyle/Medi-Cal Claims Note: Includes services provided during CY16 with the most recent DHCS processing date of May 19, 2017 Denial Reason Description Number Denied Dollars Denied Percent of Total Denied Beneficiary not eligible or aid code invalid or restricted service indicator must be 1 $478 100% Total Denied Claims 1 $478 100% Table 15: Alpine MHP Summary of CY16 Top Three Reasons for Claim Denial ---PAGE BREAK--- - 49 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 • Kings View continues to produce the annual FY Penetration Report, which is updated The MHP did not conduct any analyses of this data in FY16-17. • The MHP included analyses of demographic trends, EHR, and CSI data in their FY 2015-16 Cultural and Linguistic Competency Plan (CLCP). The MHP did not complete an FY16-17 CLCP. • Though more MHP staff have undergone Kings View training, the MHP provided no analyses and presented no special reports independent of the Kings View FY penetration report. Consumer Outcomes • The MHP implemented CANS, ANSA, MORS, PHQ-9 and GAD-7 in the EHR in 2015. However, the MHP provided no evidence of analysis of outcomes at either the individual or aggregate levels for FY16-17. ---PAGE BREAK--- - 50 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 SITE REVIEW PROCESS BARRIERS The following conditions significantly affected CalEQRO’s ability to prepare for and/or conduct a comprehensive review: • No barriers were encountered during the preparation or implementation of this review. ---PAGE BREAK--- - 51 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 CONCLUSIONS During the FY17-18 annual review, CalEQRO found in the MHP’s programs, practices, or information systems that have a significant impact on the overall delivery system and its supporting structure. In those same areas, CalEQRO also noted opportunities for quality improvement. The findings presented below relate to the operation of an effective managed care organization, reflecting the MHP’s processes for ensuring access to and timeliness of services and improving the quality of care. and Opportunities Access to Care • The MHP filled three vacancies and currently has only one vacant position, which is the clinician in Bear Valley. • The non-clinical PIP focuses on the implementation of a CIT and a CIT Workgroup that is tasked with developing triage crisis response in the county. Joint response by law enforcement, behavioral health, and other first responders fire, ambulance) is being discussed, as is mobile response with electronic capability at the point of service. However, the MHP maintains that they do not have the resources available to provide field-based joint response. • In the Spring of 2018, the MHP plans to break ground on a new/larger 5,000 square foot building that will be located across the street from the existing Markleeville clinic. In addition to MHP clinical services, this new space will be used for other community activities. • The MHP is able to electronically forward prescriptions to pharmacies in Gardnerville in Nevada and to a pharmacy in South Lake Tahoe. Opportunities: • The MHP and county Board of Supervisors are not considering adding more MHP positions. EQRO encourages the county to consider including male clinicians and creating a job classification that requires lived experience and has the potential for advancement. • Consumers must drive or be transported for clinical services and to remote pharmacies to pick up prescriptions. No public transportation is available in ---PAGE BREAK--- - 52 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 county and both staff and consumers reported that an additional driver is needed. • Crisis response remains disjointed because the MHP does not have 5150 authority and placements ERs and hospitals) of 5150s are all out-of-county. • At this time, CIT activities in the county do not include peers, and the CCC lacks designated peer positions. • Stakeholders reported that there is a cultural divide in Alpine between the two communities: Hung-A-Lel-Ti Tribal members and the Markleeville population. Cultural and mental health stigma was experienced by Tribal members, and the MHP is working with local agencies to ameliorate this issue. Timeliness of Services • Timely access to initial and ongoing services was not a problem for new or existing consumers. While the MHP is maintaining a 60-day standard for time to first appointment, actual timeliness for FY16-17 was 13 days. The MHP is tracking timeliness data from referral to to both the first offered and first kept appointment. • From their Access and Information Log, the MHP can calculate all timeliness elements described the FY17-18 CalEQRO TSA. Quality of Care • The MHP established a Leadership Team, an Integrated Health Care Team with Public Health, and a CIT with law enforcement. • The goal of the clinical PIP is to determine whether a brief intervention to reduce substance use for MHP consumers will reduce emotional and behavioral impairments and improve daily functioning. • Stakeholders report that while formal mechanisms exist, being a very small rural county results in word-of-mouth as the most common means of communication. Opportunities: • While the MHP collects data on programs and operations, they lack a truly data- driven system for program planning and quality improvement purposes. ---PAGE BREAK--- - 53 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 • The MHP did not complete an FY16-17 CLCP. The MHP states they will complete a CLCP, including both FY16-17 and 17-18, in early 2018 which will also address DHCS’s new network adequacy requirements. • The lives out of state and has only visited the county once in the past year. Staff and consumers report that it would be advantageous to have him visit at least quarterly for face-to-face discussion and clinical care. • The MHP needs to work more closely with Tribal police to forge institutional relationships and processes that endure regardless of staffing changes and the MHP needs to encourage Tribal participation in the CIT. Consumer Outcomes • The MHP has included ANSA, CANS, PHQ-9, GAD-7 and MORS in the Anasazi/Cerner EHR. The MHP conducts ANSA or CANS with all new consumers and conducts annual reviews for existing consumers. • The MHP routinely uses the GAIN-SS to screen for co-occurring mental health and SUD diagnoses in adult consumers. The MHP uses hardcopies of the ACE Questionnaire; Pediatric Checklist; and the Commercially and Sexually Exploited Children Checklist, as appropriate for consumers. • The MHP is in the process of developing processes for routine analyses at the individual and aggregate level and they are working with Kings View to configure the reports accordingly. Opportunities: • The MHP should consider the addition of (stipend) peer support positions by hiring consumers and family members. This would have the added benefit of expanding MHP capacity and adding the consumer employee voice to local initiatives. • WRAP is not currently being used as a clinical tool. Consumers need to be provided an opportunity to learn about and utilize this or a similar evidence based model to fully engage in and facilitate their recovery. • Concerns were voiced regarding consumers having to obtain the MHP’s permission to speak with a member of the Mental Health Board. ---PAGE BREAK--- - 54 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 • There was consensus among the focus group participants regarding the perception of not being liked by staff and subsequently not having a voice or being heard. Recommendations • The MHP should transition to a Continuous Quality Improvement (QIC) approach and increase its use of data to drive quality improvement activities for both administrative/operational systems and clinical care, as evidenced by: o Upgrading an existing position to or contracting with Kings View for additional data to analyze existing reports and/or create project-specific data reports. o Continue systematizing utilization of clinical outcome tools, and initiate analysis of consumer outcomes at the individual and aggregate level. o Obtaining and reviewing/utilizing medication monitoring and other reports from Kings View. • The MHP should formalize and implement a paid or stipend peer assist/volunteer program to increase its capacity to support consumer engagement, retention, clinical service delivery, recovery, and peer involvement in the QIC, cultural competency committee, crisis intervention teams and other committees and initiatives. • Implement the consumer signature in the electronic health record (EHR) at the Markleeville and Woodfords clinic sites. • Evaluate the provision of additional transportation options for consumers throughout the county add an additional driver). ---PAGE BREAK--- - 55 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 ATTACHMENTS Attachment A: CalEQRO On-site Review Agenda Attachment B: On-site Review Participants Attachment C: Approved Claims Source Data Attachment D: CalEQRO Performance Improvement Plan (PIP) Validation Tools ---PAGE BREAK--- - 56 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Attachment A—On-site Review Agenda The following sessions were held during the MHP on-site review, either individually or in combination with other sessions. Table A1—EQRO Review Sessions - Alpine MHP Opening Session – Changes in the past year; current initiatives; and status of previous year’s recommendations Use of Data to Support Program Operations Disparities and Performance Measures/ Timeliness Performance Measures Quality Improvement and Outcomes Performance Improvement Projects Clinical Line Staff Group Interview Consumer Employee Group Interview Consumer Family Member Focus Group Validation of Findings for Pathways to Mental Health Services (Katie A./CCR) ISCA/Billing/Fiscal Forensics and Law Enforcement Group Interview ---PAGE BREAK--- - 57 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Attachment B—Review Participants CalEQRO Reviewers Della Dash, Senior Quality Reviewer, Lead Reviewer Richard Hildebrand, Information Systems Reviewer Marilyn Hillerman, Consumer Family Member Consultant Additional CalEQRO staff members were involved in the review process, assessments, and recommendations. They provided significant contributions to the overall review by participating in both the pre-site and the post-site meetings and in preparing the recommendations within this report. Sites of MHP Review MHP Sites Hung-A-Lel-Ti Firehouse, 96 Washoe Blvd., Markleeville, CA 96120 ---PAGE BREAK--- - 58 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Table B1 - Participants Representing the MHP Last Name First Name Position Agency Angie Cathy Clinician Alpine County Behavioral Health Services Arnold Lisa Probation Administrator Alpine County Probation Baker John Volunteer Firefighter Eastern Alpine Fire Broadhurst Amy MHSA Coordinator Alpine County Behavioral Health Services Case Spencer Undersheriff Alpine County Sheriff’s Office Dee Misty AOD Program Specialist Alpine County Behavioral Health Services Del Vecchio Michele Assistant Conservator Alpine County District Attorney’s Office DiSalvo Tami Chief Probation Officer Alpine County Probation Ellis Nani Coordinator Alpine County Behavioral Health Services Kaner Michelle Admin. Asst. I Alpine County Behavioral Health Services Kolpacoff Tom Judge Alpine County Court Krayk MHSA Program Specialist Bear Valley, Alpine County Behavioral Health Services Leonard Charles IS Specialist Information Technology Lovell Rita MHB / Librarian Alpine County Library Lowry Brian Deputy Sheriff Alpine County Sheriff’s Office Nourse Alissa Director Alpine County Behavioral Health Services Pitts Clinician Alpine County Behavioral Health ---PAGE BREAK--- - 59 - Alpine County MHP CalEQRO Report Fiscal Year 2017–18 Table B1 - Participants Representing the MHP Last Name First Name Position Agency Services Riddle Dawn MHSA Program Specialist Alpine County Behavioral Health Services Smokey Melanie Native Wellness Advocate Alpine County Behavioral Health Services St. James Gail Clinical Coordinator Alpine County Behavioral Health Services Stevens Janet Clinician Alpine County Behavioral Health Services Sweeney Jane Board Chair, BHS Alpine County Behavioral Health Services Williamson Nicole Director Health and Human Services ---PAGE BREAK--- - 60 - County MHP CalEQRO Report Fiscal Year 2017–18 Attachment C—Approved Claims Source Data Approved Claims Summaries are provided separately to the MHP in a HIPAA-compliant manner. Values are suppressed to protect confidentiality of the individuals summarized in the data sets where beneficiary count is less than or equal to eleven Additionally, suppression may be required to prevent calculation of initially suppressed data, corresponding penetration rate percentages and cells containing zero, missing data or dollar amounts Table C1 shows the penetration rate and approved claims per beneficiary for just the CY16 ACA Penetration Rate and Approved Claims per Beneficiary. Starting with CY16 performance measures, CalEQRO has incorporated the ACA Expansion data in the total Medi-Cal enrollees and beneficiaries served. Table C2 shows the distribution of the MHP beneficiaries served by approved claims per beneficiary (ACB) range for three cost categories: under $20,000; $20,000 to $30,000, and those above $30,000. Entity Average ACA Enrollees Number of Beneficiaries Served Penetration Rate Total Approved Claims Approved Claims per Beneficiary Statewide 3,674,069 137,620 3.75% $599,045,852 $4,353 Small-Rural 30,196 2,064 6.84% $5,756,116 $2,789 Alpine 120 13 10.83% $50,713 $3,901 Table C1: Alpine MHP CY16 Medi-Cal Expansion (ACA) Penetration Rate and Approved Claims per Beneficiary Range of ACB MHP Count of Beneficiaries Served MHP Percentage of Beneficiaries Statewide Percentage of Beneficiaries MHP Total Approved Claims MHP Approved Claims per Beneficiary Statewide Approved Claims per Beneficiary MHP Percentage of Total Approved Claims Statewide Percentage of Total Approved Claims < $20K * n/a 93.97% - - $3,636 73.02% 58.96% >$20K - $30K * n/a 2.87% - - $24,284 n/a 12.02% >$30K * n/a 3.16% - - $53,219 n/a 29.02% Table C2: Alpine MHP CY16 Distribution of Beneficiaries by ACB Range ---PAGE BREAK--- - 61 - County MHP CalEQRO Report Fiscal Year 2017–18 Attachment D—PIP Validation Tools PERFORMANCE IMPROVEMENT PROJECT (PIP) VALIDATION WORKSHEET FY17-18 CLINICAL PIP GENERAL INFORMATION MHP: Alpine County Behavioral Health Services PIP Title: Seeking Safety Treatment to Reduce Co-Occurring Start Date: 10/2016 Completion Date: 10/2018 Projected Study Period: 24 Months Completed: Yes ☐ No ☒ Date(s) of On-Site Review: August 2, 2017 Name of Reviewer: Della Dash Status of PIP (Only Active and ongoing, and completed PIPs are rated): Rated ☒ Active and ongoing (baseline established and interventions started) ☐ Completed since the prior External Quality Review (EQR) Not rated. Comments provided in the PIP Validation Tool for technical assistance purposes only. ☐ Concept only, not yet active (interventions not started) ☐ Inactive, developed in a prior year ☐ Submission determined not to be a PIP ☐ No Clinical PIP was submitted Brief Description of PIP: The goal of this PIP is to determine whether a brief intervention to reduce substance use in BHS clients will reduce emotional and behavioral impairments and improve daily function. After training staff on the Seeking Safety intervention model, and screening all current adult outpatient clients using the GAIN-SS, the MHP implemented an 8- ---PAGE BREAK--- - 62 - County MHP CalEQRO Report Fiscal Year 2017–18 week Seeking Safety group-counseling program for consumers with co-occurring disorders (MH and SUD). The first 8-week session covered the following topics: safety, PTSD: taking back your power, when substances control you, detaching from emotional pain, asking for help, setting boundaries in relationships, coping with triggers, and taking good care of yourself. The program included an incentive (a $25 gift card) at the completion of each group session to ensure attendance and retention. After completing the 8-week program, participants were re-evaluated using the GAIN- SS. The total number of reported as experienced “during the past month” from the pre-test screening to the total number of reported upon completion of the 8-week Seeking Safety treatment interval were compared. Changes in GAIN-SS scores are evaluated through a comparison of mean scores from pre-treatment and post-treatment GAIN-SS assessments. Clients who continue to meet study criteria are referred to continue with the Seeking Safety group intervention. New clients who meet the criteria established for the Seeking Safety program will also be referred and may join the program at the start of each 8-week treatment interval. ACTIVITY 1: ASSESS THE STUDY METHODOLOGY STEP 1: Review the Selected Study Topic(s) Component/Standard Score Comments 1.1 Was the PIP topic selected using stakeholder input? Did the MHP develop a multi-functional team compiled of stakeholders invested in this issue? ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine Alissa Nourse, Behavioral Health Services (BHS) Director Nani Ellis, BHS Quality Improvement (QI) Coordinator Gail St. James BHS Clinical Coordinator Janet Stevens, BHS Clinician Alpine County Mental Health Board, representing stakeholders ---PAGE BREAK--- - 63 - County MHP CalEQRO Report Fiscal Year 2017–18 1.2 Was the topic selected through data collection and analysis of comprehensive aspects of enrollee needs, care, and services? ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine This problem was identified during the 2016 EQRO review process. Reviewers noted that “35.3% of FY15-16 consumers have co- occurring mental health and AOD [Alcohol and Other Drug] disorders, which equals national prevalence estimates.” Alpine County clinical staff had also directly observed significant impairments among clients with co-occurring disorders and in 2015 the Alpine County Superior Court began to more consistently refer defendants with co-occurring disorders directly to BHS for assessment. All of these factors contributed to the MHP’s awareness of the need to expand and improve the treatment options available for clients with co-occurring disorders. Between January 3 and February 3, 2017, clinicians attempted to evaluate all adult clients using the GAIN-SS to screen for co-occurring disorders in the domains of externalizing disorders, internalizing disorders, substance use, and crime/violence. Of the 31 adult clients who completed the GAIN-SS, 19 (61.3%) reported moderate or severe in substance use and 18 (58.1%) reported moderate or severe in substance use and at least one additional domain. Select the category for each PIP: Clinical: ☒ Prevention of an acute or chronic condition ☐ High volume services ☒ Care for an acute or chronic condition ☒ High risk conditions Non-Clinical: ☐ Process of accessing or delivering care 1.3 Did the Plan’s PIP, over time, address a broad spectrum of key aspects of enrollee care and services? Project must be clearly focused on identifying and correcting deficiencies in care or services, rather than on utilization or cost alone. ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine ---PAGE BREAK--- - 64 - County MHP CalEQRO Report Fiscal Year 2017–18 1.4 Did the Plan’s PIPs, over time, include all enrolled populations did not exclude certain enrollees such as those with special health care needs)? Demographics: ☒ Age Range ☐ Race/Ethnicity ☐ Gender ☐ Language ☒ Other (screening tool: GAIN-SS) ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine All adult consumers, both existing and new. Totals 4 Met 0 Partially Met 0 Not Met 0 UTD STEP 2: Review the Study Question(s) 2.1 Was the study question(s) stated clearly in writing? Does the question have a measurable impact for the defined study population? Include study question as stated in narrative: “Will the implementation of the 8-week Seeking Safety curriculum with clients having a moderate or high number of of a substance use disorder reduce emotional and behavioral impairments as measured in the GAIN-SS domains of externalizing disorders, internalizing disorders, substance use, or criminal behavior?” ☐ Met ☒ Partially Met ☐ Not Met ☐ Unable to Determine The study question as stated is not measurable “reduce”). The study question would be strengthened by adding a measurable metric reducing by X% or by X points on the GAIN-SS”). Totals 0 Met 1 Partially Met 0 Not Met 0 UTD STEP 3: Review the Identified Study Population 3.1 Did the Plan clearly define all Medi-Cal enrollees to whom the study question and indicators are relevant? Demographics: ☒ Age Range ☐ Race/Ethnicity ☐ Gender ☐ Language ☒ Other: (screening tool: GAIN-SS) ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine ---PAGE BREAK--- - 65 - County MHP CalEQRO Report Fiscal Year 2017–18 3.2 If the study included the entire population, did its data collection approach capture all enrollees to whom the study question applied? Methods of identifying participants: ☐ Utilization data ☐ Referral ☐ Self-identification ☒ Other: (Screening tool GAIN-SS score) ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine The GAIN-SS has 23 scored items, divided into four domains: Internalizing Disorders, Externalizing Disorders, Substance Disorders, and Crime/Violence. Each domain contains five to seven screening questions, each describing a specific problem or that a person might have experienced. Participants report whether they had each problem during the past month, 2 to 3 months ago, 4 to 12 months ago, more than a year ago, or never. An individual’s score is determined by the number of problems or in each domain experienced within the last year. Problems or experience more than a year ago or never experienced are not scored. Totals 2 Met 0 Partially Met 0 Not Met 0 UTD STEP 4: Review Selected Study Indicators 4.1 Did the study use objective, clearly defined, measurable indicators? List indicators: 1. Current Adult Outpatient Clients screened for co-occurring of mental health and substance use problems using GAIN-SS 2. Reduction in the Average Number of reported on the GAIN-SS as occurring “during the past month” among clients participating in Seeking Safety 3. Engagement: Average number of group sessions attended during the course of an 8-week Seeking Safety program ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine The MHP needs to add another measure that is more sensitive and is able to determine if the intervention is leading to real change in consumer. Several tools were discussed and the MHP will determine which one they will add. ---PAGE BREAK--- - 66 - County MHP CalEQRO Report Fiscal Year 2017–18 4.2 Did the indicators measure changes in: health status, functional status, or enrollee satisfaction, or processes of care with strong associations with improved outcomes? All outcomes should be consumer focused. ☒ Health Status ☒ Functional Status ☐ Member Satisfaction ☐ Provider Satisfaction Are long-term outcomes clearly stated? ☒ Yes ☐ No Are long-term outcomes implied? ☒ Yes ☐ No ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine Totals 2 Met 0 Partially Met 0 Not Met 0 UTD STEP 5: Review Sampling Methods 5.1 Did the sampling technique consider and specify the: a) True (or estimated) frequency of occurrence of the event? b) Confidence interval to be used? c) Margin of error that will be acceptable? ☐ Met ☐ Partially Met ☐ Not Met ☒ Not Applicable ☐ Unable to Determine 5.2 Were valid sampling techniques that protected against bias employed? Specify the type of sampling or census used: ☐ Met ☐ Partially Met ☐ Not Met ☒ Not Applicable ☐ Unable to Determine 5.3 Did the sample contain a sufficient number of enrollees? of enrollees in sampling frame of sample of participants (i.e. – return rate) ☐ Met ☐ Partially Met ☐ Not Met ☒ Not Applicable ☐ Unable to Determine ---PAGE BREAK--- - 67 - County MHP CalEQRO Report Fiscal Year 2017–18 Totals 0 Met 0 Partially Met 0 Not Met 3 Not Applicable 0 UTD STEP 6: Review Data Collection Procedures 6.1 Did the study design clearly specify the data to be collected? ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine 6.2 Did the study design clearly specify the sources of data? Sources of data: ☒ Member ☐ Claims ☐ Provider ☒ Other: Consumer scores using the screening tool – GAIN-SS ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine 6.3 Did the study design specify a systematic method of collecting valid and reliable data that represents the entire population to which the study’s indicators apply? ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine 6.4 Did the instruments used for data collection provide for consistent, accurate data collection over the time periods studied? Instruments used: ☐ Survey ☐ Medical record abstraction tool (attendance) ☒ Outcomes tool ☐ Level of Care tools ☒ Other: Screening Tool – GAIN-SS; sign in sheet ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine 6.5 Did the study design prospectively specify a data analysis plan? Did the plan include contingencies for untoward results? ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine ---PAGE BREAK--- - 68 - County MHP CalEQRO Report Fiscal Year 2017–18 6.6 Were qualified staff and personnel used to collect the data? Project leader: Name: Title: Role: Other team members: Names: ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine Clinical staff will administer the GAIN-SS with clients on their caseload. Gail St. James, LMFT and Clinical Coordinator, Janet Stevens, LCSW and staff clinician, Pitts, LMFT and staff clinician, and Cathy Angi, LMFT and contracted clinician, will be collecting data. Totals 6 Met 0 Partially Met 0 Not Met 0 UTD ---PAGE BREAK--- - 69 - County MHP CalEQRO Report Fiscal Year 2017–18 STEP 7: Assess Improvement Strategies 7.1 Were reasonable interventions undertaken to address causes/barriers identified through data analysis and QI processes undertaken? Describe Interventions: #3 Implementation of an 8-week Seeking Safety group-counseling program. The first 8-week session will cover the following topics: safety, PTSD: taking back your power, when substances control you, detaching from emotional pain, asking for help, setting boundaries in relationships, coping with triggers, and taking good care of yourself. #4 Include incentives: Clients will be offered a $25 gift card at the completion of each group session as an incentive to ensure attendance and retention in the program. #5 Re-evaluate using the GAIN-SS to compare total number of reported as experienced “during the past month” from the initial screening pre-test to the total number of reported upon completion of the 8-week Seeking Safety treatment interval. Changes in GAIN-SS scores will be evaluated through a comparison of mean scores from pre- treatment and post-treatment GAIN-SS assessments. #6 Refer clients who continue to meet study criteria to continue with the Seeking Safety group intervention. New clients who meet the criteria established for the Seeking Safety program will also be referred and may join the program at the start of each 8-week treatment interval. ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine Steps 1 and 2 were in preparation for the PIP: #1 Staff Training on Seeking Safety intervention: Clinical staff involved in facilitating Seeking Safety group sessions will complete an in-person, six-hour Seeking Safety training workshop offered by Treatment Innovations. #2 Screening of current, adult outpatient clients using the GAIN- SS: Evidence-based screening will be used identify those with a moderate or high number of either a moderate number (1-2) or high number (3 or more) of occurring during the past year within the “Substance Use” domain co-occurring with either a moderate number (1-2) or high number (3 or more) of occurring during the past year with another domain (Externalizing Disorders, Internalizing Disorders, or Crime/Violence). Steps 3-6 were interventions in the active PIP. While this is an initial short intervention, if the same clients are continuing the treatment modality, the MHP might find it helpful to expand the program for additional weeks (12, 16 or longer). Totals 1 Met 0 Partially Met 0 Not Met 0 NA 0 UTD ---PAGE BREAK--- - 70 - County MHP CalEQRO Report Fiscal Year 2017–18 STEP 8: Review Data Analysis and Interpretation of Study Results 8.1 Was an analysis of the findings performed according to the data analysis plan? This element is “Not Met” if there is no indication of a data analysis plan (see Step 6.5) ☐ Met ☒ Partially Met ☐ Not Met ☐ Not Applicable ☐ Unable to Determine All Clients were offered the GAIN-SS; 31 of 35 consented to complete the assessment. This component of the study went as expected. Of the 31 clients who completed the GAIN-SS assessment, 18 met criteria for participation. Scores were tabulated and analyzed using EXCEL software; this part of the data analysis process occurred as expected. Post-treatment, clients who participated in the Seeking Safety intervention (N=8) and clients who met criteria but declined to participate (N=10) were re-assessed using the GAIN-SS. At this time, the MHP noted that three clients from the non-treatment group were no longer in services. Analysis of GAIN-SS scores for clients in the non- treatment group was re-evaluated to include only those who were receiving ongoing BHS services through the 8-week treatment interval 8.2 Were the PIP results and findings presented accurately and clearly? Are tables and figures labeled? ☒ Yes ☐ No Are they labeled clearly and accurately? ☒ Yes ☐ No ☒ Met ☐ Partially Met ☐ Not Met ☐ Not Applicable ☐ Unable to Determine ---PAGE BREAK--- - 71 - County MHP CalEQRO Report Fiscal Year 2017–18 8.3 Did the analysis identify: initial and repeat measurements, statistical significance, factors that influence comparability of initial and repeat measurements, and factors that threaten internal and external validity? Indicate the time periods of measurements: Indicate the statistical analysis used: Indicate the statistical significance level or confidence level if available/known: ______Unable to determine ☒ Met ☐ Partially Met ☐ Not Met ☐ Not Applicable ☐ Unable to Determine GAIN-SS scores from clients’ pre-treatment and from post-treatment were compared for the eight clients who completed the Seeking Safety intervention. BHS anticipated that clients would report a decrease in the number of experienced “within the past month”; however, BHS did not find a decrease in the current number of GAIN-SS scores from clients’ initial assessment (pre-treatment) and from scores obtained after the same eight-week interval were compared for the seven clients who declined to participate in the Seeking Safety intervention but remained engaged in “treatment as usual” through the 8-week interval. This group reported an unexpected 22.5% increase in the number of reported “during the past month.” 8.4 Did the analysis of the study data include an interpretation of the extent to which this PIP was successful and recommend any follow-up activities? Limitations described: Conclusions regarding the success of the interpretation: Recommendations for follow-up: ☐ Met ☒ Partially Met ☐ Not Met ☐ Not Applicable ☐ Unable to Determine No follow up activities were initiated as the result of this study. Among clients who participated in the Seeking Safety session one (2/16/17 - 4/7/17), data collection occurred as expected. Among clients who did not participate in the Seeking Safety session one (2/16/17 - 4/7/17), data collection occurred later than planned (4/27/17 through 5/16/17 rather than during interval between sessions, 4/10/17 through 4/19/17). This was due to unclear identification of the “comparison group” of clients who met study criteria but declined to participate. During the Seeking Safety session two (4/20/17 – 6/15/17), data collection occurred as expected. During Seeking Safety sessions one and two, clients unexpectedly withdrew from treatment. The original data analysis plan did not include a contingency design for clients who completed a pre- treatment test but were not available to retest. The MHP decided to exclude these clients from the analysis and they also noted that 5 clients (27.8%) of the 18 who originally met study criteria withdrew from treatment. Totals 2 Met 2 Partially Met 0 Not Met 0 NA 0 UTD ---PAGE BREAK--- - 72 - County MHP CalEQRO Report Fiscal Year 2017–18 STEP 9: Assess Whether Improvement is “Real” Improvement 9.1 Was the same methodology as the baseline measurement used when measurement was repeated? Ask: At what interval(s) was the data measurement repeated? Were the same sources of data used? Did they use the same method of data collection? Were the same participants examined? Did they utilize the same measurement tools? ☒ Met ☐ Partially Met ☐ Not Met ☐ Not Applicable ☐ Unable to Determine By design, this study was not a simple evaluation of implementing a brief Seeking Safety intervention to reduce co-occurring mental health and substance use problems. The study includes the use of incentives (a $25 gift certificate for each session attended) that are not included in the Seeking Safety treatment model. From a research perspective, this is now a compounding variable and the MHP will be unable to determine which aspect(s) of the interventions, the Seeking Safety vs. “improved engagement” vs. of the incentives was actually helpful. It is possible that the weekly $25 gift certificate might have an impact on the study results independent of improved engagement which could be either positive (participants might experience a reduction in financial stress that impacts other areas of functioning) or negative (participants might have more disposable income available and therefore increased access to alcohol or other drugs). Modifications are being considered for continuation of the Seeking Safety intervention. 9.2 Was there any documented, quantitative improvement in processes or outcomes of care? Was there: ☐ Improvement ☐ Deterioration Statistical significance: ☐ Yes ☒ No Clinical significance: ☐ Yes ☐ No ☐ Met ☐ Partially Met ☐ Not Met ☐ Not Applicable ☒ Unable to Determine The PIP has only run for two 8-week cycles, with few participants. As the PIP continues, additional data will be available to determine the extent to which the PIP is having the intended impact, and is successful. 9.3 Does the reported improvement in performance have internal validity; i.e., does the improvement in performance appear to be the result of the planned quality improvement intervention? Degree to which the intervention was the reason for change: ☐ No relevance ☐ Small ☐ Fair ☐ High ☐ Met ☐ Partially Met ☐ Not Met ☐ Not Applicable ☒ Unable to Determine Among all clients who completed multiple GAIN-SS assessments, the MHP noted problems with test-retest reliability. For example, on the pre-treatment test, a client might have reported that a particular problem occurred “within the last month” and then on the post- treatment test (typically 8-10 weeks after the pre-treatment test) reported that the same problem occurred “4 to 12 months ago,” “more than a year ago,” or even “never.” Inconsistency in self- reported between clients’ test and retest occurred frequently and likely did impact the study results. ---PAGE BREAK--- - 73 - County MHP CalEQRO Report Fiscal Year 2017–18 9.4 Is there any statistical evidence that any observed performance improvement is true improvement? ☐ Weak ☐ Moderate ☐ Strong ☐ Met ☐ Partially Met ☐ Not Met ☐ Not Applicable ☒ Unable to Determine Although observed changes in the number of reported as occurring “during the past month” met the study goal of a 25% reduction, these results were not statistically significant. 9.5 Was sustained improvement demonstrated through repeated measurements over comparable time periods? ☐ Met ☐ Partially Met ☐ Not Met ☐ Not Applicable ☒ Unable to Determine Totals 1 Met 0 Partially Met 0 Not Met 0 NA 4 UTD ACTIVITY 2: VERIFYING STUDY FINDINGS (OPTIONAL) Component/Standard Score Comments Were the initial study findings verified (recalculated by CalEQRO) upon repeat measurement? ☐ Yes ☒ No ACTIVITY 3: OVERALL VALIDITY AND RELIABILITY OF STUDY RESULTS: SUMMARY OF AGGREGATE VALIDATION FINDINGS Conclusions: Although not reaching statistical significance, results do demonstrate that clients who meet study criteria are at risk for escalation over time. Clients who participated in the Seeking Safety intervention did not report an increase in while those who declined to participate did report a higher number of occurring “during the past month” when pre-treatment and post-treatment scores were compared. Overall, the use of incentives to increase attendance and engagement in the Seeking Safety intervention appears to be at least moderately successful. ---PAGE BREAK--- - 74 - County MHP CalEQRO Report Fiscal Year 2017–18 Recommendations: The study question needs to be measurable by adding, “as increased by or x score using the screening tool”. The MHP needs to add another measuring tool that is more sensitive and is able to determine if the intervention is leading to real changes in consumer outcomes. Several tools were discussed and the MHP will determine which one they will add. Additionally, some changes in the data collection process are needed. While this is an initial short intervention, if the same clients are continuing the treatment modality, the MHP should consider changes to the treatment interval, including expanding to additional weeks—12, 16 or longer. For the second year of the PIP, the MHP will need to add additional interventions for the PIP to be considered active and continuing. The MHP is encouraged to continue this PIP for a second year, making these recommendations for improvement. Check one: ☐ High confidence in reported Plan PIP results ☐ Low confidence in reported Plan PIP results ☐ Confidence in reported Plan PIP results ☐ Reported Plan PIP results not credible ☒ Confidence in PIP results cannot be determined at this time ---PAGE BREAK--- - 75 - County MHP CalEQRO Report Fiscal Year 2017–18 PERFORMANCE IMPROVEMENT PROJECT (PIP) VALIDATION WORKSHEET FY17-18 NON-CLINICAL PIP GENERAL INFORMATION MHP: Alpine County Behavioral Health Services PIP Title: Crisis Intervention Team (CIT) Start Date: 6/23/17 Completion Date: 6/30/19 Projected Study Period: 24 Months Completed: Yes ☐ No ☒ Date(s) of On-Site Review: August 2, 2017 Name of Reviewer: Della Dash Status of PIP (Only Active and ongoing, and completed PIPs are rated): Rated ☒ Active and ongoing (baseline established and interventions started) ☐ Completed since the prior External Quality Review (EQR) Not rated. Comments provided in the PIP Validation Tool for technical assistance purposes only. ☐ Concept only, not yet active (interventions not started) ☐ Inactive, developed in a prior year ☐ Submission determined not to be a PIP ☐ No Non-Clinical PIP was submitted Brief Description of PIP: Alpine County residents and Medi-Cal beneficiaries experiencing a behavioral health crisis have experienced an inconsistent response and referral system by local law enforcement to local behavioral health providers. After review of the limited data, the leadership team determined that while few consumers experience a behavioral health crisis in Alpine County, 60% of those who do, have multiple crises requiring law enforcement response. Additionally, less than 15% of that population engaged in services with ACBHS. Alpine County will implement a Crisis Intervention Team (CIT) to instigate an effective and consistent response to behavioral health crises in Alpine County. The goal of this PIP is to decrease the number of behavioral health ---PAGE BREAK--- - 76 - County MHP CalEQRO Report Fiscal Year 2017–18 crises for individuals who require response and intervention by law enforcement by increasing their engagement in local long-term behavioral health treatment after the crisis. The goal would be strengthened by focusing on improving de-escalation and provision of appropriate (safe and timely) crisis services by law enforcement for consumers through improved linkages with Alpine County Behavioral Health, and looking at prevention of as well as response to consumer crises. ACTIVITY 1: ASSESS THE STUDY METHODOLOGY STEP 1: Review the Selected Study Topic(s) Component/Standard Score Comments 1.1 Was the PIP topic selected using stakeholder input? Did the MHP develop a multi-functional team compiled of stakeholders invested in this issue? ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine As representatives of the consumer and family member stakeholder group, the Mental Health Board was instrumental in the creation, development and implementation of this PIP. The PIP team consists of Alissa Nourse, Alpine County Behavioral Health (ACBHS) Director, Gail St. James, ACBHS Clinical Coordinator and Nani Ellis, Alpine County Behavioral Health Services Coordinator. The CIT Team, which includes the Alpine County Sheriff’s Department and other law enforcement agencies (LEA) and community stakeholders is slowly becoming the PIP team. 1.2 Was the topic selected through data collection and analysis of comprehensive aspects of enrollee needs, care, and services? ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine The MHP was initially challenged in obtaining baseline data from the Sheriff’s depart, particularly for ALL CRISIS CALLS. They were able to obtain the baseline data for 5150 calls, so they started with that, and then phased in the rest as the received additional data. Of the 16 people in crisis, only 12 are Alpine County residents that remained in county, so the PIP should focus on these 12, as well as all other consumers who may potentially go into a crisis that includes a LEA response. ---PAGE BREAK--- - 77 - County MHP CalEQRO Report Fiscal Year 2017–18 Select the category for each PIP: Clinical: ☐ Prevention of an acute or chronic condition ☐ High volume services ☐ Care for an acute or chronic condition ☐ High risk conditions Non-Clinical: ☒ Process of accessing or delivering care 1.3 Did the Plan’s PIP, over time, address a broad spectrum of key aspects of enrollee care and services? Project must be clearly focused on identifying and correcting deficiencies in care or services, rather than on utilization or cost alone. ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine The PIP focuses on consumers who experience multiple, repeated crises that involve a response by a LEA. 1.4 Did the Plan’s PIPs, over time, include all enrolled populations did not exclude certain enrollees such as those with special health care needs)? Demographics: ☐ Age Range ☐ Race/Ethnicity ☐ Gender ☐ Language ☒ Other: All consumers in Alpine County (both residents and non-residents) ☐ Met ☒ Partially Met ☐ Not Met ☐ Unable to Determine While the data is for both residents and non-residents, ACBHS is only able to focus on following up with county residents who remain in county. Totals 3 Met 1 Partially Met 0 Not Met 0 UTD ---PAGE BREAK--- - 78 - County MHP CalEQRO Report Fiscal Year 2017–18 STEP 2: Review the Study Question(s) 2.1 Was the study question(s) stated clearly in writing? Does the question have a measurable impact for the defined study population? Include study question as stated in narrative: “Will establishing a Crisis Intervention Team Program in Alpine County decrease the number of behavioral health crises for individuals who require response and intervention by law enforcement during these crises by increasing their engagement in local long-term behavioral health treatment after the crisis?” ☐ Met ☒ Partially Met ☐ Not Met ☐ Unable to Determine The study question as written is not measureable, and needs to define the consumer outcomes that are targeted. What does “decrease the number” and “increase…engagement” mean. Need to have a measure of some sort – increasing from x to or decreasing the time from x to x. In addition, the MHP needs to strengthen the study question by restating it, such as: “Will consumers who experience a crisis where law enforcement is involved, have: a) More timely crisis care (same day?) by ACBHS directly rather than after an ER visit; b) Improved diversion; c) Improved engagement with ACBHS (at least four services post crisis (in a given time period?) d) Improvement clinical outcomes (as measured by a decrease in crisis calls and/or a decrease in ER visits in a 12 month period?) Totals 0 Met 1 Partially Met 0 Not Met 0 UTD STEP 3: Review the Identified Study Population 3.1 Did the Plan clearly define all Medi-Cal enrollees to whom the study question and indicators are relevant? Demographics: ☐ Age Range ☐ Race/Ethnicity ☐ Gender ☐ Language ☒ Other ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine There are currently 304 Medi-Cal eligibles in Alpine County. Forty- four of those are currently being served by ACBHS. This PIP will impact all consumers and eligibles in the county regardless of age, race, ethnicity or location of residence. ---PAGE BREAK--- - 79 - County MHP CalEQRO Report Fiscal Year 2017–18 3.2 If the study included the entire population, did its data collection approach capture all enrollees to whom the study question applied? Methods of identifying participants: ☒ Utilization data ☒ Referral ☐ Self-identification ☐ Other: ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine The MHP is working closely with the Alpine County Sheriff’s Department and other LEAs to extract and analyze all crisis call data. The data is incomplete, and not consistently coded the same way, so the PIP will focus on improving these aspects with LEA. The PIP lacks a full discussion of the 5150 process and who is designated to write/release them in county? Discussed onsite was the priority of having assistance for transportation of consumers after they are brought to the emergency room out of county. Totals 2 Met 0 Partially Met 0 Not Met 0 UTD STEP 4: Review Selected Study Indicators 4.1 Did the study use objective, clearly defined, measurable indicators? List indicators: 1. Percentage of crisis incidents that are reported to ACBHS by ACSO. 2. Percentage of crisis incidents involving Alpine County residents that are reported to ACBHS by Douglas County Sheriff’s Office. 3. Percentage of crisis incidents involving Alpine County residents that are reported to ACBHS by California Highway Patrol. 4. Percentage of crisis incidents involving Alpine County residents that are reported to ACBHS by Washoe Tribal Police. 5. Percentage of people experiencing a crisis who engage in at least four services following the crisis. 6. Percentage of ACBHS clients who continue to engage in at least four services following the crisis. ☐ Met ☒ Partially Met ☐ Not Met ☐ Unable to Determine The study indicators need to also include some linkage to the 24 hour crisis line – number of current calls, calls linked to consumers experiencing crisis. In addition, improvement in clinical outcomes, as measured by a decrease in crisis calls and/or a decrease in emergency room visits in a 12 month period for consumers and eligible. The goal is to decrease multiple crisis episodes for eligible and consumers, so the MHP will need to measure this to see if the PIP interventions are effective at decreasing crises that involve LEAs. ---PAGE BREAK--- - 80 - County MHP CalEQRO Report Fiscal Year 2017–18 4.2 Did the indicators measure changes in: health status, functional status, or enrollee satisfaction, or processes of care with strong associations with improved outcomes? All outcomes should be consumer focused. ☒ Health Status ☒ Functional Status ☐ Member Satisfaction ☐ Provider Satisfaction Are long-term outcomes clearly stated? ☐ Yes ☒ No Are long-term outcomes implied? ☒ Yes ☐ No ☐ Met ☒ Partially Met ☐ Not Met ☐ Unable to Determine Totals 0 Met 2 Partially Met 0 Not Met 0 UTD STEP 5: Review Sampling Methods 5.1 Did the sampling technique consider and specify the: a) True (or estimated) frequency of occurrence of the event? b) Confidence interval to be used? c) Margin of error that will be acceptable? ☐ Met ☐ Partially Met ☐ Not Met ☒ Not Applicable ☐ Unable to Determine 5.2 Were valid sampling techniques that protected against bias employed? Specify the type of sampling or census used: ☐ Met ☐ Partially Met ☐ Not Met ☒ Not Applicable ☐ Unable to Determine 5.3 Did the sample contain a sufficient number of enrollees? of enrollees in sampling frame of sample of participants (i.e. – return rate) ☐ Met ☐ Partially Met ☐ Not Met ☒ Not Applicable ☐ Unable to Determine ---PAGE BREAK--- - 81 - County MHP CalEQRO Report Fiscal Year 2017–18 Totals 0 Met 0 Partially Met 0 Not Met 3 Not Applicable 0 UTD STEP 6: Review Data Collection Procedures 6.1 Did the study design clearly specify the data to be collected? ☐ Met ☒ Partially Met ☐ Not Met ☐ Unable to Determine The indicators clearly state what data is to be collected. The data collected to date was gathered from July, 2015 to current date from the MHP Access Logs and from documented collaboration with the Sheriff’s Office after a crisis had already happened. The CIT team meetings have begun and minutes shared. The group is beginning to discuss the development of shared protocols and data sharing. All data is collected/analyzed manually in the Sheriff’s Department and they are not currently in a position to move to an electronic option. Two Alpine County Sheriff’s Deputies attended a 40 hour CIT training in Douglas County Nevada in March, 2017. 6.2 Did the study design clearly specify the sources of data? Sources of data: ☐ Member ☐ Claims ☐ Provider ☐ Other: ☐ Met ☒ Partially Met ☐ Not Met ☐ Unable to Determine 6.3 Did the study design specify a systematic method of collecting valid and reliable data that represents the entire population to which the study’s indicators apply? ☐ Met ☒ Partially Met ☐ Not Met ☐ Unable to Determine This needs to be improved considerably – a detailed plan for data collection, sharing, improved coding, etc. needs to be part of the interventions for this PIP. ---PAGE BREAK--- - 82 - County MHP CalEQRO Report Fiscal Year 2017–18 6.4 Did the instruments used for data collection provide for consistent, accurate data collection over the time periods studied? Instruments used: ☐ Survey ☒ Medical record abstraction tool ☐ Outcomes tool ☐ Level of Care tools ☒ Other: Crisis Log ☐ Met ☒ Partially Met ☐ Not Met ☐ Unable to Determine The data from the Sheriff’s Department was inconsistently completed and coded by officers. The MHP is working with them to improve this process. Additional data is collected from the MHP crisis log and EHR. 6.5 Did the study design prospectively specify a data analysis plan? Did the plan include contingencies for untoward results? ☐ Met ☒ Partially Met ☐ Not Met ☐ Unable to Determine The analysis plan outlined in the PIP requires much more detail, as discussed onsite during the CalEQRO review. 6.6 Were qualified staff and personnel used to collect the data? Project leader: Name: Title: Role: Other team members: Names: ☒ Met ☐ Partially Met ☐ Not Met ☐ Unable to Determine The MHP is working with the Sheriff’s Department to improve data collection. The MHP staff involved with this PIP have the skills/qualification needed for this purpose. Totals 1 Met 5 Partially Met 0 Not Met 0 UTD ---PAGE BREAK--- - 83 - County MHP CalEQRO Report Fiscal Year 2017–18 STEP 7: Assess Improvement Strategies 7.1 Were reasonable interventions undertaken to address causes/barriers identified through data analysis and QI processes undertaken? Describe Interventions: 1. Engagement with Alpine County Sheriff’s Office through the CIT that produces increased collaboration, communication and two-way reporting. 2. Engagement with California Highway Patrol through the CIT that produces increased collaboration, communication and two-way reporting. 3. Engagement with Douglas County Sheriff’s Office through the CIT that produces increased collaboration, communication and two-way reporting. 4. Engagement Washoe Tribal Police through the CIT that produces increased collaboration, communication and two- way reporting. 5. Participation by law enforcement in the 40-hour CIT training ☐ Met ☒ Partially Met ☐ Not Met ☐ Unable to Determine The terms in the interventions need to be more accurately defined so they can be measurable. What does “engagement” mean? How will that be measured? How will collaboration, communication and two- way reporting be implemented/measured? The interventions as written will only measure part of what is envisioned for PIP implementation. The MHP needs to add interventions such as: 1. Develop joint policies and/or protocols for tracking crises – describe the reporting process 2. Develop a detailed tracking log and data sharing process 3. Improve documentation and coding of crises cases and responses 4. Establish a CIT in Alpine County and implement routine interagency CIT meetings with a standing agenda, meeting minutes, case consultation, etc. (Training officers is not enough here). 5. Consider joint field response by LEAs and ACBHS – what might this look like and how could it be made a feasible option? The Sheriff’s Department is still very interested in this option and working toward this end. In addition, what other activities are needed for this to be successfully institutionalized and operationalized? And, what are the next steps for the CIT/PIP, such as engaging other partners Tribal police) and getting all data from the CHP, including jail booking data, and Tribal police? Totals 0 Met 1 Partially Met 0 Not Met 0 NA 0 UTD ---PAGE BREAK--- - 84 - County MHP CalEQRO Report Fiscal Year 2017–18 STEP 8: Review Data Analysis and Interpretation of Study Results 8.1 Was an analysis of the findings performed according to the data analysis plan? This element is “Not Met” if there is no indication of a data analysis plan (see Step 6.5) ☐ Met ☐ Partially Met ☐ Not Met ☒ Not Applicable ☐ Unable to Determine This PIP is still in the early stages and the MHP is working on data collection, analysis and improvement of interventions and indicators to strengthen their measurements. 8.2 Were the PIP results and findings presented accurately and clearly? Are tables and figures labeled? ☐ Yes ☐ No Are they labeled clearly and accurately? ☐ Yes ☐ No ☐ Met ☐ Partially Met ☐ Not Met ☒ Not Applicable ☐ Unable to Determine 8.3 Did the analysis identify: initial and repeat measurements, statistical significance, factors that influence comparability of initial and repeat measurements, and factors that threaten internal and external validity? Indicate the time periods of measurements: Indicate the statistical analysis used: Indicate the statistical significance level or confidence level if available/known: ______Unable to determine ☐ Met ☐ Partially Met ☐ Not Met ☒ Not Applicable ☐ Unable to Determine 8.4 Did the analysis of the study data include an interpretation of the extent to which this PIP was successful and recommend any follow-up activities? Limitations described: Conclusions regarding the success of the interpretation: Recommendations for follow-up: ☐ Met ☐ Partially Met ☐ Not Met ☒ Not Applicable ☐ Unable to Determine ---PAGE BREAK--- - 85 - County MHP CalEQRO Report Fiscal Year 2017–18 Totals 0 Met 0 Partially Met 0 Not Met 4 NA 0 UTD STEP 9: Assess Whether Improvement is “Real” Improvement 9.1 Was the same methodology as the baseline measurement used when measurement was repeated? Ask: At what interval(s) was the data measurement repeated? Were the same sources of data used? Did they use the same method of data collection? Were the same participants examined? Did they utilize the same measurement tools? ☐ Met ☐ Partially Met ☐ Not Met ☒ Not Applicable ☐ Unable to Determine 9.2 Was there any documented, quantitative improvement in processes or outcomes of care? Was there: ☐ Improvement ☐ Deterioration Statistical significance: ☐ Yes ☐ No Clinical significance: ☐ Yes ☐ No ☐ Met ☐ Partially Met ☐ Not Met ☒ Not Applicable ☐ Unable to Determine 9.3 Does the reported improvement in performance have internal validity; i.e., does the improvement in performance appear to be the result of the planned quality improvement intervention? Degree to which the intervention was the reason for change: ☐ No relevance ☐ Small ☐ Fair ☐ High ☐ Met ☐ Partially Met ☐ Not Met ☒ Not Applicable ☐ Unable to Determine 9.4 Is there any statistical evidence that any observed performance improvement is true improvement? ☐ Weak ☐ Moderate ☐ Strong ☐ Met ☐ Partially Met ☐ Not Met ☒ Not Applicable ☐ Unable to Determine ---PAGE BREAK--- - 86 - County MHP CalEQRO Report Fiscal Year 2017–18 9.5 Was sustained improvement demonstrated through repeated measurements over comparable time periods? ☐ Met ☐ Partially Met ☐ Not Met ☒ Not Applicable ☐ Unable to Determine Totals 0 Met 0 Partially Met 0 Not Met 5 NA 0 UTD ACTIVITY 2: VERIFYING STUDY FINDINGS (OPTIONAL) Component/Standard Score Comments Were the initial study findings verified (recalculated by CalEQRO) upon repeat measurement? ☐ Yes ☒ No ACTIVITY 3: OVERALL VALIDITY AND RELIABILITY OF STUDY RESULTS: SUMMARY OF AGGREGATE VALIDATION FINDINGS Conclusions: The PIP focuses on consumers who experience multiple, repeated crises that involve a response by a LEA. While the data is for both residents and non-residents, ACBHS is only able to focus on following up with county residents who remain in county. The MHP was initially challenged in obtaining baseline data from the Sheriff’s depart, particularly for all crisis calls, however they were able to obtain baseline data for 5150 calls, so are beginning with this and will phase in additional data as they receive it. The study question needs to be strengthened and made measurable. The data analysis plan requires much more detail, as does the plan for data collection, data sharing, improved coding, etc., and these could be added to the interventions. The indicators and interventions as written will only measure part of what is envisioned for PIP implementation. This PIP is still in the early stages and the MHP is working on data collection, analysis and improvement of interventions and indicators to strengthen their measurements. ---PAGE BREAK--- - 87 - County MHP CalEQRO Report Fiscal Year 2017–18 Recommendations: The goal of the PIP would be strengthened by focusing on “improving de-escalation and provision of appropriate (safe and timely) crisis services by law enforcement for consumers through improved linkages with Alpine County Behavioral Health, and looking at prevention of as well as response to consumer crises”. The study question needs to be strengthened by restating it, such as: “Will consumers who experience a crisis where law enforcement is involved, have: a) More timely crisis care (same day?) by ACBHS directly rather than after an ER visit; b) Improved diversion; c) Improved engagement with ACBHS (at least four services post crisis (in a given time period?) d) Improvement clinical outcomes (as measured by a decrease in crisis calls and/or a decrease in ER visits in a 12 month period?) The study indicators need to include some linkage to the 24 hour crisis line – number of current calls, calls linked to consumers experiencing crisis. In addition, improvement in clinical outcomes (as measured by a decrease in crisis calls and/or a decrease in ER visits in a 12 month period) for consumers (and eligible who do/do not become consumers). Since the goal is to decrease multiple crisis episodes for eligibles/consumers, there is a need to measure this to see if the PIP interventions are effective at decreasing crises that involve LEAs. The Data Analysis Plan needs to be expanded, with complete details. The terms in the interventions need to be more accurately defined so they can be measurable. What does “engagement” mean? How will that be measured? How will collaboration, communication and two-way reporting be implemented/measured? Additional interventions need to be added such as: 1. Develop joint policies and/or protocols for tracking crises – describe the reporting process 2. Develop a detailed tracking log and data sharing process 3. Improve documentation and coding of crises cases and responses 4. Establish a CIT in Alpine County, and implement routine interagency CIT meetings with a standing agenda, meeting minutes, case consultation, etc. (Training officers is not enough here). 5. Consider joint field response by LEAs and ACBHS – what might this look like and how could it be made a feasible option? The Sheriff’s Department is still very interested in this option and working toward this end. In addition, the MHP needs to determine what other activities are needed for this PIP to be successfully institutionalized and operationalized. Particularly for phasing in additional interventions in year two. Check one: ☐ High confidence in reported Plan PIP results ☐ Low confidence in reported Plan PIP results ☐ Confidence in reported Plan PIP results ☐ Reported Plan PIP results not credible ☒ Confidence in PIP results cannot be determined at this time