Scoring and Grading Capability Manager

The National Conference of Bar Examiners (NCBE) is a nonprofit organization dedicated to advancing a competent, ethical, and diverse legal profession. We develop and deliver high-quality assessment products, services, and research that support state courts and licensing authorities in evaluating the readiness of aspiring lawyers.

NCBE is committed to fostering a collaborative, mission-driven environment with competitive salaries, comprehensive benefits, flexible scheduling, and professional development opportunities. NCBE’s headquarters is based in Madison, Wisconsin, a dynamic community with exceptional amenities.

The Scoring and Grading Capability Manager will lead the design, coordination, and delivery of NCBE’s end-to-end grading and scoring strategy across all item types—multiple-choice and constructed-response. Working across test development, psychometrics, test operations, and product management, this role defines and executes the technical roadmap for scoring systems that ensure accuracy, reliability, and efficiency at scale. The Capability Manager will also guide research and responsible implementation of automated scoring technologies, partnering with psychometric and R&D teams to evaluate, pilot, and integrate AI-enabled scoring solutions. This position requires deep experience in large-scale assessment, strong technical and product management acumen, and an understanding of the legal licensure context in which NCBE operates. 

Essential Duties and Responsibilities

• Lead the development and execution of NCBE’s grading and scoring strategy across all item types, including multiple-choice and constructed-response components of the NextGen UBE. 
• Coordinate cross-functionally among test development, psychometrics, test operations, product management, and technology teams to ensure alignment and efficient delivery of scoring workflows and policies. 
• Define and maintain the technical roadmap for grading and scoring, detailing system enhancements, automation initiatives, and integrations with delivery and scoring platforms. 
• Oversee operational readiness for scoring activities, ensuring that all scoring systems, processes, and personnel are prepared for each administration. 
• Partner with psychometricians and data scientists to ensure scoring models and methods meet NCBE’s validity, reliability, and fairness standards.  
• Support research and development in automated scoring, including model evaluation, calibration, and pilot testing, and ensure responsible AI use consistent with NCBE’s principles. 
• Collaborate with vendor partners (e.g., scoring platforms, AI vendors, and analytics providers) to ensure seamless integration, technical quality, and adherence to NCBE’s security and compliance standards. 
• Monitor scoring quality and performance metrics, such as rater reliability, automation accuracy, turnaround time, and system uptime, and recommend continuous improvements. 
• Document and communicate product requirements and functional specifications to engineering, psychometric, and operational teams. 
• Ensure data governance, security, and compliance for all grading and scoring systems, consistent with NCBE policies and legal/licensure requirements. 
• Prepare reports and presentations summarizing grading performance, research outcomes, and roadmap progress for internal stakeholders, governance committees, and the NCBE Board. 

Qualifications

Education and Training 

• Bachelor’s degree required; master’s degree in psychometrics, measurement, data science, computer science, or a related field preferred. 

• Minimum of 5 years of experience in large-scale assessment, licensure, or certification testing, with demonstrated expertise in scoring, grading, or results reporting systems. 

• Proven experience leading technical product management, capabilities, or cross-functional initiatives involving psychometrics, operations, and technology. 

• Familiarity with automated or AI-assisted scoring systems, data pipelines, and model validation frameworks. 

• Experience managing relationships with vendors and technology partners, including software and scoring service providers. 

• Experience in regulated or high-stakes testing environments; knowledge of legal licensure or certification programs preferred. 

Technical Competencies 

• Strong understanding of scoring methodologies (human and automated), rater reliability, validity, and data flow processes in assessment environments. 

• Working knowledge of data analytics and system integration, including APIs, databases, and scoring platform interoperability. 

• Ability to define and manage a technical product roadmap, aligning requirements with psychometric and operational dependencies. 

• Familiarity with cloud-based platforms, AI/ML scoring applications, and secure data management best practices. 

• Proficiency with standard product and project management tools (e.g., Jira, Confluence, Azure DevOps, or equivalent). 

Leadership and Interpersonal Competencies 

• Exceptional cross-functional collaboration skills; able to bridge technical, psychometric, and operational domains effectively. 

• Strong analytical and systems thinking, with the ability to translate technical issues into strategic implications. 

• Excellent verbal and written communication, capable of tailoring complex information for both technical and non-technical audiences. 

• Demonstrated ability to lead through influence, manage competing priorities, and drive decision-making.  

• Commitment to fairness, equity, and validity in high-stakes assessment and to NCBE’s mission of supporting a competent, ethical, and diverse legal profession. 

The Scoring Capability Manager may work remotely. NCBE will consider applicants from the following states: FL, IA, IN, MA, MD, ME, MI, MO, MN, NE, NJ, NM, NV, OK, PA, WA, and WI.

This position may include up to 10% travel for meetings with NCBE’s Board of Trustees, committees and/or vendors, and/or for conferences, workshops, and/or outreach.   

Mission

NCBE promotes fairness, integrity, and best practices in admission to the legal profession for the benefit and protection of the public. We serve admission authorities, courts, the legal education community, and candidates by providing high-quality

• assessment products, services, and research;
• character investigations; and
• informational and educational resources and programs.

EEO Statement
NCBE is proud to be an equal employment opportunity organization. We are committed to providing equal employment opportunity to all applicants and employees.

Please note that applicants may be contacted via email throughout the hiring process. We suggest that you add BambooHR (@bamboohr.com and @app.bamboohr.com) to your Approved/Safe Sender list so that email notifications are delivered to your inbox and not marked as spam.