Description The Language AI Services Team in Amazon Web Services (AWS) is seeking a Language Data Scientist to join our data team focused on Health AI. We are looking for candidates who have a strong interest in applying their linguistic and data science expertise in collecting and analyzing natural language and human data to lead multiple data collection and data analysis efforts at a time in an exciting, fast-paced environment. In this role, you will curate, engineer, and analyze natural language datasets in the medical domain, as well as design experiments and surveys to elicit human-in-the-loop insights, which are critical for developing AI-powered language applications. You will partner closely with talented language engineers, program managers, clinical experts, applied scientists, engineers, and product managers to deliver data solutions that meet customer needs. Key job responsibilities - Translate business, modeling and ethical requirements in Health AI into executable data collection projects - Design human-in-the-loop evaluation tasks to measure the performance and usability of models in the medical domain - Develop the materials necessary to execute successful data collection efforts such as guidelines, annotation interfaces, quality assurance workflows - Support the sourcing and/or creation of high-quality language datasets and language artifacts for feature and language expansion - Analyze structured and unstructured data to provide actionable recommendations to improve data quality or model performance - Iterate and innovate on data collection methodologies to improve data turnaround time and reliability - Incorporate LLMs, prompt engineering, and ML techniques to automate repetitive annotation and data creation workflows - Stay up to date with developments in the world of AI, with a focus on model fine tuning and evaluation data needs and techniques A day in the life As a Language Data Scientist on the data team, you will take the lead on a couple of critical data projects related to AWS HealthScribe and Amazon Transcribe Medical, diving deep and developing materials to drive these projects forward. You will consult with stakeholders to understand the role data plays in developing and launching specific language services that meet customer needs. You will propose data collection and annotation strategy to curate and validate datasets, and partner with stakeholders to set the quality bar for these data projects. You will lead iterative data analysis and course correction efforts to address gaps and quality issues. You will also design experiments to elicit human-in-the-loop insights (including users who are domain experts) for model evaluation and usability testing, proposing the optimal business and evaluation metrics to use. You will gradually expand your scope by applying the principles of data-centric AI to conduct experiments to drive workflow and process improvements, with the goal to optimize the cost and quality of data. Leveraging your hands-on data analytics skills and up-to-date knowledge of machine learning (ML) and Generative AI techniques, you will collaborate with other language engineers and scientists to propose new strategies for sourcing ground truth data or generating reliable metrics, automating where appropriate. About the team The data team within AWS Language AI Services consists of language engineers, program managers, and clinical experts. We are part of a cross-functional team developing cutting-edge Language AI services within AWS. These services, ranging from subtitles and call analytics to clinical documentation and beyond, support numerous languages and enable customers across diverse industries to seamlessly integrate intelligence into their business operations and AI applications. We are customer obsessed and committed to delivering solutions with the highest quality and integrity. Diverse Experiences AWS values diverse experiences. Even if you do not meet all of the qualifications and skills listed in the job description, we encourage candidates to apply. If your career is just starting, hasn’t followed a traditional path, or includes alternative experiences, don’t let it stop you from applying. Why AWS? Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform. We pioneered cloud computing and never stopped innovating — that’s why customers from the most successful startups to Global 500 companies trust our robust suite of products and services to power their businesses. Inclusive Team Culture Here at AWS, it’s in our nature to learn and be curious. Our employee-led affinity groups foster a culture of inclusion that empower us to be proud of our differences. Ongoing events and learning experiences, including our Conversations on Race and Ethnicity (CORE) and AmazeCon (gender diversity) conferences, inspire us to never stop embracing our uniqueness. Mentorship & Career Growth We’re continuously raising our performance bar as we strive to become Earth’s Best Employer. That’s why you’ll find endless knowledge-sharing, mentorship and other career-advancing resources here to help you develop into a better-rounded professional. Work/Life Balance We value work-life harmony. Achieving success at work should never come at the expense of sacrifices at home, which is why we strive for flexibility as part of our working culture. When we feel supported in the workplace and at home, there’s nothing we can’t achieve in the cloud. Basic Qualifications - 2+ years of data scientist experience - 3+ years of data querying languages (e.g. SQL), scripting languages (e.g. Python) or statistical/mathematical software (e.g. R, SAS, Matlab, etc.) experience - PhD in a language and human behavior related field with a strong quantitative component (e.g., Cognitive Linguistics, Sociolinguistics, Human-Computer Interaction); or, a Master’s degree with 3+ years of field experience - Experience in data mining and cleaning for NLP machine learning model pipelines - Experience in language data collection for quantitative analysis, including guidelines, workflow design - Experience in research and experimental design involving human participants - Experience in statistical measures for data quality assessment and research hypotheses testing - Practical knowledge of data labeling tools and techniques (e.g., Amazon SageMaker Ground Truth, brat, ELAN) - Excellent knowledge of semantics, pragmatics, conversation analysis, and/or discourse analysis - Ability to explain complex concepts and solutions in easy-to-understand terms Preferred Qualifications - Experience with LLMs and prompt engineering techniques and other programmatic approaches to annotation, including weak supervision and active learning - Practical knowledge of version control systems (e.g. Git) - Experience with spoken data collection, speech analysis, speech transcription (from scratch or ASR-assisted) - Experience working with clinical or medical data, such as medical transcriptions, clinical notes, or electronic health records (EHRs) - Knowledge of healthcare terminology and medical ontologies (e.g., SNOMED CT, ICD, RxNorm) Amazon is committed to a diverse and inclusive workplace. Amazon is an equal opportunity employer and does not discriminate on the basis of race, national origin, gender, gender identity, sexual orientation, protected veteran status, disability, age, or other legally protected status. For individuals with disabilities who would like to request an accommodation, please visit https://www.amazon.jobs/en/disability/us. Los Angeles County applicants: Job duties for this position include: work safely and cooperatively with other employees, supervisors, and staff; adhere to standards of excellence despite stressful conditions; communicate effectively and respectfully with employees, supervisors, and staff to ensure exceptional customer service; and follow all federal, state, and local laws and Company policies. Criminal history may have a direct, adverse, and negative relationship with some of the material job duties of this position. These include the duties and responsibilities listed above, as well as the abilities to adhere to company policies, exercise sound judgment, effectively manage stress and work safely and respectfully with others, exhibit trustworthiness and professionalism, and safeguard business operations and the Company’s reputation. Pursuant to the Los Angeles County Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records. Our compensation reflects the cost of labor across several US geographic markets. The base pay for this position ranges from $125,500/year in our lowest geographic market up to $212,800/year in our highest geographic market. Pay is based on a number of factors including market location and may vary depending on job-related knowledge, skills, and experience. Amazon is a total compensation company. Dependent on the position offered, equity, sign-on payments, and other forms of compensation may be provided as part of a total compensation package, in addition to a full range of medical, financial, and/or other benefits. For more information, please visit https://www.aboutamazon.com/workplace/employee-benefits. This position will remain posted until filled. Applicants should apply via our internal or external career site.