Design and productionize state of the art tooling and open source technologies to enable the development of frontier foundation models for Databricks customers

Solve complex problems at scale around data preprocessing, model training, hyperparameter tuning and model evaluation

Implement advanced optimization techniques to reduce the resource footprint of models while preserving their performance and balancing usability for our developers and customers

Collaborate with product managers and cross-functional teams to drive technology-first initiatives that enable novel business strategies and product roadmaps

Facilitate our user community through documentation, talks, tutorials, and collaborations

Contribute to the broader AI community by publishing research, presenting at conferences, and actively participating in open-source projects, enhancing Databricks' reputation as an industry leader.

Below are some example projects:

Composer: Large-scale distributed deep learning training library

Streaming: Library for efficient data loading from cloud object storage

LLM Foundry: Framework for developing and evaluating Large Language Models

Hands on experience with the internals of deep learning frameworks (e.g. PyTorch, TensorFlow) and GenAI models (e.g. GPT, StableDiffusion)

Experience with large scale, distributed training on GPUs (e.g., Nvidia, AMD) and alternative deep learning accelerators

Strong sense of design and usability

Effective communication skills and the ability to articulate complex technical ideas to cross-disciplinary internal and external stakeholders

Prior history of contributing to or developing open source projects is a bonus but not a requirement