Xianzhi Du

I'm a Staff Research Scientist at Apple AI/ML, where I work on building and training large language model (LLM), multi-modality LLM and sparse Mixture-of-Experts (MoE).

I was previously a Senior Research Engineer at Google Brain and Google CoreML working on computer vision research and building TensorFlow official models. I also worked extensively on collaborating with Alphabet's teams, including Waymo, Google Cloud, Google Maps, Google Photos, Nest, X, to apply state-of-the-art research models to Alphabet's applications.

I did my PhD at UMIACS, University of Maryland, College Park, where I was advised by Larry Davis and David Doermann.

Email  /  Google Scholar  /  LinkedIn  /  Twitter  

Research Highlights

Introduces Apple’s on-device and server foundation models. Announced at the 2024 Worldwide Developers Conference.

Compares MoE and dense LLMs by using step time to measure model speed and designing total train budget with Chinchilla compute-optimal setting. We show MoE consistently outperforms dense models at 6B, 13B and 30B.

Builds high-performance MLLMs by identifying crucial architectural and data choices, leading to the creation of MM1 models, which excel in pre-training metrics and few-shot learning across various benchmarks.

A novel MLLM that excels in fine-grained spatial understanding and grounding descriptions within images, using a hybrid region representation and a specialized dataset, demonstrating superior performance and reduced object hallucination.

Leverages Multimodal LLMs to enhance instruction-based image editing, derive expressive instructions and provide explicit guidance.

An adaptive MoE framework that dynamically adjusts experts per task, enhancing multi-task vision recognition performance on ImageNet and COCO.

Designs a simple single-scale vision transformer architecture, achieves strong performance on object localization and instance segmentation tasks.

Revisits training recipe and model scaling strategies for ResNets, improving ResNets to be competitive with state-of-the-art.

Revisits training recipe and model scaling strategies for Object Detection, improving conventional object detectors, e.g. RetinaNet and Cascade R-CNN, to be competitive with state-of-the-art.

Designs a scale-permuted backbone dilated convolutions that is learned by Neural Architecture Search (NAS) on semantic segmentation. Achieved SoTA on Cityscape semantic segmentation benchmark on 03/2021.

Designs a scale-permuted backbone with intermediate features and cross-scale connections that is learned by Neural Architecture Search (NAS) on object detection. Achieved SoTA on COCO detection and segmentation benchmark on 12/2019.

Introduces AMNet with depthwise-separable convolutions, extended cost volume, and stacked atrous multiscale network for disparity estimation. Ranked No.1 on KITTI Stereo 2015 and 2012 benchmarks on 11/2018.

Proposes a deep neural network fusion architecture for accurate, fast and robust pedestrian detection, especially in detecting small-size and occluded pedestrians. Ranked No.1 on Caltech Pedestrian Detection benchmark on 08/2016.

Template credits.