AI Data Engineer

About the role

As a Data Engineer on the corpus team, you'll build scalable pipelines to source, process, curate, filter and prepare large-scale corpora for training and evaluating foundation models. You'll work closely with AI Researchers, ML Engineers and Platform Engineers to solve challenging data problems across web-scale datasets, unstructured data, multilingual corpora and benchmark datasets.

What you'll do

• Source and ingest large-scale datasets from Common Crawl, public web sources, open-source repositories and other data sources
• Build scalable data processing pipelines for large unstructured corpora
• Develop and optimise distributed data pipelines using tools such as Apache Spark, Ray and Dask
• Extract and process content from web pages and documents using Trafilatura and related web/content extraction libraries
• Design data cleaning, filtering, quality assessment and curation pipelines for LLM pretraining corpora

What you need

• 2–5 yrs of experience
• Machine learning
• Python
• ETL
• Spark
• LLMs
• GCP
• Any graduate

Good to have

• Physics
• HTML