source: arxiv artificial intelligence: automated data readiness for scientific ai
level: research
large scientific datasets at leadership computing facilities often need major changes before they can train ai models. no existing system combines automated transformation, readiness checks, provenance tracking, and agent-based deployment in one place. the redi framework fills this gap with a five-stage pipeline: ingest, preprocess, transform, structure, and output. each stage includes instrumentation for reproducibility, and the whole pipeline can run as a skill callable by ai agents. a companion tool called setgo handles fair compliance and catalog publication automatically.
redi was tested on datasets from climate science, proteomics, materials science, and nuclear fusion. in all cases, it turned raw data into ai-ready form, and the outputs matched domain expert references. early tests on the frontier supercomputer showed near-ideal parallel scaling up to 100 nodes for the climate dataset. the framework's design focuses on making scientific data immediately usable for machine learning without manual wrangling.
by integrating transformation, validation, and provenance in one open-source package, redi aims to speed up scientific ai workflows. the agent-native design means researchers can trigger data preparation from automated systems. the provenance tracking helps meet reproducibility standards in scientific computing. this approach could reduce the time and effort needed to prepare complex scientific data for ai training across many fields.
why it matters: it automates the tedious process of making raw scientific data ready for ai, saving time and improving reproducibility in research.
source: arxiv artificial intelligence: automated data readiness for scientific ai