level: technical
large language models can analyze single documents well but struggle with questions that need checking many records across a company's data. the problems include running out of context space, losing track of which answer came from which record, and slow performance from calling tools one after another. batchdag solves this by having an llm produce a directed acyclic graph of operations. the graph includes sql queries, semantic searches, in-memory data changes, parallel processing, and one-shot analyses.
a deterministic engine runs the graph using topological-wave parallelism and structured json data flow. this means operations that don't depend on each other execute at the same time, speeding up the whole process. a key feature is entity-aware batching, which groups rows by logical entity before sending them to the llm. this cuts the number of llm calls by up to 47 times, making the system much more efficient for large-scale tasks.
batchdag is not mainly about beating hand-tuned pipelines in accuracy. instead, it acts as a general-purpose orchestration layer. it replaces many custom-built workflows with one system that automatically creates the right execution plan. this makes it easier to handle ad-hoc analytical questions over enterprise data without building separate solutions for each type of query.
why it matters: it lets data teams ask complex, cross-entity questions over large datasets without manual pipeline engineering, reducing llm costs and latency.