A runtime layer that makes multi-agent AI systems finish their work sooner without changing the agents themselves.
An AI agent handling a research request spends most of its wall clock time waiting. It waits on web searches, on tool calls, and above all on language model responses. Two kinds of waiting are avoidable.
The first is waiting in the wrong order. An agent given ten independent cities to research will often visit them one at a time, even though nothing in the ninth city depends on the eighth.
The second is waiting behind someone else. When many agent requests share the same model serving hardware, a short request can sit behind a long one that arrived first, and both finish later than they need to.
This project addresses both. It sits between the part of an agent system that decides what to do and the part that actually does it. It works out which pieces of work are independent and lets them run at the same time. It then orders and places the resulting model calls so that the work on the longest remaining chain goes first, and so that a request lands on the machine that already holds its conversation history.
The agent code is left alone. An existing agent framework keeps its own planner, its own tools, and its own prompts, and gains shorter end to end times under load.
The domain
An agent framework turns a user request into a plan and then carries the plan out. Each unit of work in the plan calls a large language model one or more times. Under concurrency, many such requests run at once against a shared pool of model servers, and the scheduling of those calls determines how long everyone waits.
Glossary
Every term below names a specific thing in this system and is used with one meaning throughout.
-
Program One user request handled end to end by an agent framework. A program has a unique identifier, a plan, and a set of model calls it issues over its lifetime.
-
Plan The ordered list of steps a planner produces for a program. Stored as JSON with a title, a thought, and a list of steps.
-
Step One unit of agent work inside a plan. Carries a title, a description, a type such as research or processing, and once executed, a result string.
-
LLM call One request to a language model. Has a prompt measured in prefill tokens, produces output measured in decode tokens, and belongs to exactly one program.
-
Thread A chain of dependent LLM calls inside one program. A program with parallel branches has several threads at once.
-
Engine One model serving process that holds model weights in GPU memory and serves LLM calls. A deployment has one or more engines, typically one per GPU.
-
KV cache The attention key and value tensors an engine keeps in GPU memory for an in flight or recently served call. A follow up call that shares a prompt prefix can reuse the cache instead of recomputing it. Cache capacity is finite, so entries are evicted or moved to host memory under pressure.
-
Attained service The cumulative execution time a program has already received, in seconds. Starts at zero.
-
Service time For a program, the longest cumulative runtime among its threads. This is an online estimate of the program's critical path length. Measured in seconds by the workflow scheduler and in processed tokens by the resource scheduler module.
-
Waiting time The total time a program has spent sitting in scheduler queues, in seconds.
-
Starvation ratio Waiting time divided by service time. Dimensionless. Used to decide when a neglected program must be promoted.
-
Critical path The longest chain of dependent LLM calls in a program. Its length sets the floor on how quickly the program can finish.
-
Makespan Wall clock time from a program's first call to its last.
-
Head of line blocking A short program waiting behind a long one that holds the front of a queue.
-
Execution slot A permit to execute one step of one program at a time. The number of slots caps how much work runs concurrently on one host.
-
Batch A group of steps that have no dependencies on each other and therefore run at the same time.
-
Synchronization barrier The point at the end of a batch where every step in that batch must complete before the next batch starts. Barriers are what preserve the ordering that dependent steps rely on.
Why independent work is worth finding
Agent steps are dominated by network waiting rather than local computation. A process that is blocked on a socket consumes almost no CPU, so many such steps can be in flight at once on a single host. The gain comes from overlapping the waiting, not from adding compute.
The obstacle is that a plan does not usually say which steps are independent. Three ways to work it out exist, and they trade accuracy against cost:
- Explicit declarations from the framework are exact but rare, since most planners do not emit them.
- Rule based inference over step metadata is free and fast but misses dependencies expressed only in prose.
- Asking a language model to read the plan and group the steps handles prose but costs one extra model call and can be wrong.
This project uses the model based grouping by default and falls back to the rule based path when the model call fails, so a failed analysis degrades to a slower run rather than a wrong one.
Why scheduling model calls needs program awareness
A model server that schedules one call at a time has no idea that a call belongs to a longer chain. First come first served is the simplest policy and it produces head of line blocking. Shortest job first would fix that, but nobody knows a program's total length in advance, because agents decide what to do next based on what they just read.
The way around this is to infer length from behaviour rather than ask for it up front. A program that has already consumed a lot of service is probably long. Priority can therefore be set as the inverse of attained service, recomputed as the program runs. New and short programs float to the front, long ones sink, and no advance knowledge is required.
Ranking by a single continuous score means re-sorting constantly. Discretising the score into a small number of queues, each covering a service range, gives the same effect at far lower cost. A call runs for a fixed quantum, and a call that exhausts its quantum drops one level. This is a multilevel feedback queue.
Any such scheme can starve the programs it demotes. Watching each program's starvation ratio and promoting the whole program back to the top queue once the ratio crosses a threshold bounds how long any program can be neglected. Promoting every thread of the program together, rather than one call, keeps the program's branches moving in step.
Why placement matters as much as ordering
Once a call has been chosen, it still has to be sent to an engine. Spreading calls evenly keeps every GPU busy. Sending a call back to the engine that served the previous call in the same program lets it reuse that engine's KV cache instead of recomputing the prefix.
These two goals conflict, and the resolution turns on prompt length. Recomputing a short prefix is cheap, so short calls can go wherever there is capacity. Recomputing a long prefix is expensive, so long calls should follow their program's history. A single token threshold separates the two cases. The Autellix paper reports up to 1.4 times the throughput of plain round robin from this split.
Why the cache has to be moved rather than dropped
Preempting a call frees its GPU memory only if the KV cache goes somewhere. Discarding it means recomputing the whole prefix when the call resumes. Copying it to host memory preserves the work, but a naive copy issues one small transfer per cache block and leaves the PCIe bus mostly idle. Gathering many blocks into one contiguous pinned buffer and issuing a single large transfer is what makes preemption affordable.
Reference material
- Autellix: An Efficient Serving Engine for LLM Agents as General Programs, Luo et al., 2025. A copy is included at
Autellix.pdf. The process table, the attained service priority, the multilevel queues, the anti starvation promotion, the routing threshold, and the batched cache swapping all come from this paper. The relevant sections are 4.2.1 for the process table, 4.2.2 and 6.5.4 for memory management, and Algorithm 2 for the routing policy. - DeerFlow,
github.com/bytedance/deer-flow. The agent framework this project integrates with. Pulled in as a git submodule atdeerflow/and modified through patch files rather than a fork. - vLLM. The model serving engine the resource scheduler module drives. Used as a library, not rebuilt.
Working on this codebase exercises:
- Asynchronous Python. Cooperative scheduling with
asyncio, priority aware semaphores, and moving blocking calls off the event loop into thread and process pools. - Operating system scheduling concepts applied outside an operating system. Attained service, multilevel feedback queues, aging, quanta, and preemption.
- Language model serving internals. KV cache behaviour, prefill and decode phases, batching, and the memory pressure that follows from them.
- GPU memory management. Pinned host buffers, asynchronous CUDA streams, and coalesced transfers.
- Integration with an existing codebase you do not own, through patches and middleware rather than a fork.
- Performance measurement. Establishing a baseline before changing anything, then attributing a change in wall clock time to a specific mechanism.
Prior familiarity with asyncio and with GPU serving makes the resource scheduler module easier to approach, but neither is assumed.
Three subsystems sit at three different points in the path from user request to model output. They are independent: none imports another, and each can run on its own.
flowchart TB
User([User request])
subgraph AF[DeerFlow agent framework]
Server[FastAPI server]
Planner[Planner]
Executor[Step executor]
end
subgraph UDF[UDF optimizer]
WT[Workflow types]
DA[Dependency analyzer]
PE[Parallel executor]
CM[Configuration manager]
GE[Gemini executor]
end
subgraph WS[Workflow scheduler]
MW[ASGI middleware]
AS[Scheduler core]
REG[Workflow registry]
PRI[Priority calculation]
PQ[Priority queue]
SP[Priority slot pool]
MM[Memory monitor]
OP[Offload pools]
end
subgraph RSM[Resource scheduler module]
PT[Process table]
SM[Session manager]
AW[API wrapper]
SCH[ATLAS scheduler]
MLFQ[Multilevel queues]
LB[Load balancer]
MEM[Multi-engine manager]
KV[KV cache coordinator]
SWAP[Memory management]
end
GPU[(vLLM engines on GPUs)]
GEM[(Gemini API)]
DISK[(Checkpoint directory)]
User --> Server --> MW --> AS
Server --> Planner --> DA
DA --> PE --> Executor
CM --> PE
PE --> GE --> GEM
AS --> REG
AS --> PRI --> PQ --> SP
AS --> MM
MM --> DISK
AS --> OP
SP --> Executor
Executor --> AW --> SM --> PT
SCH --> PT
SCH --> MLFQ
AW --> SCH --> LB --> MEM --> GPU
LB --> PT
MEM --> KV --> SWAP
What each subsystem is responsible for
| Subsystem | Sits at | Decides |
|---|---|---|
| UDF optimizer | Between planner and executor | Which steps of one plan can run at the same time |
| Workflow scheduler | Between HTTP request and step execution | Which of many concurrent programs gets an execution slot next, and which get checkpointed to disk under memory pressure |
| Resource scheduler module | Between the agent and the model servers | Which LLM call runs next and which engine serves it |
Feature to component map
| Feature | Component |
|---|---|
| Grouping plan steps into batches | Dependency analyzer |
| Running a batch concurrently and holding the barrier | Parallel executor |
| Concurrency limits, retries, failure mode | Configuration manager |
| Model calls for both analysis and step execution | Gemini executor |
| Registering a program on request arrival | ASGI middleware |
| Tracking attained service and status per program | Workflow registry |
| Turning attained service and wait time into a score | Priority calculation |
| Handing the next slot to the highest scoring waiter | Priority queue, priority slot pool |
| Detecting host memory pressure | Memory monitor |
| Moving blocking tool calls off the event loop | Offload pools |
| Tracking service time, wait time and engine affinity per program | Process table |
| Program lifecycle and identifiers | Session manager |
| Choosing the next LLM call | ATLAS scheduler, multilevel queues |
| Choosing the engine for a call | Load balancer |
| Starting, stopping and monitoring engines | Multi-engine manager |
| Moving KV cache between GPU and host | Memory management |
A concrete flow through the UDF optimizer
A user asks for a tourist research report covering ten cities. The planner emits a twelve step plan: ten independent city research steps, one analysis step that reads all ten, and one report step that reads the analysis.
- The plan arrives as JSON and is parsed into a
Planof twelveStepobjects by the workflow types component. - The dependency analyzer sends the step titles and descriptions to the model along with the instruction in
config/parallel_prompt.md, and receives back a grouping: steps 1 to 10 in batch one, step 11 in batch two, step 12 in batch three. - The parallel executor launches all ten steps of batch one at once, bounded by
max_concurrent_tasks, which defaults to 10. It gathers the results and holds the barrier until every step has either returned or errored. - Each result is written back to its step and appended to the shared observation list, so batch two sees all ten city summaries in its prompt context.
- Batches two and three run their single steps in turn.
Measured on this plan against gemini-2.5-flash, the sequential run took about 110 seconds and the batched run about 85 seconds. The floor is set by the slowest city in batch one plus the two serial steps, not by the sum of all twelve.
A concrete flow through the resource scheduler module
A program that has already processed 900 tokens issues a follow up call with a 4000 token prompt.
- The API wrapper accepts the call, and the session manager attaches it to the program's existing identifier in the process table.
- The ATLAS scheduler reads the program's service time of 900 tokens. With a base quantum of 512, that places the program in queue level 1, since 900 is under 1024.
- The scheduler checks the program's starvation ratio against the configured threshold. It is below the threshold, so no promotion happens and the call waits its turn in level 1.
- When the call is selected, the load balancer reads its prefill length of 4000 tokens. That is above the 2048 token threshold, so the balancer routes it to the engine already recorded against this program in the process table rather than to the least loaded engine.
- The multi-engine manager submits the call to that engine process, which finds the program's prefix already resident in its KV cache and skips recomputing it.
- As tokens are processed the scheduler updates the program's service time. If the call exhausts its quantum before finishing, the program drops to level 2 and the KV cache coordinator may move its cache blocks to host memory to free room.
External connections
- vLLM engines holding model weights in GPU memory, one engine process per GPU.
- Model weights fetched from Hugging Face.
download.pypullsmeta-llama/Llama-2-7b-hf. - The Gemini API, used by the UDF optimizer for dependency analysis and step execution. Requires
GEMINI_API_KEY. - The DeerFlow FastAPI server, which the workflow scheduler attaches to as ASGI middleware.
- A checkpoint directory on local disk, default
/tmp/deerflow_checkpoints, holding serialized state for programs that have been swapped out of RAM.
How the system extends
- The Gemini executor is one provider implementation behind a common step execution interface. Adding another provider means adding one module beside it, with no change to the analyzer or the executor.
- Framework support beyond DeerFlow is a matter of adding a task adapter that maps that framework's step objects onto the shared plan and step types, plus a hook that intercepts the plan before execution. The scheduling components do not change.
- The three subsystems are currently separate. Connecting the UDF optimizer's step execution to the resource scheduler module's API wrapper, rather than to a provider API, would put the batched steps of one plan under program aware scheduling.
- The process table is in memory and single host. Replacing it with a shared store would let several scheduler hosts serve one engine pool.
src/
workflow_scheduler/ Priority scheduler and memory manager for concurrent programs
rsm/ Program-aware LLM call scheduler and multi-engine manager
udf-optimizer/ Plan parallelization with model-based dependency analysis
integration/ Patch files and apply/revert scripts for DeerFlow
scripts/ Setup, run, benchmark and test entry points
tests/ Tests for the workflow scheduler and the DeerFlow patches
deerflow/ DeerFlow git submodule, empty until initialized
Autellix.pdf The paper the scheduling design reproduces
Located at src/workflow_scheduler. Schedules whole programs on one host. Its concern is how many programs execute at once and which ones, not which model call runs.
Neighbours and communication
It attaches to the DeerFlow FastAPI server as ASGI middleware, which tells it when a program starts and ends. Patched hooks in DeerFlow's graph nodes tell it when a step is about to run and when it has finished, which is where a program blocks for a slot and where its attained service is updated. It reads host memory statistics from the operating system and writes serialized program state to a checkpoint directory. It does not talk to the resource scheduler module or to the UDF optimizer.
Components
-
Workflow registry
core/registry.py. Holds oneWorkflowMetaper active program: creation time, attained service, step count, last execution time, status, and location. Status is queued, executing, paused, or completed. Location is hot, meaning state is in RAM, or cold, meaning state has been serialized to disk. Guarded by an asyncio lock. -
Priority calculation
core/priority.py. Pure functions with no state. Base priority is the inverse of attained service,P = 1 / (A + ε)withε = 1e-6, so a new program with zero attained service starts near the maximum. The aging factor is1 + wait / θ_age, where wait is measured from the later of creation and last execution, andθ_agedefaults to 5 seconds. Effective priority is the product of the two, which grows without bound as wait time grows and so guarantees eventual execution. -
Priority queue
core/priority_queue.py. A max priority queue over a negated min heap, tiebreaking by insertion order so equal priorities are served first in first out. A bounded variant raisesCapacityErrorwhen full. Deliberately separated from the slot pool so it can be tested on its own and reused for eviction ordering. -
Priority slot pool
core/slot_pool.py. A semaphore that grants the next free slot to the highest priority waiter rather than the longest waiting one. Programs suspend here between steps, which is the point at which cooperative multitasking actually happens. Capacity defaults to 16 slots. -
Memory monitor
core/memory_monitor.py. Samples host memory throughpsutiland reports pressure as a percentage. Crossing the swap threshold, default 80 percent, triggers checkpointing of cold programs. Crossing the critical threshold, default 90 percent, blocks admission of new programs. Falls back to mock values whenpsutilis absent, so the module imports on any host, but pressure readings are then meaningless. -
Offload pools
offload/pools.py. Classifies a blocking operation as CPU bound, IO bound, or already asynchronous, then runs CPU bound work in aProcessPoolExecutorto escape the GIL and IO bound work in aThreadPoolExecutor. Defaults are 4 process workers and 10 thread workers. Without this, one synchronous tool call stalls every concurrent program on the host. -
ASGI middleware
middleware.py. Registers a program when a request reaches a workflow endpoint, stores the identifier on the request state so downstream code can find it, and unregisters on completion or error. It performs no scheduling itself. -
Scheduler core
core/atlas.py. Ties the above together and exposes the pre step and post step hooks that DeerFlow calls. -
Configuration loader
config.py. Resolves settings from environment variables first, then DeerFlow'sconf.yaml, then defaults, and validates them. Rejects a swap threshold that is not below the critical threshold, among other checks.
Located at src/rsm. Schedules individual LLM calls across a pool of engines. Reproduces the Autellix design.
Neighbours and communication
An agent program opens a session through the API wrapper and issues calls against it. Every call's arrival, wait, and service is recorded in the process table, which the scheduler and the load balancer both read. The engine layer owns the vLLM processes and reports load and cache state back. src/rsm/rsm.py wires the four groups together and imports them as top level packages, so it runs from inside src/rsm.
Frontend
-
Process table
frontend/process_table.py. The shared state everything else reads. OneProgramEntryper program holding service time, waiting time, the engines that have served it, and its live threads. OneThreadMetadataper call holding arrival time, waiting time, service time, prefill and decode token counts, parent thread identifiers, priority, and queue index. Thread safe through anRLock. Computes service time two ways: as a simple cumulative total for single threaded programs, and as the longest path among threads for multi threaded ones. -
Session manager
frontend/session_manager.py. Issues session identifiers, maps sessions to programs, and cleans up on end or timeout. Without timeout cleanup an abandoned session leaves an entry that skews the scheduler's view of load. -
API wrapper
frontend/api_wrapper.py. The client surface. Provides an OpenAI compatible adapter and a session context manager, so an agent can issue calls through the scheduler with roughly the shape it already uses.
Scheduler
-
Multilevel queue base
scheduler/mlfq_base.py. Builds the priority levels. Quantum for levelnisbase_quantum * 2^nwithbase_quantumdefaulting to 512 tokens, so higher levels get exponentially longer turns. A program's level is derived from its service time bylog2(service_time / base_quantum) + 1. -
ATLAS scheduler
scheduler/atlas_scheduler.py. Selects the next call. Runs in one of two modes per program. Program mode is first in first out within the program and suits single threaded programs. Thread mode ranks the program's threads and suits programs whose branches have diverged. The scheduler switches a program between modes based on the variance in its threads' service times, so mode is adapted rather than configured. Also handles quantum exhaustion, which demotes a program one level, and starvation promotion, which lifts every thread of a program back to the top level at once. -
Program context
scheduler/context.py. Per program scheduling state: current mode, queue level, and thread bookkeeping.
Load balancer
-
Balancer
load_balancer/balancer.py. Implements Algorithm 2 of the Autellix paper. A call with 2048 prefill tokens or fewer goes to the least used engine. A call above 2048 goes to the engine already associated with the program, to reuse its KV cache. The threshold is a configured constant, not a heuristic to tune casually, since it encodes the point where recomputing a prefix costs more than queueing. -
Engine info
load_balancer/engine_info.py. Per engine state the balancer ranks on: current load and free cache capacity.
Engine layer
-
Multi-engine manager
engine/multi_engine_manager.py. Starts and stops engine processes, routes submitted requests, collects results through futures, and cancels in flight requests either singly or by session. Cancellation matters because a preempted or abandoned program otherwise keeps consuming GPU time. -
Engine process
engine/engine_process.py. One vLLM instance in its own process, bound to one GPU, communicating over multiprocessing queues. -
KV cache coordinator
engine/kv_cache_coordinator.py. Tracks which program's cache blocks live on which engine, which is the information the balancer's affinity decision depends on. -
Lifecycle manager
engine/lifecycle_manager.py. Tracks each request through its states from submission to completion or cancellation. -
vLLM integration
engine/vllm_integration.py. The adaptation layer to vLLM, written against version 0.6.1.
Memory management
-
Batched swap manager
engine/memory/batched_swap.py. Coalesces many KV cache blocks into one contiguous pinned buffer and issues a single PCIe transfer instead of one per block, across four asynchronous CUDA streams. Detects the GPU type and sets pinned buffer size, batch size, and eviction threshold accordingly, from 16 GB pinned on an A100 down to 4 GB on a T4. -
Swap scheduler
engine/memory/swap_scheduler.py. Chooses which program's cache to evict, using program level information rather than treating every block equally. -
Swap predictor
engine/memory/swap_predictor.py. Prefetches cache back to the GPU ahead of a resume, so a resumed call does not stall on the transfer. -
Pressure manager
engine/memory/pressure_manager.py. Monitors GPU memory and triggers eviction at the configured threshold. Configured throughconfigs/memory.yaml, which supports auto detection. -
CUDA kernels
engine/kernels/. Vectorized transfer kernels using 128 bitfloat4loads and stores with grid stride loops, compiled just in time for the detected architecture, supporting compute capability 7.0 and above. A Python fallback path exists, so the absence of a compiler degrades performance rather than breaking the system. -
Multi-step scheduler
engine/scheduler/multi_step_scheduler.py. Batches several decode steps per scheduling decision to cut per step overhead.
Located at src/udf-optimizer. Turns a sequential plan into batches of independent steps and runs them.
Neighbours and communication
It takes a plan as JSON, either from a planner or from a file, and returns the plan with each step's result filled in and the shared observation list appended to. It calls a model provider for both dependency analysis and step execution. It does not know about either scheduler.
Components
-
Workflow types
core/workflow_types.py. The data model:Plan,Step,State,Configuration, andStepType.Statecarries the message list, the observation list, and the current plan, and is what threads results from one batch into the next batch's prompts. -
Dependency analyzer
core/gemini_executor.py. Sends the plan to the model with the instruction inconfig/parallel_prompt.mdand parses back a batch grouping. Groups conservatively, so an uncertain pair of steps is kept sequential rather than run together. On failure it falls back to rule based batching, and the run continues at sequential speed. -
Parallel executor
core/nodes.py.parallel_research_team_nodeorchestrates the batches._execute_batch_parallelgathers a batch with exceptions returned rather than raised, so one failed step becomes an error string on that step instead of killing the batch._execute_batch_sequentialis the fallback path. Results are written back and appended before the barrier releases. -
Graph builder
core/builder.py. Builds either the parallel or the sequential execution graph, which is what makes the whole feature switchable off. -
Configuration manager
core/config_manager.py. Loadsconfig/config.yamland offers presets named speed, balanced, reliability, and cost. Controls concurrency ceiling, launch rate, per step and per batch timeouts, retry count and backoff, and failure mode, which is eitherpartial_completionorfail_fast. -
Provider executor
core/gemini_executor.py. Executes a step against Gemini with the accumulated context in the prompt, and distinguishes research steps from processing steps. Adding another provider means adding a module beside this one. -
Sequential executor
core/sequential_executor.py. The unbatched path, used as the comparison baseline. -
Performance comparison
compare_performance.py. Runs the same plan both ways, times them, and writes a report.examples/example_performance_report.mdis a sample output.
Located at integration/ and scripts/.
Neighbours and communication
Applies a patch to the DeerFlow submodule so its server, graph nodes, and tools call into the workflow scheduler. Nothing in src/ is modified by this.
Components
-
Patch file
integration/patches/patch.patch. Touches six DeerFlow files:src/config/configuration.pyto carry a thread identifier,src/graph/checkpoint.pyandsrc/graph/nodes.pyfor step hooks and logging,src/server/app.pyfor middleware registration, andsrc/tools/crawl.pyandsrc/tools/python_repl.pyto wrap blocking tools. A disabled variant sits inintegration/patches_disabled/. -
Apply and revert scripts
integration/apply_patches.shandintegration/revert_patches.sh. Applying checks whether the patch is already present and skips rather than failing, so the script is safe to rerun. -
Setup script
scripts/setup.sh. Initializes the submodule, installs dependencies, applies patches, creates the checkpoint and log directories, and verifies that the scheduler imports and that the memory monitor reads a real value. -
Run script
scripts/run_deerflow.py. Starts the DeerFlow server with the scheduler attached, or without it whenSCHEDULER_ENABLED=false. -
Baseline benchmark
scripts/benchmark_baseline.py. Drives unmodified DeerFlow with concurrent SSE requests and records wall clock time, throughput, and success rate. No scheduler code is injected, so its numbers are the baseline that scheduler runs are compared against.
Requires Python 3.9 or later. The resource scheduler module additionally requires one or more NVIDIA GPUs and a working vLLM install. The workflow scheduler and the UDF optimizer run on CPU only.
git clone --recurse-submodules https://github.com/notming11/Agentic-Runtime-Resource-Scheduling-Optimization.git
cd Agentic-Runtime-Resource-Scheduling-Optimization
bash scripts/setup.shscripts/setup.sh initializes the DeerFlow submodule, installs the project in editable mode, applies the DeerFlow patches, and verifies the install. If the submodule was not cloned, run git submodule update --init --recursive first.
For the UDF optimizer only:
cd src/udf-optimizer
pip install -r requirements.txt
cp .env.example .envThen set GEMINI_API_KEY in .env. Keys are issued at https://makersuite.google.com/app/apikey.
DeerFlow with the workflow scheduler
export PYTHONPATH=$PWD/src:$PYTHONPATH
python scripts/run_deerflow.pyDeerFlow without the scheduler, for a baseline
SCHEDULER_ENABLED=false python scripts/run_deerflow.py
python scripts/benchmark_baseline.py --mode baseline --concurrent 5The UDF optimizer
cd src/udf-optimizer
python tests/validate.py # import and configuration check, no model calls
python main.py # interactive: load a plan, pick a mode, runWorkflow scheduler
Set through environment variables, which take precedence over DeerFlow's conf.yaml, which takes precedence over the defaults below.
| Variable | Default | Meaning |
|---|---|---|
SCHEDULER_ENABLED |
true |
Whether the scheduler runs at all |
SCHEDULER_MAX_SLOTS |
16 | Concurrent execution slots |
SCHEDULER_SWAP_THRESHOLD |
80 | Host memory percent at which checkpointing starts |
SCHEDULER_CRITICAL_THRESHOLD |
90 | Host memory percent at which new programs are refused |
SCHEDULER_AGING_THRESHOLD |
5.0 | Seconds of waiting that doubles a program's priority |
SCHEDULER_MAX_WAIT |
300.0 | Seconds before starvation intervention |
SCHEDULER_CHECKPOINT_DIR |
/tmp/deerflow_checkpoints |
Where cold program state is written |
SCHEDULER_PROCESS_WORKERS |
4 | Workers for CPU bound offload |
SCHEDULER_THREAD_WORKERS |
10 | Workers for IO bound offload |
SCHEDULER_STATS_INTERVAL |
30 | Seconds between statistics log lines |
The swap threshold must be strictly below the critical threshold. Configuration loading raises on violation rather than silently correcting.
UDF optimizer
Set in src/udf-optimizer/config/config.yaml.
| Key | Default | Meaning |
|---|---|---|
max_concurrent_tasks |
10 | Ceiling on steps in flight |
max_tasks_per_second |
5.0 | Launch rate limit |
task_timeout_seconds |
300 | Per step timeout |
batch_timeout_seconds |
900 | Per batch timeout |
dependency_strategy |
llm_based |
Also accepts heuristic and explicit |
retry_on_failure |
true |
Retry a failed step |
max_retries |
3 | Retry ceiling |
failure_mode |
partial_completion |
Also accepts fail_fast |
Lower max_concurrent_tasks to 3 or below if the provider returns rate limit errors.
Resource scheduler module
Memory behaviour is set in src/rsm/configs/memory.yaml, which supports GPU auto detection. Scheduler behaviour is set through the config dictionary passed to AutellixConfig in src/rsm/rsm.py, covering the number of queue levels, the base quantum of 512 tokens, the starvation ratio threshold, and the 2048 token routing threshold.
pytest tests/workflow_scheduler/ -v # unit tests for the scheduler
pytest tests/test_patch_integration.py -v # DeerFlow patch integration
bash scripts/run_tests.sh --unit # wrapper with coverageResource scheduler module tests live under src/rsm/tests/ and are run from that directory. The multi engine and end to end tests need GPUs.
UDF optimizer tests live under src/udf-optimizer/tests/. validate.py makes no model calls. test_e2e.py and test_real_execution.py do, and consume quota.
Continuous integration runs the workflow scheduler unit tests on Python 3.9 through 3.12, plus ruff, mypy, and a check that the scheduler enable and disable paths both work. Linting and type checking are advisory and do not fail the build.
- The three subsystems are independent. No module in one imports a module from another. Connecting them is unfinished work, not a design property.
deerflow/is an empty submodule until initialized, and the setup and run scripts skip patching when it is absent.- The resource scheduler module needs GPUs, vLLM, and model weights. On a CPU only host it imports and unit tests run, but the engine layer cannot start.
tests/scenarios/is present but empty.scripts/run_tests.shlooks fortest_patch_integration.pyundertests/workflow_scheduler/, but the file is attests/test_patch_integration.py. Invoke pytest directly to run it.- Reported speedups come from two sources. The 110 second to 85 second figure was measured on this codebase against
gemini-2.5-flash. The 1.4 times routing throughput figure is from the Autellix paper and has not been reproduced here.
Apache License 2.0. See LICENSE.