Skip to content

Latest commit

 

History

History
590 lines (400 loc) · 35.8 KB

File metadata and controls

590 lines (400 loc) · 35.8 KB

Agentic Runtime Resource Scheduling Optimization

A runtime layer that makes multi-agent AI systems finish their work sooner without changing the agents themselves.


Overview

An AI agent handling a research request spends most of its wall clock time waiting. It waits on web searches, on tool calls, and above all on language model responses. Two kinds of waiting are avoidable.

The first is waiting in the wrong order. An agent given ten independent cities to research will often visit them one at a time, even though nothing in the ninth city depends on the eighth.

The second is waiting behind someone else. When many agent requests share the same model serving hardware, a short request can sit behind a long one that arrived first, and both finish later than they need to.

This project addresses both. It sits between the part of an agent system that decides what to do and the part that actually does it. It works out which pieces of work are independent and lets them run at the same time. It then orders and places the resulting model calls so that the work on the longest remaining chain goes first, and so that a request lands on the machine that already holds its conversation history.

The agent code is left alone. An existing agent framework keeps its own planner, its own tools, and its own prompts, and gains shorter end to end times under load.


Background

The domain

An agent framework turns a user request into a plan and then carries the plan out. Each unit of work in the plan calls a large language model one or more times. Under concurrency, many such requests run at once against a shared pool of model servers, and the scheduling of those calls determines how long everyone waits.

Glossary

Every term below names a specific thing in this system and is used with one meaning throughout.

  • Program One user request handled end to end by an agent framework. A program has a unique identifier, a plan, and a set of model calls it issues over its lifetime.

  • Plan The ordered list of steps a planner produces for a program. Stored as JSON with a title, a thought, and a list of steps.

  • Step One unit of agent work inside a plan. Carries a title, a description, a type such as research or processing, and once executed, a result string.

  • LLM call One request to a language model. Has a prompt measured in prefill tokens, produces output measured in decode tokens, and belongs to exactly one program.

  • Thread A chain of dependent LLM calls inside one program. A program with parallel branches has several threads at once.

  • Engine One model serving process that holds model weights in GPU memory and serves LLM calls. A deployment has one or more engines, typically one per GPU.

  • KV cache The attention key and value tensors an engine keeps in GPU memory for an in flight or recently served call. A follow up call that shares a prompt prefix can reuse the cache instead of recomputing it. Cache capacity is finite, so entries are evicted or moved to host memory under pressure.

  • Attained service The cumulative execution time a program has already received, in seconds. Starts at zero.

  • Service time For a program, the longest cumulative runtime among its threads. This is an online estimate of the program's critical path length. Measured in seconds by the workflow scheduler and in processed tokens by the resource scheduler module.

  • Waiting time The total time a program has spent sitting in scheduler queues, in seconds.

  • Starvation ratio Waiting time divided by service time. Dimensionless. Used to decide when a neglected program must be promoted.

  • Critical path The longest chain of dependent LLM calls in a program. Its length sets the floor on how quickly the program can finish.

  • Makespan Wall clock time from a program's first call to its last.

  • Head of line blocking A short program waiting behind a long one that holds the front of a queue.

  • Execution slot A permit to execute one step of one program at a time. The number of slots caps how much work runs concurrently on one host.

  • Batch A group of steps that have no dependencies on each other and therefore run at the same time.

  • Synchronization barrier The point at the end of a batch where every step in that batch must complete before the next batch starts. Barriers are what preserve the ordering that dependent steps rely on.

Why independent work is worth finding

Agent steps are dominated by network waiting rather than local computation. A process that is blocked on a socket consumes almost no CPU, so many such steps can be in flight at once on a single host. The gain comes from overlapping the waiting, not from adding compute.

The obstacle is that a plan does not usually say which steps are independent. Three ways to work it out exist, and they trade accuracy against cost:

  • Explicit declarations from the framework are exact but rare, since most planners do not emit them.
  • Rule based inference over step metadata is free and fast but misses dependencies expressed only in prose.
  • Asking a language model to read the plan and group the steps handles prose but costs one extra model call and can be wrong.

This project uses the model based grouping by default and falls back to the rule based path when the model call fails, so a failed analysis degrades to a slower run rather than a wrong one.

Why scheduling model calls needs program awareness

A model server that schedules one call at a time has no idea that a call belongs to a longer chain. First come first served is the simplest policy and it produces head of line blocking. Shortest job first would fix that, but nobody knows a program's total length in advance, because agents decide what to do next based on what they just read.

The way around this is to infer length from behaviour rather than ask for it up front. A program that has already consumed a lot of service is probably long. Priority can therefore be set as the inverse of attained service, recomputed as the program runs. New and short programs float to the front, long ones sink, and no advance knowledge is required.

Ranking by a single continuous score means re-sorting constantly. Discretising the score into a small number of queues, each covering a service range, gives the same effect at far lower cost. A call runs for a fixed quantum, and a call that exhausts its quantum drops one level. This is a multilevel feedback queue.

Any such scheme can starve the programs it demotes. Watching each program's starvation ratio and promoting the whole program back to the top queue once the ratio crosses a threshold bounds how long any program can be neglected. Promoting every thread of the program together, rather than one call, keeps the program's branches moving in step.

Why placement matters as much as ordering

Once a call has been chosen, it still has to be sent to an engine. Spreading calls evenly keeps every GPU busy. Sending a call back to the engine that served the previous call in the same program lets it reuse that engine's KV cache instead of recomputing the prefix.

These two goals conflict, and the resolution turns on prompt length. Recomputing a short prefix is cheap, so short calls can go wherever there is capacity. Recomputing a long prefix is expensive, so long calls should follow their program's history. A single token threshold separates the two cases. The Autellix paper reports up to 1.4 times the throughput of plain round robin from this split.

Why the cache has to be moved rather than dropped

Preempting a call frees its GPU memory only if the KV cache goes somewhere. Discarding it means recomputing the whole prefix when the call resumes. Copying it to host memory preserves the work, but a naive copy issues one small transfer per cache block and leaves the PCIe bus mostly idle. Gathering many blocks into one contiguous pinned buffer and issuing a single large transfer is what makes preemption affordable.

Reference material

  • Autellix: An Efficient Serving Engine for LLM Agents as General Programs, Luo et al., 2025. A copy is included at Autellix.pdf. The process table, the attained service priority, the multilevel queues, the anti starvation promotion, the routing threshold, and the batched cache swapping all come from this paper. The relevant sections are 4.2.1 for the process table, 4.2.2 and 6.5.4 for memory management, and Algorithm 2 for the routing policy.
  • DeerFlow, github.com/bytedance/deer-flow. The agent framework this project integrates with. Pulled in as a git submodule at deerflow/ and modified through patch files rather than a fork.
  • vLLM. The model serving engine the resource scheduler module drives. Used as a library, not rebuilt.

Key skills and takeaways

Working on this codebase exercises:

  • Asynchronous Python. Cooperative scheduling with asyncio, priority aware semaphores, and moving blocking calls off the event loop into thread and process pools.
  • Operating system scheduling concepts applied outside an operating system. Attained service, multilevel feedback queues, aging, quanta, and preemption.
  • Language model serving internals. KV cache behaviour, prefill and decode phases, batching, and the memory pressure that follows from them.
  • GPU memory management. Pinned host buffers, asynchronous CUDA streams, and coalesced transfers.
  • Integration with an existing codebase you do not own, through patches and middleware rather than a fork.
  • Performance measurement. Establishing a baseline before changing anything, then attributing a change in wall clock time to a specific mechanism.

Prior familiarity with asyncio and with GPU serving makes the resource scheduler module easier to approach, but neither is assumed.


Technical overview

Three subsystems sit at three different points in the path from user request to model output. They are independent: none imports another, and each can run on its own.

flowchart TB
    User([User request])

    subgraph AF[DeerFlow agent framework]
        Server[FastAPI server]
        Planner[Planner]
        Executor[Step executor]
    end

    subgraph UDF[UDF optimizer]
        WT[Workflow types]
        DA[Dependency analyzer]
        PE[Parallel executor]
        CM[Configuration manager]
        GE[Gemini executor]
    end

    subgraph WS[Workflow scheduler]
        MW[ASGI middleware]
        AS[Scheduler core]
        REG[Workflow registry]
        PRI[Priority calculation]
        PQ[Priority queue]
        SP[Priority slot pool]
        MM[Memory monitor]
        OP[Offload pools]
    end

    subgraph RSM[Resource scheduler module]
        PT[Process table]
        SM[Session manager]
        AW[API wrapper]
        SCH[ATLAS scheduler]
        MLFQ[Multilevel queues]
        LB[Load balancer]
        MEM[Multi-engine manager]
        KV[KV cache coordinator]
        SWAP[Memory management]
    end

    GPU[(vLLM engines on GPUs)]
    GEM[(Gemini API)]
    DISK[(Checkpoint directory)]

    User --> Server --> MW --> AS
    Server --> Planner --> DA
    DA --> PE --> Executor
    CM --> PE
    PE --> GE --> GEM

    AS --> REG
    AS --> PRI --> PQ --> SP
    AS --> MM
    MM --> DISK
    AS --> OP
    SP --> Executor

    Executor --> AW --> SM --> PT
    SCH --> PT
    SCH --> MLFQ
    AW --> SCH --> LB --> MEM --> GPU
    LB --> PT
    MEM --> KV --> SWAP
Loading

What each subsystem is responsible for

Subsystem Sits at Decides
UDF optimizer Between planner and executor Which steps of one plan can run at the same time
Workflow scheduler Between HTTP request and step execution Which of many concurrent programs gets an execution slot next, and which get checkpointed to disk under memory pressure
Resource scheduler module Between the agent and the model servers Which LLM call runs next and which engine serves it

Feature to component map

Feature Component
Grouping plan steps into batches Dependency analyzer
Running a batch concurrently and holding the barrier Parallel executor
Concurrency limits, retries, failure mode Configuration manager
Model calls for both analysis and step execution Gemini executor
Registering a program on request arrival ASGI middleware
Tracking attained service and status per program Workflow registry
Turning attained service and wait time into a score Priority calculation
Handing the next slot to the highest scoring waiter Priority queue, priority slot pool
Detecting host memory pressure Memory monitor
Moving blocking tool calls off the event loop Offload pools
Tracking service time, wait time and engine affinity per program Process table
Program lifecycle and identifiers Session manager
Choosing the next LLM call ATLAS scheduler, multilevel queues
Choosing the engine for a call Load balancer
Starting, stopping and monitoring engines Multi-engine manager
Moving KV cache between GPU and host Memory management

A concrete flow through the UDF optimizer

A user asks for a tourist research report covering ten cities. The planner emits a twelve step plan: ten independent city research steps, one analysis step that reads all ten, and one report step that reads the analysis.

  1. The plan arrives as JSON and is parsed into a Plan of twelve Step objects by the workflow types component.
  2. The dependency analyzer sends the step titles and descriptions to the model along with the instruction in config/parallel_prompt.md, and receives back a grouping: steps 1 to 10 in batch one, step 11 in batch two, step 12 in batch three.
  3. The parallel executor launches all ten steps of batch one at once, bounded by max_concurrent_tasks, which defaults to 10. It gathers the results and holds the barrier until every step has either returned or errored.
  4. Each result is written back to its step and appended to the shared observation list, so batch two sees all ten city summaries in its prompt context.
  5. Batches two and three run their single steps in turn.

Measured on this plan against gemini-2.5-flash, the sequential run took about 110 seconds and the batched run about 85 seconds. The floor is set by the slowest city in batch one plus the two serial steps, not by the sum of all twelve.

A concrete flow through the resource scheduler module

A program that has already processed 900 tokens issues a follow up call with a 4000 token prompt.

  1. The API wrapper accepts the call, and the session manager attaches it to the program's existing identifier in the process table.
  2. The ATLAS scheduler reads the program's service time of 900 tokens. With a base quantum of 512, that places the program in queue level 1, since 900 is under 1024.
  3. The scheduler checks the program's starvation ratio against the configured threshold. It is below the threshold, so no promotion happens and the call waits its turn in level 1.
  4. When the call is selected, the load balancer reads its prefill length of 4000 tokens. That is above the 2048 token threshold, so the balancer routes it to the engine already recorded against this program in the process table rather than to the least loaded engine.
  5. The multi-engine manager submits the call to that engine process, which finds the program's prefix already resident in its KV cache and skips recomputing it.
  6. As tokens are processed the scheduler updates the program's service time. If the call exhausts its quantum before finishing, the program drops to level 2 and the KV cache coordinator may move its cache blocks to host memory to free room.

External connections

  • vLLM engines holding model weights in GPU memory, one engine process per GPU.
  • Model weights fetched from Hugging Face. download.py pulls meta-llama/Llama-2-7b-hf.
  • The Gemini API, used by the UDF optimizer for dependency analysis and step execution. Requires GEMINI_API_KEY.
  • The DeerFlow FastAPI server, which the workflow scheduler attaches to as ASGI middleware.
  • A checkpoint directory on local disk, default /tmp/deerflow_checkpoints, holding serialized state for programs that have been swapped out of RAM.

How the system extends

  • The Gemini executor is one provider implementation behind a common step execution interface. Adding another provider means adding one module beside it, with no change to the analyzer or the executor.
  • Framework support beyond DeerFlow is a matter of adding a task adapter that maps that framework's step objects onto the shared plan and step types, plus a hook that intercepts the plan before execution. The scheduling components do not change.
  • The three subsystems are currently separate. Connecting the UDF optimizer's step execution to the resource scheduler module's API wrapper, rather than to a provider API, would put the batched steps of one plan under program aware scheduling.
  • The process table is in memory and single host. Replacing it with a shared store would let several scheduler hosts serve one engine pool.

Repository layout

src/
  workflow_scheduler/    Priority scheduler and memory manager for concurrent programs
  rsm/                   Program-aware LLM call scheduler and multi-engine manager
  udf-optimizer/         Plan parallelization with model-based dependency analysis
integration/             Patch files and apply/revert scripts for DeerFlow
scripts/                 Setup, run, benchmark and test entry points
tests/                   Tests for the workflow scheduler and the DeerFlow patches
deerflow/                DeerFlow git submodule, empty until initialized
Autellix.pdf             The paper the scheduling design reproduces

Components

Workflow scheduler

Located at src/workflow_scheduler. Schedules whole programs on one host. Its concern is how many programs execute at once and which ones, not which model call runs.

Neighbours and communication

It attaches to the DeerFlow FastAPI server as ASGI middleware, which tells it when a program starts and ends. Patched hooks in DeerFlow's graph nodes tell it when a step is about to run and when it has finished, which is where a program blocks for a slot and where its attained service is updated. It reads host memory statistics from the operating system and writes serialized program state to a checkpoint directory. It does not talk to the resource scheduler module or to the UDF optimizer.

Components

  • Workflow registry core/registry.py. Holds one WorkflowMeta per active program: creation time, attained service, step count, last execution time, status, and location. Status is queued, executing, paused, or completed. Location is hot, meaning state is in RAM, or cold, meaning state has been serialized to disk. Guarded by an asyncio lock.

  • Priority calculation core/priority.py. Pure functions with no state. Base priority is the inverse of attained service, P = 1 / (A + ε) with ε = 1e-6, so a new program with zero attained service starts near the maximum. The aging factor is 1 + wait / θ_age, where wait is measured from the later of creation and last execution, and θ_age defaults to 5 seconds. Effective priority is the product of the two, which grows without bound as wait time grows and so guarantees eventual execution.

  • Priority queue core/priority_queue.py. A max priority queue over a negated min heap, tiebreaking by insertion order so equal priorities are served first in first out. A bounded variant raises CapacityError when full. Deliberately separated from the slot pool so it can be tested on its own and reused for eviction ordering.

  • Priority slot pool core/slot_pool.py. A semaphore that grants the next free slot to the highest priority waiter rather than the longest waiting one. Programs suspend here between steps, which is the point at which cooperative multitasking actually happens. Capacity defaults to 16 slots.

  • Memory monitor core/memory_monitor.py. Samples host memory through psutil and reports pressure as a percentage. Crossing the swap threshold, default 80 percent, triggers checkpointing of cold programs. Crossing the critical threshold, default 90 percent, blocks admission of new programs. Falls back to mock values when psutil is absent, so the module imports on any host, but pressure readings are then meaningless.

  • Offload pools offload/pools.py. Classifies a blocking operation as CPU bound, IO bound, or already asynchronous, then runs CPU bound work in a ProcessPoolExecutor to escape the GIL and IO bound work in a ThreadPoolExecutor. Defaults are 4 process workers and 10 thread workers. Without this, one synchronous tool call stalls every concurrent program on the host.

  • ASGI middleware middleware.py. Registers a program when a request reaches a workflow endpoint, stores the identifier on the request state so downstream code can find it, and unregisters on completion or error. It performs no scheduling itself.

  • Scheduler core core/atlas.py. Ties the above together and exposes the pre step and post step hooks that DeerFlow calls.

  • Configuration loader config.py. Resolves settings from environment variables first, then DeerFlow's conf.yaml, then defaults, and validates them. Rejects a swap threshold that is not below the critical threshold, among other checks.

Resource scheduler module

Located at src/rsm. Schedules individual LLM calls across a pool of engines. Reproduces the Autellix design.

Neighbours and communication

An agent program opens a session through the API wrapper and issues calls against it. Every call's arrival, wait, and service is recorded in the process table, which the scheduler and the load balancer both read. The engine layer owns the vLLM processes and reports load and cache state back. src/rsm/rsm.py wires the four groups together and imports them as top level packages, so it runs from inside src/rsm.

Frontend

  • Process table frontend/process_table.py. The shared state everything else reads. One ProgramEntry per program holding service time, waiting time, the engines that have served it, and its live threads. One ThreadMetadata per call holding arrival time, waiting time, service time, prefill and decode token counts, parent thread identifiers, priority, and queue index. Thread safe through an RLock. Computes service time two ways: as a simple cumulative total for single threaded programs, and as the longest path among threads for multi threaded ones.

  • Session manager frontend/session_manager.py. Issues session identifiers, maps sessions to programs, and cleans up on end or timeout. Without timeout cleanup an abandoned session leaves an entry that skews the scheduler's view of load.

  • API wrapper frontend/api_wrapper.py. The client surface. Provides an OpenAI compatible adapter and a session context manager, so an agent can issue calls through the scheduler with roughly the shape it already uses.

Scheduler

  • Multilevel queue base scheduler/mlfq_base.py. Builds the priority levels. Quantum for level n is base_quantum * 2^n with base_quantum defaulting to 512 tokens, so higher levels get exponentially longer turns. A program's level is derived from its service time by log2(service_time / base_quantum) + 1.

  • ATLAS scheduler scheduler/atlas_scheduler.py. Selects the next call. Runs in one of two modes per program. Program mode is first in first out within the program and suits single threaded programs. Thread mode ranks the program's threads and suits programs whose branches have diverged. The scheduler switches a program between modes based on the variance in its threads' service times, so mode is adapted rather than configured. Also handles quantum exhaustion, which demotes a program one level, and starvation promotion, which lifts every thread of a program back to the top level at once.

  • Program context scheduler/context.py. Per program scheduling state: current mode, queue level, and thread bookkeeping.

Load balancer

  • Balancer load_balancer/balancer.py. Implements Algorithm 2 of the Autellix paper. A call with 2048 prefill tokens or fewer goes to the least used engine. A call above 2048 goes to the engine already associated with the program, to reuse its KV cache. The threshold is a configured constant, not a heuristic to tune casually, since it encodes the point where recomputing a prefix costs more than queueing.

  • Engine info load_balancer/engine_info.py. Per engine state the balancer ranks on: current load and free cache capacity.

Engine layer

  • Multi-engine manager engine/multi_engine_manager.py. Starts and stops engine processes, routes submitted requests, collects results through futures, and cancels in flight requests either singly or by session. Cancellation matters because a preempted or abandoned program otherwise keeps consuming GPU time.

  • Engine process engine/engine_process.py. One vLLM instance in its own process, bound to one GPU, communicating over multiprocessing queues.

  • KV cache coordinator engine/kv_cache_coordinator.py. Tracks which program's cache blocks live on which engine, which is the information the balancer's affinity decision depends on.

  • Lifecycle manager engine/lifecycle_manager.py. Tracks each request through its states from submission to completion or cancellation.

  • vLLM integration engine/vllm_integration.py. The adaptation layer to vLLM, written against version 0.6.1.

Memory management

  • Batched swap manager engine/memory/batched_swap.py. Coalesces many KV cache blocks into one contiguous pinned buffer and issues a single PCIe transfer instead of one per block, across four asynchronous CUDA streams. Detects the GPU type and sets pinned buffer size, batch size, and eviction threshold accordingly, from 16 GB pinned on an A100 down to 4 GB on a T4.

  • Swap scheduler engine/memory/swap_scheduler.py. Chooses which program's cache to evict, using program level information rather than treating every block equally.

  • Swap predictor engine/memory/swap_predictor.py. Prefetches cache back to the GPU ahead of a resume, so a resumed call does not stall on the transfer.

  • Pressure manager engine/memory/pressure_manager.py. Monitors GPU memory and triggers eviction at the configured threshold. Configured through configs/memory.yaml, which supports auto detection.

  • CUDA kernels engine/kernels/. Vectorized transfer kernels using 128 bit float4 loads and stores with grid stride loops, compiled just in time for the detected architecture, supporting compute capability 7.0 and above. A Python fallback path exists, so the absence of a compiler degrades performance rather than breaking the system.

  • Multi-step scheduler engine/scheduler/multi_step_scheduler.py. Batches several decode steps per scheduling decision to cut per step overhead.

UDF optimizer

Located at src/udf-optimizer. Turns a sequential plan into batches of independent steps and runs them.

Neighbours and communication

It takes a plan as JSON, either from a planner or from a file, and returns the plan with each step's result filled in and the shared observation list appended to. It calls a model provider for both dependency analysis and step execution. It does not know about either scheduler.

Components

  • Workflow types core/workflow_types.py. The data model: Plan, Step, State, Configuration, and StepType. State carries the message list, the observation list, and the current plan, and is what threads results from one batch into the next batch's prompts.

  • Dependency analyzer core/gemini_executor.py. Sends the plan to the model with the instruction in config/parallel_prompt.md and parses back a batch grouping. Groups conservatively, so an uncertain pair of steps is kept sequential rather than run together. On failure it falls back to rule based batching, and the run continues at sequential speed.

  • Parallel executor core/nodes.py. parallel_research_team_node orchestrates the batches. _execute_batch_parallel gathers a batch with exceptions returned rather than raised, so one failed step becomes an error string on that step instead of killing the batch. _execute_batch_sequential is the fallback path. Results are written back and appended before the barrier releases.

  • Graph builder core/builder.py. Builds either the parallel or the sequential execution graph, which is what makes the whole feature switchable off.

  • Configuration manager core/config_manager.py. Loads config/config.yaml and offers presets named speed, balanced, reliability, and cost. Controls concurrency ceiling, launch rate, per step and per batch timeouts, retry count and backoff, and failure mode, which is either partial_completion or fail_fast.

  • Provider executor core/gemini_executor.py. Executes a step against Gemini with the accumulated context in the prompt, and distinguishes research steps from processing steps. Adding another provider means adding a module beside this one.

  • Sequential executor core/sequential_executor.py. The unbatched path, used as the comparison baseline.

  • Performance comparison compare_performance.py. Runs the same plan both ways, times them, and writes a report. examples/example_performance_report.md is a sample output.

DeerFlow integration

Located at integration/ and scripts/.

Neighbours and communication

Applies a patch to the DeerFlow submodule so its server, graph nodes, and tools call into the workflow scheduler. Nothing in src/ is modified by this.

Components

  • Patch file integration/patches/patch.patch. Touches six DeerFlow files: src/config/configuration.py to carry a thread identifier, src/graph/checkpoint.py and src/graph/nodes.py for step hooks and logging, src/server/app.py for middleware registration, and src/tools/crawl.py and src/tools/python_repl.py to wrap blocking tools. A disabled variant sits in integration/patches_disabled/.

  • Apply and revert scripts integration/apply_patches.sh and integration/revert_patches.sh. Applying checks whether the patch is already present and skips rather than failing, so the script is safe to rerun.

  • Setup script scripts/setup.sh. Initializes the submodule, installs dependencies, applies patches, creates the checkpoint and log directories, and verifies that the scheduler imports and that the memory monitor reads a real value.

  • Run script scripts/run_deerflow.py. Starts the DeerFlow server with the scheduler attached, or without it when SCHEDULER_ENABLED=false.

  • Baseline benchmark scripts/benchmark_baseline.py. Drives unmodified DeerFlow with concurrent SSE requests and records wall clock time, throughput, and success rate. No scheduler code is injected, so its numbers are the baseline that scheduler runs are compared against.


Installation

Requires Python 3.9 or later. The resource scheduler module additionally requires one or more NVIDIA GPUs and a working vLLM install. The workflow scheduler and the UDF optimizer run on CPU only.

git clone --recurse-submodules https://github.com/notming11/Agentic-Runtime-Resource-Scheduling-Optimization.git
cd Agentic-Runtime-Resource-Scheduling-Optimization
bash scripts/setup.sh

scripts/setup.sh initializes the DeerFlow submodule, installs the project in editable mode, applies the DeerFlow patches, and verifies the install. If the submodule was not cloned, run git submodule update --init --recursive first.

For the UDF optimizer only:

cd src/udf-optimizer
pip install -r requirements.txt
cp .env.example .env

Then set GEMINI_API_KEY in .env. Keys are issued at https://makersuite.google.com/app/apikey.


Running

DeerFlow with the workflow scheduler

export PYTHONPATH=$PWD/src:$PYTHONPATH
python scripts/run_deerflow.py

DeerFlow without the scheduler, for a baseline

SCHEDULER_ENABLED=false python scripts/run_deerflow.py
python scripts/benchmark_baseline.py --mode baseline --concurrent 5

The UDF optimizer

cd src/udf-optimizer
python tests/validate.py     # import and configuration check, no model calls
python main.py               # interactive: load a plan, pick a mode, run

Configuration

Workflow scheduler

Set through environment variables, which take precedence over DeerFlow's conf.yaml, which takes precedence over the defaults below.

Variable Default Meaning
SCHEDULER_ENABLED true Whether the scheduler runs at all
SCHEDULER_MAX_SLOTS 16 Concurrent execution slots
SCHEDULER_SWAP_THRESHOLD 80 Host memory percent at which checkpointing starts
SCHEDULER_CRITICAL_THRESHOLD 90 Host memory percent at which new programs are refused
SCHEDULER_AGING_THRESHOLD 5.0 Seconds of waiting that doubles a program's priority
SCHEDULER_MAX_WAIT 300.0 Seconds before starvation intervention
SCHEDULER_CHECKPOINT_DIR /tmp/deerflow_checkpoints Where cold program state is written
SCHEDULER_PROCESS_WORKERS 4 Workers for CPU bound offload
SCHEDULER_THREAD_WORKERS 10 Workers for IO bound offload
SCHEDULER_STATS_INTERVAL 30 Seconds between statistics log lines

The swap threshold must be strictly below the critical threshold. Configuration loading raises on violation rather than silently correcting.

UDF optimizer

Set in src/udf-optimizer/config/config.yaml.

Key Default Meaning
max_concurrent_tasks 10 Ceiling on steps in flight
max_tasks_per_second 5.0 Launch rate limit
task_timeout_seconds 300 Per step timeout
batch_timeout_seconds 900 Per batch timeout
dependency_strategy llm_based Also accepts heuristic and explicit
retry_on_failure true Retry a failed step
max_retries 3 Retry ceiling
failure_mode partial_completion Also accepts fail_fast

Lower max_concurrent_tasks to 3 or below if the provider returns rate limit errors.

Resource scheduler module

Memory behaviour is set in src/rsm/configs/memory.yaml, which supports GPU auto detection. Scheduler behaviour is set through the config dictionary passed to AutellixConfig in src/rsm/rsm.py, covering the number of queue levels, the base quantum of 512 tokens, the starvation ratio threshold, and the 2048 token routing threshold.


Testing

pytest tests/workflow_scheduler/ -v          # unit tests for the scheduler
pytest tests/test_patch_integration.py -v    # DeerFlow patch integration
bash scripts/run_tests.sh --unit             # wrapper with coverage

Resource scheduler module tests live under src/rsm/tests/ and are run from that directory. The multi engine and end to end tests need GPUs.

UDF optimizer tests live under src/udf-optimizer/tests/. validate.py makes no model calls. test_e2e.py and test_real_execution.py do, and consume quota.

Continuous integration runs the workflow scheduler unit tests on Python 3.9 through 3.12, plus ruff, mypy, and a check that the scheduler enable and disable paths both work. Linting and type checking are advisory and do not fail the build.


Current state

  • The three subsystems are independent. No module in one imports a module from another. Connecting them is unfinished work, not a design property.
  • deerflow/ is an empty submodule until initialized, and the setup and run scripts skip patching when it is absent.
  • The resource scheduler module needs GPUs, vLLM, and model weights. On a CPU only host it imports and unit tests run, but the engine layer cannot start.
  • tests/scenarios/ is present but empty.
  • scripts/run_tests.sh looks for test_patch_integration.py under tests/workflow_scheduler/, but the file is at tests/test_patch_integration.py. Invoke pytest directly to run it.
  • Reported speedups come from two sources. The 110 second to 85 second figure was measured on this codebase against gemini-2.5-flash. The 1.4 times routing throughput figure is from the Autellix paper and has not been reproduced here.

License

Apache License 2.0. See LICENSE.