serviceability: halt and resume a feed

CHANGELOG.md

@@ -13,6 +13,7 @@ All notable changes to this project will be documented in this file.
 - CI
   - The Agave toolchain install retries, and a failed one now fails the job. Eight workflow steps across five workflows ran `sh -c "$(curl -sSfL .../install)"` once with no retry, so a transient reset from `release.anza.xyz` killed a job before it ran anything. That form also swallowed a failed fetch: the command substitution comes back empty, `sh -c ""` exits 0, and the step passed having installed nothing. The eight are now one composite action at `.github/actions/solana-toolchain` that fetches and runs as separate steps, checks `solana --version` actually runs, clears partial state between attempts, bounds every wait so a stalled handshake or hung transfer reaches the backoff instead of sitting until the job times out, and backs off. Each caller keeps the version it used before; `solana.yml` and `offchain.local-validator.yml` are on v3.0.12 and the other three on v3.0.4, which is drift worth settling separately.
 - Serviceability
+  - `HaltFeed` (variant 120) and `ResumeFeed` (variant 121), so a feed's status can change. `FeedStatus` and the gate that reads it landed earlier, but nothing could move the status, which left a staked feed in `Pending` for life and gave no feed a way to stop publishing. Halt goes only from `Active` and resume only from `Halted`; every other transition is refused by name, `FeedNotHaltable` (124) or `FeedNotResumable` (125). `Pending` to `Active` is deliberately absent, because that step re-reads the stake mirror and belongs with the code that does. The feed's own `builder` may sign either instruction, which no other feed instruction allows: RFC-28 makes halt the builder's lever and doubles it as upstream-source rotation, so a builder that cannot halt its own feed cannot rotate either. A `FEED_AUTHORITY` or `FOUNDATION` key may sign as well, because a feed whose builder has gone quiet must still be stoppable. `Feed` gains `halted_by`, so an operator's halt can only be lifted by an operator: a builder that could undo it leaves an operator no lever at all, with `Retired` unreachable and `DeleteFeed` refusing a staked feed. Resuming a staked feed re-proves that its stake still covers its rate, because a mirror can be corrected downward while the feed sits halted.
   - `UpdateMulticastGroupRoles` authorizes only roles a user gains. An existing feed subscription no longer needs a direct subscriber allowlist entry when the user adds publishing. (malbeclabs/infra#2596)
   - `write_stake_mirror` in the instruction crate and `WriteStakeMirrorCommand` in the Rust SDK, so the relayer has a caller-side surface for the instruction A6 added. The command departs from its neighbours in one way: every other one uses `append_payer_permission_account`, which attaches the caller's `Permission` account only when it already exists, because a legacy `GlobalState` key might authorize instead. No legacy key satisfies `STAKE_ORACLE`, so a missing `Permission` account is always fatal here, and the command says so locally rather than sending a transaction that can only come back `NotAllowed`. It checks what `authorize` checks, not just ownership: an account that does not decode, a suspended `Permission`, and one lacking the flag are each refused with the reason named, which matters because suspending a `Permission` is what revoking the relayer's key looks like.
   - New `WriteStakeMirror` instruction (variant 119) and a `STAKE_ORACLE` permission, which is what a relayer will call to copy a builder's Solana stake onto the DZ ledger. Nothing could write a `StakeMirror` before this. The instruction refuses a write whose `source_slot` is not newer than the stored one: polling makes a repeated write harmless but says nothing about ordering, and a retry carrying an older read could otherwise walk the mirror backwards, so the program enforces it rather than trusting the caller. It also refuses to reassign a mirror to a different builder, since the builder is not a PDA seed, and it carries `feed_key` forward rather than taking it from the caller, because `CreateFeed` writes that to spend the stake and zeroing it would let one bond back two feeds. No legacy GlobalState key maps to `STAKE_ORACLE`, so a holder needs a real `Permission` account even while `require-permission-accounts` is clear. Gated on `allow-staked-feeds` with the rest of RFC-28. `STAKE_ORACLE` is grantable through `doublezero permission set --add stake-oracle`, named by `permission audit` and `bitmask_to_names`, listed in `AUTHORIZE_GATED_FLAGS` and in the Go SDK's flag constants. It is the first flag no legacy `GlobalState` key can satisfy, which the audit's own test now asserts through a `PERMISSION_ONLY_FLAGS` list rather than treating as a gap in the enumeration.

crates/doublezero-serviceability-instruction/src/feed.rs

@@ -28,8 +28,11 @@
 use crate::common;
 use doublezero_serviceability::{
     instructions::DoubleZeroInstruction,
-    pda::{get_feed_pda, get_globalstate_pda},
-    processors::feed::{create::FeedCreateArgs, delete::FeedDeleteArgs, update::FeedUpdateArgs},
+    pda::{get_feed_pda, get_globalstate_pda, get_stake_mirror_pda},
+    processors::feed::{
+        create::FeedCreateArgs, delete::FeedDeleteArgs, halt::FeedHaltArgs, resume::FeedResumeArgs,
+        update::FeedUpdateArgs,
+    },
 };
 use solana_program::{
     instruction::{AccountMeta, Instruction},
@@ -42,15 +45,27 @@ use solana_program::{
 pub fn create_feed(program_id: &Pubkey, payer: &Pubkey, args: FeedCreateArgs) -> Instruction {
     let (feed, _) = get_feed_pda(program_id, &args.code, &args.exchange);
     let (globalstate, _) = get_globalstate_pda(program_id);
-    common::build_with_permission(
+
+    // A staked feed needs its stake mirror: `CreateFeed` reads the tier from it and writes the
+    // feed's key onto it to claim the stake. Derived rather than taken as a parameter, because
+    // unlike `resume_feed` the args already carry the stake it is seeded on, so asking a caller
+    // for it would only create a way to get it wrong. Writable: creating the feed claims it.
+    let stake_mirror = (args.builder != Pubkey::default())
+        .then(|| get_stake_mirror_pda(program_id, &args.stake_ref).0);
+
+    let mut ix = common::build_with_permission(
         program_id,
         DoubleZeroInstruction::CreateFeed(args),
         vec![
             AccountMeta::new(feed, false),
             AccountMeta::new(globalstate, false),
         ],
         payer,
-    )
+    );
+    if let Some(stake_mirror) = stake_mirror {
+        ix.accounts.push(AccountMeta::new(stake_mirror, false));
+    }
+    ix
 }
 
 /// `UpdateFeed` (variant 113). Accounts: `[feed, globalstate]`.
@@ -91,11 +106,115 @@ pub fn delete_feed(
     )
 }
 
+/// `HaltFeed` (variant 120). Accounts: `[feed, globalstate]`.
+///
+/// Stops a feed publishing, reversibly. Unlike the other feed instructions, the feed's own
+/// `builder` may sign this one, so a builder can rotate its upstream source without an operator.
+/// A `FEED_AUTHORITY` or `FOUNDATION` key can sign it too, which is why this stays on the
+/// permission-appending path.
+pub fn halt_feed(program_id: &Pubkey, payer: &Pubkey, feed: &Pubkey) -> Instruction {
+    let (globalstate, _) = get_globalstate_pda(program_id);
+    common::build_with_permission(
+        program_id,
+        DoubleZeroInstruction::HaltFeed(FeedHaltArgs {}),
+        vec![
+            AccountMeta::new(*feed, false),
+            AccountMeta::new(globalstate, false),
+        ],
+        payer,
+    )
+}
+
+/// `ResumeFeed` (variant 121). Accounts: `[feed, globalstate]`, then the stake mirror.
+///
+/// Puts a halted feed back to publishing. Signed by the keys `halt_feed` accepts, except that an
+/// operator's halt takes an operator to lift.
+///
+/// `stake_mirror` is required for a staked feed and must be `None` for a catalog one. Resume
+/// re-proves that the stake still covers the feed's rate, because a mirror can be corrected
+/// downward while a feed sits halted, so a staked feed without its mirror is refused rather than
+/// read as having no stake to check. It rides after the payer and system program, found by its
+/// address rather than its position, so a catalog feed's caller is not forced to send one.
+pub fn resume_feed(
+    program_id: &Pubkey,
+    payer: &Pubkey,
+    feed: &Pubkey,
+    stake_mirror: Option<&Pubkey>,
+) -> Instruction {
+    let (globalstate, _) = get_globalstate_pda(program_id);
+    let mut ix = common::build_with_permission(
+        program_id,
+        DoubleZeroInstruction::ResumeFeed(FeedResumeArgs {}),
+        vec![
+            AccountMeta::new(*feed, false),
+            AccountMeta::new(globalstate, false),
+        ],
+        payer,
+    );
+    if let Some(stake_mirror) = stake_mirror {
+        ix.accounts
+            .push(AccountMeta::new_readonly(*stake_mirror, false));
+    }
+    ix
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
     use solana_system_interface::program as system_program;
 
+    /// A staked feed carries its mirror; a catalog feed must not, or a caller that sends neither
+    /// stops working.
+    #[test]
+    fn test_create_feed_appends_the_stake_mirror_only_when_staked() {
+        let pid = Pubkey::new_unique();
+        let payer = Pubkey::new_unique();
+        let exchange = Pubkey::new_unique();
+        let builder = Pubkey::new_unique();
+        let stake_ref = Pubkey::new_unique();
+
+        let staked = create_feed(
+            &pid,
+            &payer,
+            FeedCreateArgs {
+                code: "staked".to_string(),
+                name: "Staked".to_string(),
+                exchange,
+                groups: vec![Pubkey::new_unique()],
+                builder,
+                stake_ref,
+                spec_id: "top-of-book@v1.0.0".to_string(),
+                sla_hash: [9u8; 32],
+                committed_rate_bits_per_sec: 1_000_000_000,
+            },
+        );
+        let (mirror, _) = get_stake_mirror_pda(&pid, &stake_ref);
+        assert_eq!(
+            staked.accounts.last().unwrap(),
+            &AccountMeta::new(mirror, false),
+            "the mirror is writable: creating the feed claims the stake"
+        );
+
+        let catalog = create_feed(
+            &pid,
+            &payer,
+            FeedCreateArgs {
+                code: "catalog".to_string(),
+                name: "Catalog".to_string(),
+                exchange,
+                groups: vec![Pubkey::new_unique()],
+                ..Default::default()
+            },
+        );
+        assert!(
+            !catalog
+                .accounts
+                .iter()
+                .any(|a| a.pubkey == get_stake_mirror_pda(&pid, &Pubkey::default()).0),
+            "a catalog feed sends no mirror"
+        );
+    }
+
     #[test]
     fn test_create_feed_derives_pda_from_code_and_exchange() {
         let pid = Pubkey::new_unique();
@@ -151,6 +270,26 @@ mod tests {
         let delete = delete_feed(&pid, &payer, &feed, FeedDeleteArgs {});
         assert_eq!(delete.data[0], 114);
         assert_eq!(delete.accounts, expected);
+
+        // The lifecycle verbs take the same accounts. `unpack` matches the leading byte by hand
+        // with a catch-all, so a wrong tag here reaches the program as `InvalidInstructionData`
+        // rather than as a compile error.
+        let halt = halt_feed(&pid, &payer, &feed);
+        assert_eq!(halt.data[0], 120);
+        assert_eq!(halt.accounts, expected);
+        let resume = resume_feed(&pid, &payer, &feed, None);
+        assert_eq!(resume.data[0], 121);
+        assert_eq!(resume.accounts, expected);
+
+        // A staked feed's mirror rides after the payer and system program, where the processor
+        // looks for it. Without it, resume refuses with `StakeMirrorMissing`.
+        let mirror = Pubkey::new_unique();
+        let staked = resume_feed(&pid, &payer, &feed, Some(&mirror));
+        assert_eq!(staked.accounts[..expected.len()], expected[..]);
+        assert_eq!(
+            staked.accounts.last().unwrap(),
+            &AccountMeta::new_readonly(mirror, false)
+        );
     }
 
     /// Tripwire for the module-doc note: `FEED_AUTHORITY` is currently absent from

sdk/serviceability/python/serviceability/state.py

@@ -1275,6 +1275,9 @@ class Feed:
     sla_hash: bytes = b"\x00" * 32
     committed_rate_bits_per_sec: int = 0
     status: int = 0
+    # Who halted the feed, default when it is not halted. Appended after status, so a feed
+    # written before it reads as halted by nobody, which is right: it cannot have been halted.
+    halted_by: Pubkey = Pubkey.default()
     pub_key: Pubkey = Pubkey.default()  # set from account address after deserialization
 
     @classmethod
@@ -1301,4 +1304,5 @@ def from_bytes(cls, data: bytes) -> Feed:
         # Not pending: a feed written before RFC-28 has no status byte, and reading one as pending
         # would show every live catalog feed as out of service.
         f.status = r.read_u8() if has_rfc28_tail else FEED_STATUS_ACTIVE
+        f.halted_by = _read_pubkey(r)
         return f

sdk/serviceability/testdata/fixtures/generate-fixtures/src/main.rs

@@ -1516,6 +1516,7 @@ fn generate_feed(dir: &Path) {
         sla_hash: [0xE6; 32],
         committed_rate_bits_per_sec: 1_000_000_000,
         status: FeedStatus::Pending,
+        halted_by: Pubkey::default(),
     };
 
     let data = borsh::to_vec(&val).unwrap();

sdk/serviceability/typescript/serviceability/state.ts

@@ -1262,6 +1262,9 @@ export interface Feed {
   slaHash: Uint8Array;
   committedRateBitsPerSec: bigint;
   status: number;
+  // Who halted the feed, the default key when it is not halted. Appended after status, so a feed
+  // written before it reads as halted by nobody, which is right: it cannot have been halted.
+  haltedBy: PublicKey;
 }
 
 // Feed lifecycle. Matches FeedStatus in the Rust program.
@@ -1292,6 +1295,7 @@ export function deserializeFeed(data: Uint8Array): Feed {
   // Not "pending": a feed written before RFC-28 has no status byte, and reading one as pending
   // would show every live catalog feed as out of service.
   const status = hasRfc28Tail ? r.readU8() : FEED_STATUS_ACTIVE;
+  const haltedBy = readPubkey(r);
   return {
     accountType,
     owner,
@@ -1306,5 +1310,6 @@ export function deserializeFeed(data: Uint8Array): Feed {
     slaHash,
     committedRateBitsPerSec,
     status,
+    haltedBy,
   };
 }

smartcontract/programs/doublezero-serviceability/src/entrypoint.rs

@@ -38,7 +38,8 @@ use crate::{
             suspend::process_suspend_exchange, update::process_update_exchange,
         },
         feed::{
-            create::process_create_feed, delete::process_delete_feed, update::process_update_feed,
+            create::process_create_feed, delete::process_delete_feed, halt::process_halt_feed,
+            resume::process_resume_feed, update::process_update_feed,
         },
         globalconfig::set::process_set_globalconfig,
         globalstate::{
@@ -421,6 +422,10 @@ pub fn process_instruction(
         DoubleZeroInstruction::WriteStakeMirror(value) => {
             process_write_stake_mirror(program_id, accounts, &value)?
         }
+        DoubleZeroInstruction::HaltFeed(value) => process_halt_feed(program_id, accounts, &value)?,
+        DoubleZeroInstruction::ResumeFeed(value) => {
+            process_resume_feed(program_id, accounts, &value)?
+        }
         DoubleZeroInstruction::UpdateFeed(value) => {
             process_update_feed(program_id, accounts, &value)?
         }

smartcontract/programs/doublezero-serviceability/src/error.rs

@@ -257,6 +257,10 @@ pub enum DoubleZeroError {
     StakedFeedCannotBeDeleted, // variant 122
     #[error("This feed is not publishing, so it admits no new subscribers")]
     FeedNotActive, // variant 123
+    #[error("Only an active feed can be halted")]
+    FeedNotHaltable, // variant 124
+    #[error("Only a halted feed can be resumed")]
+    FeedNotResumable, // variant 125
 }
 
 impl From<DoubleZeroError> for ProgramError {
@@ -386,6 +390,8 @@ impl From<DoubleZeroError> for ProgramError {
             DoubleZeroError::StakeAlreadyBacksFeed => ProgramError::Custom(121),
             DoubleZeroError::StakedFeedCannotBeDeleted => ProgramError::Custom(122),
             DoubleZeroError::FeedNotActive => ProgramError::Custom(123),
+            DoubleZeroError::FeedNotHaltable => ProgramError::Custom(124),
+            DoubleZeroError::FeedNotResumable => ProgramError::Custom(125),
         }
     }
 }
@@ -516,6 +522,8 @@ impl From<u32> for DoubleZeroError {
             121 => DoubleZeroError::StakeAlreadyBacksFeed,
             122 => DoubleZeroError::StakedFeedCannotBeDeleted,
             123 => DoubleZeroError::FeedNotActive,
+            124 => DoubleZeroError::FeedNotHaltable,
+            125 => DoubleZeroError::FeedNotResumable,
             _ => DoubleZeroError::Custom(e),
         }
     }
@@ -550,7 +558,7 @@ mod tests {
         }
 
         // EnumIter generates Custom(0) by default, so we explicitly test values
-        // outside the known variant range (currently 0-123) to ensure the conversion
+        // outside the known variant range (currently 0-125) to ensure the conversion
         // logic handles arbitrary custom codes correctly.
         for code in [1000u32, 100_000, u32::MAX] {
             let err = DoubleZeroError::Custom(code);

smartcontract/programs/doublezero-serviceability/src/instructions.rs

@@ -27,7 +27,10 @@ use crate::processors::{
         create::ExchangeCreateArgs, delete::ExchangeDeleteArgs, resume::ExchangeResumeArgs,
         setdevice::ExchangeSetDeviceArgs, suspend::ExchangeSuspendArgs, update::ExchangeUpdateArgs,
     },
-    feed::{create::FeedCreateArgs, delete::FeedDeleteArgs, update::FeedUpdateArgs},
+    feed::{
+        create::FeedCreateArgs, delete::FeedDeleteArgs, halt::FeedHaltArgs, resume::FeedResumeArgs,
+        update::FeedUpdateArgs,
+    },
     globalconfig::set::SetGlobalConfigArgs,
     globalstate::{
         setairdrop::SetAirdropArgs, setauthority::SetAuthorityArgs,
@@ -261,6 +264,9 @@ pub enum DoubleZeroInstruction {
     UnsubscribeFeed(UnsubscribeFeedArgs), // variant 118
 
     WriteStakeMirror(StakeMirrorWriteArgs), // variant 119
+
+    HaltFeed(FeedHaltArgs),     // variant 120
+    ResumeFeed(FeedResumeArgs), // variant 121
 }
 
 impl DoubleZeroInstruction {
@@ -414,6 +420,8 @@ impl DoubleZeroInstruction {
             119 => Ok(Self::WriteStakeMirror(
                 StakeMirrorWriteArgs::try_from(rest).unwrap(),
             )),
+            120 => Ok(Self::HaltFeed(FeedHaltArgs::try_from(rest).unwrap())),
+            121 => Ok(Self::ResumeFeed(FeedResumeArgs::try_from(rest).unwrap())),
 
             _ => Err(ProgramError::InvalidInstructionData),
         }
@@ -561,6 +569,8 @@ impl DoubleZeroInstruction {
 
             Self::CreateFeed(_) => "CreateFeed".to_string(), // variant 112
             Self::WriteStakeMirror(_) => "WriteStakeMirror".to_string(), // variant 119
+            Self::HaltFeed(_) => "HaltFeed".to_string(),     // variant 120
+            Self::ResumeFeed(_) => "ResumeFeed".to_string(), // variant 121
             Self::UpdateFeed(_) => "UpdateFeed".to_string(), // variant 113
             Self::DeleteFeed(_) => "DeleteFeed".to_string(), // variant 114
             Self::SetAccessPassFeeds(_) => "SetAccessPassFeeds".to_string(), // variant 115
@@ -706,6 +716,8 @@ impl DoubleZeroInstruction {
 
             Self::CreateFeed(args) => format!("{args:?}"), // variant 112
             Self::WriteStakeMirror(args) => format!("{args:?}"), // variant 119
+            Self::HaltFeed(args) => format!("{args:?}"),   // variant 120
+            Self::ResumeFeed(args) => format!("{args:?}"), // variant 121
             Self::UpdateFeed(args) => format!("{args:?}"), // variant 113
             Self::DeleteFeed(args) => format!("{args:?}"), // variant 114
             Self::SetAccessPassFeeds(args) => format!("{args:?}"), // variant 115

smartcontract/programs/doublezero-serviceability/src/processors/feed/create.rs

@@ -153,6 +153,7 @@ pub fn process_create_feed(
         } else {
             FeedStatus::Pending
         },
+        halted_by: Pubkey::default(),
     };
 
     try_acc_create(