serviceability: halt and resume a feed

CHANGELOG.md

@@ -13,6 +13,7 @@ All notable changes to this project will be documented in this file.
 - CI
   - The Agave toolchain install retries, and a failed one now fails the job. Eight workflow steps across five workflows ran `sh -c "$(curl -sSfL .../install)"` once with no retry, so a transient reset from `release.anza.xyz` killed a job before it ran anything. That form also swallowed a failed fetch: the command substitution comes back empty, `sh -c ""` exits 0, and the step passed having installed nothing. The eight are now one composite action at `.github/actions/solana-toolchain` that fetches and runs as separate steps, checks `solana --version` actually runs, clears partial state between attempts, bounds every wait so a stalled handshake or hung transfer reaches the backoff instead of sitting until the job times out, and backs off. Each caller keeps the version it used before; `solana.yml` and `offchain.local-validator.yml` are on v3.0.12 and the other three on v3.0.4, which is drift worth settling separately.
 - Serviceability
+  - `HaltFeed` (variant 120) and `ResumeFeed` (variant 121), so a feed's status can change. `FeedStatus` and the gate that reads it landed earlier, but nothing could move the status, which left a staked feed in `Pending` for life and gave no feed a way to stop publishing. Halt goes only from `Active` and resume only from `Halted`; every other transition is refused by name, `FeedNotHaltable` (124) or `FeedNotResumable` (125). `Pending` to `Active` is deliberately absent, because that step re-reads the stake mirror and belongs with the code that does. The feed's own `builder` may sign either instruction, which no other feed instruction allows: RFC-28 makes halt the builder's lever and doubles it as upstream-source rotation, so a builder that cannot halt its own feed cannot rotate either. A `FEED_AUTHORITY` or `FOUNDATION` key may sign as well, because a feed whose builder has gone quiet must still be stoppable. `Feed` gains `halted_by`, so an operator's halt can only be lifted by an operator: a builder that could undo it leaves an operator no lever at all, with `Retired` unreachable and `DeleteFeed` refusing a staked feed. Resuming a staked feed re-proves that its stake still covers its rate, because a mirror can be corrected downward while the feed sits halted.
   - `UpdateMulticastGroupRoles` authorizes only roles a user gains. An existing feed subscription no longer needs a direct subscriber allowlist entry when the user adds publishing. (malbeclabs/infra#2596)
   - `write_stake_mirror` in the instruction crate and `WriteStakeMirrorCommand` in the Rust SDK, so the relayer has a caller-side surface for the instruction A6 added. The command departs from its neighbours in one way: every other one uses `append_payer_permission_account`, which attaches the caller's `Permission` account only when it already exists, because a legacy `GlobalState` key might authorize instead. No legacy key satisfies `STAKE_ORACLE`, so a missing `Permission` account is always fatal here, and the command says so locally rather than sending a transaction that can only come back `NotAllowed`. It checks what `authorize` checks, not just ownership: an account that does not decode, a suspended `Permission`, and one lacking the flag are each refused with the reason named, which matters because suspending a `Permission` is what revoking the relayer's key looks like.
   - New `WriteStakeMirror` instruction (variant 119) and a `STAKE_ORACLE` permission, which is what a relayer will call to copy a builder's Solana stake onto the DZ ledger. Nothing could write a `StakeMirror` before this. The instruction refuses a write whose `source_slot` is not newer than the stored one: polling makes a repeated write harmless but says nothing about ordering, and a retry carrying an older read could otherwise walk the mirror backwards, so the program enforces it rather than trusting the caller. It also refuses to reassign a mirror to a different builder, since the builder is not a PDA seed, and it carries `feed_key` forward rather than taking it from the caller, because `CreateFeed` writes that to spend the stake and zeroing it would let one bond back two feeds. No legacy GlobalState key maps to `STAKE_ORACLE`, so a holder needs a real `Permission` account even while `require-permission-accounts` is clear. Gated on `allow-staked-feeds` with the rest of RFC-28. `STAKE_ORACLE` is grantable through `doublezero permission set --add stake-oracle`, named by `permission audit` and `bitmask_to_names`, listed in `AUTHORIZE_GATED_FLAGS` and in the Go SDK's flag constants. It is the first flag no legacy `GlobalState` key can satisfy, which the audit's own test now asserts through a `PERMISSION_ONLY_FLAGS` list rather than treating as a gap in the enumeration.

crates/doublezero-serviceability-instruction/src/feed.rs

@@ -29,7 +29,10 @@ use crate::common;
 use doublezero_serviceability::{
     instructions::DoubleZeroInstruction,
     pda::{get_feed_pda, get_globalstate_pda},
-    processors::feed::{create::FeedCreateArgs, delete::FeedDeleteArgs, update::FeedUpdateArgs},
+    processors::feed::{
+        create::FeedCreateArgs, delete::FeedDeleteArgs, halt::FeedHaltArgs, resume::FeedResumeArgs,
+        update::FeedUpdateArgs,
+    },
 };
 use solana_program::{
     instruction::{AccountMeta, Instruction},
@@ -91,6 +94,41 @@ pub fn delete_feed(
     )
 }
 
+/// `HaltFeed` (variant 120). Accounts: `[feed, globalstate]`.
+///
+/// Stops a feed publishing, reversibly. Unlike the other feed instructions, the feed's own
+/// `builder` may sign this one, so a builder can rotate its upstream source without an operator.
+/// A `FEED_AUTHORITY` or `FOUNDATION` key can sign it too, which is why this stays on the
+/// permission-appending path.
+pub fn halt_feed(program_id: &Pubkey, payer: &Pubkey, feed: &Pubkey) -> Instruction {
+    let (globalstate, _) = get_globalstate_pda(program_id);
+    common::build_with_permission(
+        program_id,
+        DoubleZeroInstruction::HaltFeed(FeedHaltArgs {}),
+        vec![
+            AccountMeta::new(*feed, false),
+            AccountMeta::new(globalstate, false),
+        ],
+        payer,
+    )
+}
+
+/// `ResumeFeed` (variant 121). Accounts: `[feed, globalstate]`.
+///
+/// Puts a halted feed back to publishing. Signed by the same keys `halt_feed` accepts.
+pub fn resume_feed(program_id: &Pubkey, payer: &Pubkey, feed: &Pubkey) -> Instruction {
+    let (globalstate, _) = get_globalstate_pda(program_id);
+    common::build_with_permission(
+        program_id,
+        DoubleZeroInstruction::ResumeFeed(FeedResumeArgs {}),
+        vec![
+            AccountMeta::new(*feed, false),
+            AccountMeta::new(globalstate, false),
+        ],
+        payer,
+    )
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -151,6 +189,16 @@ mod tests {
         let delete = delete_feed(&pid, &payer, &feed, FeedDeleteArgs {});
         assert_eq!(delete.data[0], 114);
         assert_eq!(delete.accounts, expected);
+
+        // The lifecycle verbs take the same accounts. `unpack` matches the leading byte by hand
+        // with a catch-all, so a wrong tag here reaches the program as `InvalidInstructionData`
+        // rather than as a compile error.
+        let halt = halt_feed(&pid, &payer, &feed);
+        assert_eq!(halt.data[0], 120);
+        assert_eq!(halt.accounts, expected);
+        let resume = resume_feed(&pid, &payer, &feed);
+        assert_eq!(resume.data[0], 121);
+        assert_eq!(resume.accounts, expected);
     }
 
     /// Tripwire for the module-doc note: `FEED_AUTHORITY` is currently absent from

smartcontract/programs/doublezero-serviceability/src/entrypoint.rs

@@ -38,7 +38,8 @@ use crate::{
             suspend::process_suspend_exchange, update::process_update_exchange,
         },
         feed::{
-            create::process_create_feed, delete::process_delete_feed, update::process_update_feed,
+            create::process_create_feed, delete::process_delete_feed, halt::process_halt_feed,
+            resume::process_resume_feed, update::process_update_feed,
         },
         globalconfig::set::process_set_globalconfig,
         globalstate::{
@@ -421,6 +422,10 @@ pub fn process_instruction(
         DoubleZeroInstruction::WriteStakeMirror(value) => {
             process_write_stake_mirror(program_id, accounts, &value)?
         }
+        DoubleZeroInstruction::HaltFeed(value) => process_halt_feed(program_id, accounts, &value)?,
+        DoubleZeroInstruction::ResumeFeed(value) => {
+            process_resume_feed(program_id, accounts, &value)?
+        }
         DoubleZeroInstruction::UpdateFeed(value) => {
             process_update_feed(program_id, accounts, &value)?
         }

smartcontract/programs/doublezero-serviceability/src/error.rs

@@ -257,6 +257,10 @@ pub enum DoubleZeroError {
     StakedFeedCannotBeDeleted, // variant 122
     #[error("This feed is not publishing, so it admits no new subscribers")]
     FeedNotActive, // variant 123
+    #[error("Only an active feed can be halted")]
+    FeedNotHaltable, // variant 124
+    #[error("Only a halted feed can be resumed")]
+    FeedNotResumable, // variant 125
 }
 
 impl From<DoubleZeroError> for ProgramError {
@@ -386,6 +390,8 @@ impl From<DoubleZeroError> for ProgramError {
             DoubleZeroError::StakeAlreadyBacksFeed => ProgramError::Custom(121),
             DoubleZeroError::StakedFeedCannotBeDeleted => ProgramError::Custom(122),
             DoubleZeroError::FeedNotActive => ProgramError::Custom(123),
+            DoubleZeroError::FeedNotHaltable => ProgramError::Custom(124),
+            DoubleZeroError::FeedNotResumable => ProgramError::Custom(125),
         }
     }
 }
@@ -516,6 +522,8 @@ impl From<u32> for DoubleZeroError {
             121 => DoubleZeroError::StakeAlreadyBacksFeed,
             122 => DoubleZeroError::StakedFeedCannotBeDeleted,
             123 => DoubleZeroError::FeedNotActive,
+            124 => DoubleZeroError::FeedNotHaltable,
+            125 => DoubleZeroError::FeedNotResumable,
             _ => DoubleZeroError::Custom(e),
         }
     }
@@ -550,7 +558,7 @@ mod tests {
         }
 
         // EnumIter generates Custom(0) by default, so we explicitly test values
-        // outside the known variant range (currently 0-123) to ensure the conversion
+        // outside the known variant range (currently 0-125) to ensure the conversion
         // logic handles arbitrary custom codes correctly.
         for code in [1000u32, 100_000, u32::MAX] {
             let err = DoubleZeroError::Custom(code);

smartcontract/programs/doublezero-serviceability/src/instructions.rs

@@ -27,7 +27,10 @@ use crate::processors::{
         create::ExchangeCreateArgs, delete::ExchangeDeleteArgs, resume::ExchangeResumeArgs,
         setdevice::ExchangeSetDeviceArgs, suspend::ExchangeSuspendArgs, update::ExchangeUpdateArgs,
     },
-    feed::{create::FeedCreateArgs, delete::FeedDeleteArgs, update::FeedUpdateArgs},
+    feed::{
+        create::FeedCreateArgs, delete::FeedDeleteArgs, halt::FeedHaltArgs, resume::FeedResumeArgs,
+        update::FeedUpdateArgs,
+    },
     globalconfig::set::SetGlobalConfigArgs,
     globalstate::{
         setairdrop::SetAirdropArgs, setauthority::SetAuthorityArgs,
@@ -261,6 +264,9 @@ pub enum DoubleZeroInstruction {
     UnsubscribeFeed(UnsubscribeFeedArgs), // variant 118
 
     WriteStakeMirror(StakeMirrorWriteArgs), // variant 119
+
+    HaltFeed(FeedHaltArgs),     // variant 120
+    ResumeFeed(FeedResumeArgs), // variant 121
 }
 
 impl DoubleZeroInstruction {
@@ -414,6 +420,8 @@ impl DoubleZeroInstruction {
             119 => Ok(Self::WriteStakeMirror(
                 StakeMirrorWriteArgs::try_from(rest).unwrap(),
             )),
+            120 => Ok(Self::HaltFeed(FeedHaltArgs::try_from(rest).unwrap())),
+            121 => Ok(Self::ResumeFeed(FeedResumeArgs::try_from(rest).unwrap())),
 
             _ => Err(ProgramError::InvalidInstructionData),
         }
@@ -561,6 +569,8 @@ impl DoubleZeroInstruction {
 
             Self::CreateFeed(_) => "CreateFeed".to_string(), // variant 112
             Self::WriteStakeMirror(_) => "WriteStakeMirror".to_string(), // variant 119
+            Self::HaltFeed(_) => "HaltFeed".to_string(),     // variant 120
+            Self::ResumeFeed(_) => "ResumeFeed".to_string(), // variant 121
             Self::UpdateFeed(_) => "UpdateFeed".to_string(), // variant 113
             Self::DeleteFeed(_) => "DeleteFeed".to_string(), // variant 114
             Self::SetAccessPassFeeds(_) => "SetAccessPassFeeds".to_string(), // variant 115
@@ -706,6 +716,8 @@ impl DoubleZeroInstruction {
 
             Self::CreateFeed(args) => format!("{args:?}"), // variant 112
             Self::WriteStakeMirror(args) => format!("{args:?}"), // variant 119
+            Self::HaltFeed(args) => format!("{args:?}"),   // variant 120
+            Self::ResumeFeed(args) => format!("{args:?}"), // variant 121
             Self::UpdateFeed(args) => format!("{args:?}"), // variant 113
             Self::DeleteFeed(args) => format!("{args:?}"), // variant 114
             Self::SetAccessPassFeeds(args) => format!("{args:?}"), // variant 115

smartcontract/programs/doublezero-serviceability/src/processors/feed/create.rs

@@ -153,6 +153,7 @@ pub fn process_create_feed(
         } else {
             FeedStatus::Pending
         },
+        halted_by: Pubkey::default(),
     };
 
     try_acc_create(

smartcontract/programs/doublezero-serviceability/src/processors/feed/halt.rs

@@ -0,0 +1,77 @@
+use crate::{
+    error::DoubleZeroError,
+    processors::feed::require_feed_writer,
+    serializer::try_acc_write,
+    state::{
+        feed::{Feed, FeedStatus},
+        globalstate::GlobalState,
+    },
+};
+use borsh::BorshSerialize;
+use borsh_incremental::BorshDeserializeIncremental;
+use solana_program::{
+    account_info::{next_account_info, AccountInfo},
+    entrypoint::ProgramResult,
+    msg,
+    pubkey::Pubkey,
+};
+
+#[derive(BorshSerialize, BorshDeserializeIncremental, PartialEq, Debug, Clone, Default)]
+#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
+pub struct FeedHaltArgs {}
+
+/// Stop a feed publishing, reversibly.
+///
+/// Halt is the builder's own lever and doubles as a way to rotate the upstream source without
+/// redeploying, so it is reversible and touches neither the stake nor the seats already sold.
+/// Retirement is the terminal path and is not this.
+pub fn process_halt_feed(
+    program_id: &Pubkey,
+    accounts: &[AccountInfo],
+    _value: &FeedHaltArgs,
+) -> ProgramResult {
+    let accounts_iter = &mut accounts.iter();
+
+    let feed_account = next_account_info(accounts_iter)?;
+    let globalstate_account = next_account_info(accounts_iter)?;
+    let payer_account = next_account_info(accounts_iter)?;
+    let _system_program = next_account_info(accounts_iter)?;
+
+    assert!(payer_account.is_signer, "Payer must be a signer");
+    assert_eq!(feed_account.owner, program_id, "Invalid PDA Account Owner");
+    assert_eq!(
+        globalstate_account.owner, program_id,
+        "Invalid GlobalState Account Owner"
+    );
+    assert!(feed_account.is_writable, "PDA Account is not writable");
+
+    let mut feed = Feed::try_from(feed_account)?;
+    let globalstate = GlobalState::try_from(globalstate_account)?;
+    require_feed_writer(
+        program_id,
+        accounts_iter,
+        payer_account.key,
+        &globalstate,
+        &feed,
+    )?;
+
+    // Only a publishing feed can stop publishing. Pending has not started and Retired is terminal,
+    // so neither is a halt this instruction can honor.
+    if feed.status != FeedStatus::Active {
+        msg!(
+            "Feed {} is {}, so there is nothing to halt",
+            feed_account.key,
+            feed.status
+        );
+        return Err(DoubleZeroError::FeedNotHaltable.into());
+    }
+
+    feed.status = FeedStatus::Halted;
+    // Recorded so resume can tell an operator's halt from the builder's own.
+    feed.halted_by = *payer_account.key;
+    try_acc_write(&feed, feed_account, payer_account, accounts)?;
+
+    msg!("Halted feed: {}", feed_account.key);
+
+    Ok(())
+}

smartcontract/programs/doublezero-serviceability/src/processors/feed/mod.rs

@@ -1,12 +1,18 @@
 pub mod create;
 pub mod delete;
+pub mod halt;
+pub mod resume;
 pub mod update;
 
 use crate::{
-    error::DoubleZeroError,
+    authorize::authorize,
+    error::{DoubleZeroError, Validate},
     state::{
         accesspass::AccessPass,
         feed::{Feed, FeedStatus},
+        globalstate::GlobalState,
+        permission::permission_flags,
+        stake_mirror::StakeMirror,
     },
 };
 use solana_program::{
@@ -120,3 +126,80 @@ pub fn require_feed_admits(feed_key: &Pubkey, feed: &Feed) -> Result<(), DoubleZ
     }
     Ok(())
 }
+
+/// Whether `payer` may change this feed's lifecycle.
+///
+/// Two ways in, for different reasons. The feed's own builder, because RFC-28 makes halt the
+/// builder's lever and a builder that cannot halt its own feed cannot rotate its upstream source.
+/// A `FEED_AUTHORITY` or `FOUNDATION` key, because a feed whose builder has gone quiet must still
+/// be stoppable, and every other feed instruction already authorizes that way.
+///
+/// A catalog feed has no builder, so only the second way applies to it. The default pubkey is not
+/// a signer anyone can produce, but the check is explicit rather than relying on that.
+pub fn require_feed_writer<'a, 'b: 'a, I>(
+    program_id: &Pubkey,
+    accounts_iter: &mut I,
+    payer: &Pubkey,
+    globalstate: &GlobalState,
+    feed: &Feed,
+) -> ProgramResult
+where
+    I: Iterator<Item = &'a AccountInfo<'b>>,
+{
+    if feed.builder != Pubkey::default() && &feed.builder == payer {
+        return Ok(());
+    }
+
+    authorize(
+        program_id,
+        accounts_iter,
+        payer,
+        globalstate,
+        permission_flags::FEED_AUTHORITY | permission_flags::FOUNDATION,
+    )
+}
+
+/// Whether the stake behind `feed` still covers the rate it publishes at.
+///
+/// Not the same question `CreateFeed` asks. Creation claims an unspent stake, so it requires
+/// `feed_key` to be empty; here the feed already holds the claim, so the mirror must name this
+/// feed and no other. What both check is the tier, because a mirror can be corrected downward
+/// while a feed sits halted.
+pub fn require_stake_still_covers(
+    program_id: &Pubkey,
+    mirror_account: &AccountInfo,
+    feed_key: &Pubkey,
+    feed: &Feed,
+) -> Result<(), DoubleZeroError> {
+    if mirror_account.data_is_empty() || mirror_account.owner != program_id {
+        msg!("No stake mirror written for stake {}", feed.stake_ref);
+        return Err(DoubleZeroError::StakeMirrorMissing);
+    }
+
+    let mirror =
+        StakeMirror::try_from(mirror_account).map_err(|_| DoubleZeroError::InvalidAccountType)?;
+    mirror.validate()?;
+
+    if mirror.stake_ref != feed.stake_ref || mirror.builder != feed.builder {
+        msg!("Stake mirror names a different stake or builder");
+        return Err(DoubleZeroError::InvalidArgument);
+    }
+    // The claim has to point back at this feed. A mirror claimed by another feed is not this
+    // feed's cover, whatever its tier says.
+    if &mirror.feed_key != feed_key {
+        msg!("Stake mirror is claimed by feed {}", mirror.feed_key);
+        return Err(DoubleZeroError::InvalidArgument);
+    }
+
+    if !mirror.tier.covers(feed.committed_rate_bits_per_sec) {
+        msg!(
+            "Tier {} covers up to {} bits/sec, feed commits to {}",
+            mirror.tier,
+            mirror.tier.max_rate_bits_per_sec(),
+            feed.committed_rate_bits_per_sec
+        );
+        return Err(DoubleZeroError::StakeDoesNotCoverRate);
+    }
+
+    Ok(())
+}