Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CREDITS.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,4 @@ The following software libraries are utilized in this repository.

* [c2pa-node](https://github.com/contentauth/c2pa-node) test fixtures (`XCA.jpg`, `A.jpg`) by Adobe, licensed MIT. Used as real-world signed JPEG assets in `tests/fixtures/c2pa/` to verify the JPEG APP11 manifest reader.
* [HTML to MD](https://github.com/dmsnell/html-to-md) by Dennis Snell. Used to convert HTML to Markdown.
* Wapuu working sprites (`src/experiments/ai-workspace/assets/wapuu/`), from the set [Jeffrey Paul (@jeffpaul)](https://github.com/jeffpaul) shared on [#1287](https://github.com/WordPress/ai/issues/1287). The sprites are derived from the [original Wapuu](https://wapu.us/wapuu-history-origin/) by Kazuko Kaneuchi, licensed under GPL-2.0-or-later. Used for the AI Workspace's busy, done and error states.
2 changes: 1 addition & 1 deletion docs/experiments/ai-request-logging.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ Writes an entry to the same log the experiment populates. Returns the log identi
) );
```

`type`, `operation`, and `status` are validated: anything else triggers `_doing_it_wrong()` and no row is written. `type` must be one of the values `AI_Request_Log_Manager::get_types()` returns — `ai_client` (generations made through the AI Client SDK), `mcp_tool`, or `ability`. A row carrying an unlisted type could never be filtered through the REST API, and `operation` and `status` are stored in columns that cannot be null.
`type`, `operation`, and `status` are validated: anything else triggers `_doing_it_wrong()` and no row is written. `type` must be one of the values `AI_Request_Log_Manager::get_types()` returns — `ai_client` (generations made through the AI Client SDK), `mcp_tool`, `ability`, or `agent` (the AI Workspace agent's tool calls and proposal events; see [AI Workspace](ai-workspace.md#auditing-with-the-ai-request-log)). A row carrying an unlisted type could never be filtered through the REST API, and `operation` and `status` are stored in columns that cannot be null.

The REST `type` filter enum is derived from `get_types()`, so the read and write sides cannot drift apart.

Expand Down
42 changes: 35 additions & 7 deletions docs/experiments/ai-workspace.md
Original file line number Diff line number Diff line change
Expand Up @@ -404,6 +404,8 @@ A turn is capped at five rounds by default and terminates with a status of `comp

**A chat belongs to one user.** Every read matches the post author as well as the conversation ID, and a `map_meta_cap` filter grants `read_post`, `edit_post` and `delete_post` on a chat to its author alone, with no override for other administrators. A conversation ID that does not resolve for the requesting user is reported as not found rather than silently starting a new conversation. Deleting a user, or removing them from a site, deletes their chats before WordPress can reassign their content to someone else. The ownership filter and that cleanup are hooked whether or not the experiment is on.

**Chats are personal data.** `Tools → Export Personal Data` includes an **AI chats** group: one item per chat, archived ones included, with its name, when it was created and last updated, whether it is archived, and each turn's message and reply. Tool results are left out. It pages at 20 chats. `Tools → Erase Personal Data` deletes all of the person's chats. A proposal from an erased chat can no longer be approved, because an approval is refused once its chat is gone. Like the ownership filter, the exporter and eraser (`Conversation_Privacy`) are hooked whether or not the experiment is on, and they find archived chats even while the `wpai_archived` status is not registered.

**A turn replays a bounded window.** The whole transcript is kept for display, but a turn sends the model only the last 40 stored messages (see [`wpai_workspace_replay_messages`](#wpai_workspace_replay_messages)), opening on a message the person sent. Within that window, tool results from before the latest turn are replaced by their retrieval summary, so a long chat does not outgrow the provider's context limit.

Cancellation markers and proposals still live in **user-scoped transients** sharing the plugin's `wpai_` prefix:
Expand Down Expand Up @@ -458,16 +460,41 @@ curl -X POST "https://yoursite.com/wp-json/wp-abilities/v1/abilities/ai/search-c

## Auditing with the AI Request Log

Enable the **AI Request Logging** experiment and every workspace tool call and every write attempt appears at `Tools → AI Request Logs`. The shape is fixed so workspace rows join with rows any other ability consumer writes:
Every agent event is recorded in one place, `Agent_Audit`: each tool call, including a withheld one; each item an approval writes; and each event in a proposal's life. Enable the **AI Request Logging** experiment and they appear at `Tools → AI Request Logs`, where the **Type** filter's `agent` value shows only them and the **Status** filter's `denied` value shows what was refused.

- `type` is always `agent`. Rows written before this type existed keep the `ability` type.
- `operation` is one of:
- for a tool call, the ability name, such as `ai/search-content` or `ai/propose-post-changes`;
- for an item an approval wrote, the kind's write operation: `ai/create-drafts`, `ai/create-template`, `ai/update-post-fields` or `ai/update-site-identity`;
- for a proposal event, `proposal_created`, `proposal_edited` (the person changed a value before approving), `proposal_declined`, `proposal_expired` (recorded once, when an expired proposal is next read), `proposal_failed` (a write that threw, or one interrupted for longer than the approval lock) or `proposal_refused` (an approval after the chat was deleted).
- `status` is `success`, `error`, `denied`, `skipped` (an item left as it was: unchanged, stale, locked or a duplicate), or `timeout` for an expired proposal. A **denial is recorded separately from a failure**: a refusal sets `denied` and records `denial_reason` in the context.
- `context` carries `surface` (always `ai-workspace`) and `conversation_id`.
- Tool rows add the `round` index and a `tool` object naming the ability, the function name and the call ID.
- Item rows add a `proposal` object with the proposal ID and item key, plus the post ID, post type, status and field, or the option.
- Proposal event rows add a `proposal` object with the proposal ID, its kind and its item keys, and for an edit the edited item keys. Failures and refusals add `error_code`.
- `user_id` is the person the tool ran as, or whose proposal it is.

- `type` is always `ability`.
- `operation` is the ability name — `ai/search-content`, `ai/read-content-bodies`, `ai/propose-drafts`, `ai/propose-template` — or `ai/create-drafts` or `ai/create-template` for a write attempt.
- `status` is `success`, `error`, `denied`, or `skipped` for an idempotent duplicate. A **denial is recorded separately from a failure**: a refusal from the Abilities API sets `denied` and records `denial_reason` in the context.
- `context` carries `surface` (always `ai-workspace`), `conversation_id`, the `round` index, and a `tool` object naming the ability, the function name and the call ID. Write rows add a `proposal` object with the proposal ID, item key, post type and status.
- `user_id` is the person the tool ran as.
No row carries post content or option values: a row names the field or option that changed, never what it changed to.

Between them, those fields answer "what did the assistant do on this site, for whom, and was it allowed" without reading application code. Logging is best-effort: `log_ai_request()` returns `false` harmlessly when the logging experiment is off, so it is called unconditionally and costs nothing when nobody is watching.

### `wpai_agent_event`

Fires once for every event above, before its row is written, whether or not AI Request Logging is on, so an audit plugin need not depend on that experiment.

```php
add_action(
'wpai_agent_event',
function ( string $operation, string $status, array $context, string $error ) {
// $operation, $status and $context are the row's fields; $error is its error message, or ''.
},
10,
4
);
```

It covers agent turns and approvals in the AI Workspace only. A direct MCP call to an ability does not fire it.

## Extending the Experiment

### `wpai_workspace_tool_candidates`
Expand Down Expand Up @@ -627,7 +654,8 @@ Filters whether a function-calling-capable model is available, for connectors th
- As a non-administrator, confirm the Tools menu entry is absent and the routes return 403
- With a limited role that can reach the screen via a filtered capability, confirm search results never contain another author's private or draft posts
6. **Auditing:**
- Enable **AI Request Logging** and confirm each tool call and write attempt appears with `surface: ai-workspace`, its conversation ID and round
- Enable **AI Request Logging** and confirm each tool call and write attempt appears with type `agent`, `surface: ai-workspace`, its conversation ID and round
- Propose a change, edit it and approve it; decline a second one. Confirm `proposal_created`, `proposal_edited` and `proposal_declined` rows appear, and filter the log by type `agent` and status `denied`

### Automated tests

Expand Down
148 changes: 148 additions & 0 deletions includes/Experiments/AI_Workspace/Agent_Audit.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
<?php
/**
* Audit trail for agent events.
*
* @package WordPress\AI\Experiments\AI_Workspace
*/

declare( strict_types=1 );

namespace WordPress\AI\Experiments\AI_Workspace;

use function WordPress\AI\log_ai_request;

// Exit if accessed directly.
defined( 'ABSPATH' ) || exit;

/**
* Records every agent event in one place.
*
* Each tool call, including a withheld one, each proposal item written, and
* each proposal event (created, edited, declined, expired, failed, refused) is
* one `agent` row in the AI Request Log and one call of the
* {@see self::ACTION} hook. The hook fires whether or not the AI Request
* Logging experiment is on, so an audit plugin need not depend on it (KTD11).
*
* The hook covers agent turns and approvals only. A direct MCP call to an
* ability does not pass through here.
*
* Context never carries post content or option values: tool rows name the
* tool, item rows name the post and field or the option, and proposal rows
* name the proposal and its item keys.
*
* @since x.x.x
*/
final class Agent_Audit {

/**
* The action fired for every agent event.
*
* @since x.x.x
*
* @var string
*/
public const ACTION = 'wpai_agent_event';

/**
* The AI Request Log type of every agent row.
*
* @since x.x.x
*
* @var string
*/
public const LOG_TYPE = 'agent';

/**
* Records one event: fires the hook, then writes the log row.
*
* @since x.x.x
*
* @param array{
* operation: string,
* status: string,
* user_id?: int,
* context?: array<string, mixed>,
* duration_ms?: int,
* error_message?: string
* } $data The event, in the shape {@see log_ai_request()} takes, without a type.
*/
public static function record( array $data ): void {
$context = $data['context'] ?? array();
$error = $data['error_message'] ?? '';

/**
* Fires for every agent event.
*
* Events are tool calls (operation: the ability name; status `success`,
* `error` or `denied`), proposal items written (operation: the kind's
* write operation, such as `ai/update-post-fields`), and proposal events
* (operation `proposal_created`, `proposal_edited`, `proposal_declined`,
* `proposal_expired`, `proposal_failed` or `proposal_refused`). It fires
* whether or not AI Request Logging is on.
*
* @since x.x.x
*
* @param string $operation The operation.
* @param string $status The status: `success`, `error`, `denied`, `skipped` or `timeout`.
* @param array<string, mixed> $context The event context. It carries no post content or option values.
* @param string $error The error message, or an empty string.
*/
do_action( 'wpai_agent_event', $data['operation'], $data['status'], $context, $error );

log_ai_request( array_merge( $data, array( 'type' => self::LOG_TYPE ) ) );
}

/**
* Records an event in a proposal's life.
*
* @since x.x.x
*
* @param string $event The operation, such as `proposal_created`.
* @param string $status The status.
* @param array<string, mixed> $proposal The stored proposal.
* @param array<string, mixed> $extra Optional. More proposal fields, such as the edited item keys. Default empty array.
* @param string $error_code Optional. The error code, when there is one. Default empty string.
*/
public static function proposal( string $event, string $status, array $proposal, array $extra = array(), string $error_code = '' ): void {
$items = isset( $proposal['items'] ) && is_array( $proposal['items'] ) ? $proposal['items'] : array();
$keys = array();

foreach ( $items as $item ) {
if ( ! is_array( $item ) || ! isset( $item['key'] ) || ! is_string( $item['key'] ) ) {
continue;
}

$keys[] = $item['key'];
}

$context = array(
'surface' => Turn_Runner::LOG_SURFACE,
'conversation_id' => isset( $proposal['conversation_id'] ) && is_string( $proposal['conversation_id'] ) ? $proposal['conversation_id'] : '',
'proposal' => array_merge(
array(
'id' => isset( $proposal['id'] ) && is_string( $proposal['id'] ) ? $proposal['id'] : '',
'kind' => isset( $proposal['kind'] ) && is_string( $proposal['kind'] ) ? $proposal['kind'] : Draft_Writer::KIND,
'items' => $keys,
),
$extra
),
);

if ( '' !== $error_code ) {
$context['error_code'] = $error_code;
}

if ( Turn_Runner::LOG_STATUS_DENIED === $status ) {
$context['denial_reason'] = $error_code;
}

self::record(
array(
'operation' => $event,
'status' => $status,
'user_id' => isset( $proposal['user_id'] ) ? (int) $proposal['user_id'] : get_current_user_id(),
'context' => $context,
)
);
}
}
Loading
Loading