diff --git a/CREDITS.md b/CREDITS.md index cc89600f4..a863ce954 100644 --- a/CREDITS.md +++ b/CREDITS.md @@ -20,3 +20,4 @@ The following software libraries are utilized in this repository. * [c2pa-node](https://github.com/contentauth/c2pa-node) test fixtures (`XCA.jpg`, `A.jpg`) by Adobe, licensed MIT. Used as real-world signed JPEG assets in `tests/fixtures/c2pa/` to verify the JPEG APP11 manifest reader. * [HTML to MD](https://github.com/dmsnell/html-to-md) by Dennis Snell. Used to convert HTML to Markdown. +* Wapuu working sprites (`src/experiments/ai-workspace/assets/wapuu/`), from the set [Jeffrey Paul (@jeffpaul)](https://github.com/jeffpaul) shared on [#1287](https://github.com/WordPress/ai/issues/1287). The sprites are derived from the [original Wapuu](https://wapu.us/wapuu-history-origin/) by Kazuko Kaneuchi, licensed under GPL-2.0-or-later. Used for the AI Workspace's busy, done and error states. diff --git a/docs/experiments/ai-request-logging.md b/docs/experiments/ai-request-logging.md index b46683252..1b178153a 100644 --- a/docs/experiments/ai-request-logging.md +++ b/docs/experiments/ai-request-logging.md @@ -62,7 +62,7 @@ Writes an entry to the same log the experiment populates. Returns the log identi ) ); ``` -`type`, `operation`, and `status` are validated: anything else triggers `_doing_it_wrong()` and no row is written. `type` must be one of the values `AI_Request_Log_Manager::get_types()` returns — `ai_client` (generations made through the AI Client SDK), `mcp_tool`, or `ability`. A row carrying an unlisted type could never be filtered through the REST API, and `operation` and `status` are stored in columns that cannot be null. +`type`, `operation`, and `status` are validated: anything else triggers `_doing_it_wrong()` and no row is written. `type` must be one of the values `AI_Request_Log_Manager::get_types()` returns — `ai_client` (generations made through the AI Client SDK), `mcp_tool`, `ability`, or `agent` (the AI Workspace agent's tool calls and proposal events; see [AI Workspace](ai-workspace.md#auditing-with-the-ai-request-log)). A row carrying an unlisted type could never be filtered through the REST API, and `operation` and `status` are stored in columns that cannot be null. The REST `type` filter enum is derived from `get_types()`, so the read and write sides cannot drift apart. diff --git a/docs/experiments/ai-workspace.md b/docs/experiments/ai-workspace.md index f709b7747..f64a45ef2 100644 --- a/docs/experiments/ai-workspace.md +++ b/docs/experiments/ai-workspace.md @@ -404,6 +404,8 @@ A turn is capped at five rounds by default and terminates with a status of `comp **A chat belongs to one user.** Every read matches the post author as well as the conversation ID, and a `map_meta_cap` filter grants `read_post`, `edit_post` and `delete_post` on a chat to its author alone, with no override for other administrators. A conversation ID that does not resolve for the requesting user is reported as not found rather than silently starting a new conversation. Deleting a user, or removing them from a site, deletes their chats before WordPress can reassign their content to someone else. The ownership filter and that cleanup are hooked whether or not the experiment is on. +**Chats are personal data.** `Tools → Export Personal Data` includes an **AI chats** group: one item per chat, archived ones included, with its name, when it was created and last updated, whether it is archived, and each turn's message and reply. Tool results are left out. It pages at 20 chats. `Tools → Erase Personal Data` deletes all of the person's chats. A proposal from an erased chat can no longer be approved, because an approval is refused once its chat is gone. Like the ownership filter, the exporter and eraser (`Conversation_Privacy`) are hooked whether or not the experiment is on, and they find archived chats even while the `wpai_archived` status is not registered. + **A turn replays a bounded window.** The whole transcript is kept for display, but a turn sends the model only the last 40 stored messages (see [`wpai_workspace_replay_messages`](#wpai_workspace_replay_messages)), opening on a message the person sent. Within that window, tool results from before the latest turn are replaced by their retrieval summary, so a long chat does not outgrow the provider's context limit. Cancellation markers and proposals still live in **user-scoped transients** sharing the plugin's `wpai_` prefix: @@ -458,16 +460,41 @@ curl -X POST "https://yoursite.com/wp-json/wp-abilities/v1/abilities/ai/search-c ## Auditing with the AI Request Log -Enable the **AI Request Logging** experiment and every workspace tool call and every write attempt appears at `Tools → AI Request Logs`. The shape is fixed so workspace rows join with rows any other ability consumer writes: +Every agent event is recorded in one place, `Agent_Audit`: each tool call, including a withheld one; each item an approval writes; and each event in a proposal's life. Enable the **AI Request Logging** experiment and they appear at `Tools → AI Request Logs`, where the **Type** filter's `agent` value shows only them and the **Status** filter's `denied` value shows what was refused. + +- `type` is always `agent`. Rows written before this type existed keep the `ability` type. +- `operation` is one of: + - for a tool call, the ability name, such as `ai/search-content` or `ai/propose-post-changes`; + - for an item an approval wrote, the kind's write operation: `ai/create-drafts`, `ai/create-template`, `ai/update-post-fields` or `ai/update-site-identity`; + - for a proposal event, `proposal_created`, `proposal_edited` (the person changed a value before approving), `proposal_declined`, `proposal_expired` (recorded once, when an expired proposal is next read), `proposal_failed` (a write that threw, or one interrupted for longer than the approval lock) or `proposal_refused` (an approval after the chat was deleted). +- `status` is `success`, `error`, `denied`, `skipped` (an item left as it was: unchanged, stale, locked or a duplicate), or `timeout` for an expired proposal. A **denial is recorded separately from a failure**: a refusal sets `denied` and records `denial_reason` in the context. +- `context` carries `surface` (always `ai-workspace`) and `conversation_id`. + - Tool rows add the `round` index and a `tool` object naming the ability, the function name and the call ID. + - Item rows add a `proposal` object with the proposal ID and item key, plus the post ID, post type, status and field, or the option. + - Proposal event rows add a `proposal` object with the proposal ID, its kind and its item keys, and for an edit the edited item keys. Failures and refusals add `error_code`. +- `user_id` is the person the tool ran as, or whose proposal it is. -- `type` is always `ability`. -- `operation` is the ability name — `ai/search-content`, `ai/read-content-bodies`, `ai/propose-drafts`, `ai/propose-template` — or `ai/create-drafts` or `ai/create-template` for a write attempt. -- `status` is `success`, `error`, `denied`, or `skipped` for an idempotent duplicate. A **denial is recorded separately from a failure**: a refusal from the Abilities API sets `denied` and records `denial_reason` in the context. -- `context` carries `surface` (always `ai-workspace`), `conversation_id`, the `round` index, and a `tool` object naming the ability, the function name and the call ID. Write rows add a `proposal` object with the proposal ID, item key, post type and status. -- `user_id` is the person the tool ran as. +No row carries post content or option values: a row names the field or option that changed, never what it changed to. Between them, those fields answer "what did the assistant do on this site, for whom, and was it allowed" without reading application code. Logging is best-effort: `log_ai_request()` returns `false` harmlessly when the logging experiment is off, so it is called unconditionally and costs nothing when nobody is watching. +### `wpai_agent_event` + +Fires once for every event above, before its row is written, whether or not AI Request Logging is on, so an audit plugin need not depend on that experiment. + +```php +add_action( + 'wpai_agent_event', + function ( string $operation, string $status, array $context, string $error ) { + // $operation, $status and $context are the row's fields; $error is its error message, or ''. + }, + 10, + 4 +); +``` + +It covers agent turns and approvals in the AI Workspace only. A direct MCP call to an ability does not fire it. + ## Extending the Experiment ### `wpai_workspace_tool_candidates` @@ -627,7 +654,8 @@ Filters whether a function-calling-capable model is available, for connectors th - As a non-administrator, confirm the Tools menu entry is absent and the routes return 403 - With a limited role that can reach the screen via a filtered capability, confirm search results never contain another author's private or draft posts 6. **Auditing:** - - Enable **AI Request Logging** and confirm each tool call and write attempt appears with `surface: ai-workspace`, its conversation ID and round + - Enable **AI Request Logging** and confirm each tool call and write attempt appears with type `agent`, `surface: ai-workspace`, its conversation ID and round + - Propose a change, edit it and approve it; decline a second one. Confirm `proposal_created`, `proposal_edited` and `proposal_declined` rows appear, and filter the log by type `agent` and status `denied` ### Automated tests diff --git a/includes/Experiments/AI_Workspace/Agent_Audit.php b/includes/Experiments/AI_Workspace/Agent_Audit.php new file mode 100644 index 000000000..6612b01f7 --- /dev/null +++ b/includes/Experiments/AI_Workspace/Agent_Audit.php @@ -0,0 +1,148 @@ +, + * duration_ms?: int, + * error_message?: string + * } $data The event, in the shape {@see log_ai_request()} takes, without a type. + */ + public static function record( array $data ): void { + $context = $data['context'] ?? array(); + $error = $data['error_message'] ?? ''; + + /** + * Fires for every agent event. + * + * Events are tool calls (operation: the ability name; status `success`, + * `error` or `denied`), proposal items written (operation: the kind's + * write operation, such as `ai/update-post-fields`), and proposal events + * (operation `proposal_created`, `proposal_edited`, `proposal_declined`, + * `proposal_expired`, `proposal_failed` or `proposal_refused`). It fires + * whether or not AI Request Logging is on. + * + * @since x.x.x + * + * @param string $operation The operation. + * @param string $status The status: `success`, `error`, `denied`, `skipped` or `timeout`. + * @param array $context The event context. It carries no post content or option values. + * @param string $error The error message, or an empty string. + */ + do_action( 'wpai_agent_event', $data['operation'], $data['status'], $context, $error ); + + log_ai_request( array_merge( $data, array( 'type' => self::LOG_TYPE ) ) ); + } + + /** + * Records an event in a proposal's life. + * + * @since x.x.x + * + * @param string $event The operation, such as `proposal_created`. + * @param string $status The status. + * @param array $proposal The stored proposal. + * @param array $extra Optional. More proposal fields, such as the edited item keys. Default empty array. + * @param string $error_code Optional. The error code, when there is one. Default empty string. + */ + public static function proposal( string $event, string $status, array $proposal, array $extra = array(), string $error_code = '' ): void { + $items = isset( $proposal['items'] ) && is_array( $proposal['items'] ) ? $proposal['items'] : array(); + $keys = array(); + + foreach ( $items as $item ) { + if ( ! is_array( $item ) || ! isset( $item['key'] ) || ! is_string( $item['key'] ) ) { + continue; + } + + $keys[] = $item['key']; + } + + $context = array( + 'surface' => Turn_Runner::LOG_SURFACE, + 'conversation_id' => isset( $proposal['conversation_id'] ) && is_string( $proposal['conversation_id'] ) ? $proposal['conversation_id'] : '', + 'proposal' => array_merge( + array( + 'id' => isset( $proposal['id'] ) && is_string( $proposal['id'] ) ? $proposal['id'] : '', + 'kind' => isset( $proposal['kind'] ) && is_string( $proposal['kind'] ) ? $proposal['kind'] : Draft_Writer::KIND, + 'items' => $keys, + ), + $extra + ), + ); + + if ( '' !== $error_code ) { + $context['error_code'] = $error_code; + } + + if ( Turn_Runner::LOG_STATUS_DENIED === $status ) { + $context['denial_reason'] = $error_code; + } + + self::record( + array( + 'operation' => $event, + 'status' => $status, + 'user_id' => isset( $proposal['user_id'] ) ? (int) $proposal['user_id'] : get_current_user_id(), + 'context' => $context, + ) + ); + } +} diff --git a/includes/Experiments/AI_Workspace/Conversation_Privacy.php b/includes/Experiments/AI_Workspace/Conversation_Privacy.php new file mode 100644 index 000000000..11d27c835 --- /dev/null +++ b/includes/Experiments/AI_Workspace/Conversation_Privacy.php @@ -0,0 +1,235 @@ +> $exporters The registered exporters. + * @return array> The exporters. + */ + public static function register_exporter( $exporters ): array { + $exporters = is_array( $exporters ) ? $exporters : array(); + + $exporters[ self::ID ] = array( + 'exporter_friendly_name' => __( 'AI chats', 'ai' ), + 'callback' => array( self::class, 'export' ), + ); + + return $exporters; + } + + /** + * Registers the AI chats eraser. + * + * @since x.x.x + * + * @param array> $erasers The registered erasers. + * @return array> The erasers. + */ + public static function register_eraser( $erasers ): array { + $erasers = is_array( $erasers ) ? $erasers : array(); + + $erasers[ self::ID ] = array( + 'eraser_friendly_name' => __( 'AI chats', 'ai' ), + 'callback' => array( self::class, 'erase' ), + ); + + return $erasers; + } + + /** + * Exports one page of a person's chats. + * + * Each chat is one item: its name, when it was created and last updated, + * whether it is archived, and each turn's message and reply. Tool results + * are left out; they are site content the person can export on its own. + * + * @since x.x.x + * + * @param string $email_address The person's email address. + * @param int $page The one-based page. + * @return array{data: list>, done: bool} The export page. + */ + public static function export( $email_address, $page = 1 ): array { + $user = get_user_by( 'email', (string) $email_address ); + + if ( false === $user ) { + return array( + 'data' => array(), + 'done' => true, + ); + } + + $page = max( 1, (int) $page ); + $ids = Conversation_Store::chat_ids( $user->ID, self::PER_PAGE, ( $page - 1 ) * self::PER_PAGE ); + $data = array(); + + // One query for the page's posts and one for their meta, rather than two per chat. + _prime_post_caches( $ids, false, true ); + + foreach ( $ids as $id ) { + $post = get_post( $id ); + + if ( ! $post instanceof WP_Post ) { + continue; + } + + $data[] = array( + 'group_id' => self::ID, + 'group_label' => __( 'AI chats', 'ai' ), + 'group_description' => __( 'Your saved chats with the AI Workspace assistant.', 'ai' ), + 'item_id' => 'wpai-chat-' . $post->post_name, + 'data' => self::fields( $post ), + ); + } + + return array( + 'data' => $data, + 'done' => count( $ids ) < self::PER_PAGE, + ); + } + + /** + * Erases every chat a person owns. + * + * @since x.x.x + * + * @param string $email_address The person's email address. + * @param int $page The one-based page. Every chat is erased on the first. + * @return array{items_removed: bool, items_retained: bool, messages: list, done: bool} The erase result. + */ + public static function erase( $email_address, $page = 1 ): array { + unset( $page ); + + $user = get_user_by( 'email', (string) $email_address ); + $had = false !== $user && array() !== Conversation_Store::chat_ids( $user->ID, 1 ); + + if ( $had ) { + Conversation_Store::delete_user_chats( $user->ID ); + } + + return array( + 'items_removed' => $had, + // A chat that could not be deleted is still there. + 'items_retained' => $had && array() !== Conversation_Store::chat_ids( $user->ID, 1 ), + 'messages' => array(), + 'done' => true, + ); + } + + /** + * Builds the exported fields of one chat. + * + * @since x.x.x + * + * @param \WP_Post $post The chat post. + * @return list The fields. + */ + private static function fields( WP_Post $post ): array { + $fields = array( + array( + 'name' => __( 'Name', 'ai' ), + 'value' => $post->post_title, + ), + array( + 'name' => __( 'Created', 'ai' ), + 'value' => $post->post_date_gmt, + ), + array( + 'name' => __( 'Updated', 'ai' ), + 'value' => $post->post_modified_gmt, + ), + array( + 'name' => __( 'Archived', 'ai' ), + 'value' => Conversation_Store::STATUS_ARCHIVED === $post->post_status ? __( 'Yes', 'ai' ) : __( 'No', 'ai' ), + ), + ); + + foreach ( Conversation_Store::entries_of( $post->ID ) as $index => $entry ) { + if ( ! is_array( $entry ) ) { + continue; + } + + $turn = $index + 1; + + // A follow-up after an approval has no message of the person's. + if ( isset( $entry['prompt'] ) && is_string( $entry['prompt'] ) && '' !== $entry['prompt'] ) { + $fields[] = array( + /* translators: %d: the turn's number in the chat. */ + 'name' => sprintf( __( 'Turn %d: your message', 'ai' ), $turn ), + 'value' => $entry['prompt'], + ); + } + + if ( ! isset( $entry['text'] ) || ! is_string( $entry['text'] ) || '' === $entry['text'] ) { + continue; + } + + $fields[] = array( + /* translators: %d: the turn's number in the chat. */ + 'name' => sprintf( __( 'Turn %d: reply', 'ai' ), $turn ), + 'value' => $entry['text'], + ); + } + + return $fields; + } +} diff --git a/includes/Experiments/AI_Workspace/Conversation_Store.php b/includes/Experiments/AI_Workspace/Conversation_Store.php index 7b71b5158..d199127ec 100644 --- a/includes/Experiments/AI_Workspace/Conversation_Store.php +++ b/includes/Experiments/AI_Workspace/Conversation_Store.php @@ -337,15 +337,15 @@ public function get( string $conversation_id, int $user_id ): ?array { return null; } - $conversation = $this->read_json( $post->ID, self::STATE_META ); + $conversation = self::read_json( $post->ID, self::STATE_META ); return array_merge( $conversation, $this->summary_of( $post ), array( 'user_id' => (int) $post->post_author, - 'messages' => array_values( $this->read_json( $post->ID, self::MESSAGES_META ) ), - 'entries' => array_values( $this->read_json( $post->ID, self::ENTRIES_META ) ), + 'messages' => array_values( self::read_json( $post->ID, self::MESSAGES_META ) ), + 'entries' => array_values( self::read_json( $post->ID, self::ENTRIES_META ) ), ) ); } @@ -805,40 +805,71 @@ private function list_of( array $conversation, string $key ): array { * @param string $key The meta key. * @return array The value, or an empty array when it is missing or unreadable. */ - private function read_json( int $post_id, string $key ): array { + private static function read_json( int $post_id, string $key ): array { $raw = get_post_meta( $post_id, $key, true ); $decoded = is_string( $raw ) && '' !== $raw ? json_decode( $raw, true ) : null; return is_array( $decoded ) ? $decoded : array(); } + /** + * Reads the turns stored for a chat post. + * + * @since x.x.x + * + * @param int $post_id The chat post ID. + * @return list The stored entry records, oldest first. + */ + public static function entries_of( int $post_id ): array { + return array_values( self::read_json( $post_id, self::ENTRIES_META ) ); + } + + /** + * Reads the post IDs of chats, every one or one user's, oldest first. + * + * The IDs are read straight from the posts table, not through WP_Query. The + * readers run while the experiment is off (a user is deleted, or their + * personal data exported or erased) and during uninstall, when the post type + * and its archived status are not registered, and WP_Query silently drops a + * status that is not registered: archived chats would be skipped. + * + * @since x.x.x + * + * @param int $user_id The owner whose chats to read, or 0 for every chat. + * @param int $limit The most IDs to read. + * @param int $offset Optional. How many IDs to skip. Default 0. + * @return list The post IDs. + */ + public static function chat_ids( int $user_id, int $limit, int $offset = 0 ): array { + global $wpdb; + + $ids = 0 < $user_id + // phpcs:ignore WordPress.DB.DirectDatabaseQuery -- Read live, regardless of which statuses are registered; see above. + ? $wpdb->get_col( $wpdb->prepare( "SELECT ID FROM {$wpdb->posts} WHERE post_type = %s AND post_author = %d ORDER BY ID ASC LIMIT %d OFFSET %d", self::POST_TYPE, $user_id, $limit, $offset ) ) + // phpcs:ignore WordPress.DB.DirectDatabaseQuery -- Read live, regardless of which statuses are registered; see above. + : $wpdb->get_col( $wpdb->prepare( "SELECT ID FROM {$wpdb->posts} WHERE post_type = %s ORDER BY ID ASC LIMIT %d OFFSET %d", self::POST_TYPE, $limit, $offset ) ); + + return array_values( array_map( 'intval', $ids ) ); + } + /** * Deletes chats, every one or one user's, in batches. * - * The IDs are read straight from the posts table, not through WP_Query. These - * deletions run while the experiment is off (a user is deleted) and during - * uninstall, when the post type and its archived status are not registered, - * and WP_Query silently drops a status that is not registered: archived chats - * would be skipped, then reassigned with the rest of a deleted user's content. + * The IDs come from {@see self::chat_ids()}, so archived chats are deleted + * too while the post type is not registered, rather than reassigned with the + * rest of a deleted user's content. * * @since x.x.x * * @param int $user_id The owner whose chats to delete, or 0 for every chat. */ private static function delete_chats( int $user_id ): void { - global $wpdb; - do { - $ids = 0 < $user_id - // phpcs:ignore WordPress.DB.DirectDatabaseQuery -- Read live, regardless of which statuses are registered; see above. - ? $wpdb->get_col( $wpdb->prepare( "SELECT ID FROM {$wpdb->posts} WHERE post_type = %s AND post_author = %d LIMIT 100", self::POST_TYPE, $user_id ) ) - // phpcs:ignore WordPress.DB.DirectDatabaseQuery -- Read live, regardless of which statuses are registered; see above. - : $wpdb->get_col( $wpdb->prepare( "SELECT ID FROM {$wpdb->posts} WHERE post_type = %s LIMIT 100", self::POST_TYPE ) ); - + $ids = self::chat_ids( $user_id, 100 ); $deleted = 0; foreach ( $ids as $id ) { - if ( ! wp_delete_post( (int) $id, true ) ) { + if ( ! wp_delete_post( $id, true ) ) { continue; } diff --git a/includes/Experiments/AI_Workspace/Proposal_Store.php b/includes/Experiments/AI_Workspace/Proposal_Store.php index 5c1d75d3a..c16c16bcb 100644 --- a/includes/Experiments/AI_Workspace/Proposal_Store.php +++ b/includes/Experiments/AI_Workspace/Proposal_Store.php @@ -291,6 +291,8 @@ public function create( int $user_id, string $conversation_id, array $items, str ); } + Agent_Audit::proposal( 'proposal_created', 'success', $proposal ); + return $proposal; } @@ -313,7 +315,7 @@ public function get( string $proposal_id, int $user_id ): ?array { } if ( $this->is_expired( $stored ) ) { - $this->delete( $proposal_id, $user_id ); + $this->expire( $stored ); return null; } @@ -360,11 +362,32 @@ public function find( string $proposal_id, int $user_id, bool $fresh = false ): if ( isset( $stored['status'] ) && 'executing' === $stored['status'] && $since + self::LOCK_TTL <= time() ) { $stored['status'] = 'failed'; $this->save( $stored ); + + // Saved as `failed`, so this is recorded once, by the first read past the window. + Agent_Audit::proposal( 'proposal_failed', 'error', $stored, array(), 'workspace_proposal_interrupted' ); } return $stored; } + /** + * Deletes an expired proposal and records that it expired. + * + * It is deleted here, so the expiry is recorded once: the next read finds nothing. + * + * @since x.x.x + * + * @param array $proposal The stored proposal. + */ + public function expire( array $proposal ): void { + $proposal_id = isset( $proposal['id'] ) && is_string( $proposal['id'] ) ? $proposal['id'] : ''; + $user_id = isset( $proposal['user_id'] ) ? (int) $proposal['user_id'] : 0; + + $this->delete( $proposal_id, $user_id ); + + Agent_Audit::proposal( 'proposal_expired', 'timeout', $proposal, array(), 'workspace_proposal_expired' ); + } + /** * Reports whether a stored proposal has passed its expiry. * diff --git a/includes/Experiments/AI_Workspace/Proposal_Writes.php b/includes/Experiments/AI_Workspace/Proposal_Writes.php index f6b7b564c..c220efc73 100644 --- a/includes/Experiments/AI_Workspace/Proposal_Writes.php +++ b/includes/Experiments/AI_Workspace/Proposal_Writes.php @@ -9,8 +9,6 @@ namespace WordPress\AI\Experiments\AI_Workspace; -use function WordPress\AI\log_ai_request; - // Exit if accessed directly. defined( 'ABSPATH' ) || exit; @@ -177,7 +175,6 @@ protected function log( } $log_data = array( - 'type' => 'ability', 'operation' => $operation, 'status' => $status, 'user_id' => get_current_user_id(), @@ -189,7 +186,7 @@ protected function log( $log_data['error_message'] = $error_message; } - log_ai_request( $log_data ); + Agent_Audit::record( $log_data ); } /** diff --git a/includes/Experiments/AI_Workspace/REST/Proposal_Controller.php b/includes/Experiments/AI_Workspace/REST/Proposal_Controller.php index 436f13157..ec535b6ba 100644 --- a/includes/Experiments/AI_Workspace/REST/Proposal_Controller.php +++ b/includes/Experiments/AI_Workspace/REST/Proposal_Controller.php @@ -14,10 +14,12 @@ use WP_REST_Request; use WP_REST_Response; use WP_REST_Server; +use WordPress\AI\Experiments\AI_Workspace\Agent_Audit; use WordPress\AI\Experiments\AI_Workspace\Conversation_Store; use WordPress\AI\Experiments\AI_Workspace\Draft_Writer; use WordPress\AI\Experiments\AI_Workspace\Proposal_Kind; use WordPress\AI\Experiments\AI_Workspace\Proposal_Store; +use WordPress\AI\Experiments\AI_Workspace\Turn_Runner; use WordPress\AiClient\Messages\DTO\MessagePart; use WordPress\AiClient\Messages\DTO\UserMessage; @@ -262,6 +264,8 @@ public function decline_proposal( WP_REST_Request $request ) { $this->store->delete( $proposal_id, $user_id ); $this->store->release( $proposal_id, $user_id ); + Agent_Audit::proposal( 'proposal_declined', 'success', $proposal ); + $conversation_id = isset( $proposal['conversation_id'] ) && is_string( $proposal['conversation_id'] ) ? $proposal['conversation_id'] : ''; @@ -337,6 +341,8 @@ public function execute_proposal( WP_REST_Request $request ) { if ( null === $this->conversations->get( $conversation_id, $user_id ) ) { $this->store->delete( $proposal_id, $user_id ); + Agent_Audit::proposal( 'proposal_refused', Turn_Runner::LOG_STATUS_DENIED, $proposal, array(), 'workspace_proposal_chat_deleted' ); + return new WP_Error( 'workspace_proposal_chat_deleted', __( 'The chat this proposal belongs to was deleted, so it can no longer be approved.', 'ai' ), @@ -419,6 +425,10 @@ public function execute_proposal( WP_REST_Request $request ) { $proposal = $edited; + if ( is_array( $edits ) && array() !== $edits ) { + Agent_Audit::proposal( 'proposal_edited', 'success', $proposal, array( 'edited' => array_map( 'strval', array_keys( $edits ) ) ) ); + } + $proposal['status'] = 'executing'; $proposal['executing_since'] = time(); $proposal['results'] = array(); @@ -488,6 +498,8 @@ private function write_failed( array $proposal, Proposal_Kind $kind ): WP_Error $this->store->save( $proposal ); $this->store->release( $proposal_id, $user_id ); + Agent_Audit::proposal( 'proposal_failed', 'error', $proposal, array(), 'workspace_proposal_write_failed' ); + $results = $this->stored_results( $proposal ); $counts = array_intersect_key( $results, array_flip( Proposal_Kind::OUTCOMES ) ); @@ -610,7 +622,7 @@ private function load( string $proposal_id ) { * still lapses on its own, {@see Proposal_Store::GRACE} after its last save. */ if ( $this->is_pending( $proposal ) && $this->store->is_expired( $proposal ) ) { - $this->store->delete( $proposal_id, $user_id ); + $this->store->expire( $proposal ); return new WP_Error( 'workspace_proposal_expired', diff --git a/includes/Experiments/AI_Workspace/Turn_Runner.php b/includes/Experiments/AI_Workspace/Turn_Runner.php index b98cee6f0..ba40b4cb7 100644 --- a/includes/Experiments/AI_Workspace/Turn_Runner.php +++ b/includes/Experiments/AI_Workspace/Turn_Runner.php @@ -18,8 +18,6 @@ use WordPress\AiClient\Tools\DTO\FunctionCall; use WordPress\AiClient\Tools\DTO\FunctionResponse; -use function WordPress\AI\log_ai_request; - // Exit if accessed directly. defined( 'ABSPATH' ) || exit; @@ -873,9 +871,10 @@ private function retrieval_query( FunctionCall $call, string $key ): string { /** * Writes the single log row for one tool invocation. * - * The shape is fixed so workspace rows join with the rows any other ability - * consumer writes (KTD10): `type` is always `ability`, `operation` is always - * the ability name, and the surface, conversation and round live in `context`. + * The shape is fixed so a conversation's tool calls and writes join in one + * log view (KTD10): the row is an {@see Agent_Audit} `agent` row, `operation` + * is always the ability name, and the surface, conversation and round live + * in `context`. * * @since x.x.x * @@ -918,7 +917,6 @@ private function log_invocation( } $log_data = array( - 'type' => 'ability', 'operation' => '' === $ability_name ? 'unknown' : $ability_name, 'status' => $status, 'user_id' => $user_id, @@ -931,7 +929,7 @@ private function log_invocation( $log_data['error_message'] = $error['message']; } - log_ai_request( $log_data ); + Agent_Audit::record( $log_data ); } /** diff --git a/includes/Logging/AI_Request_Log_Manager.php b/includes/Logging/AI_Request_Log_Manager.php index feeb32750..2c902ae51 100644 --- a/includes/Logging/AI_Request_Log_Manager.php +++ b/includes/Logging/AI_Request_Log_Manager.php @@ -204,14 +204,17 @@ public function log( array $data ) { * * `ai_client` covers generations made through the AI Client SDK. `mcp_tool` * and `ability` are for consumers that surface abilities themselves, such as - * an MCP server or a direct ability invocation. + * an MCP server or a direct ability invocation. `agent` records the AI + * Workspace agent's tool calls and proposal events, so its audit trail can + * be filtered on its own. * * @since 1.3.0 + * @since x.x.x Added the `agent` type. * * @return list Supported log types. */ public static function get_types(): array { - return array( 'ai_client', 'mcp_tool', 'ability' ); + return array( 'ai_client', 'mcp_tool', 'ability', 'agent' ); } /** diff --git a/includes/Logging/REST/AI_Request_Log_Controller.php b/includes/Logging/REST/AI_Request_Log_Controller.php index e82f8385a..e133e9325 100644 --- a/includes/Logging/REST/AI_Request_Log_Controller.php +++ b/includes/Logging/REST/AI_Request_Log_Controller.php @@ -269,7 +269,7 @@ public function get_collection_params(): array { 'status' => array( 'description' => __( 'Filter by status.', 'ai' ), 'type' => 'string', - 'enum' => array( '', 'success', 'error', 'timeout' ), + 'enum' => array( '', 'success', 'error', 'timeout', 'denied' ), 'default' => '', ), 'provider' => array( diff --git a/includes/Main.php b/includes/Main.php index c9f79fb51..499f785c4 100644 --- a/includes/Main.php +++ b/includes/Main.php @@ -22,6 +22,7 @@ use WordPress\AI\CLI\Embeddings_Command; use WordPress\AI\Connectors\Featured_Connectors; use WordPress\AI\Embeddings\Sync\Embedding_Sync; +use WordPress\AI\Experiments\AI_Workspace\Conversation_Privacy; use WordPress\AI\Experiments\AI_Workspace\Conversation_Store; use WordPress\AI\Experiments\Experiments; use WordPress\AI\Features\Loader; @@ -100,6 +101,9 @@ public function load(): void { // Keep AI Workspace chats private to their owner regardless of experiment state. Conversation_Store::register_lifecycle_hooks(); + // Export and erase AI Workspace chats with personal data regardless of experiment state. + Conversation_Privacy::register(); + // List featured Connector plugins on the Connectors screen. ( new Featured_Connectors() )->init(); diff --git a/src/admin/ai-request-logs/components/LogDetailModal.tsx b/src/admin/ai-request-logs/components/LogDetailModal.tsx index d7031e6e7..e64d17288 100644 --- a/src/admin/ai-request-logs/components/LogDetailModal.tsx +++ b/src/admin/ai-request-logs/components/LogDetailModal.tsx @@ -130,6 +130,8 @@ const LogDetailModal: React.FC< LogDetailModalProps > = ( { return '\u2717'; // X mark case 'timeout': return '\u23F1'; // Stopwatch + case 'denied': + return '\u2298'; // Circled division slash default: return ''; } diff --git a/src/admin/ai-request-logs/components/LogsTable.tsx b/src/admin/ai-request-logs/components/LogsTable.tsx index 137c62841..fd2bf091a 100644 --- a/src/admin/ai-request-logs/components/LogsTable.tsx +++ b/src/admin/ai-request-logs/components/LogsTable.tsx @@ -95,6 +95,8 @@ const getStatusClass = ( status: string ): string => { return 'ai-request-logs__status--error'; case 'timeout': return 'ai-request-logs__status--timeout'; + case 'denied': + return 'ai-request-logs__status--denied'; default: return ''; } @@ -108,7 +110,13 @@ const formatSelectLabel = ( value: string ): string => const getRequestKind = ( entry: LogEntry ): string => { const raw = entry.context?.request_kind; - return typeof raw === 'string' ? raw : 'text'; + + if ( typeof raw === 'string' ) { + return raw; + } + + // Agent rows record tool calls and proposal events, not generations. + return 'agent' === entry.type ? 'agent' : 'text'; }; const getSourceLabel = ( entry: LogEntry ): string | null => { diff --git a/src/admin/ai-request-logs/index.scss b/src/admin/ai-request-logs/index.scss index d787aabe1..18e150cad 100644 --- a/src/admin/ai-request-logs/index.scss +++ b/src/admin/ai-request-logs/index.scss @@ -325,6 +325,11 @@ color: #856404; } +.ai-request-logs__status--denied { + background: #e2e3e5; + color: #383d41; +} + .ai-request-logs__kind { display: inline-flex; align-items: center; diff --git a/src/admin/ai-request-logs/types.ts b/src/admin/ai-request-logs/types.ts index f5ab228e5..fd16021e9 100644 --- a/src/admin/ai-request-logs/types.ts +++ b/src/admin/ai-request-logs/types.ts @@ -30,7 +30,7 @@ export interface LogEntryContext extends Record< string, unknown > { export interface LogEntry { id: string; timestamp: string; - type: 'ai_client' | 'mcp_tool' | 'ability'; + type: 'ai_client' | 'mcp_tool' | 'ability' | 'agent'; operation: string; provider: string | null; model: string | null; @@ -39,7 +39,7 @@ export interface LogEntry { tokens_output: number | null; tokens_total: number | null; tokens_per_second: number | null; - status: 'success' | 'error' | 'timeout'; + status: 'success' | 'error' | 'timeout' | 'denied'; error_message: string | null; user_id: number | null; context: LogEntryContext | null; diff --git a/src/experiments/ai-workspace/assets/wapuu/svg.d.ts b/src/experiments/ai-workspace/assets/wapuu/svg.d.ts new file mode 100644 index 000000000..85fe5f32d --- /dev/null +++ b/src/experiments/ai-workspace/assets/wapuu/svg.d.ts @@ -0,0 +1,11 @@ +/** + * Type declaration for SVG imports. + * + * `@wordpress/scripts` runs an SVG imported from script through `url-loader`, + * so its default export is the file as a data URI. + */ + +declare module '*.svg' { + const src: string; + export default src; +} diff --git a/src/experiments/ai-workspace/assets/wapuu/wapuu-celebrating.svg b/src/experiments/ai-workspace/assets/wapuu/wapuu-celebrating.svg new file mode 100644 index 000000000..2aae87478 --- /dev/null +++ b/src/experiments/ai-workspace/assets/wapuu/wapuu-celebrating.svg @@ -0,0 +1 @@ +Wapuu done diff --git a/src/experiments/ai-workspace/assets/wapuu/wapuu-oops.svg b/src/experiments/ai-workspace/assets/wapuu/wapuu-oops.svg new file mode 100644 index 000000000..baf94a1b3 --- /dev/null +++ b/src/experiments/ai-workspace/assets/wapuu/wapuu-oops.svg @@ -0,0 +1 @@ +Wapuu oops diff --git a/src/experiments/ai-workspace/assets/wapuu/wapuu-working.svg b/src/experiments/ai-workspace/assets/wapuu/wapuu-working.svg new file mode 100644 index 000000000..36d5b8fb8 --- /dev/null +++ b/src/experiments/ai-workspace/assets/wapuu/wapuu-working.svg @@ -0,0 +1 @@ +Wapuu working diff --git a/src/experiments/ai-workspace/components/Transcript.tsx b/src/experiments/ai-workspace/components/Transcript.tsx index 84cc922b8..9822ceff0 100644 --- a/src/experiments/ai-workspace/components/Transcript.tsx +++ b/src/experiments/ai-workspace/components/Transcript.tsx @@ -5,7 +5,7 @@ /** * WordPress dependencies */ -import { Button, Notice, Spinner } from '@wordpress/components'; +import { Button, Notice } from '@wordpress/components'; import { __, _n, sprintf } from '@wordpress/i18n'; /** @@ -16,6 +16,7 @@ import ConfirmProposal from './ConfirmProposal'; import EmptyState from './EmptyState'; import Markdown from './Markdown'; import ResultsTable from './ResultsTable'; +import WapuuStatus, { WapuuDone } from './WapuuStatus'; import { toPostResultSet } from '../utils/post-results'; import { PROPOSAL_ABILITIES, toProposalId } from '../utils/proposal'; import { toQuestion } from '../utils/question'; @@ -346,12 +347,18 @@ function TurnState( { case 'error': // A follow-up is never retried: the server owes exactly one. if ( entry.followUp ) { - return

{ entry.detail }

; + return ( +

+ + { entry.detail } +

+ ); } return ( -

+

+ { '' === entry.detail ? __( 'The assistant could not complete this turn.', @@ -468,11 +475,13 @@ export default function Transcript( { { 'streaming' === entry.status && (

- + { __( 'Responding…', 'ai' ) }

) } + + { '' !== conversationId && proposalIds( entry.toolCalls ).map( ( id ) => ( = { + busy: busySprite, + done: doneSprite, + error: errorSprite, +}; + +/** + * The prefix of the data URI the bundler inlines each sprite as. + */ +const DATA_URI = 'data:image/svg+xml;base64,'; + +/** + * Turns a sprite into its still frame. + * + * Each sprite keeps its motion inside its own reduced-motion query, but an + * `` evaluates that query in a separate document, which does not reliably + * follow the page's setting. Switching every animation off in the markup is the + * still pose the query would have shown. A source that is not an inlined SVG is + * returned as it is, so this can only ever fall back to the moving sprite. + * + * @param src The sprite's source. + * @return The still frame's source. + */ +function toStill( src: string ): string { + if ( ! src.startsWith( DATA_URI ) ) { + return src; + } + + const markup = window + .atob( src.slice( DATA_URI.length ) ) + .replace( + '', + '' + ); + + return 'data:image/svg+xml,' + encodeURIComponent( markup ); +} + +/** + * Each sprite's still frame, built once per page the first time it is needed. + */ +const STILLS: Partial< Record< WapuuState, string > > = {}; + +/** + * Returns a sprite's still frame. + * + * @param state The state whose sprite to return. + * @return The still frame's source. + */ +function stillOf( state: WapuuState ): string { + STILLS[ state ] ??= toStill( SPRITES[ state ] ); + + return STILLS[ state ]; +} + +/** + * Renders one Wapuu sprite. + * + * The sprite is decorative: the text beside it is what says how the turn is + * going, so it has an empty alternative and is never announced on its own. + * + * @param props Component props. + * @param props.state The state to show. + * @return The rendered sprite. + */ +export default function WapuuStatus( { state }: { state: WapuuState } ) { + const isReduced = useReducedMotion(); + const src = isReduced ? stillOf( state ) : SPRITES[ state ]; + + return ( + + ); +} + +/** + * Shows the done sprite for a moment after a turn finishes. + * + * Only a turn seen running gets one, so a chat reopened from history does not + * celebrate every turn it already had. A turn that ends any other way, stopped, + * capped or failed, shows nothing here. + * + * @param props Component props. + * @param props.status The turn's status. + * @return The rendered sprite, or null. + */ +export function WapuuDone( { status }: { status: EntryStatus } ) { + const previous = useRef( status ); + const [ isShown, setIsShown ] = useState( false ); + const isDone = 'complete' === status || 'awaiting_user' === status; + + useEffect( () => { + if ( 'streaming' === previous.current && isDone ) { + setIsShown( true ); + } + + previous.current = status; + }, [ status, isDone ] ); + + // The timer has its own effect so that re-running the one above can never + // leave the sprite up without something to take it down. + useEffect( () => { + if ( ! isShown ) { + return undefined; + } + + const timer = window.setTimeout( + () => setIsShown( false ), + DONE_DURATION + ); + + return () => window.clearTimeout( timer ); + }, [ isShown ] ); + + if ( ! isShown || ! isDone ) { + return null; + } + + return ; +} diff --git a/src/experiments/ai-workspace/index.scss b/src/experiments/ai-workspace/index.scss index 49b9918bf..78b743e3e 100644 --- a/src/experiments/ai-workspace/index.scss +++ b/src/experiments/ai-workspace/index.scss @@ -438,6 +438,20 @@ body.ai-workspace-frame { color: var(--wpds-color-foreground-content-neutral-weak); } +// The Wapuu sits beside the text that says how the turn is going. +.ai-workspace__wapuu { + flex: none; + width: 48px; + height: 48px; +} + +.ai-workspace__state--error, +.ai-workspace__failure { + display: flex; + align-items: center; + gap: var(--wpds-dimension-gap-sm); +} + .ai-workspace__markdown { width: 100%; min-width: 0; diff --git a/tests/Integration/Includes/Experiments/AI_Workspace/Agent_AuditTest.php b/tests/Integration/Includes/Experiments/AI_Workspace/Agent_AuditTest.php new file mode 100644 index 000000000..7f6b01ed8 --- /dev/null +++ b/tests/Integration/Includes/Experiments/AI_Workspace/Agent_AuditTest.php @@ -0,0 +1,278 @@ +original_shared_manager = Logging_Integration::get_log_manager(); + } + + /** + * Tear down test case. + * + * @since x.x.x + */ + public function tearDown(): void { + $this->set_shared_manager( $this->original_shared_manager ); + + delete_option( 'wpai_request_logs_schema_version' ); + wp_clear_scheduled_hook( 'wpai_request_logs_cleanup' ); + + parent::tearDown(); + } + + /** + * Replaces the manager the logging integration shares. + * + * @since x.x.x + * + * @param \WordPress\AI\Logging\AI_Request_Log_Manager|null $manager Manager, or null for a disabled experiment. + */ + private function set_shared_manager( ?AI_Request_Log_Manager $manager ): void { + $property = new ReflectionProperty( Logging_Integration::class, 'log_manager' ); + $property->setAccessible( true ); + $property->setValue( null, $manager ); + } + + /** + * Boots the request log table and shares a fresh manager with the integration. + * + * @since x.x.x + */ + private function boot_logging(): void { + delete_option( 'wpai_request_logs_schema_version' ); + + $manager = new AI_Request_Log_Manager(); + $manager->init(); + + global $wpdb; + $table = $wpdb->prefix . AI_Request_Log_Schema::TABLE_NAME; + $wpdb->query( "DELETE FROM {$table} WHERE 1=1" ); // phpcs:ignore + + $this->set_shared_manager( $manager ); + } + + /** + * Returns the agent log rows written so far. + * + * @since x.x.x + * + * @return list> The rows, oldest first. + */ + private function rows(): array { + global $wpdb; + $table = $wpdb->prefix . AI_Request_Log_Schema::TABLE_NAME; + + $rows = $wpdb->get_results( $wpdb->prepare( "SELECT * FROM {$table} WHERE type = %s ORDER BY id ASC", Agent_Audit::LOG_TYPE ), ARRAY_A ); // phpcs:ignore + + return is_array( $rows ) ? $rows : array(); + } + + /** + * Records the hook calls made from here on. + * + * @since x.x.x + * + * @return \ArrayObject> The arguments of each call, in order. + */ + private function capture(): \ArrayObject { + $calls = new \ArrayObject(); + + add_action( + Agent_Audit::ACTION, + static function ( ...$args ) use ( $calls ): void { + $calls[] = $args; + }, + 10, + 4 + ); + + return $calls; + } + + /** + * The agent type is one the log accepts and the log screen can filter on. + * + * @since x.x.x + */ + public function test_agent_is_a_log_type(): void { + $this->assertContains( Agent_Audit::LOG_TYPE, AI_Request_Log_Manager::get_types() ); + } + + /** + * An event writes one agent row and fires the hook once with its operation, status, context and error. + * + * @since x.x.x + */ + public function test_an_event_writes_one_row_and_fires_the_hook(): void { + $user_id = self::factory()->user->create( array( 'role' => 'administrator' ) ); + $this->boot_logging(); + $calls = $this->capture(); + + $context = array( + 'surface' => 'ai-workspace', + 'conversation_id' => 'c1', + ); + + Agent_Audit::record( + array( + 'operation' => 'ai/search-content', + 'status' => 'error', + 'user_id' => $user_id, + 'context' => $context, + 'duration_ms' => 12, + 'error_message' => 'It broke.', + ) + ); + + $this->assertSame( array( array( 'ai/search-content', 'error', $context, 'It broke.' ) ), $calls->getArrayCopy() ); + + $rows = $this->rows(); + + $this->assertCount( 1, $rows ); + $this->assertSame( 'ai/search-content', $rows[0]['operation'] ); + $this->assertSame( 'error', $rows[0]['status'] ); + $this->assertSame( (string) $user_id, (string) $rows[0]['user_id'] ); + $this->assertSame( '12', (string) $rows[0]['duration_ms'] ); + $this->assertSame( 'It broke.', $rows[0]['error_message'] ); + $this->assertSame( $context, json_decode( (string) $rows[0]['context'], true ) ); + } + + /** + * With the AI Request Logging experiment off, the hook still fires and nothing is written. + * + * @since x.x.x + */ + public function test_the_hook_fires_with_logging_off(): void { + $this->boot_logging(); + $this->set_shared_manager( null ); + $calls = $this->capture(); + + Agent_Audit::record( + array( + 'operation' => 'proposal_created', + 'status' => 'success', + 'context' => array(), + ) + ); + + $this->assertCount( 1, $calls ); + $this->assertSame( '', $calls[0][3], 'An event without an error passes an empty string.' ); + $this->assertSame( array(), $this->rows() ); + } + + /** + * A proposal event names the proposal and its items, never their values. + * + * @since x.x.x + */ + public function test_a_proposal_event_names_the_proposal_and_its_items_only(): void { + $user_id = self::factory()->user->create( array( 'role' => 'administrator' ) ); + $calls = $this->capture(); + + Agent_Audit::proposal( + 'proposal_created', + 'success', + array( + 'id' => 'p1', + 'user_id' => $user_id, + 'conversation_id' => 'c1', + 'kind' => 'post-field', + 'items' => array( + array( + 'key' => 'k1', + 'value' => 'A secret body', + ), + array( + 'key' => 'k2', + 'value' => 'Another secret', + ), + ), + ) + ); + + $this->assertCount( 1, $calls ); + + [ $operation, $status, $context ] = $calls[0]; + + $this->assertSame( 'proposal_created', $operation ); + $this->assertSame( 'success', $status ); + $this->assertSame( + array( + 'surface' => 'ai-workspace', + 'conversation_id' => 'c1', + 'proposal' => array( + 'id' => 'p1', + 'kind' => 'post-field', + 'items' => array( 'k1', 'k2' ), + ), + ), + $context + ); + $this->assertStringNotContainsString( 'secret', (string) wp_json_encode( $calls[0] ) ); + } + + /** + * A proposal stored without a kind is a set of drafts. + * + * @since x.x.x + */ + public function test_a_proposal_without_a_kind_is_recorded_as_drafts(): void { + $calls = $this->capture(); + + Agent_Audit::proposal( + 'proposal_declined', + 'success', + array( + 'id' => 'p1', + 'items' => array(), + ) + ); + + $this->assertSame( Draft_Writer::KIND, $calls[0][2]['proposal']['kind'] ); + $this->assertSame( '', $calls[0][2]['conversation_id'] ); + } +} diff --git a/tests/Integration/Includes/Experiments/AI_Workspace/Conversation_PrivacyTest.php b/tests/Integration/Includes/Experiments/AI_Workspace/Conversation_PrivacyTest.php new file mode 100644 index 000000000..c497599f6 --- /dev/null +++ b/tests/Integration/Includes/Experiments/AI_Workspace/Conversation_PrivacyTest.php @@ -0,0 +1,420 @@ + + */ + private static $user_ids = array(); + + /** + * Manager the logging integration held before the test replaced it. + * + * @since x.x.x + * + * @var \WordPress\AI\Logging\AI_Request_Log_Manager|null + */ + private $original_shared_manager = null; + + /** + * Creates the shared users. + * + * @since x.x.x + * + * @param \WP_UnitTest_Factory $factory The unit test factory. + */ + public static function wpSetUpBeforeClass( $factory ): void { + self::$user_ids = array( + 'owner' => $factory->user->create( + array( + 'role' => 'administrator', + 'user_email' => 'owner@example.org', + ) + ), + 'peer' => $factory->user->create( + array( + 'role' => 'administrator', + 'user_email' => 'peer@example.org', + ) + ), + ); + } + + /** + * Set up test case. + * + * @since x.x.x + */ + public function setUp(): void { + parent::setUp(); + + Conversation_Store::register_post_type(); + + // Storing a proposal records an agent event; nothing here reads the log, so none is written. + $this->original_shared_manager = Logging_Integration::get_log_manager(); + $this->set_shared_manager( null ); + } + + /** + * Tear down test case. + * + * @since x.x.x + */ + public function tearDown(): void { + $this->set_shared_manager( $this->original_shared_manager ); + wp_set_current_user( 0 ); + + parent::tearDown(); + } + + /** + * Replaces the manager the logging integration shares. + * + * @since x.x.x + * + * @param \WordPress\AI\Logging\AI_Request_Log_Manager|null $manager Manager, or null for a disabled experiment. + */ + private function set_shared_manager( ?\WordPress\AI\Logging\AI_Request_Log_Manager $manager ): void { + $property = new ReflectionProperty( Logging_Integration::class, 'log_manager' ); + $property->setAccessible( true ); + $property->setValue( null, $manager ); + } + + /** + * Saves a chat for a user and returns its conversation. + * + * @since x.x.x + * + * @param int $user_id The owner. + * @param list>|null $entries Optional. The turns. Default one turn. + * @param bool $archive Optional. Whether to archive it. Default false. + * @return array The conversation. + */ + private function save_chat( int $user_id, ?array $entries = null, bool $archive = false ): array { + $store = new Conversation_Store(); + $conversation = $store->create( $user_id, 'site' ); + + $conversation['entries'] = $entries ?? array( + array( + 'prompt' => 'Hello there', + 'text' => 'Hi.', + 'tool_calls' => array(), + ), + ); + + $this->assertTrue( $store->save( $conversation ) ); + + if ( $archive ) { + $this->assertTrue( $store->set_archived( $conversation['id'], $user_id, true ) ); + } + + return $conversation; + } + + /** + * Returns the registered callback of the AI chats exporter or eraser. + * + * @since x.x.x + * + * @param string $filter `wp_privacy_personal_data_exporters` or `wp_privacy_personal_data_erasers`. + * @return callable The callback. + */ + private function privacy_callback( string $filter ): callable { + $registered = apply_filters( $filter, array() ); // phpcs:ignore WordPress.NamingConventions.PrefixAllGlobals.DynamicHooknameFound -- Core's privacy filters. + $key = 'wp_privacy_personal_data_exporters' === $filter ? 'exporter_friendly_name' : 'eraser_friendly_name'; + + $this->assertArrayHasKey( Conversation_Privacy::ID, $registered ); + $this->assertSame( 'AI chats', $registered[ Conversation_Privacy::ID ][ $key ] ); + + return $registered[ Conversation_Privacy::ID ]['callback']; + } + + /** + * Unregisters the chat post type and archived status, as when the experiment is off. + * + * The next test's set-up registers them again. + * + * @since x.x.x + */ + private function unregister_chat_types(): void { + unregister_post_type( Conversation_Store::POST_TYPE ); + unset( $GLOBALS['wp_post_statuses'][ Conversation_Store::STATUS_ARCHIVED ] ); + } + + /** + * Counts a user's chats in any status, straight from the posts table. + * + * @since x.x.x + * + * @param int $user_id The owner. + * @return int The count. + */ + private function chat_count( int $user_id ): int { + global $wpdb; + + return (int) $wpdb->get_var( // phpcs:ignore WordPress.DB.DirectDatabaseQuery.DirectQuery, WordPress.DB.DirectDatabaseQuery.NoCaching -- Counting fixtures in any status. + $wpdb->prepare( "SELECT COUNT(*) FROM {$wpdb->posts} WHERE post_type = %s AND post_author = %d", Conversation_Store::POST_TYPE, $user_id ) + ); + } + + /** + * Reads one export item's fields as a name to value map. + * + * @since x.x.x + * + * @param array $item The export item. + * @return array The fields. + */ + private function fields( array $item ): array { + return array_column( $item['data'], 'value', 'name' ); + } + + /** + * The exporter and eraser are registered by the plugin itself, not by the experiment. + * + * @since x.x.x + */ + public function test_the_exporter_and_eraser_are_registered_without_the_experiment(): void { + $this->assertNotFalse( has_filter( 'wp_privacy_personal_data_exporters', array( Conversation_Privacy::class, 'register_exporter' ) ) ); + $this->assertNotFalse( has_filter( 'wp_privacy_personal_data_erasers', array( Conversation_Privacy::class, 'register_eraser' ) ) ); + } + + /** + * Exporting returns every chat, archived ones included, with each turn's prompt and reply. + * + * @since x.x.x + */ + public function test_export_returns_every_chat_with_its_prompts_and_replies(): void { + $owner = self::$user_ids['owner']; + $active = $this->save_chat( + $owner, + array( + array( + 'prompt' => 'Find posts about remote work', + 'text' => 'I found three.', + 'tool_calls' => array( + array( + 'ability' => 'ai/search-content', + 'result' => array( 'secret' => 'a tool result' ), + ), + ), + ), + array( + 'prompt' => '', + 'text' => 'The draft was created.', + 'follow_up' => true, + ), + ) + ); + $archived = $this->save_chat( $owner, null, true ); + $this->save_chat( self::$user_ids['peer'] ); + + $export = $this->privacy_callback( 'wp_privacy_personal_data_exporters' )( 'owner@example.org', 1 ); + + $this->assertTrue( $export['done'] ); + $this->assertCount( 2, $export['data'] ); + + $by_id = array(); + foreach ( $export['data'] as $item ) { + $this->assertSame( Conversation_Privacy::ID, $item['group_id'] ); + $this->assertSame( 'AI chats', $item['group_label'] ); + $by_id[ $item['item_id'] ] = $this->fields( $item ); + } + + $first = $by_id[ 'wpai-chat-' . $active['id'] ]; + + $this->assertSame( 'Find posts about remote work', $first['Name'] ); + $this->assertSame( 'No', $first['Archived'] ); + $this->assertSame( 'Find posts about remote work', $first['Turn 1: your message'] ); + $this->assertSame( 'I found three.', $first['Turn 1: reply'] ); + $this->assertArrayNotHasKey( 'Turn 2: your message', $first, 'A follow-up has no message of the person\'s.' ); + $this->assertSame( 'The draft was created.', $first['Turn 2: reply'] ); + $this->assertArrayHasKey( 'Created', $first ); + $this->assertArrayHasKey( 'Updated', $first ); + $this->assertStringNotContainsString( 'a tool result', (string) wp_json_encode( $export ), 'Tool results are left out.' ); + + $this->assertSame( 'Yes', $by_id[ 'wpai-chat-' . $archived['id'] ]['Archived'] ); + } + + /** + * With the experiment off, archived chats are exported too. + * + * @since x.x.x + */ + public function test_export_with_the_experiment_off_includes_archived_chats(): void { + $owner = self::$user_ids['owner']; + $this->save_chat( $owner ); + $this->save_chat( $owner, null, true ); + + $this->unregister_chat_types(); + + $export = $this->privacy_callback( 'wp_privacy_personal_data_exporters' )( 'owner@example.org', 1 ); + + $this->assertCount( 2, $export['data'] ); + } + + /** + * The export pages through a person's chats twenty at a time. + * + * @since x.x.x + */ + public function test_export_pages_at_twenty_chats(): void { + $owner = self::$user_ids['owner']; + + for ( $index = 0; $index < Conversation_Privacy::PER_PAGE + 1; $index++ ) { + $this->save_chat( $owner ); + } + + $exporter = $this->privacy_callback( 'wp_privacy_personal_data_exporters' ); + $first = $exporter( 'owner@example.org', 1 ); + $second = $exporter( 'owner@example.org', 2 ); + + $this->assertCount( Conversation_Privacy::PER_PAGE, $first['data'] ); + $this->assertFalse( $first['done'] ); + $this->assertCount( 1, $second['data'] ); + $this->assertTrue( $second['done'] ); + $this->assertSame( array(), array_intersect( array_column( $first['data'], 'item_id' ), array_column( $second['data'], 'item_id' ) ) ); + } + + /** + * Erasing removes every chat the person owns, archived ones included, and no one else's. + * + * @since x.x.x + */ + public function test_erase_removes_the_persons_chats_only(): void { + $owner = self::$user_ids['owner']; + $peer = self::$user_ids['peer']; + $this->save_chat( $owner ); + $this->save_chat( $owner, null, true ); + $this->save_chat( $peer ); + + $this->unregister_chat_types(); + + $result = $this->privacy_callback( 'wp_privacy_personal_data_erasers' )( 'owner@example.org', 1 ); + + $this->assertTrue( $result['items_removed'] ); + $this->assertFalse( $result['items_retained'] ); + $this->assertTrue( $result['done'] ); + $this->assertSame( 0, $this->chat_count( $owner ) ); + $this->assertSame( 1, $this->chat_count( $peer ), 'Another user\'s chats are untouched.' ); + + $again = $this->privacy_callback( 'wp_privacy_personal_data_erasers' )( 'owner@example.org', 1 ); + + $this->assertFalse( $again['items_removed'], 'Nothing is left to remove.' ); + } + + /** + * A proposal in an erased chat can no longer be approved. + * + * @since x.x.x + */ + public function test_a_proposal_in_an_erased_chat_cannot_be_approved(): void { + $owner = self::$user_ids['owner']; + wp_set_current_user( $owner ); + ( new Show_In_Abilities() )->register(); + + $chat = $this->save_chat( $owner ); + $proposal = ( new Proposal_Store() )->create( + $owner, + $chat['id'], + array( + array( + 'post_type' => 'post', + 'status' => 'draft', + 'title' => 'Never written', + 'content' => 'Body', + ), + ) + ); + + $this->assertIsArray( $proposal ); + + $this->privacy_callback( 'wp_privacy_personal_data_erasers' )( 'owner@example.org', 1 ); + + global $wp_rest_server; + $wp_rest_server = null; // phpcs:ignore WordPress.WP.GlobalVariablesOverride.Prohibited, WordPress.NamingConventions.PrefixAllGlobals.NonPrefixedVariableFound -- Rebuilding the REST server for this test's routes. + rest_get_server(); + ( new Proposal_Controller() )->register_routes(); + + $request = new WP_REST_Request( 'POST', '/ai/v1/workspace/proposals/' . $proposal['id'] . '/execute' ); + $request->set_header( 'Content-Type', 'application/json' ); + $request->set_body( + (string) wp_json_encode( + array( + 'conversation_id' => $chat['id'], + 'selected' => array_column( $proposal['items'], 'key' ), + ) + ) + ); + + $response = rest_get_server()->dispatch( $request ); + + $this->assertSame( 410, $response->get_status() ); + $this->assertSame( 'workspace_proposal_chat_deleted', $response->get_data()['code'] ); + $written = get_posts( // phpcs:ignore WordPressVIPMinimum.Functions.RestrictedFunctions.get_posts_get_posts + array( + 'title' => 'Never written', + 'post_status' => 'any', + ) + ); + + $this->assertSame( array(), $written ); + } + + /** + * An email with no matching user exports nothing and erases nothing. + * + * @since x.x.x + */ + public function test_an_unknown_email_exports_and_erases_nothing(): void { + $this->save_chat( self::$user_ids['owner'] ); + + $export = $this->privacy_callback( 'wp_privacy_personal_data_exporters' )( 'nobody@example.org', 1 ); + $erase = $this->privacy_callback( 'wp_privacy_personal_data_erasers' )( 'nobody@example.org', 1 ); + + $this->assertSame( + array( + 'data' => array(), + 'done' => true, + ), + $export + ); + $this->assertFalse( $erase['items_removed'] ); + $this->assertFalse( $erase['items_retained'] ); + $this->assertTrue( $erase['done'] ); + $this->assertSame( 1, $this->chat_count( self::$user_ids['owner'] ) ); + } +} diff --git a/tests/Integration/Includes/Experiments/AI_Workspace/Post_Field_ProposalTest.php b/tests/Integration/Includes/Experiments/AI_Workspace/Post_Field_ProposalTest.php index 848fc1bfe..dffc41ea5 100644 --- a/tests/Integration/Includes/Experiments/AI_Workspace/Post_Field_ProposalTest.php +++ b/tests/Integration/Includes/Experiments/AI_Workspace/Post_Field_ProposalTest.php @@ -12,6 +12,7 @@ use WP_UnitTestCase; use WordPress\AI\Abilities\Content\Read_Content_Bodies; use WordPress\AI\Abilities\Show_In_Abilities; +use WordPress\AI\Experiments\AI_Workspace\Agent_Audit; use WordPress\AI\Experiments\AI_Workspace\Conversation_Store; use WordPress\AI\Experiments\AI_Workspace\Post_Field_Writer; use WordPress\AI\Experiments\AI_Workspace\Proposal_Kind; @@ -1337,4 +1338,54 @@ public function test_content_proposed_from_a_current_read_or_without_one_is_acce $this->assertIsArray( $later ); } + + /** + * A content update records the post, the field and the outcome, never the content (R14). + * + * @since x.x.x + */ + public function test_a_content_update_records_the_field_not_the_content(): void { + $post_id = $this->make_post( array( 'post_content' => '

Private old body.

' ) ); + + $events = array(); + add_action( + Agent_Audit::ACTION, + static function ( string $operation, string $status, array $context ) use ( &$events ): void { + $events[] = array( + 'operation' => $operation, + 'status' => $status, + 'context' => $context, + ); + }, + 10, + 3 + ); + + $proposal = $this->store_changes( + array( + array( + 'post_id' => $post_id, + 'field' => 'content', + 'value' => '

Private new body.

', + ), + ) + ); + + $this->assertSame( 200, $this->execute( $proposal )->get_status() ); + + $writes = array_values( + array_filter( + $events, + static function ( array $event ): bool { + return Post_Field_Writer::LOG_OPERATION === $event['operation']; + } + ) + ); + + $this->assertCount( 1, $writes ); + $this->assertSame( 'success', $writes[0]['status'] ); + $this->assertSame( 'content', $writes[0]['context']['proposal']['field'] ); + $this->assertSame( $post_id, $writes[0]['context']['proposal']['post_id'] ); + $this->assertStringNotContainsString( 'Private', (string) wp_json_encode( $events ), 'No event carries the old or new content.' ); + } } diff --git a/tests/Integration/Includes/Experiments/AI_Workspace/Proposal_ControllerTest.php b/tests/Integration/Includes/Experiments/AI_Workspace/Proposal_ControllerTest.php index 14ae60c4f..2a4f6eed6 100644 --- a/tests/Integration/Includes/Experiments/AI_Workspace/Proposal_ControllerTest.php +++ b/tests/Integration/Includes/Experiments/AI_Workspace/Proposal_ControllerTest.php @@ -12,6 +12,7 @@ use WP_UnitTestCase; use WP_User; use WordPress\AI\Abilities\Show_In_Abilities; +use WordPress\AI\Experiments\AI_Workspace\Agent_Audit; use WordPress\AI\Experiments\AI_Workspace\Conversation_Store; use WordPress\AI\Experiments\AI_Workspace\Draft_Writer; use WordPress\AI\Experiments\AI_Workspace\Proposal_Store; @@ -36,6 +37,7 @@ * @covers \WordPress\AI\Experiments\AI_Workspace\Draft_Writer * @covers \WordPress\AI\Experiments\AI_Workspace\Proposal_Writer * @covers \WordPress\AI\Experiments\AI_Workspace\Proposal_Store + * @covers \WordPress\AI\Experiments\AI_Workspace\Agent_Audit */ class Proposal_ControllerTest extends WP_UnitTestCase { @@ -993,7 +995,7 @@ static function ( $maybe_empty, $postarr ) { $this->assertSame( 'ai-workspace', $context['surface'] ); $this->assertSame( self::CONVERSATION, $context['conversation_id'] ); $this->assertSame( $proposal['id'], $context['proposal']['id'] ); - $this->assertSame( 'ability', $rows[0]['type'] ); + $this->assertSame( Agent_Audit::LOG_TYPE, $rows[0]['type'] ); } /** @@ -1698,4 +1700,258 @@ public function test_a_proposal_of_a_gated_off_kind_cannot_be_approved(): void { $this->assertSame( 'workspace_proposal_invalid_kind', $response->get_data()['code'] ); $this->assertSame( array(), $kind->written ); } + + /** + * Records every agent event the audit hook fires from here on. + * + * @since x.x.x + * + * @return \ArrayObject, error: string}> The events, in order. + */ + private function capture_agent_events(): \ArrayObject { + $events = new \ArrayObject(); + + add_action( + Agent_Audit::ACTION, + static function ( string $operation, string $status, array $context, string $error ) use ( $events ): void { + $events[] = array( + 'operation' => $operation, + 'status' => $status, + 'context' => $context, + 'error' => $error, + ); + }, + 10, + 4 + ); + + return $events; + } + + /** + * Returns the agent log rows written so far. + * + * @since x.x.x + * + * @return list> The rows, oldest first. + */ + private function agent_log_rows(): array { + global $wpdb; + $table = $wpdb->prefix . AI_Request_Log_Schema::TABLE_NAME; + + $rows = $wpdb->get_results( // phpcs:ignore + $wpdb->prepare( "SELECT * FROM {$table} WHERE type = %s ORDER BY id ASC", Agent_Audit::LOG_TYPE ), // phpcs:ignore + ARRAY_A + ); + + return is_array( $rows ) ? $rows : array(); + } + + /** + * Dispatches a discard request for a proposal. + * + * @since x.x.x + * + * @param string $proposal_id The proposal ID. + * @return \WP_REST_Response The response. + */ + private function decline( string $proposal_id ) { + return rest_get_server()->dispatch( new WP_REST_Request( 'DELETE', '/ai/v1/workspace/proposals/' . $proposal_id ) ); + } + + /** + * Proposing, discarding, editing and approving each record one event, in order (R14). + * + * @since x.x.x + */ + public function test_each_proposal_event_is_recorded_once_in_order(): void { + $user_id = $this->login_as( 'owner' ); + $this->boot_logging(); + $events = $this->capture_agent_events(); + + $kind = new Test_Update_Kind(); + $update = $this->store_update_proposal( $kind ); + $drafts = $this->store_proposal( 1 ); + $updates = $this->keys( $update ); + + $this->assertSame( 200, $this->decline( $drafts['id'] )->get_status() ); + $this->assertSame( 200, $this->execute( $update['id'], $updates, self::CONVERSATION, array( $updates[0] => array( 'value' => 'Edited' ) ) )->get_status() ); + + $expected = array( + array( 'proposal_created', 'success', $update['id'] ), + array( 'proposal_created', 'success', $drafts['id'] ), + array( 'proposal_declined', 'success', $drafts['id'] ), + array( 'proposal_edited', 'success', $update['id'] ), + ); + + $summarize = static function ( string $operation, string $status, array $context ): array { + return array( $operation, $status, $context['proposal']['id'] ); + }; + + $fired = array(); + foreach ( $events as $event ) { + $fired[] = $summarize( $event['operation'], $event['status'], $event['context'] ); + } + + $this->assertSame( $expected, $fired, 'The hook fires once per event, in order.' ); + + $logged = array(); + foreach ( $this->agent_log_rows() as $row ) { + $this->assertSame( (string) $user_id, (string) $row['user_id'] ); + $logged[] = $summarize( $row['operation'], $row['status'], json_decode( (string) $row['context'], true ) ); + } + + $this->assertSame( $expected, $logged, 'Each event writes one row of the agent type.' ); + + $this->assertSame( Test_Update_Kind::KIND, $events[0]['context']['proposal']['kind'] ); + $this->assertSame( $updates, $events[0]['context']['proposal']['items'] ); + $this->assertSame( array( $updates[0] ), $events[3]['context']['proposal']['edited'] ); + $this->assertSame( self::CONVERSATION, $events[2]['context']['conversation_id'] ); + $this->assertSame( 'ai-workspace', $events[2]['context']['surface'] ); + $this->assertStringNotContainsString( 'Edited', (string) wp_json_encode( $events[3] ), 'An edited value is not logged.' ); + $this->assertStringNotContainsString( 'First', (string) wp_json_encode( $events[0] ), 'A proposed value is not logged.' ); + } + + /** + * With the AI Request Logging experiment off, every event still fires the hook (R15). + * + * @since x.x.x + */ + public function test_events_fire_the_hook_with_logging_off(): void { + $this->login_as( 'owner' ); + $this->boot_logging(); + $this->set_shared_manager( null ); + $events = $this->capture_agent_events(); + + $proposal = $this->store_proposal( 1 ); + $this->decline( $proposal['id'] ); + + $this->assertSame( array( 'proposal_created', 'proposal_declined' ), array_column( $events->getArrayCopy(), 'operation' ) ); + $this->assertSame( array(), $this->agent_log_rows(), 'Nothing is written while logging is off.' ); + } + + /** + * An expired proposal records one `expired` event when it is next read, and none after. + * + * @since x.x.x + */ + public function test_an_expired_proposal_is_recorded_once(): void { + $this->login_as( 'owner' ); + $store = new Proposal_Store(); + $proposal = $this->store_proposal( 1 ); + + $proposal['expires'] = time() - 1; + $store->save( $proposal ); + + $events = $this->capture_agent_events(); + + $first = rest_get_server()->dispatch( new WP_REST_Request( 'GET', '/ai/v1/workspace/proposals/' . $proposal['id'] ) ); + $second = rest_get_server()->dispatch( new WP_REST_Request( 'GET', '/ai/v1/workspace/proposals/' . $proposal['id'] ) ); + + $this->assertSame( 410, $first->get_status() ); + $this->assertSame( 404, $second->get_status() ); + $this->assertCount( 1, $events ); + $this->assertSame( 'proposal_expired', $events[0]['operation'] ); + $this->assertSame( 'timeout', $events[0]['status'] ); + $this->assertSame( $proposal['id'], $events[0]['context']['proposal']['id'] ); + } + + /** + * A write interrupted for longer than the lock records one `failed` event when it is next read. + * + * @since x.x.x + */ + public function test_an_interrupted_write_is_recorded_failed_once(): void { + $user_id = $this->login_as( 'owner' ); + $store = new Proposal_Store(); + $proposal = $this->store_update_proposal( new Test_Update_Kind() ); + + $proposal['status'] = 'executing'; + $proposal['executing_since'] = time() - Proposal_Store::LOCK_TTL - 1; + $store->save( $proposal ); + + $events = $this->capture_agent_events(); + + $store->find( $proposal['id'], $user_id ); + $store->find( $proposal['id'], $user_id ); + + $this->assertCount( 1, $events ); + $this->assertSame( 'proposal_failed', $events[0]['operation'] ); + $this->assertSame( 'error', $events[0]['status'] ); + $this->assertSame( 'workspace_proposal_interrupted', $events[0]['context']['error_code'] ); + } + + /** + * A writer that throws part way records one `failed` event. + * + * @since x.x.x + */ + public function test_a_writer_that_throws_is_recorded_failed(): void { + $this->login_as( 'owner' ); + + $kind = new Test_Update_Kind(); + $kind->throw_on = 2; + $proposal = $this->store_update_proposal( $kind ); + $events = $this->capture_agent_events(); + + $this->assertSame( 500, $this->execute( $proposal['id'], $this->keys( $proposal ) )->get_status() ); + + $this->assertSame( array( 'proposal_failed' ), array_column( $events->getArrayCopy(), 'operation' ) ); + $this->assertSame( 'workspace_proposal_write_failed', $events[0]['context']['error_code'] ); + } + + /** + * An approval refused because its chat was deleted is recorded as denied. + * + * @since x.x.x + */ + public function test_an_approval_for_a_deleted_chat_is_recorded_denied(): void { + $user_id = $this->login_as( 'owner' ); + $proposal = $this->store_proposal( 1 ); + + $this->ensure_conversation(); + ( new Conversation_Store() )->delete( self::CONVERSATION, $user_id ); + + $events = $this->capture_agent_events(); + + $request = new WP_REST_Request( 'POST', '/ai/v1/workspace/proposals/' . $proposal['id'] . '/execute' ); + $request->set_header( 'Content-Type', 'application/json' ); + $request->set_body( + (string) wp_json_encode( + array( + 'conversation_id' => self::CONVERSATION, + 'selected' => $this->keys( $proposal ), + ) + ) + ); + + $this->assertSame( 410, rest_get_server()->dispatch( $request )->get_status() ); + + $this->assertSame( array( 'proposal_refused' ), array_column( $events->getArrayCopy(), 'operation' ) ); + $this->assertSame( 'denied', $events[0]['status'] ); + $this->assertSame( 'workspace_proposal_chat_deleted', $events[0]['context']['error_code'] ); + $this->assertSame( 'workspace_proposal_chat_deleted', $events[0]['context']['denial_reason'] ); + } + + /** + * An expired proposal read through the store records one `expired` event, and none after. + * + * @since x.x.x + */ + public function test_an_expired_proposal_read_through_the_store_is_recorded_once(): void { + $user_id = $this->login_as( 'owner' ); + $store = new Proposal_Store(); + $proposal = $this->store_proposal( 1 ); + + $proposal['expires'] = time() - 1; + $store->save( $proposal ); + + $events = $this->capture_agent_events(); + + $this->assertNull( $store->get( $proposal['id'], $user_id ) ); + $this->assertNull( $store->get( $proposal['id'], $user_id ) ); + + $this->assertSame( array( 'proposal_expired' ), array_column( $events->getArrayCopy(), 'operation' ) ); + $this->assertSame( 'workspace_proposal_expired', $events[0]['context']['error_code'] ); + } } diff --git a/tests/Integration/Includes/Experiments/AI_Workspace/Proposal_StoreTest.php b/tests/Integration/Includes/Experiments/AI_Workspace/Proposal_StoreTest.php index b9e91f854..520279d83 100644 --- a/tests/Integration/Includes/Experiments/AI_Workspace/Proposal_StoreTest.php +++ b/tests/Integration/Includes/Experiments/AI_Workspace/Proposal_StoreTest.php @@ -7,6 +7,7 @@ namespace WordPress\AI\Tests\Integration\Includes\Experiments\AI_Workspace; +use ReflectionProperty; use WP_UnitTestCase; use WordPress\AI\Abilities\Show_In_Abilities; use WordPress\AI\Experiments\AI_Workspace\Draft_Writer; @@ -18,6 +19,7 @@ use WordPress\AI\Experiments\AI_Workspace\Propose_Template; use WordPress\AI\Experiments\AI_Workspace\Site_Identity_Writer; use WordPress\AI\Experiments\AI_Workspace\Template_Writer; +use WordPress\AI\Logging\Logging_Integration; use WordPress\AI\Tests\Integration\Includes\Experiments\AI_Workspace\Fixtures\Test_Update_Kind; /** @@ -52,6 +54,15 @@ class Proposal_StoreTest extends WP_UnitTestCase { */ private int $user_id = 0; + /** + * Manager the logging integration held before the test replaced it. + * + * @since x.x.x + * + * @var \WordPress\AI\Logging\AI_Request_Log_Manager|null + */ + private $original_shared_manager = null; + /** * Set up test case. * @@ -60,6 +71,10 @@ class Proposal_StoreTest extends WP_UnitTestCase { public function setUp(): void { parent::setUp(); + // Storing a proposal records an agent event; nothing here reads the log, so none is written. + $this->original_shared_manager = Logging_Integration::get_log_manager(); + $this->set_shared_manager( null ); + ( new Show_In_Abilities() )->register(); $this->user_id = self::factory()->user->create( array( 'role' => 'administrator' ) ); @@ -89,9 +104,24 @@ public function tearDown(): void { wp_set_current_user( 0 ); + $this->set_shared_manager( $this->original_shared_manager ); + parent::tearDown(); } + /** + * Replaces the manager the logging integration shares. + * + * @since x.x.x + * + * @param \WordPress\AI\Logging\AI_Request_Log_Manager|null $manager Manager, or null for a disabled experiment. + */ + private function set_shared_manager( ?\WordPress\AI\Logging\AI_Request_Log_Manager $manager ): void { + $property = new ReflectionProperty( Logging_Integration::class, 'log_manager' ); + $property->setAccessible( true ); + $property->setValue( null, $manager ); + } + /** * Stores a test-kind proposal with two items. * diff --git a/tests/Integration/Includes/Experiments/AI_Workspace/Propose_DraftsTest.php b/tests/Integration/Includes/Experiments/AI_Workspace/Propose_DraftsTest.php index 4fa1b7f0a..940131263 100644 --- a/tests/Integration/Includes/Experiments/AI_Workspace/Propose_DraftsTest.php +++ b/tests/Integration/Includes/Experiments/AI_Workspace/Propose_DraftsTest.php @@ -7,12 +7,14 @@ namespace WordPress\AI\Tests\Integration\Includes\Experiments\AI_Workspace; +use ReflectionProperty; use WP_Ability; use WP_UnitTestCase; use WordPress\AI\Abilities\Show_In_Abilities; use WordPress\AI\Experiments\AI_Workspace\Proposal_Store; use WordPress\AI\Experiments\AI_Workspace\Propose_Drafts; use WordPress\AI\Experiments\AI_Workspace\Turn_Context; +use WordPress\AI\Logging\Logging_Integration; /** * Propose_Drafts test case. @@ -57,6 +59,15 @@ public static function wpSetUpBeforeClass( $factory ): void { ); } + /** + * Manager the logging integration held before the test replaced it. + * + * @since x.x.x + * + * @var \WordPress\AI\Logging\AI_Request_Log_Manager|null + */ + private $original_shared_manager = null; + /** * Set up test case. * @@ -65,6 +76,10 @@ public static function wpSetUpBeforeClass( $factory ): void { public function setUp(): void { parent::setUp(); + // Storing a proposal records an agent event; nothing here reads the log, so none is written. + $this->original_shared_manager = Logging_Integration::get_log_manager(); + $this->set_shared_manager( null ); + ( new Show_In_Abilities() )->register(); $this->ensure_ability_category( 'content' ); $this->register_ability(); @@ -97,9 +112,24 @@ public function tearDown(): void { wp_set_current_user( 0 ); + $this->set_shared_manager( $this->original_shared_manager ); + parent::tearDown(); } + /** + * Replaces the manager the logging integration shares. + * + * @since x.x.x + * + * @param \WordPress\AI\Logging\AI_Request_Log_Manager|null $manager Manager, or null for a disabled experiment. + */ + private function set_shared_manager( ?\WordPress\AI\Logging\AI_Request_Log_Manager $manager ): void { + $property = new ReflectionProperty( Logging_Integration::class, 'log_manager' ); + $property->setAccessible( true ); + $property->setValue( null, $manager ); + } + /** * Ensures an ability category exists for an ability to attach to. * diff --git a/tests/Integration/Includes/Experiments/AI_Workspace/Turn_ControllerTest.php b/tests/Integration/Includes/Experiments/AI_Workspace/Turn_ControllerTest.php index eaddd1c8e..5228c9be8 100644 --- a/tests/Integration/Includes/Experiments/AI_Workspace/Turn_ControllerTest.php +++ b/tests/Integration/Includes/Experiments/AI_Workspace/Turn_ControllerTest.php @@ -14,6 +14,7 @@ use WordPress\AI\Abilities\Content\Read_Content_Bodies; use WordPress\AI\Abilities\Content\Search_Content; use WordPress\AI\Abilities\Show_In_Abilities; +use WordPress\AI\Experiments\AI_Workspace\Agent_Audit; use WordPress\AI\Experiments\AI_Workspace\Ask_User; use WordPress\AI\Experiments\AI_Workspace\Conversation_Store; use WordPress\AI\Experiments\AI_Workspace\Model_Client_Interface; @@ -364,7 +365,7 @@ private function ability_log_rows(): array { global $wpdb; $table = $wpdb->prefix . AI_Request_Log_Schema::TABLE_NAME; - $rows = $wpdb->get_results( "SELECT * FROM {$table} WHERE type = 'ability' ORDER BY id ASC", ARRAY_A ); // phpcs:ignore + $rows = $wpdb->get_results( $wpdb->prepare( "SELECT * FROM {$table} WHERE type = %s ORDER BY id ASC", Agent_Audit::LOG_TYPE ), ARRAY_A ); // phpcs:ignore return is_array( $rows ) ? $rows : array(); } @@ -1675,7 +1676,7 @@ public function test_every_tool_call_writes_exactly_one_log_row(): void { $first = $rows[0]; $second = $rows[1]; - $this->assertSame( 'ability', $first['type'] ); + $this->assertSame( Agent_Audit::LOG_TYPE, $first['type'] ); $this->assertSame( self::SEARCH_ABILITY, $first['operation'] ); $this->assertSame( 'success', $first['status'] ); $this->assertSame( (string) $user_id, (string) $first['user_id'] ); @@ -1695,6 +1696,56 @@ public function test_every_tool_call_writes_exactly_one_log_row(): void { $this->assertArrayNotHasKey( 'denial_reason', $first_context ); } + /** + * A turn records its withheld call, its proposal and its proposing call, in that order (R14, R15). + * + * The proposal is stored while the propose call runs, so its `created` event + * comes before the row for the call itself. + * + * @since x.x.x + */ + public function test_a_turn_records_a_withheld_call_and_a_proposal_in_order(): void { + $this->login_as( 'administrator' ); + $this->boot_logging(); + $this->register_proposal_ability(); + + $fired = array(); + add_action( + Agent_Audit::ACTION, + static function ( string $operation, string $status ) use ( &$fired ): void { + $fired[] = array( $operation, $status ); + }, + 10, + 2 + ); + + $client = new Scripted_Model_Client( + array( + $this->tool_call_message( self::DENIED_ABILITY, array( 'value' => 'x' ), 'call_denied' ), + $this->proposal_message(), + ) + ); + + $this->dispatch_turn( $client, array( 'message' => 'Draft me a post.' ) ); + + $expected = array( + array( self::DENIED_ABILITY, Turn_Runner::LOG_STATUS_DENIED ), + array( 'proposal_created', 'success' ), + array( Propose_Drafts::ABILITY, 'success' ), + ); + + $this->assertSame( $expected, $fired ); + + $rows = array_map( + static function ( array $row ): array { + return array( $row['operation'], $row['status'] ); + }, + $this->ability_log_rows() + ); + + $this->assertSame( $expected, $rows ); + } + /** * Logging being switched off does not break a turn. * diff --git a/tests/Integration/Includes/Logging/AI_Request_Log_ControllerTest.php b/tests/Integration/Includes/Logging/AI_Request_Log_ControllerTest.php index d5327373c..508ae7688 100644 --- a/tests/Integration/Includes/Logging/AI_Request_Log_ControllerTest.php +++ b/tests/Integration/Includes/Logging/AI_Request_Log_ControllerTest.php @@ -452,4 +452,46 @@ public function test_get_logs_rejects_non_string_operation(): void { $this->assertSame( 400, $response->get_status() ); } + + /** + * Tests that agent rows can be filtered by type and by the `denied` status. + * + * @since x.x.x + */ + public function test_get_logs_filters_agent_rows_by_type_and_denied_status(): void { + $admin_id = $this->factory()->user->create( array( 'role' => 'administrator' ) ); + wp_set_current_user( $admin_id ); + + $this->insert_log(); + $this->insert_log( + array( + 'type' => 'agent', + 'operation' => 'ai/search-content', + 'status' => 'success', + ) + ); + $this->insert_log( + array( + 'type' => 'agent', + 'operation' => 'core/get-site-info', + 'status' => 'denied', + ) + ); + + $by_type = new WP_REST_Request( 'GET', '/ai/v1/logs' ); + $by_type->set_param( 'type', 'agent' ); + $response = rest_get_server()->dispatch( $by_type ); + + $this->assertSame( 200, $response->get_status() ); + $this->assertSame( '2', $response->get_headers()['X-WP-Total'] ); + + $denied = new WP_REST_Request( 'GET', '/ai/v1/logs' ); + $denied->set_param( 'type', 'agent' ); + $denied->set_param( 'status', 'denied' ); + $response = rest_get_server()->dispatch( $denied ); + + $this->assertSame( 200, $response->get_status() ); + $this->assertSame( '1', $response->get_headers()['X-WP-Total'] ); + $this->assertSame( 'core/get-site-info', $response->get_data()[0]['operation'] ); + } } diff --git a/tests/e2e/specs/experiments/ai-workspace.spec.js b/tests/e2e/specs/experiments/ai-workspace.spec.js index e40ddce45..0d127615f 100644 --- a/tests/e2e/specs/experiments/ai-workspace.spec.js +++ b/tests/e2e/specs/experiments/ai-workspace.spec.js @@ -191,6 +191,197 @@ test.describe( 'AI Workspace transcript', () => { */ } ); +/** + * Holds every turn request open until it is released. + * + * The mocked provider answers at once, so without this there is no window in + * which a turn is running to look at. + * + * @param {import('@playwright/test').Page} page The page. + * @return {Promise<{ release: () => Promise }>} Lets the held turns go on. + */ +async function holdTurns( page ) { + const held = []; + let isReleased = false; + + await page.route( + ( url ) => + url.href.includes( 'workspace/messages' ) && + ! url.href.includes( 'cancel' ), + ( route ) => { + if ( isReleased ) { + return route.continue(); + } + + held.push( route ); + } + ); + + return { + release: async () => { + isReleased = true; + await Promise.all( held.map( ( route ) => route.continue() ) ); + }, + }; +} + +/** + * Reads a sprite's source as text, so a still frame can be told from a moving one. + * + * @param {import('@playwright/test').Locator} sprite The sprite image. + * @return {Promise} The decoded source. + */ +function spriteSource( sprite ) { + return sprite.evaluate( ( node ) => decodeURIComponent( node.src ) ); +} + +/** + * Whether a sprite image loaded and decoded. + * + * @param {import('@playwright/test').Locator} sprite The sprite image. + * @return {Promise} True when the image rendered. + */ +function spriteLoaded( sprite ) { + return sprite.evaluate( + ( node ) => node.complete && node.naturalWidth > 0 + ); +} + +test.describe( 'AI Workspace Wapuu', () => { + test.beforeEach( async ( { admin, page } ) => { + await enableExperiment( admin, page, EXPERIMENT_LABEL ); + await admin.visitAdminPage( WORKSPACE_PAGE ); + } ); + + test.afterEach( async ( { admin, page } ) => { + await disableExperiment( admin, page, EXPERIMENT_LABEL ); + } ); + + test( 'a running turn shows the busy Wapuu beside "Responding…"', async ( { + page, + } ) => { + // The shared Playwright config asks for reduced motion; this test needs motion on. + await page.emulateMedia( { reducedMotion: 'no-preference' } ); + await holdTurns( page ); + + await page + .getByLabel( 'Message', { exact: true } ) + .fill( 'What should I write about next?' ); + await page.getByRole( 'button', { name: 'Send' } ).click(); + + const progress = page.locator( '.ai-workspace__progress' ); + const sprite = progress.locator( '.ai-workspace__wapuu--busy' ); + + await expect( progress ).toContainText( 'Responding…' ); + await expect( sprite ).toBeVisible(); + await expect( sprite ).toHaveAttribute( 'alt', '' ); + expect( await spriteLoaded( sprite ) ).toBe( true ); + + // The spinner it replaces is gone, and motion is on. + await expect( progress.locator( '.components-spinner' ) ).toHaveCount( + 0 + ); + expect( await spriteSource( sprite ) ).not.toContain( + 'animation:none' + ); + } ); + + test( 'with reduced motion the busy Wapuu is a still frame', async ( { + page, + } ) => { + await page.emulateMedia( { reducedMotion: 'reduce' } ); + await holdTurns( page ); + + await page + .getByLabel( 'Message', { exact: true } ) + .fill( 'What should I write about next?' ); + await page.getByRole( 'button', { name: 'Send' } ).click(); + + const sprite = page.locator( + '.ai-workspace__progress .ai-workspace__wapuu--busy' + ); + + await expect( sprite ).toBeVisible(); + + /* + * An image's own reduced-motion query is evaluated in a separate + * document and cannot be inspected from here, so the still frame is + * checked where it is made: the source switches every animation off, + * and it still renders. + */ + expect( await spriteSource( sprite ) ).toContain( 'animation:none' ); + expect( await spriteLoaded( sprite ) ).toBe( true ); + } ); + + test( 'a failed turn shows the error Wapuu with the error', async ( { + page, + } ) => { + await page.route( + ( url ) => + url.href.includes( 'workspace/messages' ) && + ! url.href.includes( 'cancel' ), + ( route ) => route.abort( 'failed' ) + ); + + await page + .getByLabel( 'Message', { exact: true } ) + .fill( 'What should I write about next?' ); + await page.getByRole( 'button', { name: 'Send' } ).click(); + + const failure = page + .locator( '.ai-workspace__turn' ) + .first() + .locator( '.components-notice.is-error .ai-workspace__failure' ); + + await expect( failure ).toBeVisible(); + await expect( failure ).not.toHaveText( '' ); + + const sprite = failure.locator( '.ai-workspace__wapuu--error' ); + + await expect( sprite ).toBeVisible(); + await expect( sprite ).toHaveAttribute( 'alt', '' ); + expect( await spriteLoaded( sprite ) ).toBe( true ); + + // Nothing is left looking busy. + await expect( + page.locator( '.ai-workspace__wapuu--busy' ) + ).toHaveCount( 0 ); + } ); + + test( 'a finished turn shows the done Wapuu briefly', async ( { + page, + } ) => { + const turns = await holdTurns( page ); + + await page + .getByLabel( 'Message', { exact: true } ) + .fill( 'What should I write about next?' ); + await page.getByRole( 'button', { name: 'Send' } ).click(); + + await expect( + page.locator( '.ai-workspace__wapuu--busy' ) + ).toBeVisible(); + + await turns.release(); + + const turn = page.locator( '.ai-workspace__turn' ).first(); + await expect( turn ).toContainText( MOCKED_RESPONSE, { + timeout: 30000, + } ); + + const done = turn.locator( '.ai-workspace__wapuu--done' ); + + await expect( done ).toBeVisible(); + await expect( done ).toHaveAttribute( 'alt', '' ); + await expect( + page.locator( '.ai-workspace__wapuu--busy' ) + ).toHaveCount( 0 ); + + // It goes away on its own. + await expect( done ).toHaveCount( 0, { timeout: 5000 } ); + } ); +} ); + test.describe( 'AI Workspace frame', () => { test.beforeEach( async ( { admin, page } ) => { await enableExperiment( admin, page, EXPERIMENT_LABEL ); diff --git a/tests/e2e/specs/experiments/command-palette-ai-search.spec.js b/tests/e2e/specs/experiments/command-palette-ai-search.spec.js index e45ff4d10..67c75c53a 100644 --- a/tests/e2e/specs/experiments/command-palette-ai-search.spec.js +++ b/tests/e2e/specs/experiments/command-palette-ai-search.spec.js @@ -220,10 +220,12 @@ test.describe( 'Command Palette AI Search', () => { await route.continue(); } ); - const input = await searchPalette( page, 'q' ); + // Characters no command label contains, so the palette has no results of its own + // whichever experiments (and so admin screens) are on. + const input = await searchPalette( page, '~' ); await expect( page.getByText( 'No results found.' ) ).toBeVisible(); - await input.fill( 'qz' ); + await input.fill( '~^' ); await expect( page.getByText( 'No results found.' ) ).toBeVisible(); // Longer than the loader's debounce.