Feature: New Experiment Abilities Toggle by hbhalodia · Pull Request #881 · WordPress/ai

hbhalodia · 2026-07-17T10:23:48Z

What?

Closes #863

Why?

Need to provide an option to hide the custom abilities behind the toggle experiment button.

How?

Added new Admin Custom experiment, Which gates the plugin core abilities behing a single opt-in toggle, so enabling it exposes all of them at once via the abilities.

Use of AI Tools

Yes, Cluade Code, Opus 5.
Used for inital implementation related to gating abilities and adding a custom ability experiment.
Optimization and structuring related changes.
Implementation was reviewed by me.

Testing Instructions

Checkout this branch and go to AI plugin settings.
Scroll to Admin Experiments --> Custom Abilities.
Before enabling check the number of abilities registered from Abilities Explorer page.
Enable the settings, check again.
You would see when toggled is on, the custom abilities are registered else they are not.

Changelog Entry

Added - New Experiment: Custom Abilities. Gates the plugin's custom WordPress Abilities (ai/get-post-details, ai/get-post-terms, core/read-settings, core/read-users, core/read-content) behind a single opt-in toggle, so enabling it exposes all of them at once via the Abilities API. Note for anyone that is using these Abilities, you'll need to enable this new experiment first for those to be available.

AI Summary

Implements #863 — wraps the plugin's "always-on" custom WordPress Abilities behind a single opt-in experiment. Previously these abilities were registered unconditionally on every request and exposed via the Abilities API / MCP. Now a site owner enables the Custom Abilities experiment and gains access to all of them at once; when it's off, none are registered.

Motivation

Per #863, ability APIs (some of which go beyond read-only operations) shouldn't be automatically exposed — they should require purposeful enablement. This gates the current custom abilities behind an experiment toggle, giving admins explicit control and establishing a scalable pattern for future abilities.

What changed

New: `Custom Abilities` experiment

A standard experiment (custom-abilities, Admin category) nested under the global Enable AI toggle. When enabled, it registers every gated ability together — no per-ability toggles.
Show_In_Abilities (infrastructure, not a user-facing ability) runs once before registration when any enabled ability needs core objects exposed to the Abilities API.

New: gated-ability framework (scalable)

Abstract_Gated_Ability — minimal contract: register() + requires_core_object_exposure().
Four thin wrappers around the existing ability classes:
- Post_Utilities → ai/get-post-details, ai/get-post-terms
- Read_Settings → core/read-settings
- Read_Users → core/read-users
- Read_Content → core/read-content
Gated_Abilities registry + wpai_gated_abilities filter for third-party extensibility.
Adding a new gated ability = one small class + one registry line (or via the filter). No closures — each ability is independently unit-testable.

Removed always-on registration

Main.php no longer registers Posts, Show_In_Abilities, Settings, Users, and Content abilities directly; they're now owned by the experiment.

Decoupling guard (no internal regressions)

Extracted Posts::get_post_details() / Posts::get_post_terms() static methods; the abilities' execute_callbacks delegate to them.
get_post_context() now calls these static methods directly instead of via wp_get_ability(), so the six generation features that rely on post context (excerpt, title, meta-description, image-prompt, summarization, content-classification) keep working regardless of whether the Posts ability is toggled on. The toggle only controls external (REST/MCP) exposure.

Files

Added

includes/Experiments/Custom_Abilities/Custom_Abilities.php
includes/Abstracts/Abstract_Gated_Ability.php
includes/Abilities/Gated/Gated_Abilities.php
includes/Abilities/Gated/Post_Utilities.php
includes/Abilities/Gated/Read_Settings.php
includes/Abilities/Gated/Read_Users.php
includes/Abilities/Gated/Read_Content.php

Modified

includes/Main.php — removed always-on ability registration + unused imports
includes/Experiments/Experiments.php — registered Custom_Abilities
includes/helpers.php — get_post_context() uses the new static methods
includes/Abilities/Utilities/Posts.php — extracted static methods; callbacks delegate

Behavior

Opt-in, no migration. After update, these abilities are OFF until an admin enables Enable AI → Custom Abilities. This is intentional per New Experiment: Abilities toggle #863 (explicit enablement).
Internal plugin features are unaffected thanks to the get_post_context() decoupling.

Notes for reviewers

@since x.x.x docblock tags are placeholders to be stamped at release.
No JS/build changes — this is PHP-only. The experiment renders as a standard toggle on the settings screen.
Third parties can register their own gated abilities via the wpai_gated_abilities filter.

…xperiment

codecov · 2026-08-06T09:57:26Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.76%. Comparing base (d3a3d5e) to head (ddd4f01).

Additional details and impacted files

@@              Coverage Diff              @@
##             develop     #881      +/-   ##
=============================================
- Coverage      75.95%   75.76%   -0.20%     
- Complexity      2989     3013      +24     
=============================================
  Files            124      131       +7     
  Lines          11850    11915      +65     
=============================================
+ Hits            9001     9027      +26     
- Misses          2849     2888      +39

Flag	Coverage Δ
unit	`75.76% <100.00%> (-0.20%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-08-10T07:15:11Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: hbhalodia <hbhalodia@git.wordpress.org>
Co-authored-by: dkotter <dkotter@git.wordpress.org>
Co-authored-by: jeffpaul <jeffpaul@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

jeffpaul

Tests well via Playground, will await @dkotter's review before merging

dkotter

Left some comments that need looked at, in addition to a few other things:

We should look to add documentation here matching our other experiments
It doesn't appear we have E2E tests specific to this new experiment, ensuring the experiment can be turned on/off and abilities are available/not available depending on that state
Technically this is a breaking change for any site that is relying on these Abilities. Once they upgrade, they'll need to enable this new experiment before those Abilities are available. That's likely fine but will need to ensure that is called out when we put together the release (cc @jeffpaul). Could also make this experiment enabled by default though that goes against how everything else is set up right now

hbhalodia · 2026-08-12T04:50:36Z

Thanks for the review @dkotter, I will update the PR to address those.

hbhalodia · 2026-08-12T06:57:13Z

Left some comments that need looked at, in addition to a few other things:

We should look to add documentation here matching our other experiments

Added the documentation in the commit - 6e4af35
Have created the write up with AI who was having the context regarding the change. Have reviewed myself, let me know if anything needs to be added or re-written. Used the context from other experiments.

It doesn't appear we have E2E tests specific to this new experiment, ensuring the experiment can be turned on/off and abilities are available/not available depending on that state

Added the E2E tests for the experiment in commit - d37e079

Technically this is a breaking change for any site that is relying on these Abilities. Once they upgrade, they'll need to enable this new experiment before those Abilities are available. That's likely fine but will need to ensure that is called out when we put together the release (cc @jeffpaul). Could also make this experiment enabled by default though that goes against how everything else is set up right now

I guess it would be added while we do the release, so have not added anything here.

Thanks,

dkotter

Couple more minor things I've noted.

And I don't know if related to this or just a general bug in our plugin but I'm finding that if you toggle the experiment on and off a few times, we end up losing credentials, which is really weird. May not be related to this PR but figured I'd flag it:

missing-credentials.mov

jeffpaul · 2026-08-12T16:00:24Z

Let's not enable these abilities by default, but instead call out in the changelogs and with an upgrade notice that they'll need to be enabled for anyone making use of abilities previously.

dkotter · 2026-08-12T16:20:06Z

And I don't know if related to this or just a general bug in our plugin but I'm finding that if you toggle the experiment on and off a few times, we end up losing credentials, which is really weird. May not be related to this PR but figured I'd flag it

Quick followup to this bug. I have verified the same behavior on develop so this is not related to this PR.

In diving in, seems this is an upstream issue within WordPress Core. Any request to /wp/v2/settings will trigger core to re-validate all Connector API keys, even if you aren't specifically saving an API key. So in our case, we make a request to that endpoint when we save our plugin settings.

If anything goes wrong in that validation (in my case, seems like occasionally requests to the OpenAI models endpoint is timing out at 5 seconds), Core sets the API key to a blank string.

We can look to fix on our end by introducing a custom endpoint to handle saving our settings, bypassing the /wp/v2/settings all together. But also seems ideal to fix this upstream to avoid having Core re-validate credentials on every request, even if credentials aren't being passed in.

dkotter · 2026-08-12T20:42:19Z

Note that E2E and Plugin Check tests aren't running right know due to WordPress 7.0.4 not being properly tagged on GitHub:

fatal: couldn't find remote ref 7.0.4

I've run those locally and they are all passing so going to merge this in

hbhalodia · 2026-08-13T05:36:32Z

Couple more minor things I've noted.

And I don't know if related to this or just a general bug in our plugin but I'm finding that if you toggle the experiment on and off a few times, we end up losing credentials, which is really weird. May not be related to this PR but figured I'd flag it:

missing-credentials.mov

Thanks @dkotter, I noticed that too as well in earlier days while I was contributing to AI plugin, but it happen so randomly that I was not able to reproduce with the exact steps again and again, hence have'nt logged the ticket 😅.

Will create the ticket on core or either GB and will add the reproduction steps.

Thanks for exact reproduction steps.

Update: Raised the trac ticket and PR.

Cc: @jeffpaul

hbhalodia added 5 commits July 17, 2026 15:52

Add Abilities toggle for individual abilities in Abilities Explorer e…

7b7bf5c

…xperiment

Fix phpcs feedback

8338183

Merge branch 'develop' into fix/issue-863

ebd53b3

Update to add custom experiment for custom abilities for registeration

3eb5191

Fix tests

9824a43

hbhalodia mentioned this pull request Aug 6, 2026

New Experiment: Abilities toggle #863

Closed

1 task

hbhalodia added 7 commits August 10, 2026 10:39

Merge branch 'develop' into fix/issue-863

f24c49c

Add the test cases for new files added

e57b1d4

Fix e2e test spec and enable custom abilities experiment

bcd433d

Update tests for missing lines

9cc7b40

Update the code to fix the tests

1188da9

Add missing test coverage

6c0dff6

Update test to fix last line coverage

d58dcbc

hbhalodia marked this pull request as ready for review August 10, 2026 07:15

hbhalodia requested a review from a team August 10, 2026 07:15

jeffpaul added this to the 1.3.0 milestone Aug 10, 2026

jeffpaul previously approved these changes Aug 10, 2026

View reviewed changes

jeffpaul mentioned this pull request Aug 10, 2026

Release version 1.3.0 #924

Closed

48 tasks

dkotter assigned hbhalodia Aug 11, 2026

dkotter requested changes Aug 11, 2026

View reviewed changes

Comment thread includes/Experiments/Custom_Abilities/Custom_Abilities.php Outdated

Comment thread includes/Abilities/Gated/Gated_Abilities.php Outdated

Comment thread includes/helpers.php

Comment thread includes/Abilities/Utilities/Posts.php Outdated

dkotter changed the title ~~Feature: Nex Experiment Abilities Toggle~~ Feature: New Experiment Abilities Toggle Aug 11, 2026

hbhalodia added 2 commits August 12, 2026 10:20

Merge branch 'develop' into fix/issue-863

18571d5

Address feedbacks

75d86d9

hbhalodia dismissed jeffpaul’s stale review via 75d86d9 August 12, 2026 05:09

hbhalodia added 2 commits August 12, 2026 11:50

Add the experiment readme.md file

6e4af35

Add the e2e test for custom ability experiment

d37e079

hbhalodia requested a review from dkotter August 12, 2026 06:57

dkotter requested changes Aug 12, 2026

View reviewed changes

Comment thread README.md

Comment thread docs/experiments/custom-abilities.md Outdated

dkotter added 2 commits August 12, 2026 14:21

Merge branch 'develop' into fix/issue-863

dc9d195

Update docs and readme

ddd4f01

dkotter approved these changes Aug 12, 2026

View reviewed changes

dkotter merged commit 4ff447f into WordPress:develop Aug 12, 2026
25 of 28 checks passed

hbhalodia deleted the fix/issue-863 branch August 13, 2026 07:10

hbhalodia mentioned this pull request Aug 14, 2026

Introduce new custom endpoint for settings save and by pass core /wp/v2/settings endpoint for saving AI options #940

Closed

6 tasks

Uh oh!

Conversation

hbhalodia commented Jul 17, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

How?

Use of AI Tools

Testing Instructions

Changelog Entry

AI Summary

Motivation

What changed

New: Custom Abilities experiment

New: gated-ability framework (scalable)

Removed always-on registration

Decoupling guard (no internal regressions)

Files

Behavior

Notes for reviewers

Uh oh!

codecov Bot commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jeffpaul left a comment

Choose a reason for hiding this comment

Uh oh!

dkotter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hbhalodia commented Aug 12, 2026

Uh oh!

hbhalodia commented Aug 12, 2026

Uh oh!

dkotter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jeffpaul commented Aug 12, 2026

Uh oh!

dkotter commented Aug 12, 2026

Uh oh!

dkotter commented Aug 12, 2026

Uh oh!

Uh oh!

hbhalodia commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hbhalodia commented Jul 17, 2026 •

edited by github-actions Bot

Loading

New: `Custom Abilities` experiment

codecov Bot commented Aug 6, 2026 •

edited

Loading

github-actions Bot commented Aug 10, 2026 •

edited

Loading

hbhalodia commented Aug 13, 2026 •

edited

Loading