notifications/tools/list_changed doesnt reload codex? codex bug?

This commit is contained in:
lda
2026-05-17 01:16:12 +07:00 Verified
parent a55879ea2b
commit 3437ec24e8
10 changed files with 118 additions and 22 deletions
+6 -2
View File
@@ -36,7 +36,9 @@ async def downstream_message_handler(message: object) -> None:
downstream_seen.append(message.root.method) downstream_seen.append(message.root.method)
upstream_transport = FastMCPTransport(server) # any transport will do, tried python stdio, and fastmcp, both reproduce. upstream_transport = FastMCPTransport(
server
) # any transport will do, tried python stdio, and fastmcp, both reproduce.
async def prog() -> None: async def prog() -> None:
@@ -60,7 +62,9 @@ async def prog() -> None:
"notifications/message", "notifications/message",
] ]
assert upstream_seen == seen assert upstream_seen == seen
assert downstream_seen == seen, "bug here: notifications should be forwarded by the proxy" assert downstream_seen == seen, (
"bug here: notifications should be forwarded by the proxy"
)
asyncio.run(prog()) asyncio.run(prog())
+11
View File
@@ -93,6 +93,14 @@ Tool simulate-research-query requires task augmentation (taskSupport: 'required'
- Sampling was not visible as a direct listed tool in the current Everything - Sampling was not visible as a direct listed tool in the current Everything
server inventory. server inventory.
- Dynamic tool-list adoption is harness-dependent. On 2026-05-17, after
enabling `playwright.default` and reloading `wf-mcp`, the server-side admin
inventory showed the new proxy tools immediately. After restarting Codex, the
UI could show `playwright.default.browser_navigate`, but the current model turn
still did not receive a callable tool binding for it. That means a host can
observe an updated `tools/list` without rebuilding the tool schema supplied to
an already-running model interaction.
### Current Classification ### Current Classification
- Proxy tool projection: healthy for ordinary tools. - Proxy tool projection: healthy for ordinary tools.
@@ -109,6 +117,9 @@ Tool simulate-research-query requires task augmentation (taskSupport: 'required'
MCP task support is implemented. MCP task support is implemented.
- Notifications/logging/subscriptions: unverified; current client did not show - Notifications/logging/subscriptions: unverified; current client did not show
forwarded notifications. forwarded notifications.
- Dynamic tool-list adoption: server-side updates work, but LLM harnesses may
keep a stale or separately-materialized callable tool schema even after a UI
refresh or reconnect.
## Next Test Targets ## Next Test Targets
+30
View File
@@ -99,6 +99,36 @@ classified as a proxy bug.
Task-required tools are discoverable but not usable through our current surface. Task-required tools are discoverable but not usable through our current surface.
This is a real protocol-support gap, not an ordinary tool-call issue. This is a real protocol-support gap, not an ordinary tool-call issue.
### 5. LLM harnesses may not adopt dynamic tool changes reliably
`wf-mcp` can expose new tools after reload, but that does not guarantee an agent
harness rebuilds the callable tool schema for the model turn that is already in
flight.
Observed on 2026-05-17 with Codex:
- `playwright.default` was enabled
- `wf.admin.reload_config` remounted it successfully
- `wf.admin.list_proxy_tools` showed the new tools
- after reconnect, the Codex UI showed `playwright.default.browser_navigate`
- the model turn still did not receive a callable binding for that new tool
This is a common Codex / Claude Code / general LLM harness class of problem, not
just a proxy-server problem. The host may refresh `tools/list` for display or
inspection while model invocation still uses a previously materialized tool
schema.
Design consequence: do not make core workflows depend on newly registered MCP
tools becoming callable immediately. Prefer stable control tools plus explicit
inspection/call paths when the client must work reliably across harnesses.
When the exposed tool catalog grows large or changes often, FastMCP's search
transform is a good mitigation: keep a stable pinned control/workflow spine
visible, and use `search_tools` plus its synthetic `call_tool` for the changing
rest of the catalog. Do not confuse that synthetic raw-tool caller with a
future workflow-capability test tool; testing a normalized `NodeSpec` contract
is a separate operation and should also remain pinned once it exists.
## What Is Probably Not Worth Owning Yet ## What Is Probably Not Worth Owning Yet
Do **not** rush to implement a custom full protocol relay for: Do **not** rush to implement a custom full protocol relay for:
+5 -3
View File
@@ -47,9 +47,11 @@ wf.workflow.run_deployment
``` ```
This is important because MCP clients may not reliably refresh `tools/list` when This is important because MCP clients may not reliably refresh `tools/list` when
new workflow artifacts are saved. A stable `run_deployment` tool lets an LLM new workflow artifacts are saved. More specifically, LLM harnesses may refresh
test saved workflows immediately without requiring dynamic tool registration or or display a changed tool list without rebuilding the callable tool schema used
tool-list notifications to work perfectly. for the current model turn. A stable `run_deployment` tool lets an LLM test saved
workflows immediately without requiring dynamic tool registration or tool-list
notifications to work perfectly.
This control surface should not fork between backend service layers. The public This control surface should not fork between backend service layers. The public
server now has one exposure style, but internally it still combines: server now has one exposure style, but internally it still combines:
+3 -1
View File
@@ -32,7 +32,9 @@ def register_service_admin_tools(
} }
def name(local_name: str) -> str: def name(local_name: str) -> str:
visible_name = legacy_name_map.get(local_name, local_name) if legacy_names else local_name visible_name = (
legacy_name_map.get(local_name, local_name) if legacy_names else local_name
)
return visible_name if namespace is None else f"{namespace}.{visible_name}" return visible_name if namespace is None else f"{namespace}.{visible_name}"
if include_connection_tools: if include_connection_tools:
-1
View File
@@ -24,4 +24,3 @@ def create_broker_server(service: WfMcpService) -> FastMCP:
register_broker_resources(server, service) register_broker_resources(server, service)
register_broker_prompts(server, service) register_broker_prompts(server, service)
return server return server
+11 -8
View File
@@ -24,18 +24,21 @@ from .tools import (
) )
from .reload_events import ProxyReloadResult, reload_change_events from .reload_events import ProxyReloadResult, reload_change_events
_ADMIN_TOOL_NAMES = [ _SEARCH_ALWAYS_VISIBLE_TOOL_NAMES = [
# Stable discovery/control spine.
f"{ADMIN_NAMESPACE}.list_sources",
f"{ADMIN_NAMESPACE}.list_connections", f"{ADMIN_NAMESPACE}.list_connections",
f"{ADMIN_NAMESPACE}.get_connection_statuses", f"{ADMIN_NAMESPACE}.get_connection_statuses",
f"{ADMIN_NAMESPACE}.get_config",
f"{ADMIN_NAMESPACE}.reload_config", f"{ADMIN_NAMESPACE}.reload_config",
f"{ADMIN_NAMESPACE}.list_proxy_tools", f"{ADMIN_NAMESPACE}.list_proxy_tools",
f"{ADMIN_NAMESPACE}.get_proxy_tool", f"{ADMIN_NAMESPACE}.get_proxy_tool",
f"{ADMIN_NAMESPACE}.add_connection", # Stable workflow control surface. Keep future workflow-capability test
f"{ADMIN_NAMESPACE}.update_connection", # tools pinned here too; they are distinct from raw MCP tool execution.
f"{ADMIN_NAMESPACE}.enable_connection", "wf.workflow.list_artifacts",
f"{ADMIN_NAMESPACE}.disable_connection", "wf.workflow.inspect_artifact",
f"{ADMIN_NAMESPACE}.remove_connection", "wf.workflow.list_deployments",
"wf.workflow.validate_deployment",
"wf.workflow.run_deployment",
] ]
@@ -81,7 +84,7 @@ class ProxyRuntime:
self.server.add_transform(PromptsAsTools(self.server)) self.server.add_transform(PromptsAsTools(self.server))
if search_tools: if search_tools:
self.server.add_transform( self.server.add_transform(
BM25SearchTransform(always_visible=_ADMIN_TOOL_NAMES) BM25SearchTransform(always_visible=_SEARCH_ALWAYS_VISIBLE_TOOL_NAMES)
) )
def current_config(self) -> BrokerConfig: def current_config(self) -> BrokerConfig:
+43
View File
@@ -101,6 +101,49 @@ def test_server_can_hide_admin_tools() -> None:
asyncio.run(run_proxy()) asyncio.run(run_proxy())
def test_server_search_mode_pins_stable_control_and_workflow_tools() -> None:
config = BrokerConfig(
store_root=local_temp_root() / "server_search_mode_store",
connections=[
ConnectionConfig(
id="fixture.personal",
server="fixture",
account="personal",
metadata={
"transport": "stdio",
"command": sys.executable,
"args": [fixture_server_path()],
},
)
],
)
async def run_proxy() -> None:
client = create_server_client(config, search_tools=True)
async with client:
tools = await client.list_tools()
names = [tool.name for tool in tools]
assert "search_tools" in names
assert "call_tool" in names
assert "wf.admin.list_sources" in names
assert "wf.admin.list_connections" in names
assert "wf.admin.get_connection_statuses" in names
assert "wf.admin.reload_config" in names
assert "wf.admin.list_proxy_tools" in names
assert "wf.admin.get_proxy_tool" in names
assert "wf.workflow.list_artifacts" in names
assert "wf.workflow.inspect_artifact" in names
assert "wf.workflow.list_deployments" in names
assert "wf.workflow.validate_deployment" in names
assert "wf.workflow.run_deployment" in names
assert "wf.admin.call_tool" not in names
assert "fixture.personal.echo_tool" not in names
asyncio.run(run_proxy())
def test_workflow_tools_have_human_metadata() -> None: def test_workflow_tools_have_human_metadata() -> None:
config = BrokerConfig( config = BrokerConfig(
store_root=local_temp_root() / "unified_metadata_store", store_root=local_temp_root() / "unified_metadata_store",
+8 -6
View File
@@ -88,12 +88,14 @@ def test_service_rejects_reserved_connection_ids() -> None:
def test_service_installs_builtin_stdlib_specs_by_default() -> None: def test_service_installs_builtin_stdlib_specs_by_default() -> None:
service = WfMcpService(store=FileStore(local_temp_root() / "builtin_store")) service = WfMcpService(store=FileStore(local_temp_root() / "builtin_store"))
assert "wf.std.runtime_error" in service.capability_sources[ assert (
"wf.std" "wf.std.runtime_error"
].capabilities.node_specs in service.capability_sources["wf.std"].capabilities.node_specs
assert "wf.mcp.call_tool" in service.capability_sources[ )
"wf.mcp" assert (
].capabilities.node_specs "wf.mcp.call_tool"
in service.capability_sources["wf.mcp"].capabilities.node_specs
)
def test_service_lists_all_capability_sources_with_owned_capability_names() -> None: def test_service_lists_all_capability_sources_with_owned_capability_names() -> None:
+1 -1
View File
@@ -19,7 +19,7 @@
"id": "playwright.default", "id": "playwright.default",
"server": "playwright", "server": "playwright",
"account": "default", "account": "default",
"enabled": false, "enabled": true,
"metadata": { "metadata": {
"transport": "stdio", "transport": "stdio",
"command": "pnpx", "command": "pnpx",