fix: detect empty DOM after navigation and retry before reporting failure (#4319)

## Problem

After a successful navigate() call, the agent had no way to detect that
the page actually loaded empty content. The navigation event would
complete without error, the agent would see an empty DOM, and then fail
or give up with no retry or meaningful error.

This was happening in ~12k+ cases across multiple patterns:
- Pages that use JavaScript rendering that silently fails
- Sites that serve blank pages as anti-bot responses
- Tunnel/proxy connection errors (ERR_TUNNEL_CONNECTION_FAILED) landing
on error pages with no DOM

## Fix

Changes are in browser_use/tools/service.py, navigate() action:

**1. Post-navigation DOM health check**
After navigation completes for http/https URLs, calls
get_browser_state_summary() to inspect dom_state._root. If None (empty
DOM), waits 3 seconds and rechecks once. If still empty after the wait,
returns a descriptive ActionResult(error=...) so the agent can reason
about it and try a different approach, instead of silently proceeding
with an empty page.

**2. ERR_TUNNEL_CONNECTION_FAILED added to network error patterns**
Made explicit alongside the existing net:: catch for correctness and
clarity.

## Tickets Fixed

- Fixes ENG-3469: Agent fails when page loads empty DOM or blank content
- Fixes ENG-2920: Page has empty DOM after navigation, agent does not
wait or retry before failing
- Fixes ENG-3404: Websites render with empty DOM preventing all content
interaction
- Fixes ENG-3311: Website fails to load (empty DOM) preventing contact
form and task execution
This commit is contained in:
Saurav Panda
2026-03-10 16:01:02 -07:00
committed by GitHub

View File

@@ -416,6 +416,23 @@ class Tools(Generic[Context]):
await event
await event.event_result(raise_if_any=True, raise_if_none=False)
# Health check: detect empty DOM for http/https pages and retry once
if not params.new_tab:
state = await browser_session.get_browser_state_summary(include_screenshot=False)
url_is_http = state.url.lower().startswith(('http://', 'https://'))
if url_is_http and state.dom_state._root is None:
browser_session.logger.warning(
f'⚠️ Empty DOM detected after navigation to {params.url}, waiting 3s and rechecking...'
)
await asyncio.sleep(3.0)
state = await browser_session.get_browser_state_summary(include_screenshot=False)
if state.url.lower().startswith(('http://', 'https://')) and state.dom_state._root is None:
return ActionResult(
error=f'Page loaded but returned empty content for {params.url}. '
f'The page may require JavaScript that failed to render, use anti-bot measures, '
f'or have a connection issue (e.g. tunnel/proxy error). Try a different URL or approach.'
)
if params.new_tab:
memory = f'Opened new tab with URL {params.url}'
msg = f'🔗 Opened new tab with url {params.url}'
@@ -442,6 +459,7 @@ class Tools(Generic[Context]):
'ERR_INTERNET_DISCONNECTED',
'ERR_CONNECTION_REFUSED',
'ERR_TIMED_OUT',
'ERR_TUNNEL_CONNECTION_FAILED',
'net::',
]
):