CodeVetter · Repo Unpack · Testing signals
browser-use/browser-use
The project uses pytest 9 with pytest-asyncio in auto mode, a 300s default timeout, and xdist. The CI-gated suite is tests/ci, discovered as one GitHub Actions job per test file with a 4 minute job timeout. Tests are expected to use a real browser plus pytest-httpserver; the LLM is the only allowed mock, via conftest.create_mock_llm. Large areas outside tests/ci (examples, most LLM live tests, the beta agent, and the browser_harness CLI daemon) are not part of that matrix. A separate eval workflow can POST the PR commit to an external eval platform for owner, member, or collaborator PRs.
- pytest is configured in pyproject.toml: timeout 300, asyncio_mode auto, testpaths tests, addopts -svx --strict-markers --tb=short --dist=loadscope, markers slow, integration, unit, and asyncio. bin/test.sh runs uv run pytest --numprocesses auto tests/ci.
- tests/ci is grouped into browser/, infrastructure/, interactions/, models/, security/, plus many top-level test_*.py files covering agent planning, tools, DOM, filesystem, MCP, sandbox structured output, variable substitution, and cloud events.
- tests/ci/conftest.py autouse fixture forces SKIP_LLM_API_KEY_VERIFICATION, ANONYMIZED_TELEMETRY=false, and placeholder cloud URLs so tests do not talk to production. create_mock_llm builds a BaseChatModel that returns scripted AgentOutput actions. The module docstring says tests must not connect to production services.
- CLAUDE.md requires real objects in tests, forbids mocking except the LLM, and forbids real remote URLs such as google.com; HTML fixtures should be served with pytest-httpserver. It also says files that pass move into tests/ci and that @pytest.mark.asyncio is unnecessary because asyncio_mode is auto.
- The GitHub Actions test workflow checks out the repo, caches Playwright Chromium, lists tests/ci test_*.py into a JSON matrix, and runs each file on ubuntu-latest with timeout-minutes 4. It injects OPENAI, ANTHROPIC, GOOGLE, GROQ, AZURE, BROWSER_USE, and OPENROUTER API keys from secrets when present.
- Security-focused tests live in tests/ci/security and include domain filtering, download filename sanitization, IP blocking, MCP allowed domains, security flags, sensitive data, and upload path containment.
- pyright include is browser_use, examples, and tests, but excludes tests/scripts, tests/old, browser_use/dom/playground, browser_use/llm/oci_raw, browser_use/llm/tests/test_chat_models.py, browser_use/llm/tests/test_single_step.py, and browser_use/beta/service.py. Those paths are not held to the type checker.
- Lint CI is three jobs: ruff check --select PLE, pre-commit run --all-files on Python 3.11, and pyright after uv sync --dev --all-extras. Dev dependencies pin pytest==9.0.2, pytest-asyncio==1.3.0, pytest-httpserver==1.1.3, pyright==1.1.408, and ruff==0.14.14.
- eval-on-pr.yml, for PRs whose author association is OWNER, MEMBER, or COLLABORATOR, POSTs the head SHA to secrets.EVAL_PLATFORM_URL/api/triggerInteractionTasksV6 with a default test case of InteractionTasks_v8. This is an external eval, not the local pytest suite.
- browser_use/llm/tests contains live-model tests (test_chat_models.py, test_single_step.py, provider cache and schema tests) that are not under tests/ci and are partly excluded from pyright, so they do not run in the per-file CI matrix.
- There is no tests/ci coverage of the browser_harness daemon itself; CLI tests that exist (tests/ci/test_browser_use_cli.py is in the tree) can only cover this repo's dispatcher. The harness package is an external dependency (browser-harness==0.1.13).