Web

v2 extracts public page/feed image, video, audio, and document pointers in addition to text and links. Authenticated agents can call GET /v1/primitives/web/search?q=... for bounded transient SearXNG JSON; the response is not persisted. Agents may also pass engines, categories, language, time_range, and pageno; Argus rejects other parameters and always forces format=json.

What it collects

Direct URLs fetch a public HTML page and extract readable text. RSS and Atom feeds yield their entries. Queries ask the trusted, managed SearXNG endpoint for discovery results; a query is not a general browser crawl.

Prerequisites

Use public http or https URLs without credentials. Configure a managed SearXNG endpoint before adding Web queries. Direct URLs and feeds do not require SearXNG.

Configure the source

sources:
  web:
    enabled: true
    searchEndpoint: http://searxng:8080
    searchEndpointTrust: trusted
    userAgent: Argus/0.1

Browser automation is not supported as a fallback.

Use searchEndpointTrust: trusted only for an operator-controlled private service such as Argus-managed SearXNG. External endpoints default to public, which validates and pins every public DNS answer again on redirect hops.

Configure watches

Keep direct pages, feeds, and query discovery explicit.

watches:
  - id: web-signals
    schedule: "*/15 * * * *"
    inputs:
      web:
        urls: [https://example.com/news]
        feeds: [https://example.com/feed.xml]
        queries: ["critical infrastructure"]

Validate and apply

argus config validate /opt/argus/argus.yaml
argus config apply /opt/argus/argus.yaml

Verify ingestion

Use argus doctor --json, then query GET /v1/records?source=web with bearer authentication. A POST /v1/watches/web-signals/ingest queues the configured targets immediately.

Limits and safety

Web requests enforce public-network policy: only http/https, no URL credentials, public DNS answers, and public redirects with no HTTPS downgrade. The resolver is bounded at 2 seconds; a request is bounded to 10 seconds (maximum configurable timeout 20 seconds), follows at most five redirects, and reads at most 2 MiB by default (never more than 10 MiB). SSRF targets such as loopback, private, link-local, documentation, multicast, and non-global IPv6 addresses are rejected.

Troubleshooting

• A Web query needs sources.web.searchEndpoint; managed private SearXNG also needs searchEndpointTrust: trusted, while external public endpoints keep the public default.
• A rejected direct URL is usually non-public DNS, a credentialed URL, a private redirect, or an HTTPS downgrade.
• Check argus logs argus for bounded request failures or non-success feed/page responses.