Agent

This is the primary class to interact with SecretAgent. The following is a simple example:

const agent = require('secret-agent');

(async () => {
  await agent.goto('https://www.google.com');
  // other actions...
  await agent.close();
})();

An Agent instance can be thought of as a single user-browsing session. A default instance is automatically initialized and available as the default export of secret-agent. Each additional instance you create has the following attributes:

Replayable

An instance has a replayable Session that will record all commands, dom changes, interaction and page events.

Lightweight

Instances are very lightweight, sharing a pool of browsers underneath. To manage concurrent scrapes in a single script, you can create one Agent for each scrape, or manage load and concurrency with a Handler.

Single Active Tab

Agent instances can have multiple Tabs, but only a single tab can be focused at a time. Clicks and other user interaction will go to the active tab (interacting with multiple tabs at once by a single user is easily detectable).

Sandboxed

Each Agent instance creates a private environment with its own cache, cookies, session data and BrowserEmulator. No data is shared between instances -- each operates within an airtight sandbox to ensure no identities leak across requests.

Constructor

new Agent(options)

Creates a new sandboxed browser instance with unique user session and fingerprints. Or pass in an existing UserProfile to reconstruct a previously used user session.

You can optionally await an instance (or constructor) to cause the connection to the underlying SecretAgent to be initialized. If you don't await, the connection will be established on the first call.

Note: If you provide a name that has already been used to name another instance then a counter will be appended to your string to ensure its uniqueness. However, it's only unique within a single NodeJs process (i.e., rerunning your script will reset the counter).

const { Agent } = require('secret-agent');

(async () => {
  // connection established here
  const agent = await new Agent();
})();

Arguments:

  • options object Accepts any of the following:
    • connectionToCore options | ConnectionToCore. An object containing IConnectionToCoreOptions used to connect, or an already created ConnectionToCore instance. Defaults to automatically booting up and connecting to a local Core.
    • name string. This is used to generate a unique sessionName.
    • browserEmulatorId string defaults to chrome-83. Emulates a specific browser engine version.
    • humanEmulatorId string. Drives human-like mouse/keyboard movements.
    • timezoneId string. Overrides the host timezone. A list of valid ids are available at unicode.org
    • locale string. Overrides the host languages settings (eg, en-US). Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting rules.
    • viewport IViewport. Sets the emulated screen size, window position in the screen, inner/outer width and height. If not provided, the most popular resolution is used from statcounter.com.
    • blockedResourceTypes BlockedResourceType[]. Controls browser resource loading. Valid options are listed here.
    • userProfile IUserProfile. Previous user's cookies, session, etc.
    • showReplay boolean. Whether or not to show the Replay UI. Can also be set with an env variable: SA_SHOW_REPLAY=true.
    • upstreamProxyUrl string. A socks5 or http proxy url (and optional auth) to use for all HTTP requests in this session. Dns over Tls requests will also use this proxy, if provided. The optional "auth" should be included in the UserInfo section of the url, eg: http://username:password@proxy.com:80.

Properties

agent.activeTab

Returns a reference to the currently active tab.

Type: Tab

agent.coreHost

The connectionToCore host address to which this Agent has connected. This is useful in scenarios where a Handler is round-robining connections between multiple hosts.

Type: Promise<string>

agent.document
W3C

Returns a reference to the main Document for the active tab.

Type: SuperDocument

Alias for activeTab.document

agent.mainFrameEnvironment

Returns a reference to the document of the mainFrameEnvironment of the active tab.

Alias for tab.mainFrameEnvironment.document.

Type: SuperDocument

agent.frameEnvironments

Returns a list of FrameEnvironments loaded for the active tab.

Type: Promise<Frame[]>.

agent.lastCommandId

An execution point that refers to a command run on this instance (waitForElement, click, type, etc). Command ids can be passed to select waitFor* methods to indicate a starting point to listen for changes.

Type: Promise<number>

Alias for activeTab.lastCommandId

agent.meta

Retrieves metadata about the agent configuration:

  • sessionId string. The session identifier.
  • sessionName string. The unique session name that will be visible in Replay.
  • browserEmulatorId string. The id of the Browser Emulator in use.
  • humanEmulatorId string. The id of the Human Emulator in use.
  • timezoneId string. The configured unicode TimezoneId or host default (eg, America/New_York).
  • locale string. The configured locale in use (eg, en-US).
  • viewport IViewport. The emulated viewport size and location.
  • blockedResourceTypes BlockedResourceType[]. The blocked resource types.
  • upstreamProxyUrl string. The proxy url in use for this agent.
  • userAgentString string. The user agent string used in Http requests and within the DOM.

Type: Promise<IAgentMeta>

agent.sessionId

An identifier used for storing logs, snapshots, and other assets associated with the current session.

Type: Promise<string>

agent.sessionName

A human-readable identifier of the current Agent session.

You can set this property when calling Handler.dispatchAgent() or Handler.createAgent().

Type: Promise<string>

agent.tabs

Returns all open browser tabs.

Type: Promise<Tab[]>

agent.url

The url of the active tab.

Type: Promise<string>

Alias for Tab.url

agent.Request
W3C

Returns a constructor for a Request object bound to the activeTab. Proxies to tab.Request. These objects can be used to run browser-native tab.fetch requests from the context of the Tab document.

Type: Request

Alias for Tab.Request

Methods

agent.click(mousePosition)

Executes a click interaction. This is a shortcut for agent.interact({ click: mousePosition }). See the Interactions page for more details.

Arguments:

Returns: Promise

agent.close()

Closes the current instance and any open tabs.

Returns: Promise

agent.closeTab(tab)

Close a single Tab. The first opened Tab will become the focused tab.

Arguments:

  • tab Tab The Tab to close.

Returns: Promise<void>

Alias for Tab.close()

agent.configure(options)

Update existing configuration settings.

Arguments:

  • options object Accepts any of the following:
    • userProfile IUserProfile. Previous user's cookies, session, etc.
    • timezoneId string. Overrides the host timezone. A list of valid ids are available at unicode.org
    • locale string. Overrides the host languages settings (eg, en-US). Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting rules.
    • viewport IViewport. Sets the emulated screen size, window position in the screen, inner/outer width.
    • blockedResourceTypes BlockedResourceType[]. Controls browser resource loading. Valid options are listed here.
    • upstreamProxyUrl string. A socks5 or http proxy url (and optional auth) to use for all HTTP requests in this session. The optional "auth" should be included in the UserInfo section of the url, eg: http://username:password@proxy.com:80.
    • connectionToCore options | ConnectionToCore. An object containing IConnectionToCoreOptions used to connect, or an already created ConnectionToCore instance. Defaults to booting up and connecting to a local Core.

Returns: Promise

See the Configuration page for more details on options and its defaults. You may also want to explore BrowserEmulators and HumanEmulators.

agent.exportUserProfile()

Returns a json representation of the underlying browser state for saving. This can later be restored into a new instance using agent.configure({ userProfile: serialized }). See the UserProfile page for more details.

Returns: Promise<IUserProfile>

agent.interact(interaction[, interaction, ...])

Executes a series of mouse and keyboard interactions.

Arguments:

Returns: Promise

Refer to the Interactions page for details on how to construct an interaction.

agent.scrollTo(mousePosition)

Executes a scroll interaction. This is a shortcut for agent.interact({ scroll: mousePosition }). See the Interactions page for more details.

Arguments:

  • mousePosition MousePosition

Returns: Promise

agent.type(keyboardInteraction[, keyboardInteraction, ...])

Executes a keyboard interactions. This is a shortcut for agent.interact({ type: string | KeyName[] }).

Arguments:

Returns: Promise

Refer to the Interactions page for details on how to construct keyboard interactions.

agent.focusTab(tab)

Bring a tab to the forefront. This will route all interaction (click, type, etc) methods to the tab provided as an argument.

Arguments:

  • tab Tab The Tab which will become the activeTab.

Returns: Promise<void>

Alias for Tab.focus()

agent.waitForNewTab()

Wait for a new tab to be created. This can occur either via a window.open from within the page javascript, or a Link with a target opening in a new tab or window.

Returns: Promise<Tab>

const url = 'https://dataliberationfoundation.org/nopost';
const { document, activeTab } = agent;

await agent.goto('http://example.com');

// ...
// <a id="newTabLink" href="/newPage" target="_blank">Link to new target</a>
// ...

await document.querySelector('#newTabLink').click();
const newTab = await agent.waitForNewTab();

await newTab.waitForPaintingStable();

Aliased Tab Methods

Agent instances have aliases to all top-level Tab methods. They will be routed to the activeTab.

agent.fetch(requestInput, requestInit)
W3C

Alias for Tab.fetch()

agent.getFrameEnvironment(frameElement)

Alias for Tab.getFrameEnvironment()

agent.getComputedStyle(element, pseudoElement)
W3C

Alias for Tab.getComputedStyle()

agent.getJsValue(path)

Alias for Tab.getJsValue()

agent.goBack(timeoutMs?)

Alias for Tab.goBack

agent.goForward(timeoutMs?)

Alias for Tab.goForward

agent.goto(href, timeoutMs?)

Alias for Tab.goto

agent.isElementVisible(element)

Alias for Tab.isElementVisible

agent.reload(timeoutMs?)

Alias for Tab.reload

agent.takeScreenshot(options?)

Alias for Tab.takeScreenshot

agent.waitForPaintingStable()

Alias for Tab.waitForLoad(PaintingStable)

agent.waitForResource(filter, options)

Alias for Tab.waitForResource

agent.waitForElement(element, options)

Alias for Tab.waitForElement

agent.waitForLocation(trigger, options)

Alias for Tab.waitForLocation

agent.waitForMillis(millis)

Alias for Tab.waitForMillis

Edit this page on GitHub