SecretAgent operates out of the box over WebSockets. You'll eventually want to launch Core on a server where clients can remotely access it.
You'll need a simple script to start the server on the machine where the secret-agent
npm package is installed. Make sure to open the port you allocate on any firewall that a client might have to pass through:
Below is code you can use to start Core in your own server process. NOTE: you can also simply run the 'start.js' script that is packaged with @secret-agent/core
at @secret-agent/core/start
;
// SERVER ip is 122.22.232.1
const Core = require('@secret-agent/core');
(async () => {
Core.onShutdown = () => {
log.info('Exiting Core Process');
process.exit();
};
await Core.start({ coreServerPort: 7007 });
})();
Your Agent or Handler must be configured to point at this Remote Core (and any others you've set up).
NOTE: you can use the @secret-agent/client
npm package if you don't want to install a full browser engine on the machine coordinating all your scrapes. That example is shown below.
const agent = require('@secret-agent/client');
(async () => {
await agent.configure({
connectionToCore: {
host: 'localhost:7007',
},
});
await agent.goto('https://ulixee.org');
})().catch(console.log);