Agentic Scraper API

Table of Content

Subscribe on AWS Marketplace
Code sample
API endpoint: /crawl
Troubleshooting

Subscribe on AWS Marketplace

Navigate to our product listing on the AWS Marketplace. Select the "View purchase options" button to proceed.

On the following page, click the "Subscribe" button to initiate the subscription process.

After subscribing, you will need to set up your account on our platform. Do this by clicking the "Set up your account" link provided.

You will be directed to the registration page. Here, fill in your account details in the provided fields. Once you have entered all necessary information, click the 'Subscribe' button to proceed.

Upon successful registration, your unique API key will be displayed. It is crucial to store this key in a secure location for future use.

Code Samples

The code samples below calls the '/crawl' endpoint in order to retrieve the current price of the iPhone 17 on Amazon. It uses a proxy from the US.

API endpoint: /crawl

URL endpoint: https://scraper-api.silverlining.cloud/v1/crawl


Method: POST


"header": {
  "x-api-key": STRING,
  "Content-Type": "application/json"
}


"body": {
  "url": STRING,
  "action": STRING,
  "proxy_country_code": STRING
}

Required
"url": The full URL of the web page you want to scrape. The URL must be accessible from the public internet. Must include the protocol (http:// or https://). Example: "https://www.silverlining.cloud/products"
"action": A natural language instruction describing the data you want to extract from the page. The AI model will use this instruction to visually identify and return the requested information in a JSON format. Be as specific as possible. A good example could look like: "Extract all product titles and product description from the 'All products by SilverLining.Cloud' section in this format: ['product1': {'title': '<title>', 'description': '<description>'}]"

Optional
"proxy_country_code": The two-letter ISO 3166-1 alpha-2 country code for the proxy server. Use this to make the request appear as if it's coming from a specific country, which is useful for accessing geo-restricted content. If this parameter is omitted, the request will be made from the US.

Response:

"body": {
  "input": {
    "url": STRING,
    "action": STRING,
    "proxy_country_code": STRING
  },
  "metadata": {
    "cookie_banner_accepted": BOOLEAN,
    "full_page_screenshot": STRING
  },
  "data_extracted": OBJECT
}   

Input Object

This object simply reflects the parameters you sent in your request. It's useful for logging and confirming what instructions the API executed.

• "url": The target URL that was crawled.
• "action": The natural language instruction you provided.
• "proxy_country_code": The country code used for the request, or null if none was specified.

Metadata Object

This object provides additional information about the crawling process.

• "cookie_banner_accepted": A boolean (true or false) indicating if the API detected and automatically accepted a cookie consent banner on the page.
• "full_page_screenshot": An object containing the s3_url, which is a direct, public link to a PNG image of the full web page captured before data extraction.

Data Extracted Object

This is the main payload containing the data you requested.

• The structure of this JSON object is determined by the AI based on your action prompt. It will attempt to format the response exactly as you requested.
• If the requested data cannot be found or an error occurs (e.g., a login wall is encountered), this object will typically contain an error key with a descriptive message.