Solving the "Unstructured Directory" Problem with Agentic Scraping
Published on 25.03.2026
We recently helped a client who needed to extract contact information from a business directory. While the task sounds simple, the data was buried in a way that made traditional CSS-selector-based scraping a nightmare.
The Challenge: Unstructured Text
The client provided a list of over 1,000 company profile URLs. Unlike a clean database, the email addresses weren't stored in a specific mailto: link or a labeled "Email" field. Instead, they were buried inside long, unstructured "About Us" paragraphs.
In a traditional setup, you would have to:
- Write a script to fetch the HTML.
- Use Regex to hunt for email patterns (which often misses obfuscated emails like "info [at] company.com").
- Manually update the script every time the directory changed its layout.
The Solution: An Agentic Loop
Instead of writing custom parsing logic for each listing, we used our Agentic Scraper API to handle the heavy lifting. We treated the browser like an intelligent assistant rather than a rigid set of rules.
The Workflow:
- The Input: A CSV of 1,000+ listing URLs.
- The Instruction: We sent a simple, plain-English prompt to the API for every URL:
"Find the primary contact email address hidden within the text descriptions. Return the result in a JSON format: {'email': 'address'}. If no email is found, return null."- The Execution: The API's vision model rendered the page, bypassed the cookie consent pop-ups, and read the text exactly as a human would.
Why This Worked Better Than Standard Tools
- Resilience: The directory used dynamic JavaScript loading. Since the API uses a headless browser with full rendering, we didn't have to worry about missing content that hadn't "popped up" yet.
- Context Awareness: Traditional scrapers often grab the "Support" email from the footer by mistake. Because we used a natural language command, the agent focused specifically on the company description area we defined.
- Zero Infrastructure: We didn't need to rotate proxies or manage a fleet of headless Chrome instances. We just looped through the URLs with a simple Python script and collected the JSON responses.
The Result
Within a couple of hours, the client had a clean spreadsheet with the direct contact emails for over 1,000 companies. No Regex, no broken selectors, and no manual data entry.
Try the API out yourself directly in your browser at https://www.silverlining.cloud/products/agentic-scraper