# Lead Scraping Basics for Recruitment Agencies: From Job Ads to CRM-Ready Leads

> Lead scraping for a recruitment agency means collecting public job ads automatically and turning them into company leads with a decision maker. The basics are to pull cheap overviews first and details only for new ads, deduplicate by company rather than by ad, remove other agencies, enrich with website and contact, drop expired ads and respect each source’s terms and data protection rules.

Source: https://hiring-intelligence.io/en/blog/lead-scraping-basics-for-recruitment-agencies
Author: Philipp Kaming · Published: 2026-11-08

Every recruitment consultant knows the routine. Open Reed, Totaljobs, Indeed and LinkedIn, search your niche, copy company names into a spreadsheet, google the website, hunt for the hiring manager, paste it all into the CRM. An hour later you have fifteen leads, half of which were posted by other agencies.

Lead scraping is the attempt to automate that routine. Done well, it gives you a daily list of companies that are hiring in your niche, without duplicates, without competitors and with a contact attached. Done badly, it gives you ten thousand rows of noise and a legal headache.

This article covers the basics. It is not a technical tutorial and it will not tell you how to get around any website's protections. If a site does not want to be read automatically, that is your answer.

## What is lead scraping in recruitment?

In agency business development, lead scraping means collecting publicly available job ads and company data automatically and turning them into sales leads. The job ad is the trigger: a company needs people now. The value comes afterwards: filtering, deduplicating, enriching and getting the result into your CRM while the vacancy is still open.

If you are new to the idea of using vacancies as a trigger, our guide on [how to find companies that are hiring](/en/blog/how-to-find-companies-that-are-hiring) covers the manual version first.

## Which sources are worth collecting?

| Source | What you get | Things to know |
|---|---|---|
| Large job boards (Indeed, Reed, Totaljobs, LinkedIn, ZipRecruiter) | Volume, many niches | Terms of use often restrict automated access, check before you start; lots of agency ads |
| Niche boards | Highly relevant roles in one field | Smaller, but fewer competitors |
| Company careers pages | Roles that never reach a board, direct employer | Every site looks different, more effort per company |
| Company registers (Companies House in the UK, state business registries in the US) | Legal name, address, officers, status | Not a hiring signal on its own, but useful for enrichment and checking |

A good rule: start with the sources where your target companies actually advertise. A healthcare agency in Leeds needs different boards from an engineering agency in Dallas.

### Check the terms of use and robots.txt first

Before you collect anything automatically, read the site's terms of use and its robots.txt file. Many large job boards restrict automated access in their terms, and some offer official feeds, APIs or partner programmes instead. The robots.txt standard (RFC 9309) describes rules that crawlers are asked to follow. The standard itself says it is not a form of access authorisation, so it does not replace the terms of use. In short: if either one says no, do not do it, and look for an official route.

## Step 1: Overview first, details only for new ads

The most common beginner mistake is opening every single ad, every single day. That is slow, it puts unnecessary load on the source and most of it is wasted, because the bulk of ads you see today were already there yesterday.

A better structure has two layers:

1. **Overview.** Collect the result lists: title, company, location, date and link. This is light and fast.
2. **Detail.** Open the full ad only when it is new or has visibly changed since your last run.

That alone cuts your effort dramatically and keeps you a polite visitor. Keep a record of every ad you have seen, with its link and the date you first saw it, so the next run knows what is new.

## Step 2: Deduplicate by company, not by ad

One company can post the same role on four boards, plus its careers page, plus a reposted version two weeks later. If you count ads, you will call the same HR manager five times.

Your unit is the company, not the ad:

- **Match on the company first.** Clean up the name (remove legal forms like Ltd, LLC or Inc and extra spaces), then match on name plus location, and ideally on the website domain once you have it.
- **Group ads under the company.** One company, several open roles. That is a stronger signal, not a duplicate.
- **Keep the history.** A role that comes back after a few weeks usually means it was not filled. That is worth knowing on the call.

## Step 3: Filter out other agencies

On large boards a big share of ads comes from recruitment agencies advertising on behalf of clients. Calling them is wasted time, and in the worst case you pitch to a competitor.

Typical signs of an agency ad:

| Sign | Example |
|---|---|
| Company name contains recruitment words | "Recruitment", "Staffing", "Talent", "Resourcing", "Personnel" |
| Vague employer description | "Our client, a leading manufacturer in the North West …" |
| Same advertiser, wildly different industries | Welders, nurses and accountants from one "company" |
| Board marks the ad as agency | Some boards label who posted the ad |

Build a blocklist of known agencies and keep adding to it; the website check in the next step catches the rest. Also drop roles outside your niche, companies far too large or small for your service, and existing clients, who belong in account management. Our guide to [no agencies please job ads](/en/blog/no-agencies-please-job-ads) explains how to handle employers who explicitly ask not to be contacted by recruiters.

## Step 4: Enrich with website and decision maker

A company name and a job title are not yet a lead. To call, you need:

1. **The website.** Confirms the company is real, shows its size and what it does, and gives you the domain for matching.
2. **The decision maker.** For most agencies that is the hiring manager for the role, the HR lead or, in smaller businesses, the owner or managing director. Our article on [who to contact in recruitment business development](/en/blog/who-to-contact-recruitment-business-development) goes through this in detail.
3. **A way to reach them.** Phone number for the switchboard or direct line. The job ad itself often names a contact person.

Do not go straight to the CEO of a group with thousands of employees. They will not own your vacancy.

## Step 5: Keep it fresh

Job ads expire. A lead for a role that was filled three weeks ago is worse than no lead, because it makes your opener wrong.

- **Recheck ads regularly** and mark them as expired once they disappear from the source.
- **Prioritise by age.** Ads from the last seven days go to the top of the call list.
- **Keep expired roles as history,** not as leads. If the company hires again, you know they hire.

## Step 6: Get it into the CRM

Many home-made setups end in a spreadsheet nobody opens. Push leads straight into your CRM with the fields you call from (open role, source, date posted, location, decision maker) and a saved filter turns them into a daily call list.

The call itself uses the vacancy as the opener:

> "Hi {first name}, it's {your name} from {agency}. I'm calling about the {role} you're hiring for in {city} – that's still open, right?"

From there, one or two questions on urgency, a yes question ("If we put a couple of strong candidates from your area in front of you, would you want to meet them?") and straight to a meeting.

## What about data protection?

This is general information, not legal advice.

- **Company data and personal data are different.** A company name and an open role are not personal data. A named hiring manager with a phone number is.
- **UK GDPR and EU GDPR.** If you collect personal data from a source other than the person, Article 14 requires you to tell them, among other things, where the data came from. If you use the data to contact them, this has to happen at the latest at the first contact; otherwise within a reasonable period and at the latest within one month.
- **US.** There is no single federal privacy law, rules vary by state. In California, the exemption for business contact data under the CCPA expired on 1 January 2023, so B2B contacts have the same rights as other consumers there.
- **Database rights.** In the UK and EU, databases can be protected by a separate database right, so copying large parts of someone else's database can be a problem even if each single entry is public.
- **Do not contact means never.** Anyone who asks not to be called again goes on a blocklist that every future import checks.

When in doubt, speak to a data protection specialist before you build anything at scale.

## Build it or have it built?

A small version is doable: a few sources, a spreadsheet, a weekly run. Keeping it running every day across dozens of sources, with deduplication, agency filters, enrichment and freshness checks, is a product of its own. That is exactly what [LeadFlow](/en/services/leadflow) does: it delivers companies that are hiring in your niche, with the decision maker, straight into your CRM every day, so your consultants spend their time on the phone instead of on research.

## Key takeaways

- Lead scraping is only step one. The value is in filtering, deduplication, enrichment and freshness.
- Collect overviews first and open details only for new or changed ads.
- Your unit is the company, not the ad. Group ads under one company.
- Filter out other agencies early, with a name filter and a website check.
- Respect terms of use and robots.txt, never work around protections, and handle personal data in line with GDPR or local law.
- Push leads straight into the CRM and call while the role is still open.

## FAQ

### Is lead scraping legal for recruitment agencies?
It depends on the source, the data and the country. Respect each site's terms of use and robots.txt, do not work around technical protections, and handle personal data such as a named contact in line with UK GDPR, EU GDPR or the relevant US state law. This is not legal advice, so check your specific setup with a specialist.

### What is the best source for recruitment leads?
The best source is wherever your target companies advertise. For most agencies that is a mix of large job boards, niche boards for their specialism and company careers pages, because many roles never reach a board. Company registers help with checking and enriching, but are not a hiring signal on their own.

### How do I avoid calling the same company twice?
Deduplicate by company instead of by job ad. Clean up the company name, match on name plus location and later on the website domain, and group all ads from one company under one CRM record.

### How do I remove recruitment agencies from scraped job ads?
Use a name filter for typical words such as recruitment, staffing or resourcing, keep a growing blocklist of known agencies and check the website of anything that looks vague. Ads that say "our client" are almost always from an agency.