Creator database of 800M+ profiles split by the has_email filter into 300M with email and 500M without
Data infrastructure

The challenges of opening search to 800M+ creators

Discovery used to show only creators with an email. What the data side already had, what the product assumed, and what a database this size does to a query.

Until this month, Discovery only showed creators with an email. The rest of the creator database, more than 500M profiles, was there but unsearchable.

Opening search to all 800M+ creators took less data work than you’d expect, and more product work. Here’s what the data side already had, what the product quietly assumed, and what a database this size does to a query.

Built for outreach

Influencers.Club started as an outreach tool. You found creators, got their contact details and reached out. A creator you couldn’t email was a creator you couldn’t use, so email became more than one field among many. Search, lists, exports and billing were all built around it.

Then people started using the product differently: looking up creators they already work with, tracking campaigns and pulling profile data into their own systems through the API. For that work, an email is useful but optional. We kept hearing the same complaint: “I searched for three creators I work with and none of them are in your database.” They were there. They just didn’t have an email.

Coverage
Same database, different gate
The profiles were already in the database. Turning off the email gate made them searchable.
Beforeemail required
300M500M+ not searchable
AfterHas email off
300M+500M without email

Why the data side was ready

Serving 500M more creators means those profiles have to be complete, current and linked across platforms, not just stored. Four pieces of earlier work made that true.

  • A pipeline that scales out.Rebuilding our pipeline on Kafka and Prefect replaced one-at-a-time processing with parallel workers and cut a 100k-creator enrichment job from about 13 hours to about one.
  • Profiles built from independent writers.Moving profile processing to ClickHouse made every metric its own datapoint, merged into one current profile per creator per platform. Those datapoints already covered every profile, contact details or not, so opening 500M more to search added no write load and no new jobs.
  • An identity that isn’t an email.Identity resolution links a creator’s accounts across 9 platforms, keyed on platform accounts. A creator without an email is still a complete record, with their other platforms attached.
  • One contract for what a creator is.The data team and the product share a versioned schema library of typed Pydantic models. It generates the Elasticsearch mapping, validates every record during the build and stamps each index with its schema version. A breaking change can’t be promoted until every service that reads the index has upgraded. So when the index more than doubled, both sides still agreed on exactly what a creator record contains.

The profiles were ready. The product built on top of them wasn’t.

Email was the primary key, and nobody had written that down

We audited every code path that touches a creator and asked how it tells one creator from another. Almost everywhere, the answer was the email address.

WhereWhat it assumedWhat broke without an email
Export deduplicationSame email = same creatorTwo creators sharing a manager’s address collapsed into one row
Exclusion listsExcluded creators are stored as emailsCreators without an email couldn’t be reliably excluded
List import“Email is mandatory”Customers couldn’t upload a list of handles or account IDs
Bulk exportInvalid email = swap in another matching creatorCreators the customer picked were silently replaced

None of these were bugs when we wrote them. Each was a reasonable shortcut, since every creator reaching that code had an email.

A new identity order

We replaced all of those shortcuts with one rule for deciding whether two references point to the same creator. Every one of those code paths now uses it:

# simplified
def identity_key(ref):
    if ref.platform and ref.user_id:
        return ("id", ref.platform, ref.user_id)
    if ref.email:
        return ("email", ref.email.lower())
    if ref.platform and ref.handle:
        return ("handle", ref.platform, normalize_handle(ref.handle))
    raise InvalidCreatorRef(ref)

The platform account ID comes first because it’s the only identifier that doesn’t change. Handles get renamed. Emails get shared. That’s why every export now includes platform and user_id.

has_email: one filter, a default and a contract

The visible part of the change is a single Discovery filter. Deciding exactly what it should mean took longer than building it.

Turning off the Has email filter in Discovery and exporting creators with and without an email
  • On by default.Search works exactly as before, so no saved search changed on launch day.
  • Off means “everyone”, not “only creators without an email”.People want every creator who fits their search.
  • No more replacements.With the filter off, a creator whose email fails validation is exported without it instead of being swapped for someone else.
  • AI search understands it.“Creators who have an email” turns into the filter automatically.

The API default was the hardest call. Integrations that never send has_email were written against a contract where every result had an email. Flipping the default would have changed their data without changing their code. So the default stays on, every result includes has_email, and we told API customers before the change shipped. The whole build sat behind a per-account feature flag through a full testing cycle before it went live.

Enrichment, too

Enrichment follows the same rules. Enrich by handle without marking the email as required, single or in batch, and you get the full profile whether or not we hold an email.

Enrichment also reads the creator database directly. A handle lookup combines the stored profile with a live refresh: fresh values win where they exist, the stored profile fills the rest, and the stored profile answers on its own if the live refresh comes back empty or is unavailable. Imported lists are matched to creators in the background the same way, by account ID, email or handle.

What 800M does to a query

Before launch, we replayed 200 real saved searches against the full index. The disks were mostly idle. The bottleneck was memory, once the index grew much larger than each node’s RAM. That’s why creator search now runs on its own cluster.

The most expensive lesson came from what customers send along with a search. Every search goes to all 60 primary shards, and each shard builds its own copy of the query. On November 4, a burst of searches each carried a customer’s email suppression list inline: about 848,000 addresses, split into chunks just under Elasticsearch’s terms limit and sent twice. That’s about 48 MB per request, and about 2.9 GB once copied to every shard. Around ten requests in flight filled the heap, and the nodes died with OutOfMemoryError. No guardrail tripped, because each request stayed just under every limit.

Eight days later, a bug in our own query builder copied an 80,000-handle exclusion list into the rescore phase. The alerts added after November 4 flagged it 85 minutes before search went down, and the fix shipped the same day.

A bigger database doesn’t just mean bigger results. It means bigger questions, and the questions are what fill the heap.

What we’d still change

  • Exclusions by reference, not by value.Stored exclusion lists should be applied with a terms lookup, or written onto creator documents as a keyword field like suppressed_for. Then a search sends a customer ID instead of a list. A 48 MB request becomes a few bytes, and the cost is paid once when the list is saved, not on every query.
  • Limits per search and per client.A combined cap on excluded values per search, plus a concurrency limit per client, so one integration can’t overload the cluster for everyone.
  • Tests for what goes into a query.Nothing checked the contents of the rescore query, which is how a duplicated list reached production. Tests on query structure are cheap compared to an outage.

The data side was the easy part, because the pipeline, the ClickHouse profile layer and identity resolution were already built for every creator. What took a quarter was finding every place the product had quietly decided what counts as a creator, and changing each one without affecting customers who never asked.

What this means for customers

  • The creator you’re looking for is there.Turn off Has email and search covers 800M+ creators.
  • Nothing changes unless you want it to.Has email stays on by default in the dashboard and the API.
  • Bring your own list, in any shape.Upload and enrich creators by handle or account ID, not just email.
  • Same data, same price.A creator without an email exports with their full profile and platform user ID, for the same number of credits.

Build on the creator data layer

800M+ creators, 300M with verified emails, 100+ data points per profile via API, dashboard or MCP server.

More from engineering

All articles →