API reference

All symbols below are exported from @chrisdothtml/job-scraper, including the types. Start with guides for task-oriented examples. Network APIs may fail because of transport errors or board changes; the error classes below describe failures callers can distinguish.

Jobs

listCompanyJobs

listCompanyJobs(input: CompanyInput | string): Promise<ListedJob[]>

Lists all currently posted jobs by company name, careers URL, or explicit scraper and slug. Resolves the company first and disposes its scraper afterward. Throws UnresolvedCompanyError if resolution fails; invalid input can throw TypeError. Board requests can throw HttpError or CorsError.

import { listCompanyJobs } from '@chrisdothtml/job-scraper';

// resolved from the registry, or discovered and remembered for next time
await listCompanyJobs('Airbnb');

import { listCompanyJobs } from '@chrisdothtml/job-scraper';

// a board URL, or a company's own careers page with a board embedded in it
await listCompanyJobs('https://jobs.ashbyhq.com/zapier');
await listCompanyJobs('https://careers.duolingo.com');

import { listCompanyJobs } from '@chrisdothtml/job-scraper';

// pin the board yourself
await listCompanyJobs({ scraper: 'GreenhouseScraper', slug: 'figma' });

// or pin just the board, and let the slug be discovered
await listCompanyJobs({ company: 'Ramp', scraper: 'GreenhouseScraper' });

fetchJob

fetchJob(input: FetchJobInput | string): Promise<Job>

Fetches one full posting. A string must be a posting URL carrying an id, or use object input with id. An explicit id overrides the id parsed from a URL. Throws TypeError when no id can be recovered, plus resolution and request errors. A landing page does not select an arbitrary job linked from its markup. Missing postings can raise JobNotFoundError or an HTTP 404/410.

import { fetchJob } from '@chrisdothtml/job-scraper';

await fetchJob('https://job-boards.greenhouse.io/airbnb/jobs/7712345');

import { fetchJob } from '@chrisdothtml/job-scraper';

// boards embedded on the company's own domain work too
await fetchJob('https://careers.airbnb.com/positions/8184174?gh_jid=8184174');

import { fetchJob } from '@chrisdothtml/job-scraper';

// a company (or `url`, or `scraper` + `slug`) plus the job's id
await fetchJob({ company: 'Airbnb', id: '7712345' });

FetchJobInput

interface FetchJobInput extends CompanyInput {
  id?: string;
}

The board-specific job id is required unless the supplied URL carries it. Inherits all resolution options from CompanyInput.

ListedJob

interface ListedJob {
  id: string;
  title: string;
  location: string;
  url: string;
}

A listing with its board-specific stable id, title, location and URL. Fields unavailable from source data can be empty strings.

Job

interface Job extends ListedJob {
  content: string;
}

The full posting. content is the board’s original string: often serialized JSON, sometimes Markdown, plaintext or HTML. Use the rendering helpers to create readable text and retain the original for provenance.

JobNotFoundError

class JobNotFoundError extends Error {
  jobId: string;
  companySlug: string;
}

A scraper could not find the requested id in its board’s postings. Distinct from network, quota or server failures; those do not prove a posting has closed.

Rendering and text

renderScrapedPosting

renderScrapedPosting(job: Job, options?: RenderPostingOptions): RenderedPosting

Pure, runtime-independent renderer returning { markdown, body, raw }. The default complete mode accounts for every parsed JSON field, including nested containers, empty values and scalars, with JSON-pointer labels and an Additional posting data section. It also includes exact outer job fields in Source job metadata. HTML fields become readable Markdown. Compact mode selects posting prose, filters application/legal/metadata-like keys, and deduplicates overlapping sections; it can omit useful source fields. Plain strings need at least 200 characters and whitespace to count as compact body prose; rich HTML sections can be shorter. Non-JSON content is converted as rich text. postingUrl overrides the displayed posting link, useful when the scraper’s URL is an application link.

raw is the original job object; raw.content preserves the exact input string and the renderer does not mutate it. Markdown is a readable representation, not a lossless serialization or HTML sanitizer. JSON parsing itself can normalize numeric representation and duplicate object keys. Preserve raw whenever exact source fidelity matters. See rendering for LLM input.

// A self-contained example: replace this synthetic job with await fetchJob(url).
import { renderScrapedPosting, type Job } from '@chrisdothtml/job-scraper';

const job: Job = {
  id: 'example-1',
  title: 'Software Engineer',
  location: 'Remote',
  url: 'https://example.com/apply/example-1',
  content: JSON.stringify({
    company_name: 'Example Company',
    description: '<h2>About the role</h2><p>Build useful software.</p>',
    benefits: ['Flexible hours'],
    application_questions: [],
  }),
};

const complete = renderScrapedPosting(job); // mode: 'complete' by default
const compact = renderScrapedPosting(job, {
  mode: 'compact',
  postingUrl: 'https://example.com/jobs/example-1',
});

console.log(complete.markdown); // readable posting plus all parsed source fields
console.log(compact.body); // selected, deduplicated posting prose
console.log(complete.raw.content === job.content); // true: exact source retained

RenderPostingOptions

type RenderPostingOptions = {
  mode?: 'complete' | 'compact';
  postingUrl?: string;
};

mode defaults to complete. postingUrl defaults to job.url and affects the displayed link, not raw.

RenderedPosting

type RenderedPosting = { markdown: string; body: string; raw: Job };

markdown includes the title, available company/location/compensation/posting metadata, and body, with a trailing newline. body contains the prose and complete-mode source accounting. raw is the unchanged input object.

richTextToMarkdown

richTextToMarkdown(text: string): string

Accepts HTML, escaped HTML, Markdown or plain text. Decodes HTML entities when escaped markup is detected, converts markup, normalizes line endings and excess whitespace, and trims the result.

htmlToMarkdown

htmlToMarkdown(html: string): string

Forgiving HTML conversion retaining headings, paragraphs, lists, links, emphasis, code, blockquotes and tables as readable Markdown. Drops scripts/styles and other non-posting elements; this is not a sanitizer and does not preserve exact markup.

htmlToText

htmlToText(html: string): string

Renders HTML to plain text using the same structural walk: lists and table rows remain readable, links keep their text. Plain text has whitespace tidied. Intended for text consumers, not sanitizing HTML for insertion into a page.

decodeHtmlEntities

decodeHtmlEntities(text: string): string

Decodes supported named and numeric HTML entities. Unsupported entities remain unchanged.

looksLikeHtml

looksLikeHtml(text: string): boolean

Heuristic markup detection used by rich-text conversion; not validation of an HTML document.

looksLikeEscapedHtml

looksLikeEscapedHtml(text: string): boolean

Heuristic detection of entity-escaped HTML such as <p>. Does not convert the input itself.

Resolution

resolveCompany

resolveCompany(input: CompanyInput | string): Promise<ResolvedCompany>

Resolves a name, URL, or explicit board to a company without fetching a full job. Discovery can still make board-list, page and search requests and persist registry entries. Throws TypeError for missing input or an unknown scraper. If no usable board is found, a recorded fallback page-shape failure (BoardShapeError) is propagated separately; otherwise resolution throws UnresolvedCompanyError. See resolution order.

import {
  listCompanies,
  registerCompany,
  resolveCompany,
} from '@chrisdothtml/job-scraper';

await resolveCompany('Klaviyo');
// { name: 'Klaviyo', scraper: 'GreenhouseScraper', slug: 'klaviyo', discovered: true }

await listCompanies(); // everything currently known
await registerCompany({ name: 'Acme', scraper: 'LeverScraper', slug: 'acme' });

resolveJob

resolveJob(input: CompanyInput | string): Promise<ResolvedJob>

Runs the same resolution process and includes a recovered job id, or null for company-level input. Does not fetch a full posting. Embedded page links identify the board; their ids do not override the caller’s posting id.

CompanyInput

interface CompanyInput {
  company?: string;
  url?: string;
  scraper?: ScraperName;
  slug?: string;
  search?: boolean;
  minJobs?: number;
}

Use company, url, or a board slug; { scraper, slug } skips discovery. scraper can also restrict resolution to a board. search defaults to allowed when a SerpApi key is configured; false skips it for this call. minJobs defaults to 3 real postings for probing/search only; explicit URLs and registry entries are trusted without that threshold.

ResolvedCompany

interface ResolvedCompany extends Company {
  discovered: boolean;
}

A company with a concrete scraper and slug. discovered reports discovery rather than a direct URL, explicit board or registry lookup.

ResolvedJob

interface ResolvedJob extends ResolvedCompany {
  jobId: string | null;
}

A resolved company plus an id when the input addresses a single posting.

UnresolvedCompanyError

class UnresolvedCompanyError extends Error {
  input: string;
  blocked: boolean;
}

No scraper could serve the input. In browsers, blocked indicates at least one attempted page or board request failed before a response, so server-side resolution may succeed. A recorded board fallback page-shape failure is thrown separately as BoardShapeError before this error is constructed. BoardShapeError is not a package-root export.

parseJobUrl

parseJobUrl(input: string, only?: ScraperName): (ResolvedJob & { discovered: false }) | null

Pure URL parsing through the supported scrapers, optionally restricted to one scraper. Returns null for invalid or unrecognized URLs. Does not verify that the board or posting exists.

createScraper

createScraper(company: Company): Scraper

Constructs the board scraper from the record’s scraper name and slug. Caller owns disposal. Throws for an unknown scraper.

boardHints

boardHints(url: URL): ScraperName[]

Returns scrapers hinted at by recognized query parameters, such as Greenhouse’s gh_jid. A hint is not verification.

companyNameCandidates

companyNameCandidates(name: string): string[]

Returns the original name and, when different, a version stripped of careers/jobs prefixes and suffixes, in order.

companyNameFromUrl

companyNameFromUrl(url: URL): string | null

Best-effort name from the hostname after removing common board subdomains. It is a heuristic, not public-suffix parsing or verified identity.

jobIdFromUrl

jobIdFromUrl(url: URL): string | null

Best-effort id from gh_jid, pid, jobId, job_id, or lever-job-id, then an all-digit final path segment. Returns null otherwise.

Registry

Company

interface Company {
  name: string;
  scraper: ScraperName;
  slug: string;
  domains?: string[];
  homepage?: string;
}

Display name and board identity. domains are company-owned careers hostnames, never shared board hosts. homepage is the company’s website origin, distinct from its careers board.

StoredCompany

interface StoredCompany extends Company {
  source: 'seed' | 'discovered';
  homepageMiss?: HomepageMiss;
}

Persistent registry record with provenance and optionally a remembered failed homepage lookup.

HomepageMiss

interface HomepageMiss {
  ts: number;
  searched: boolean;
  version: string;
}

Homepage miss timestamp (milliseconds), whether web search was included, and package version. Used to avoid repeated lookups while allowing retries after expiry or an upgrade.

seedCompanies

const seedCompanies: Company[];

Bundled company records used to initialize and reconcile the registry. Runtime discoveries are stored separately, not written into this shipped list.

loadCompanies

loadCompanies(): Promise<Map<string, StoredCompany>>

Loads, seeds or reconciles persistent state, then returns the in-memory map keyed by normalized company names. Treat returned records as registry state; use the mutation APIs to persist changes.

listCompanies

listCompanies(): Promise<StoredCompany[]>

Returns all known companies sorted by display name, including bundled and discovered entries.

findCompany

findCompany(name: string): Promise<StoredCompany | null>

Looks up a display name or normalized key without probing boards. Returns null when absent.

findCompanyByDomain

findCompanyByDomain(hostname: string): Promise<StoredCompany | null>

Looks up a known careers hostname, checking parents from most specific to least. Shared job-board hosts are excluded.

registerCompany

registerCompany(company: Company, source?: StoredCompany['source']): Promise<StoredCompany>

Persists a record; source defaults to discovered. Re-registering replaces the existing normalized-name entry. Throws Error for an invalid company record.

addCompanyDomain

addCompanyDomain(name: string, hostname: string): Promise<StoredCompany | null>

Normalizes and remembers a company-owned careers domain. Returns null for invalid/shared board hosts or an unknown company. Existing domain ownership is retained; repeat additions do not duplicate entries.

setCompanyHomepage

setCompanyHomepage(name: string, outcome: { homepage: string } | { miss: HomepageMiss }): Promise<StoredCompany | null>

Persists either a homepage or a failed lookup. A successful homepage clears the previous miss. Returns null for an unknown company.

clearCompaniesCache

clearCompaniesCache(): void

Drops the in-memory registry and domain index so the next load rereads storage. Does not delete the persisted registry.

mergeRegistry

mergeRegistry(stored: StoredCompany[]): StoredCompany[]

Pure reconciliation against bundled records. Shipped board identities win for covered names; user discoveries absent from the seed survive; removed seed entries disappear. Preserves supported stored enrichment when merging. See registry upgrades.

normalizeCompanyName

normalizeCompanyName(input: string): string

Builds the registry key by folding case/diacritics, removing punctuation and trailing legal suffixes. For example, Apollo.io, Inc. becomes apolloio.

normalizeDomain

normalizeDomain(input: string): string

Accepts a URL or hostname; lowercases the host, removes a port and leading www., and strips outer dots.

isBoardDomain

isBoardDomain(hostname: string): boolean

Checks a normalized host against shared board domains, including tenant subdomains. Prevents a company claiming an entire shared board host.

Homepages

resolveHomepage

resolveHomepage(input: Company | string, options?: HomepageOptions): Promise<string | null>

Returns the company website origin, not its careers URL. String input is looked up/resolved first, so unresolved-company errors can propagate. Uses remembered homepages/misses and persists new outcomes for registered companies. Individual homepage sources fail best-effort; null means no supported source settled it. The object input is a full Company record, not arbitrary CompanyInput.

import { configureSearch, resolveHomepage } from '@chrisdothtml/job-scraper';

await resolveHomepage('Ramp'); // 'https://ramp.com'
await resolveHomepage('https://jobs.ashbyhq.com/zapier'); // anything resolveCompany takes

// let it fall back to a web search when the free sources aren't conclusive
configureSearch({ apiKey: 'your SerpApi key', homepages: true });

// e.g. a company's icon, via Google's favicon service
const homepage = await resolveHomepage('Ramp');
const icon =
  homepage &&
  `https://www.google.com/s2/favicons?domain=${new URL(homepage).hostname}&sz=64`;

findHomepage

findHomepage(company: Company, options?: HomepageOptions): Promise<FoundHomepage | null>

Runs homepage discovery directly and returns its source. Does not use or persist registry outcomes itself. Sources are best-effort; see homepage verification and browser limits.

HomepageOptions

interface HomepageOptions {
  search?: boolean;
}

Controls optional web search. Defaults to SearchConfig.homepages (false); needs a configured key and remaining quota.

HomepageSource

type HomepageSource =
  | 'board'
  | 'careers-domain'
  | 'clearbit'
  | 'wikidata'
  | 'guess'
  | 'search';

Source of a discovered homepage, distinguishing a board-declared website from candidate lookups.

FoundHomepage

interface FoundHomepage {
  url: string;
  source: HomepageSource;
}

Website origin and where it came from. Returned by findHomepage.

Caching and request errors

configureCache

configureCache(config: Partial<CacheConfig>): void

Merges process-wide overrides. Explicit undefined drops an override, restoring environment/default fallback. Configuration can disable caching, but cannot force caching for calls that opted out.

import { clearCache, configureCache, time } from '@chrisdothtml/job-scraper';

configureCache({ ttl: time.hour }); // override every scraper's TTL
configureCache({ enabled: false }); // always hit the network
configureCache({ dir: '/tmp/jobs' }); // put it somewhere else
configureCache({ ttl: undefined }); // drop the override again

await clearCache(); // returns how many entries went

getCacheConfig

getCacheConfig(): CacheConfig

Returns effective settings: explicit override → Node environment → defaults. See cache policy.

CacheConfig

interface CacheConfig {
  enabled: boolean;
  ttl: number | null;
  dir: string;
}

enabled defaults true. ttl is milliseconds; null preserves each scraper’s choice (usually one day). dir defaults to the cache path; in browsers it names the Cache API cache.

clearCache

clearCache(): Promise<number>

Deletes cached board responses from the currently configured cache directory/name and returns the number removed. Does not clear registry or search state.

HttpError

class HttpError extends Error {
  status: number;
  statusText: string;
  url: string;
}

A board request returned a non-2xx status. Use status to distinguish a missing posting from rate limits or server failures. Failed responses are not cached.

CorsError

class CorsError extends Error {
  url: string;
}

Browser request failed before a response, most likely CORS; plain network failure looks identical. Not thrown under Node, which does not enforce browser CORS.

Search

configureSearch

configureSearch(config: Partial<SearchConfig>): void

Merges process-wide SerpApi settings; explicit undefined removes an override. Search remains optional and needs an API key.

import { configureSearch, resolveCompany } from '@chrisdothtml/job-scraper';

configureSearch({
  apiKey: 'your SerpApi key',
  limit: 100,
  period: 'month',
});

await resolveCompany('Whatnot');
// { name: 'Whatnot', scraper: 'WorkdayScraper', slug: 'whatnot.wd1.Whatnot', discovered: true }

getSearchConfig

getSearchConfig(): SearchConfig

Returns effective overrides, Node environment and defaults. Browsers only use explicit settings/defaults.

SearchConfig

interface SearchConfig {
  apiKey: string | null;
  limit: number;
  period: 'day' | 'month';
  resultCount: number;
  resultTtl: number;
  batchSize: number;
  homepages: boolean;
}

Defaults: no key (unless Node environment supplies one), 250/month, 10 results, 30 days result TTL in milliseconds, 5 companies per batch, homepage search false. See search quota and environment.

canSearch

canSearch(): Promise<boolean>

Reports whether a key is configured and the current quota period has searches remaining. Does not make a web search or reserve quota.

searchForBoards

searchForBoards(companyName: string): Promise<SearchedBoard[]>

Searches supported board domains for one company. New queries are skipped when disabled or out of quota; already remembered results can still be reused. Returns an empty array when no matching results are available. Results are candidates: resolution additionally verifies board activity before registration.

searchForBoardsBatch

searchForBoardsBatch(companyNames: string[]): Promise<BatchedSearch[]>

Reuses cached results and searches remaining names in OR queries using batchSize names and up to 100 results per query. Results are attributed by board slug. Concurrent equivalent searches share work.

import {
  clearSearchCache,
  searchForBoardsBatch,
} from '@chrisdothtml/job-scraper';

await searchForBoardsBatch(['Klaviyo', 'Notion', 'Whatnot']); // one search
await clearSearchCache(); // start over

searchForHomepage

searchForHomepage(companyName: string): Promise<string[]>

Searches for official-website URL candidates. Does not verify ownership or persist a homepage outcome. Returns no results when search cannot run; homepage resolution performs verification.

SearchedBoard

interface SearchedBoard extends ParsedUrl {
  scraper: ScraperName;
  url: string;
}

Parsed board candidate and original search-result URL, including nullable posting id.

BatchedSearch

interface BatchedSearch {
  companyName: string;
  boards: SearchedBoard[];
}

Per-company board-search results, including an empty list for a miss.

clearSearchCache

clearSearchCache(): Promise<number>

Clears remembered board-search results and in-flight bookkeeping, returning the number of cached company keys removed. Does not reset quota accounting.

Page discovery

findBoardsInPage

findBoardsInPage(html: string): SniffedBoard[]

Pure markup inspection: extracts board URLs using scraper parsers, groups by scraper/slug, and sorts by reference count. Examines up to two million characters. Does not execute JavaScript or verify board activity.

findNamesInPage

findNamesInPage(html: string): string[]

Pure extraction of company-name candidates from og:site_name, hiring-organization JSON-LD and application-name metadata, adding careers-suffix-stripped forms. Results are ordered and deduplicated; arbitrary page titles are not treated as company identity.

sniffPage

sniffPage(url: string): Promise<SniffedPage>

Fetches a careers page best-effort and extracts boards/names. Fetch failures return empty arrays; blocked distinguishes browser pre-response failures. Does not run page JavaScript.

sniffBoards

sniffBoards(url: string): Promise<SniffedBoard[]>

Convenience wrapper returning only sniffPage(url).boards; failure yields an empty array without the blocked diagnostic.

SniffedBoard

interface SniffedBoard extends ParsedUrl {
  scraper: ScraperName;
  hits: number;
}

A board reference recovered from markup with its count and nullable job id.

SniffedPage

interface SniffedPage {
  boards: SniffedBoard[];
  names: string[];
  blocked: boolean;
}

Boards most-referenced first, name candidates most trustworthy first, and whether a browser fetch failed before any response.

Scraper classes and registry

Scraper

abstract class Scraper {
  constructor(companySlug: string);
  static readonly discoverable: boolean;
  static readonly boardDomains: readonly string[];
  static readonly urlHintParams: readonly string[];
  static slugCandidates(companyName: string): string[];
  static parseUrl(url: URL): ParsedUrl | null;
  static fetchHomepage(
    slug: string,
    signal?: AbortSignal
  ): Promise<string | null>;
  getJobsList(testing?: boolean): Promise<ListedJob[]>;
  getJobContent(id: string): Promise<string>;
  getJob(id: string): Promise<Job>;
  jobUrl(id: string): string | null;
  hasCompanyBoard(minJobs?: number): Promise<boolean>;
  dispose(): void;
}

Base scraper class. getJob combines full content with available listing fields. Reuse one instance for bulk jobs so board-list caches can be shared, and dispose in finally. testing requests a minimal listing for probes, not necessarily exactly one job. hasCompanyBoard defaults to 1 real posting (the resolution pipeline explicitly passes its separate default of 3), returns false for ordinary board misses, and lets fallback-shape/browser-block failures propagate. Static capabilities include discoverable, boardDomains, urlHintParams, slugCandidates, parseUrl and best-effort fetchHomepage. Symbol.dispose is attached only when the runtime supports it; otherwise call dispose() explicitly. See supported boards.

import { scrapers } from '@chrisdothtml/job-scraper';

using scraper = new scrapers.AshbyScraper('zapier');
const jobs = await scraper.getJobsList();
for (const job of jobs) await scraper.getJob(job.id); // one list fetch, not one per job

ScraperSubclass

interface ScraperSubclass {
  new (companySlug: string): Scraper;
  readonly name: string;
  readonly discoverable: boolean;
  readonly boardDomains: readonly string[];
  readonly urlHintParams: readonly string[];
  slugCandidates(companyName: string): string[];
  parseUrl(url: URL): ParsedUrl | null;
}

Constructor/static-capability shape used by the scraper registry.

ParsedUrl

interface ParsedUrl {
  slug: string;
  jobId: string | null;
}

Board identity recovered from a URL, with an optional posting id. Parsing alone does not confirm existence.

ScraperName

type ScraperName = keyof typeof scrapers;

Case-sensitive union of supported constructor names, such as GreenhouseScraper and WorkdayScraper. Individual constructors are available as properties of scrapers.

scrapers

const scrapers: {
  /* each ScraperName maps to its constructor */
};

Supported constructors keyed by scraper name. Use new scrapers.AshbyScraper("zapier"), or getScraper for a dynamic name. The boards guide lists every supported constructor and its capabilities.

import { scrapers } from '@chrisdothtml/job-scraper';

using scraper = new scrapers.WorkdayScraper('autodesk.wd1.Ext');
const jobs = await scraper.getJobsList();
const job = await scraper.getJob(jobs[0]!.id);

scraperNames

const scraperNames: ScraperName[];

All registered scraper names, in registry order.

discoverableScrapers

const discoverableScrapers: ScraperName[];

Names of scrapers whose board slugs can be guessed from company names. Other scrapers remain usable by URL, registry or explicit board identity.

getScraper

getScraper(name: string): ScraperSubclass

Returns a constructor for a supported name. Throws Error with Unknown scraper: ... for an unknown name.

isScraperName

isScraperName(name: string): name is ScraperName

Type guard for supported scraper names.

Paths and miscellaneous

dataDir

const dataDir: string;

Mutable-state root: Node defaults to ~/.job-scraper, overridden by JOB_SCRAPER_DATA_DIR. In browsers this is a storage identifier, not a filesystem path. See state storage.

cacheDir

const cacheDir: string;

Default board-response cache directory under the data root. In browsers this is a storage identifier, not a filesystem path. See state storage.

companiesFile

const companiesFile: string;

Persistent registry location under the data root. In browsers this is a storage identifier, not a filesystem path. See state storage.

time

const time: {
  readonly second: 1000;
  readonly minute: number;
  readonly hour: number;
  readonly day: number;
};

Millisecond durations: second = 1,000; minute = 60,000; hour = 3,600,000; day = 86,400,000. Useful for cache/search TTL configuration.