ten portals, one morning

every weekday, before anyone at landinc opens their laptop, ten procurement portals have already been read. it took a few months to get there, and almost none of the work was the scraping itself.

every portal is its own country

one portal paginates with a ?page= parameter. another loads results over a websocket. one logs you out after four minutes; one puts the closing date in the title, the body, and an attached pdf, and they don't always agree. treating them as one problem was my first mistake.

so each portal gets its own watcher: a small playwright job that knows that site's login, its search, and its markup, and nothing else. they share one contract, a list of raw postings, and one table to write to.

@watcher("bids-and-tenders", every="1d")
async def run(page: Page) -> list[RawPosting]:
    await login(page, secrets.portal("bids-and-tenders"))
    await page.goto(SEARCH_URL, wait_until="networkidle")
    rows = await page.locator("table.results tr").all()
    return [await parse_row(r) for r in rows[1:]]

fail loudly, fail alone

portals change their markup without warning. the rule is that a watcher which finds zero rows, or rows without a closing date, fails instead of returning an empty list. an empty list looks like a quiet day. a failure pages me.

an empty result and a broken scraper look the same in a dashboard. make them look different.

because each watcher runs alone, one broken portal is one red row in the morning report, and the other nine still feed the digest.

what i'd tell myself at the start

• write the per-site code first; abstract only what three sites turned out to share.
• store the raw html next to every parsed row. you will want to re-parse history.
• count rows per portal per day and alert on the drop, not just on exceptions.

the scoring and the digest get the attention in demos. the watchers are why anyone trusts them.