scrape-kdl: Documentation for hsblabs/scrape-kdl.
# Scraping KDL
> A declarative language and runtime for HTML extraction. Write an extractor in KDL, let the compiler check it before any network operation, then execute it with HTTP or a browser adapter from Go, Node.js, or Bun.
Scraping KDL is a declarative language and a runtime for HTML extraction. You write an extractor in [KDL](https://kdl.dev/). The compiler resolves the imports, checks the types, and makes a language-neutral Validated IR. A runtime then executes the IR with HTTP or a live browser. Scraping KDL is for developers who keep their extraction rules in source control. The compiler finds the errors before the runtime sends a request. The reference implementation is available for Go, Node.js, and Bun. The license is Apache-2.0. ## How it operates [Section titled “How it operates”](#how-it-operates)
```text
KDL source
-> parser
-> semantic validation and type checking
-> Validated IR
-> HTTP runtime or browser adapter
-> structured result
```
The compiler does not send a request, start a browser, or call an external transform. These operations occur only after the validation is correct. A bad selector, an unknown transform, or a wrong type becomes a diagnostic with a source location. ## What you write [Section titled “What you write”](#what-you-write)
```kdl
extractor "basic-http" version="2026-07-15" language-version="2026-07-15" {
source "html" {
fetch mode="http" url="https://example.invalid/{id}"
}
input "id" type="string" required=#true
field "title" type="string" required=#true {
select "h1" match="one"
value "text"
apply "normalize-whitespace"
}
collection "items" min-items=1 {
select "ul.items > li"
field "value" type="u8" required=#true {
select ".value" match="one"
value "text"
apply "trim"
apply "parse-int" as="u8"
}
}
}
```
## What you get [Section titled “What you get”](#what-you-get)
```json
{
"value": {
"items": [{ "value": 1 }, { "value": 2 }, { "value": 3 }],
"title": "Scraping KDL Runtime"
},
"warnings": [],
"partial": false
}
```
The runtime checks each value against its declared type. The `partial` flag tells you if the runtime recovered an error. Thus you cannot mistake a degraded result for a correct result. ## Main properties [Section titled “Main properties”](#main-properties) * **The compiler operates first.** It checks the selectors, the transform signatures, the capabilities, and the output types. No data leaves the process before this step. * **The browser is a capability.** Browser mode needs an adapter that you supply. JavaScript stays off until you set the option. * **One program, two runtimes.** Go and TypeScript accept the same documents. They give the same diagnostics and the same values. * **Portable selectors.** A documented subset of CSS operates in the same manner in the internal DOM and in a live browser. * **Offline execution.** A program without a workflow and without JavaScript can operate on a saved HTML file. No network operation occurs. ## Start here [Section titled “Start here”](#start-here) * [Installation](./getting-started/installation/) — the Go CLI, the Go modules, npm, and Bun. * [Quick Start](./getting-started/quick-start/) — validate, compile, and extract from a saved HTML file. * [How It Operates](./guides/how-it-operates/) — the compiler stages and their sequence. * [Write an Extractor](./guides/write-an-extractor/) — the structure of an extractor document. Then select your runtime: [CLI](./cli/), [TypeScript and Bun](./npm/), or [Go](./golang/). ## Before you use a live service [Section titled “Before you use a live service”](#before-you-use-a-live-service) Scraping KDL is an extraction tool. It does not give you permission to access or to re-use content of other persons. Read [Security and Responsible Use](./guides/security-and-responsible-use/) before you send a request to a service that you do not operate. The project does not supply functions that bypass access controls, rate limits, or bot detection.
# CLI
> The contract of the scrape-kdl command line — the four commands, the options, the standard streams, the JSON envelopes, the exit statuses, and the handling of the secrets.
The Go binary `scrape-kdl` has four commands: `validate`, `compile`, `extract`, and `version`. It is the only command-line distribution of version 1. The TypeScript packages have no CLI. The go-rod adapter has its own binary, `scrape-kdl-rod`, for browser mode. Refer to [go-rod Adapter](../golang/rod/). ## The commands [Section titled “The commands”](#the-commands)
```bash
scrape-kdl validate extractor.kdl
scrape-kdl compile extractor.kdl --out extractor.ir.json
scrape-kdl extract extractor.kdl --input id=42
scrape-kdl version
```
Each command and also the root accept `-h` and `--help`. The help goes to the standard output and the exit status is 0. `scrape-kdl help ` is equivalent. An absent command or an absent argument writes a short usage text to the standard error and gives the exit status 2. The CLI never asks you for the absent input. ## `validate` [Section titled “validate”](#validate)
```text
scrape-kdl validate [--json]
```
It parses the document, resolves the symbols, checks the types, and calculates the capabilities. It causes no network activity and no browser activity. ## `compile` [Section titled “compile”](#compile)
```text
scrape-kdl compile [--json] [-o file.json|-]
```
| Option | Function | | ------------------ | ---------------------------------------------------------------- | | `-o`, `--out PATH` | Writes the bare IR to the path. Use `-` for the standard output. | | `--json` | Writes one JSON document to the standard output. | | `--emit-ir` | A compatibility spelling. `compile` always makes the IR. | ## `extract` [Section titled “extract”](#extract)
```text
scrape-kdl extract [options]
```
| Option | Function | | ----------------------- | --------------------------------------------------------------------------------------------------- | | `--input NAME=VALUE` | A runtime input. Repeat the option for more than one input. | | `--html PATH` | Uses the decoded HTML from the path. Use `-` for the standard input. | | `--session-file PATH` | Reads the headers and the cookies from a JSON file. Use `-` for the standard input. | | `--session` | Supplies an explicit empty session. | | `--allow-private-hosts` | Permits a target that is not globally accessible, and gives the ordinary behavior of a proxy again. | | `--timeout DURATION` | The timeout of the HTTP request. The default is 30 seconds. | | `--max-body BYTES` | The maximum size of the body of the response. The default is 33554432. | | `--user-agent VALUE` | The User-Agent of the HTTP request. | | `--json` | Writes one JSON document to the standard output. | | `-o`, `--out PATH` | Writes the bare result to the path. Use `-` for the standard output. | With `--html`, the runtime does no acquisition. There is no URL expansion, no URL policy, no session, and no request. Thus the declared inputs are not necessary. Refer to [Offline Snapshots](../guides/offline-snapshots/). ## The streams [Section titled “The streams”](#the-streams) The primary result goes to the standard output. A diagnostic, a warning, a confirmation of a write, and an error go to the standard error. The commands `compile` and `extract` write the bare IR or the bare result as formatted JSON, by default. An invocation with a redirection or a pipe has no color escape sequence, no animation of the progress, no rendering with a carriage return, and no question. ## The standard input [Section titled “The standard input”](#the-standard-input) The character `-` selects the standard input where one stream is not ambiguous:
```bash
cat extractor.kdl | scrape-kdl validate -
cat extractor.kdl | scrape-kdl compile - --out -
cat page.html | scrape-kdl extract extractor.kdl --html -
cat session.json | scrape-kdl extract extractor.kdl --session-file -
```
One invocation can give the standard input to the KDL source, to `--html -`, or to `--session-file -`. It cannot give it to two of them. An ambiguous combination is a usage error, and the CLI does not wait for more input. The option `--out -` selects the standard output and does not use the standard input. ## The JSON envelopes [Section titled “The JSON envelopes”](#the-json-envelopes) The option `--json` writes exactly one JSON document to the standard output. It does this after a success, after a processing failure, and after a usage failure that occurs after the CLI recognized the flag. A diagnostic for a person stays on the standard error. | Command | Envelope | | ---------- | ------------------------------------------------------------------------------------------- | | `validate` | `{"ok": boolean, "diagnostics": [...]}` | | `compile` | `{"ok": true, "diagnostics": [...], "ir": {...}}`, or `{"ok": false, "diagnostics": [...]}` | | `extract` | `{"ok": true, "result": {...}}`, or `{"ok": false, "error": {...}}` | | `version` | `{"version": "...", "commit": "...", "built": "..."}` | You cannot use `--json` with `--out FILE`. Use `--out -`, or do not use `--out`. In an automated procedure, use `--json` and examine the field `ok` and also the exit status. ## The exit statuses [Section titled “The exit statuses”](#the-exit-statuses) | Status | Meaning | | ------ | -------------------------------------------------------------------------------------------------------------- | | 0 | Success. | | 1 | A failure of the validation, the compilation, the extraction, the input and output, or a different processing. | | 2 | An error in the use of a command or a flag. | | 130 | `SIGINT` stopped the process. | | 143 | `SIGTERM` stopped the process. | `SIGINT` and `SIGTERM` cancel the active context of the extraction. The HTTP request and the work of the runtime see the cancellation before the process stops with the status of the signal. The CLI writes no partial primary document after an interrupted extraction. ## The network policy [Section titled “The network policy”](#the-network-policy) By default, `extract` rejects an address that the IANA special-purpose registries do not mark as globally accessible. This includes the loopback, private, link-local, carrier-grade NAT, documentation, benchmarking, multicast, unspecified, and reserved ranges. The CLI examines the declared host and also the address that it selects at connection time, and it examines each redirect again. A rejection gives `E_URL_POLICY`. The guarded HTTP client makes a direct connection and does not use the proxy settings of the environment, because a proxy resolves the target itself and the client then cannot examine the selected address. Use `--allow-private-hosts` for a local, an intranet, or an explicitly proxied extraction. An offline execution with `--html` causes no network activity, thus this option has no effect on it. ## The secrets [Section titled “The secrets”](#the-secrets) The CLI accepts a header and a cookie only from `--session-file FILE` or from `--session-file -`:
```json
{
"headers": {"Authorization": ["Bearer example"]},
"cookies": [{"name": "session", "value": "example"}]
}
```
The flags `--header` and `--cookie` were removed at the contract boundary of version 0.5, because a command argument can go into the history of a shell or become visible in a list of the processes. Put a repeated flag in the array `headers` or in an entry of `cookies`. The CLI rejects a removed flag and does not write its value. ## The compatibility [Section titled “The compatibility”](#the-compatibility) The contracts of the help, the streams, the JSON, the exit statuses, the signals, and the input of the secrets are frozen for version 1. A change follows Semantic Versioning and needs a black-box test and a compatibility note. ## Next step [Section titled “Next step”](#next-step) * [Patterns](../guides/patterns/) — a loop with `--json` and `jq`. * [Diagnostics](../guides/diagnostics/) — how to read the output of a failure.
# Installation
> Install the Scraping KDL CLI, the Go modules, or the npm packages, and check that your Go, Node.js, Bun, and operating system versions are supported.
Install the CLI with `go install`. Install the libraries with `go get` or with `npm install`. The current release is `v1.0.4` for the Go modules and `1.0.4` for the npm packages. The two ecosystems have one release train and one version number. ## Supported versions [Section titled “Supported versions”](#supported-versions) | Component | Supported | | --------------------- | ----------------------------------------------------- | | Go | 1.26 or later | | Node.js | 22 or later | | Bun | 1.3 or later, for `@hsblabs/scrape-kdl` | | Operating systems | Linux and macOS | | Release architectures | amd64 and arm64 | | Playwright browser | Chromium. Firefox and WebKit have best-effort status. | Windows is not supported. This is not a temporary condition. The project has no Windows CI jobs, no Windows release targets, and no Windows compatibility code. A successful compilation on Windows is accidental and is not a contract. ## Command line [Section titled “Command line”](#command-line) The Go binary is the only CLI. The npm packages do not contain a CLI.
```bash
go install github.com/hsblabs/scrape-kdl/cmd/scrape-kdl@v1.0.4
```
For browser mode with go-rod, install the separate CLI of the adapter:
```bash
go install github.com/hsblabs/scrape-kdl/adapters/rod/cmd/scrape-kdl-rod@v1.0.4
```
To confirm the installation, use this command:
```bash
scrape-kdl version
```
## Go modules [Section titled “Go modules”](#go-modules)
```bash
go get github.com/hsblabs/scrape-kdl@v1.0.4
```
The go-rod adapter is a **separate module**. It is not a package in the core module.
```bash
go get github.com/hsblabs/scrape-kdl/adapters/rod@v1.0.4
```
The adapter depends on the core module. The core module never imports a browser library. Thus an HTTP-only application does not get Chromium, CDP, or go-rod in its dependency graph. For more data, refer to [Go](../golang/). ## Node.js [Section titled “Node.js”](#nodejs)
```bash
npm install @hsblabs/scrape-kdl@1.0.4
```
The core package supports only ESM. It has three entry points: * `@hsblabs/scrape-kdl` — the compiler, the diagnostics, the IR, the HTTP runtime, the offline snapshot runtime, and the browser adapter types; * `@hsblabs/scrape-kdl/node` — `compileFile` and `validateFile`. These functions stay outside the core package. Thus the core package gets no automatic access to the file system. * `@hsblabs/scrape-kdl/authoring` — the bounded authoring model and the catalog of the built-in transforms. The official Playwright adapter is a separate package:
```bash
npm install @hsblabs/scrape-kdl-playwright@1.0.4 playwright
npx playwright install chromium
```
For more data, refer to [Playwright Adapter](../npm/playwright/). ## Bun [Section titled “Bun”](#bun)
```bash
bun add @hsblabs/scrape-kdl@1.0.4
```
Bun 1.3 or later supports the core package. The tests of the Playwright adapter use Node.js 22 or later. ## Next step [Section titled “Next step”](#next-step) Continue with the [Quick Start](./quick-start/). It executes an extractor against a saved HTML file and sends no request.
# Quick Start
> Write your first Scraping KDL extractor, then run validate, compile, and extract against a saved HTML file. This procedure sends no network request.
This procedure starts with an empty directory and ends with a structured result. It uses only the CLI. It sends **no network request**, because the extractor operates on a saved HTML file. Test each new extractor in this manner before you point it at a live service. You must have the `scrape-kdl` binary. Refer to [Installation](./installation/). ## 1. Save an HTML file [Section titled “1. Save an HTML file”](#1-save-an-html-file) Save this text as `page.html`. The spaces in the heading are intentional. They show you the function of a transform.
```html
Scraping KDL Runtime
```
## 2. Write the extractor [Section titled “2. Write the extractor”](#2-write-the-extractor) Save this text as `extractor.kdl`.
```kdl
extractor "basic-http" version="2026-07-15" language-version="2026-07-15" {
source "html" {
fetch mode="http" url="https://example.invalid/{id}"
}
input "id" type="string" required=#true
field "title" type="string" required=#true {
select "h1" match="one"
value "text"
apply "normalize-whitespace"
}
collection "items" min-items=1 {
select "ul.items > li"
field "value" type="u8" required=#true {
select ".value" match="one"
value "text"
apply "trim"
apply "parse-int" as="u8"
}
}
}
```
Read the document from the top to the bottom: * `version` identifies this revision of the document. `language-version` selects the language contract. Its value must be `2026-07-15`. Both properties are necessary. * `source` declares the method to get the document. The value `mode="http"` selects the static HTTP runtime. The runtime puts a declared input in the position of `{id}`. * The node `field "title"` selects one `h1`, reads its text, and removes the unwanted spaces. * The node `collection "items"` makes one row from each `li` that agrees with the selector. It also requires a minimum of one row. * The property `type="u8"` is a true constraint. The runtime parses the text into an 8-bit unsigned integer. A value that is too large causes an extraction error. The runtime does not truncate the value. ## 3. Validate [Section titled “3. Validate”](#3-validate)
```bash
scrape-kdl validate ./extractor.kdl
```
```text
valid: ./extractor.kdl
```
Validation is analysis only. It parses the document, resolves the symbols, checks the types, and calculates the capabilities. It does not open a socket. The exit status is `0` for a correct document and `1` when the diagnostics contain an error. Now cause an error. Change the selector of the title to `h1:has(a)` and validate the document again:
```text
extractor.kdl:9:5: error E_SELECTOR_UNSUPPORTED: selector byte 9: unsupported pseudo-class "has" [output.title.selection]
```
The pseudo-class `:has()` is outside of the portable selector profile. Thus the compiler rejects it and gives you a source location and an output path. You get this error immediately, not at the twentieth page of a crawl. Change the selector to `h1` again before you continue. Refer to [Diagnostics](../guides/diagnostics/). ## 4. Compile [Section titled “4. Compile”](#4-compile)
```bash
scrape-kdl compile ./extractor.kdl --out ./extractor.ir.json
```
```text
wrote: ./extractor.ir.json
```
The Validated IR is the language-neutral contract between the compiler and each runtime. If you do not give `--out`, the CLI writes the IR to the standard output. Examine these fields first:
```json
{
"irVersion": "2026-07-15",
"languageVersion": "2026-07-15",
"capabilities": ["http.fetch"]
}
```
The array `capabilities` contains the exact set of the capabilities that this program needs. This program only fetches with HTTP. If you add a browser workflow or an `evaluate-js` field, the set becomes larger. Thus a host can decide what to permit before it executes the program. Refer to [How It Operates](../guides/how-it-operates/). ## 5. Extract from the saved HTML [Section titled “5. Extract from the saved HTML”](#5-extract-from-the-saved-html)
```bash
scrape-kdl extract ./extractor.kdl --html ./page.html
```
```json
{
"value": {
"items": [
{
"value": 1
},
{
"value": 2
},
{
"value": 3
}
],
"title": "Scraping KDL Runtime"
},
"warnings": [],
"partial": false
}
```
The heading has no unwanted spaces. The item values are numbers, not strings. The flag `partial: false` shows you that the runtime recovered no error. The option `--html` does no acquisition. There is no URL expansion, no URL policy, no session, and no HTTP request. Thus you do not have to supply the necessary `id` input. Refer to [Offline Snapshots](../guides/offline-snapshots/). ## 6. Get a machine-readable envelope [Section titled “6. Get a machine-readable envelope”](#6-get-a-machine-readable-envelope) For a script, the option `--json` puts the result in an envelope with an explicit success flag:
```bash
scrape-kdl extract ./extractor.kdl --html ./page.html --json
```
```json
{
"ok": true,
"result": {
"value": {
"items": [
{
"value": 1
},
{
"value": 2
},
{
"value": 3
}
],
"title": "Scraping KDL Runtime"
},
"warnings": [],
"partial": false
}
}
```
In an automated procedure, examine `ok` and also the exit status of the process. Refer to [CLI](../cli/). ## Execute against a live URL [Section titled “Execute against a live URL”](#execute-against-a-live-url) The same extractor operates on a live URL. Supply the declared input in the place of `--html`:
```bash
scrape-kdl extract ./extractor.kdl --input id=123
```
Read [Security and Responsible Use](../guides/security-and-responsible-use/) before you do this against a real service. By default the CLI rejects a target that is not globally accessible. It accepts a session only from `--session-file`. Thus your credentials do not go into the history of your shell. ## Next step [Section titled “Next step”](#next-step) * [Write an Extractor](../guides/write-an-extractor/) — the fields, the collections, the inputs, and the error policy. * [HTTP Execution](../guides/http-execution/) — the sessions, the redirects, the limits, and the URL policy. * [TypeScript and Bun](../npm/) or [Go](../golang/) — execute the same program from a library.
# Go
> The Go modules of Scraping KDL — the core module, the separate go-rod adapter module, the reason the core never imports a browser library, and the first compilation.
The core module gives you the compiler, the diagnostics, the IR, the HTTP runtime, the offline snapshot runtime, and the interface of the browser adapter. It needs Go 1.26 or later.
```bash
go get github.com/hsblabs/scrape-kdl@v1.0.4
```
## The two modules [Section titled “The two modules”](#the-two-modules) The go-rod adapter is a **separate module**. It is not a package inside the core module.
```bash
go get github.com/hsblabs/scrape-kdl/adapters/rod@v1.0.4
```
The core module must not import go-rod. This is an invariant of the project, not a preference. A browser library belongs in an adapter. Thus an application with HTTP only does not get Chromium, CDP, or go-rod in its dependency graph. Add the adapter module only when you need browser mode. Refer to [go-rod Adapter](./rod/). ## The first compilation [Section titled “The first compilation”](#the-first-compilation)
```go
package main
import (
"context"
"time"
scrapekdl "github.com/hsblabs/scrape-kdl"
)
func main() {
ctx := context.Background()
program, diagnostics, err := scrapekdl.CompileFile(ctx, "extractor.kdl")
if err != nil {
panic(err)
}
if diagnostics.HasErrors() {
panic(diagnostics)
}
result, err := program.Extract(ctx, map[string]any{"id": "123"}, scrapekdl.Options{
RequestTimeout: 15 * time.Second,
})
if err != nil {
panic(err)
}
var output struct {
Title string `json:"title"`
}
if err := result.Decode(&output); err != nil {
panic(err)
}
}
```
The compilation gives three values: a program, the ordered diagnostics, and an operational error. Examine `diagnostics.HasErrors()` before you use the program. An operational error is different from a diagnostic. A cancellation, a failure of the file system, and a failure of an injected loader are operational errors. The functions keep the cause for `errors.Is` and `errors.As`. Refer to [Compile and Extract in Go](./compile-and-extract/). ## The functions of the compilation [Section titled “The functions of the compilation”](#the-functions-of-the-compilation) | Function | Source | | -------------------------------------- | ------------------------------- | | `Compile(ctx, Source, CompileOptions)` | A source in the memory. | | `CompileFile(ctx, path)` | A file of the operating system. | | `CompileFS(ctx, fsys, path)` | An `fs.FS` of your application. | `Validate`, `ValidateFile`, and `ValidateFS` are the equivalents that give only the diagnostics. The functions `CompileFS` and `ValidateFS` limit the root and each import to a valid path of `io/fs` and reject a lexical escape to a parent. Your `fs.FS` still defines the true authority. `os.DirFS` can follow a symbolic link outside of its directory. Use `os.Root.FS` when you need containment. ## The metadata of a program [Section titled “The metadata of a program”](#the-metadata-of-a-program)
```go
metadata := program.Metadata()
metadata.Capabilities // the exact capabilities that the program needs
metadata.LanguageVersion
metadata.IRVersion
metadata.Files // each source file, with its SHA-256
descriptor := program.Descriptor()
descriptor.Source.FetchMode // "http" or "browser"
descriptor.Source.SessionPolicy // "none", "optional", or "required"
```
A program is immutable and has a reusable execution plan. Use the capabilities to permit or to refuse a program in your host. Use the descriptor for a decision about the acquisition, without a decode of the full IR. `Program.IRJSON()` gives the full Validated IR for an interchange or a tool. ## The concurrency [Section titled “The concurrency”](#the-concurrency) A program is safe for a concurrent extraction, if your adapters obey the documented contract of the ownership. The mutable state stays inside one extraction. A browser adapter that controls one page is the exception. It must implement `BrowserAdapterLease`, and the lease then prevents an interleaved operation. Use more than one page or more than one adapter for a true parallelism. ## The command line [Section titled “The command line”](#the-command-line) The core module also has the CLI:
```bash
go install github.com/hsblabs/scrape-kdl/cmd/scrape-kdl@v1.0.4
```
Refer to [CLI](../cli/). ## Next step [Section titled “Next step”](#next-step) * [Compile and Extract in Go](./compile-and-extract/) — the full API of the execution. * [go-rod Adapter](./rod/) — browser mode with an official adapter.
# Compile and Extract in Go
> The Go execution API — the compile options, the injected loaders, the execution options, the three extraction entry points, the URL policy, and the typed decode of a result.
This page shows you the API of the compilation and the execution in the module `github.com/hsblabs/scrape-kdl`. ## Compile [Section titled “Compile”](#compile)
```go
source := scrapekdl.Source{
Path: "extractor.kdl",
Data: data,
}
program, diagnostics, err := scrapekdl.Compile(ctx, source, scrapekdl.CompileOptions{})
if err != nil {
return err
}
if diagnostics.HasErrors() {
return fmt.Errorf("compilation failed: %v", diagnostics)
}
```
The field `Path` is a logical identity for the diagnostics, the source locations, the resolution of the imports, and the file identities in the IR. It does not have to name a file of the operating system. The type `Diagnostics` has a deterministic order. It contains a warning also. Use `HasErrors()` to find an error. ## The imports [Section titled “The imports”](#the-imports) A source with an import needs a loader:
```go
options := scrapekdl.CompileOptions{
Loader: func(ctx context.Context, path string) ([]byte, error) {
if !allowed[path] {
return nil, fmt.Errorf("refused: %s", path)
}
return os.ReadFile(path)
},
}
```
The compiler resolves each import lexically, relative to the source that imports it, before it calls your loader. Your loader gives only the bytes. The compiler makes the parse, the validation, the detection of the cycles, the hash, and the deterministic order. A compilation of a source with an import and without a loader fails before it gives an IR. Your loader is an authority boundary: limit the paths, obey the cancellation, and do not put the content of a source or a credential in an error. For a file system of your application, use `CompileFS(ctx, fsys, path)`. It resolves each nested import inside the same file system and rejects a lexical escape to a parent. It examines the cancellation before and after each `fs.ReadFile`. The interface `fs.FS` cannot interrupt a read that is in progress. ## The three entry points of the extraction [Section titled “The three entry points of the extraction”](#the-three-entry-points-of-the-extraction)
```go
result, err := program.Extract(ctx, inputs, options)
result, err := program.ExtractHTML(ctx, html, options)
result, err := program.ExtractSnapshot(ctx, html, options)
```
| Method | Acquisition | Accepted modes | | ----------------- | ------------------------------- | ---------------------------------------------------------- | | `Extract` | Follows the mode of the source. | Each mode. A browser-mode program needs `Options.Browser`. | | `ExtractHTML` | None. | HTTP mode only. | | `ExtractSnapshot` | None. | Each mode, if the program is eligible for a snapshot. | Refer to [Offline Snapshots](../guides/offline-snapshots/). ## The execution options [Section titled “The execution options”](#the-execution-options)
```go
type Options struct {
Browser BrowserAdapter
AllowJavaScript bool
HTTPClient *http.Client
Session *Session
ExternalTransforms map[string]ExternalTransform
CharsetDecoder CharsetDecoder
RequestTimeout time.Duration
MaxResponseBytes int64
UserAgent string
URLPolicy URLPolicy
}
```
Each `Options` value configures one extraction. The mutable state stays inside that extraction. Thus you can execute one immutable program more than one time, at the same time, with different options. The field `AllowJavaScript` is off by default. A program with an `evaluate-js` node then fails with `E_JAVASCRIPT_DISABLED` before the navigation. ## The result [Section titled “The result”](#the-result)
```go
type Result struct {
Value map[string]any `json:"value"`
Warnings []Warning `json:"warnings"`
Partial bool `json:"partial"`
}
```
Use `Decode` for a typed value:
```go
var output struct {
Title string `json:"title"`
Items []struct {
Value uint8 `json:"value"`
} `json:"items"`
}
if err := result.Decode(&output); err != nil {
return err
}
```
A `Warning` has a `Code`, a `Message`, an optional `Path`, and an optional `Row`. The flag `Partial` is `true` only after the runtime recovered an error or dropped a row. A failure gives an `*ExecutionError` with a `Code`, a `Message`, a `Path`, and a `Cause`:
```go
var execErr *scrapekdl.ExecutionError
if errors.As(err, &execErr) && execErr.Code == "E_REQUIRED_VALUE_MISSING" {
// handle the missing value
}
```
Examine the `Code`. The codes are stable. The messages are not. Refer to [Diagnostics](../guides/diagnostics/). ## The URL policy [Section titled “The URL policy”](#the-url-policy)
```go
options := scrapekdl.Options{
URLPolicy: scrapekdl.PublicInternetURLPolicy(),
HTTPClient: scrapekdl.NewPublicInternetHTTPClient(),
}
```
Use the two together. The policy examines the initial target and each redirect. The guarded client resolves the address again at connection time and examines it again. Thus DNS rebinding cannot defeat the check. The guarded client makes a direct connection and does not use the proxy settings of the environment, because a proxy resolves the target itself and the client then cannot examine the selected address. The library applies no policy until you configure one. The CLI applies the two by default. Refer to [HTTP Execution](../guides/http-execution/). ## The external transforms [Section titled “The external transforms”](#the-external-transforms)
```go
options := scrapekdl.Options{
ExternalTransforms: map[string]scrapekdl.ExternalTransform{
"decrypt_payload": func(ctx context.Context, input any) (any, error) {
return decrypt(input)
},
},
}
```
If the registry does not have a symbol that the program needs, the validation fails before the fetch. After your function gives a result, the runtime immediately examines the result against the declared output type. ## The cancellation [Section titled “The cancellation”](#the-cancellation) The runtime propagates the context to the HTTP request, to the operations of the adapter, and to the traversal of the output. It examines the context before the parse of the HTML in the memory, and also before each output member and each collection row. A cancellation gives `E_EXECUTION_CANCELED` and keeps `context.Canceled` or `context.DeadlineExceeded` as the cause. A field policy or a row policy cannot recover it. ## The compatibility [Section titled “The compatibility”](#the-compatibility)
```go
scrapekdl.SupportedLanguageVersions()
scrapekdl.SupportedIRVersions()
```
The two functions give the exact versions that this build accepts. ## Next step [Section titled “Next step”](#next-step) * [go-rod Adapter](./rod/) — how to execute a browser-mode program. * [Patterns](../guides/patterns/) — a loop over more than one page.
# go-rod Adapter
> The official go-rod browser adapter for Go — the separate module, the ownership of a page, the lease that serializes an extraction, and the scrape-kdl-rod command line.
The go-rod integration is an independent nested Go module:
```text
adapters/rod/
```
```bash
go get github.com/hsblabs/scrape-kdl/adapters/rod@v1.0.4
```
The main module has no dependency on a browser library. You select the adapter explicitly. Thus you control the installation of Chromium, the options of the launch, the sandbox, the network policy, and the lifecycle of the process. ## The lifecycle [Section titled “The lifecycle”](#the-lifecycle) | Function | Behavior | | -------------------------------- | ------------------------------------------------------------ | | `rodadapter.New(page)` | Uses a page that you own. | | `rodadapter.NewBrowser(browser)` | Makes one page and owns it. | | `Adapter.Close` | Closes a page that the adapter owns. It closes nothing else. | The adapter never closes a `*rod.Browser` that you own. One adapter represents one mutable browser page. The adapter implements `BrowserAdapterLease`. Thus the lease puts the full sequence of the navigation, the workflow, and the extraction in a series, between the concurrent calls. For a parallel extraction, use a separate adapter and a separate page for each thread of the work. ## JavaScript [Section titled “JavaScript”](#javascript) The core rejects JavaScript until `AllowJavaScript` is `true`. The adapter executes a script with the scope `document` through `Page.Evaluate`. It executes a script with the scope `current` through `Element.Evaluate` and gives the current element to your KDL function. ## The command line [Section titled “The command line”](#the-command-line) The module has its own binary. It compiles one extractor and executes it in browser mode.
```bash
go install github.com/hsblabs/scrape-kdl/adapters/rod/cmd/scrape-kdl-rod@v1.0.4
```
```bash
scrape-kdl-rod --spec extractor.kdl --input race_id=202401010101 --json
scrape-kdl-rod --spec extractor.kdl --session-file session.json -o result.json
```
| Option | Function | | ------------------------ | --------------------------------------------------------------- | | `--spec FILE` | The KDL source. The CLI always uses this option for the source. | | `--input NAME=VALUE` | A runtime input. Repeat the option for more than one input. | | `--session-file FILE\|-` | The JSON session schema of the core CLI. | | `--timeout` | The timeout of the operations. | | `--user-agent` | The User-Agent. | | `--headless` | The mode of the browser. | | `--allow-js` | The explicit opt-in for JavaScript. | | `--allow-private-hosts` | Disables the default limit of the initial target. | | `--json` | One JSON document for a success or for a failure. | | `-o`, `--out FILE\|-` | The bare result of the extraction. | The CLI rejects the flags `--header` and `--cookie` and does not write their values. The standard input is reserved for `--session-file -`. ## The output and the exit status [Section titled “The output and the exit status”](#the-output-and-the-exit-status) Without `--json`, a success writes the bare formatted result to the standard output or to the selected file. A warning and a failure for a person go to the standard error. With `--json`, the standard output has exactly one of these documents: * `{"ok": true, "result": {...}}` after a successful extraction; * `{"ok": false, "error": {...}}` after a failure of the compilation, the execution, the input and output, or the use; * `{"version": "...", "commit": "...", "built": "..."}` for `--version --json`. You cannot use `--json` with `--out FILE`. Use `--out -`, or do not use `--out`. The exit status is 0 for a success, 1 for a processing failure, 2 for an error of the use, 130 for `SIGINT`, and 143 for `SIGTERM`. The two signals cancel the active context before the exit. ## The sessions and the URL policy [Section titled “The sessions and the URL policy”](#the-sessions-and-the-url-policy) The headers and the cookies of the session, the User-Agent, the runtime inputs, the timeout, and the opt-in for JavaScript use the same public contract `scrapekdl.Options` as a library. The CLI applies `PublicInternetURLPolicy` to the initial navigation target, by default. It rejects a scheme, a credential, and an address that the IANA special-purpose registries do not mark as globally accessible. The option `--allow-private-hosts` disables it. This initial check is not a network sandbox for the browser. A redirect inside Chromium, a subresource, a service worker, and a request that the page starts are outside of the hook. A production host must apply the necessary egress policy at the boundary of the browser context, the process, the container, or the network. ## The verification [Section titled “The verification”](#the-verification) The contract tests use a local stub and do not download go-rod:
```bash
make test-rod-contract
```
The build with the true dependency and its tests use this command:
```bash
make test-rod
```
The end-to-end suite with Chromium uses this command:
```bash
make test-rod-e2e
```
The end-to-end suite needs a runtime that is compatible with Chromium. The core tests and the contract tests do not need one. ## Next step [Section titled “Next step”](#next-step) * [Browser Mode](../guides/browser-mode/) — the workflow steps and the rules of the JavaScript. * [Compile and Extract in Go](./compile-and-extract/) — the options of the execution.
# Browser Mode
> How browser mode operates in Scraping KDL — the adapter contract, the workflow steps, the JavaScript opt-in, the extraction lease, and the parts that are common to each adapter.
Browser mode is a capability, not a different method to fetch a page. A program with `mode="browser"` needs an adapter that you supply. The core module does not depend on Playwright, Puppeteer, go-rod, or chromedp. If you do not supply an adapter, the extraction fails with `E_BROWSER_RUNTIME_MISSING` before the navigation. This page describes the contract that is common to each adapter. For an installation and a lifecycle, refer to [Playwright Adapter](../npm/playwright/) or [go-rod Adapter](../golang/rod/). ## When you need it [Section titled “When you need it”](#when-you-need-it) Use browser mode only for a condition that a static DOM cannot give you: * a script writes the content after the load; * the content becomes visible only after a click, an input, or a scroll; * the data is in the memory of the page and not in the markup. For each other condition, use HTTP mode. It is faster, it needs no Chromium, and it has a smaller attack surface. ## The execution order [Section titled “The execution order”](#the-execution-order)
```text
capability and output preflight
-> input resolution and URL expansion
-> session policy
-> optional lease acquisition
-> navigation
-> workflow steps, in source order
-> extraction from the live DOM
-> validation of the JavaScript results and the transforms
-> lease release, after a success or a failure
```
The runtime checks the availability of the external transforms, the kinds and the selectors of the workflow steps, the portable output selectors, the kinds of the output members, and the kinds of the value sources before it acquires the lease and before the navigation. Thus a program with an error does not start a browser. ## The workflow [Section titled “The workflow”](#the-workflow) A workflow executes after the navigation and before the extraction. The steps execute in source order.
```kdl
source "html" {
fetch mode="browser" url="https://example.com/race/{race_id}"
workflow {
wait-for ".content" state="visible" timeout-ms=5000
click ".load-more" timeout-ms=3000
wait-for-network-idle idle-ms=500 timeout-ms=5000
}
}
```
| Step | Function | | ---------------------------------------------- | -------------------------------------------------------------------------------------------- | | `wait-for selector [state] [timeout-ms]` | Waits for a state: `attached`, `visible`, `hidden`, or `detached`. The default is `visible`. | | `click selector [timeout-ms]` | Clicks the element. | | `fill selector value [timeout-ms]` | Puts a value in the element. | | `press selector key [timeout-ms]` | Sends a key to the element. | | `scroll x y` | Scrolls the window. The two numbers are CSS pixels. | | `wait-for-network-idle [idle-ms] [timeout-ms]` | Waits until no tracked HTTP request is active for `idle-ms`. The default is 500 ms. | | `evaluate-js script [timeout-ms]` | Executes a function. The runtime discards the result. | A workflow step is available in browser mode only. In HTTP mode the compiler rejects the node `workflow` with `E_BROWSER_CAPABILITY_REQUIRED`. The values `timeout-ms` and `idle-ms` must be from 1 to 9,223,372,036,854 milliseconds. An expired timeout is an extraction error. A WebSocket connection and an EventSource connection are not tracked requests for `wait-for-network-idle`. ## JavaScript [Section titled “JavaScript”](#javascript) JavaScript is off by default. You must give an explicit opt-in: `AllowJavaScript: true` in Go, or `allowJavaScript: true` in TypeScript. Without the opt-in, a program with JavaScript fails with `E_JAVASCRIPT_DISABLED` before the navigation.
```kdl
field "race" type="object?" {
evaluate-js #"""
() => window.__INITIAL_STATE__?.race ?? null
"""# scope="document" returns="object?" timeout-ms=3000
}
```
The rules are strict: * The script must give a callable function. An async function is permitted. * The property `scope` is `document` or `current`. With `document`, the function gets no argument. With `current`, it gets the current element as its first argument. * The property `returns` declares the type of the raw result. * The result must be JSON-compatible: `null`, a boolean, a string, a finite number, an array, or a plain object with string keys. * The values `undefined`, `NaN`, an infinity, a bigint, a symbol, a function, a DOM node, a handle of the runtime, a cyclic object, a `Map`, a `Set`, and a `Date` are forbidden. The result fails with `E_JAVASCRIPT_RESULT_TYPE`. Treat `evaluate-js` as trusted code of the specification. It executes with the full permissions of the page. It is also an intentional escape from the portable behavior: an offline snapshot cannot reproduce it. ## The adapter contract [Section titled “The adapter contract”](#the-adapter-contract) An adapter implements a small interface. The Go interface has these operations:
```go
type BrowserAdapter interface {
Navigate(context.Context, string, BrowserNavigateOptions) error
WaitFor(context.Context, string, string, time.Duration) error
Click(context.Context, string, time.Duration) error
Fill(context.Context, string, string, time.Duration) error
Press(context.Context, string, string, time.Duration) error
Scroll(context.Context, float64, float64) error
WaitForNetworkIdle(context.Context, time.Duration, time.Duration) error
Evaluate(context.Context, string, BrowserEvaluateOptions) (any, error)
QueryAll(context.Context, BrowserElement, string) ([]BrowserElement, error)
Text(context.Context, BrowserElement) (string, error)
HTML(context.Context, BrowserElement) (string, error)
Attribute(context.Context, BrowserElement, string) (string, bool, error)
}
```
The TypeScript contract has the same operations with promises, timeout fields in milliseconds, and an optional `AbortSignal`. A `BrowserElement` is an opaque handle. A Playwright adapter can keep a Locator or an ElementHandle. A go-rod adapter can keep a `*rod.Element`. The core never examines the content of a handle. If the adapter does not have an operation that the program needs, the validation fails with `E_BROWSER_CAPABILITY_MISSING` before the navigation. ## The lease [Section titled “The lease”](#the-lease) An adapter that controls one mutable page must implement a lease:
```go
type BrowserAdapterLease interface {
Acquire(context.Context) (release func(), err error)
}
```
The runtime holds the lease during the navigation, the workflow, and each output read. Thus the operations of two concurrent extractions cannot interleave on one page. A lease does not give parallelism. For a parallel execution, use more than one page or more than one adapter. ## The scope of a query [Section titled “The scope of a query”](#the-scope-of-a-query) A nil element for `QueryAll` means the scope of the document. For `evaluate-js`, the scope `document` gives a nil scope, and the scope `current` gives the selected element of the field or the row of the collection. An adapter can also implement a bounded query, `BrowserAdapterQueryLimit` in Go or `queryLimit` in TypeScript. The runtime uses it for `match="first"` and `match="one"`, where one or two handles are sufficient. An adapter without this function uses `QueryAll`. ## The sessions and the URL policy [Section titled “The sessions and the URL policy”](#the-sessions-and-the-url-policy) With `session policy="none"`, the runtime gives no explicit session to `Navigate`. It does not clear the cookies, the storage, or the authentication that the browser context already has. For an execution without a state, supply an isolated context. The hook `Options.URLPolicy` executes before the lease acquisition and before the navigation. A rejection gives `E_URL_POLICY` and does not use the browser. The policy controls the initial target only. A redirect of the browser, a subresource, a service worker, and a request that the page starts are outside of this hook. Control them with the browser context or with a network policy of the host. ## Next step [Section titled “Next step”](#next-step) * [Playwright Adapter](../npm/playwright/) — the official adapter for TypeScript and Node.js. * [go-rod Adapter](../golang/rod/) — the official adapter for Go. * [Offline Snapshots](./offline-snapshots/) — how to test a browser-mode program without a browser.
# Diagnostics
> How to read a Scraping KDL diagnostic — the stable codes, the deterministic order, the severities, the warnings, and the most frequent compile and runtime errors.
A diagnostic tells you what is wrong, where it is, and which part of the output it affects. The codes are a public compatibility surface. A code keeps the same meaning between the releases. Thus you can write a test or an alert that examines a code. The full list is in [diagnostics.md](https://github.com/hsblabs/scrape-kdl/blob/main/docs/spec/diagnostics.md). ## The format [Section titled “The format”](#the-format) The CLI writes one line for each diagnostic:
```text
extractor.kdl:9:5: error E_SELECTOR_UNSUPPORTED: selector byte 9: unsupported pseudo-class "has" [output.title.selection]
```
The line has five parts: | Part | Meaning | | -------------------------- | ----------------------------------------------------------- | | `extractor.kdl:9:5` | The file, the line, and the column. The numbers start at 1. | | `error` | The severity. | | `E_SELECTOR_UNSUPPORTED` | The stable code. | | The text after the code | The message for a person. It is not stable. | | `[output.title.selection]` | The path in the output that the error affects. | Examine the code, not the message. The codes and their conditions are normative. The messages are not, and they can change. ## The machine-readable form [Section titled “The machine-readable form”](#the-machine-readable-form) The option `--json` gives one JSON document:
```bash
scrape-kdl validate ./extractor.kdl --json
```
```json
{
"ok": false,
"diagnostics": [
{
"code": "E_SELECTOR_UNSUPPORTED",
"severity": "error",
"message": "selector byte 13: unsupported pseudo-class \"has\"",
"span": {
"file": "extractor.kdl",
"start": { "offset": 197, "line": 7, "column": 5 },
"end": { "offset": 219, "line": 7, "column": 27 }
},
"path": "output.title.selection"
}
]
}
```
The `span` uses the same definition as the Validated IR. The line and the column start at 1. The `offset` is a 0-based UTF-8 byte offset. The end position is exclusive. The `path` is absent when the diagnostic affects no output member. The envelope is different for each command: * `validate`: `{"ok": boolean, "diagnostics": [...]}`; * `compile`: `{"ok": true, "diagnostics": [...], "ir": {...}}`, or `{"ok": false, "diagnostics": [...]}`; * `extract`: `{"ok": true, "result": {...}}`, or `{"ok": false, "error": {...}}`. ## The order [Section titled “The order”](#the-order) The compiler orders the static diagnostics by: 1. the depth-first resolution order of the imports; 2. the lexical order of the file paths, for an equal position; 3. the start offset in the source; 4. the lexical order of the codes. The runtime orders the warnings by the sequence of the execution. The order is deterministic. Two executions of the same program on the same input give the same sequence. Thus you can compare the full output of a diagnostic in a golden file. ## The severities [Section titled “The severities”](#the-severities) There are two severities: * `error` — the process stops. The compiler makes no IR, or the runtime gives no result. * `warning` — the extraction continues. The result has the warning in its array `warnings`. A warning frequently sets the flag `partial` to `true`. Then you know that the result is not complete. ## The frequent compile errors [Section titled “The frequent compile errors”](#the-frequent-compile-errors) | Code | Cause | Correction | | -------------------------------- | ------------------------------------------------------------------------------------------ | -------------------------------------------------------------- | | `E_SELECTOR_UNSUPPORTED` | The selector is outside of the portable profile, for example `:has()`. | Refer to [Selectors](./selectors/). | | `E_TRANSFORM_TYPE_MISMATCH` | The output type of one transform does not agree with the input type of the next transform. | Add a conversion, for example `parse-int`. | | `E_TRANSFORM_UNKNOWN` | The name is not a built-in, a local transform, or a qualified imported transform. | Examine the spelling and the alias of the import. | | `E_BROWSER_CAPABILITY_REQUIRED` | A browser-only node is in an HTTP-mode program. | Change the mode to `browser`, or remove the node. | | `E_LANGUAGE_VERSION_UNSUPPORTED` | The value of `language-version` is well formed but not supported. | Use `2026-07-15`. | | `E_DUPLICATE_PROPERTY` | One node has the same property two times. | Remove one property. The compiler does not select one for you. | | `E_IMPORT_CYCLE` | The import graph has a cycle. | Divide the shared transforms into a module without an import. | | `E_REMOTE_IMPORT_UNSUPPORTED` | The path of the import is not relative. | Copy the module into your repository. | ## The frequent runtime errors [Section titled “The frequent runtime errors”](#the-frequent-runtime-errors) | Code | Cause | | --------------------------- | ------------------------------------------------------------------------------------ | | `E_REQUIRED_VALUE_MISSING` | A field with `required=#true` found no value. | | `E_SELECTOR_CARDINALITY` | A selector with `match="one"` found more than one element. | | `E_URL_POLICY` | The URL policy rejected the initial target or a redirect. | | `E_HTTP_STATUS` | The status of the response is outside of the range 200 to 299. | | `E_HTTP_BODY_TOO_LARGE` | The response is larger than the limit of the body. | | `E_JAVASCRIPT_DISABLED` | The program has JavaScript, but you gave no opt-in. | | `E_SNAPSHOT_UNSUPPORTED` | A snapshot execution was requested for a program with a workflow or with JavaScript. | | `E_BROWSER_RUNTIME_MISSING` | A browser-mode program has no adapter. | | `E_EXECUTION_CANCELED` | The context or the `AbortSignal` stopped the execution. | The error `E_JAVASCRIPT_DISABLED`, the error `E_BROWSER_RUNTIME_MISSING`, and the error `E_SNAPSHOT_UNSUPPORTED` occur before the acquisition. Thus a bad configuration does not cause traffic. ## The warnings [Section titled “The warnings”](#the-warnings) | Code | Meaning | | ---------------------- | ---------------------------------------------------------- | | `W_ROW_SKIPPED` | The policy `on-row-error="skip"` dropped a collection row. | | `W_ERROR_RECOVERED` | The policy `on-error="warn"` recovered an error. | | `W_PARTIAL_EXTRACTION` | A summary warning for a partial result. | | `W_JAVASCRIPT_PRESENT` | A static examination found the trusted-code capability. | ## The exit statuses [Section titled “The exit statuses”](#the-exit-statuses) | Status | Meaning | | ------ | -------------------------------------------------------------------------------------- | | 0 | Success. | | 1 | A failure of the validation, the compilation, the extraction, or the input and output. | | 2 | An error in the use of a command or a flag. | | 130 | `SIGINT` stopped the process. | | 143 | `SIGTERM` stopped the process. | In an automated procedure, examine the exit status and also the field `ok` of the envelope `--json`. ## Next step [Section titled “Next step”](#next-step) * [Patterns](./patterns/) — the shapes that prevent the frequent errors. * [CLI](../cli/) — the full contract of the commands.
# How It Operates
> The seven compiler stages of Scraping KDL, the capability set, the Validated IR, and the rule that no network or browser operation occurs before the validation is correct.
Scraping KDL divides the work into two parts. The compiler reads your document and makes a Validated IR. A runtime executes that IR. The two parts are fully separate. Thus you can examine, store, or transmit a program before it touches a network. ## The stages [Section titled “The stages”](#the-stages) The compiler executes these stages in this sequence: 1. KDL syntax parse; 2. base restriction validation; 3. application grammar validation; 4. import graph resolution; 5. symbol resolution; 6. type check; 7. capability derivation and validation. An error in one stage stops the program before the subsequent stages. Each error becomes a diagnostic with a code and a source location. A conforming implementation must complete the stages 1 to 7 before it sends a network request, starts a browser, navigates a page, changes a session, or calls an external transform. This is a rule of the language, not a property of one implementation. ## What the compiler does not do [Section titled “What the compiler does not do”](#what-the-compiler-does-not-do) The compiler package contains no network client, no browser control, and no external transform call. It reads the entry KDL file and the relative modules that the file imports. It does nothing more. The TypeScript core has no automatic access to the file system and no automatic network loader. It resolves a relative path lexically and asks a `SourceLoader` for the bytes. The `@hsblabs/scrape-kdl/node` entry point supplies the loader for the file system. Thus the core package cannot read a file that you did not permit. The parser keeps the sequence of the properties and also the duplicates. The semantic stage then rejects a duplicate property with `E_DUPLICATE_PROPERTY`. The compiler does not silently use the last property, although KDL permits this behavior. ## Capabilities [Section titled “Capabilities”](#capabilities) The Validated IR contains a sorted set of the capabilities that the program needs. The compiler calculates this set from the content of the document. | Capability | Cause | | ------------------------------------------------------------------------- | ------------------------------- | | `http.fetch` | An HTTP source. | | `browser.navigate` | A browser source. | | `browser.query` | A selection in browser mode. | | `browser.read-text`, `browser.read-html`, `browser.read-attr` | A value source in browser mode. | | `browser.wait`, `browser.input`, `browser.scroll`, `browser.network-idle` | A workflow step. | | `browser.evaluate-js` | An `evaluate-js` value source. | | `transform.external:` | An external transform. | The set is a contract. A host can read it and refuse a program before the execution. For example, a host can permit `http.fetch` and refuse each `browser.*` capability. The exact list is in [language-v0.1.md](https://github.com/hsblabs/scrape-kdl/blob/main/docs/spec/language-v0.1.md). ## The Validated IR [Section titled “The Validated IR”](#the-validated-ir) The IR is the language-neutral contract between the compiler and each runtime. The Go runtime and the TypeScript runtime read the same IR and give the same values. The IR contains three version fields with a related function: * `irVersion` — the format of the IR document. The current value is `2026-07-15`. * `languageVersion` — the language contract that the document selected with `language-version`. The current value is `2026-07-15`. * The `version` property of your document — your revision identifier. The compiler does not interpret it, but the value must be a real calendar date in the form `YYYY-MM-DD`. The schema of the IR is in [schema.json](https://github.com/hsblabs/scrape-kdl/blob/main/docs/ir/schema.json). ## The runtimes [Section titled “The runtimes”](#the-runtimes) One IR has three execution boundaries: * **HTTP** — the runtime sends a request, decodes the body, parses the HTML, and extracts the values. Refer to [HTTP Execution](./http-execution/). * **Browser** — the runtime uses an adapter that you supply to navigate a live page, execute the workflow, and read the values. Refer to [Browser Mode](./browser-mode/). * **Offline snapshot** — the runtime extracts from HTML that you supply and does no acquisition. Refer to [Offline Snapshots](./offline-snapshots/). For an extraction without JavaScript, the HTTP runtime and the browser runtime must agree. The same conformance fixtures test the two runtimes. A difference between them is a defect, not a permitted variation. ## Deterministic diagnostics [Section titled “Deterministic diagnostics”](#deterministic-diagnostics) The compiler orders the static diagnostics by the resolution sequence of the imports, then by the path of the file, then by the position in the source, then by the code. The runtime orders the warnings by the sequence of the execution. Thus two executions of the same program on the same input give the same output. You can compare the diagnostics in a test or in a CI job. Refer to [Diagnostics](./diagnostics/). ## Next step [Section titled “Next step”](#next-step) * [Write an Extractor](./write-an-extractor/) — the structure of a document. * [Selectors](./selectors/) — the portable subset of CSS. * [Transforms](./transforms/) — the value pipeline and the types.
# HTTP Execution
> How the HTTP runtime gets a document — the execution order, the limits, the charset decoding, the sessions, the redirects, and the URL policy that protects a private network.
The HTTP runtime executes a program with `mode="http"`. It sends one GET request, decodes the body, parses the HTML into an internal DOM, and extracts the values. It does not execute JavaScript and does not start a browser. ## The execution order [Section titled “The execution order”](#the-execution-order)
```text
compiler validation
-> runtime capability preflight
-> input and default resolution
-> validation of a required session
-> URL expansion
-> HTTP request
-> validation of the response size
-> charset decode
-> DOM parse
-> output extraction
```
The runtime checks the selectors, the availability of the external transforms, the browser-only value sources, and the fetch mode before it sends the request. Thus a program that cannot operate correctly does not cause traffic. ## The behavior of the request [Section titled “The behavior of the request”](#the-behavior-of-the-request) | Property | Value | | ------------------ | ----------------------------------------- | | Method | GET | | Accepted status | 200 to 299 | | Default timeout | 30 seconds | | Default body limit | 32 MiB | | Default User-Agent | `scrape-kdl/1.0` | | Redirects | The supplied `http.Client` controls them. | A response that is too large fails with `E_HTTP_BODY_TOO_LARGE`. A status outside of the accepted range fails with `E_HTTP_STATUS`. The limits are not advisory. The runtime applies them before it reads the full body into the memory. ## The charset [Section titled “The charset”](#the-charset) The runtime selects the charset from the HTTP header `Content-Type`, then from an early declaration of ``, with UTF-8 as the default. A recognized byte-order mark of UTF-8 or UTF-16 has priority over the declared charset. UTF-8, ASCII, ISO-8859-1, Windows-1252, UTF-16LE, and UTF-16BE are built in. The runtime resolves a different label through the WHATWG encoding index. Thus a legacy encoding such as Shift\_JIS or EUC-JP decodes without configuration. Two conditions cause an error: * A bad byte sequence fails with `E_HTML_DECODE`. The Go runtime and the TypeScript runtime agree on this behavior. * The replacement encoding, and a label outside of the WHATWG index, fail with `E_HTML_CHARSET_UNSUPPORTED`. ## The sessions [Section titled “The sessions”](#the-sessions) The property `policy` of the node `session` controls the behavior: | Policy | Behavior | | ---------------- | ------------------------------------------------------------------- | | `none` (default) | The runtime ignores the session that you supply explicitly. | | `optional` | The runtime uses the session when you supply one. | | `required` | The extraction stops before the fetch when no session is available. | The runtime adds the headers and the cookies of the session to the initial request only. It does not put them in a redirected request again. Thus the redirect rules of your `http.Client` control the propagation. By default the client copies a sensitive header such as `Authorization` or `Cookie` only to the same domain or to a subdomain. A configured cookie jar applies the scope of each cookie. The policy `none` does not clear the ambient state of the host. Your `http.Client` and its cookie jar continue to operate, and a `Set-Cookie` response header continues to have an effect. For an execution without a state, supply an isolated client that has no jar and no ambient authentication. ## The URL policy [Section titled “The URL policy”](#the-url-policy) The hook `Options.URLPolicy` executes before the initial request and also before each HTTP redirect. An error from the hook stops the extraction with `E_URL_POLICY`. `PublicInternetURLPolicy` is a prepared policy. It rejects a scheme that is not HTTP or HTTPS, a URL with userinfo, and an address that the IANA special-purpose registries do not mark as globally accessible. This includes the loopback, private, link-local, carrier-grade NAT, documentation, benchmarking, multicast, unspecified, and reserved ranges. It keeps the globally accessible exceptions of the registries. A policy check occurs before the connection. Thus DNS rebinding can defeat the check alone. Use `NewPublicInternetHTTPClient` with the policy. Its dialer resolves the address again at connection time, examines it again, and reports a rejection as `E_URL_POLICY`. This guarded client makes a direct connection and does not use the proxy settings of the environment, because a proxy resolves the target itself and the client then cannot examine the selected address. The CLI applies the policy and the guarded client together, by default. The option `--allow-private-hosts` disables them. The library has different defaults: it applies no policy until you configure one. A `CheckRedirect` function of your own executes after the policy of Scraping KDL, not in the place of it. The policy is not a substitute for a network-level egress control. It is one layer. ## The HTML parser [Section titled “The HTML parser”](#the-html-parser) The Go runtime uses a pinned version of `golang.org/x/net/html` for the tree construction, after it decodes the bounded bytes to UTF-8. The internal DOM keeps the document order and supplies the portable selectors, the decoded text, a deterministic inner HTML, the attributes, and the missing-value behavior. A compatibility manifest in the repository covers foster parenting in a bad table, the active formatting elements, the integration of foreign content, the raw text, the RCDATA, the optional end tags, and truncated input. It has no approved divergence. The parser does not execute a script and does not calculate a layout. Use browser mode when you need a mutation by a script, a layout value, or a different behavior of a live DOM. Refer to [Browser Mode](./browser-mode/). ## The error recovery [Section titled “The error recovery”](#the-error-recovery) A missing selector or a missing attribute follows the properties `required` and `default` of the field. The node `on-error` does not control this condition. The node `on-error` controls a transform error, a selector cardinality error, an output type error, and an external transform error. A collection with `on-row-error="skip"` drops only a row that contains an error that no policy recovered. Each dropped row adds the warning `W_ROW_SKIPPED` and sets `partial` to `true`. ## The cancellation [Section titled “The cancellation”](#the-cancellation) The runtime examines the context before it parses the HTML in the memory, and also before each output member and each collection row. A cancellation at one of these boundaries gives `E_EXECUTION_CANCELED` and keeps `context.Canceled` or `context.DeadlineExceeded` as the cause. A field policy or a row policy cannot recover a cancellation. The runtime does not interrupt one parser call that is in progress. It sees the cancellation at the next boundary. The TypeScript runtime has the same boundaries with an `AbortSignal`. ## Next step [Section titled “Next step”](#next-step) * [Offline Snapshots](./offline-snapshots/) — extraction without an acquisition. * [Browser Mode](./browser-mode/) — a live page and a workflow. * [Security and Responsible Use](./security-and-responsible-use/) — the rules before a live target.
# Offline Snapshots
> Execute an extractor against saved HTML with no network operation and no browser — the three execution boundaries, the rules of eligibility, and how to make a test that is stable.
An offline snapshot executes the extraction against HTML that you supply. It gets nothing. There is no URL expansion, no URL policy, no session, no HTTP request, no browser lease, and no JavaScript. Use a snapshot for a test in CI, for the development of a selector, and for a regression test after a change of a page. ## The three execution boundaries [Section titled “The three execution boundaries”](#the-three-execution-boundaries) One compiled program has three entry points. Their differences are important: | Entry point | Acquisition | Accepted modes | | ---------------------------------------------------- | ----------------------------------------------------------- | -------------------------------------- | | `Program.Extract`, `program.extract` | Follows the mode of the source. HTTP, or a browser adapter. | Each mode. | | `Program.ExtractHTML` | None. | HTTP mode only. Go only. | | `Program.ExtractSnapshot`, `program.extractSnapshot` | None. | Each mode, if the program is eligible. | `Program.ExtractHTML` is the original entry point of Go for saved HTML. It accepts an HTTP-mode program only. `ExtractSnapshot` is the general form. It accepts a browser-mode program also, but only when the program is eligible. ## The eligibility [Section titled “The eligibility”](#the-eligibility) The runtime calculates the eligibility for a snapshot from the full program, when it prepares the immutable program. A program that has a `workflow` node or an `evaluate-js` field value source **is not eligible**. It fails with `E_SNAPSHOT_UNSUPPORTED`. The runtime never ignores these operations. Static HTML cannot reproduce a mutation of a browser or a result of JavaScript. Thus a false success is not possible. If your browser-mode program needs a snapshot test, keep the JavaScript in a small number of fields and test the other fields offline. ## What a snapshot keeps [Section titled “What a snapshot keeps”](#what-a-snapshot-keeps) An eligible snapshot keeps the full behavior of the extraction: * the selectors and their cardinality; * the transforms, and also the external transforms; * the error recovery with `on-error` and `on-row-error`; * the warnings and the flag `partial`; * the cancellation. Thus a snapshot test examines the true behavior. It is not an approximation. ## The CLI [Section titled “The CLI”](#the-cli) Give the HTML with the option `--html`:
```bash
scrape-kdl extract ./extractor.kdl --html ./page.html
```
The CLI also accepts the standard input:
```bash
cat page.html | scrape-kdl extract ./extractor.kdl --html -
```
The declared inputs are not necessary here, because the runtime does no URL expansion. An offline execution causes no network activity. The option `--allow-private-hosts` has no effect on it. ## A stable test [Section titled “A stable test”](#a-stable-test) Save the HTML in your repository with the extractor. Then a test tells you about a change of your code, not about a change of the network. 1. Save the page one time. Use a page that has the conditions that you must control: an absent optional value, a collection with more than one row, and a bad row. 2. Put the file in the version control with the extractor. 3. Execute `extract` with `--html` in your CI job. 4. Compare the JSON result with a golden file. The result is deterministic. The same HTML and the same program always give the same value, the same warnings, and the same flag `partial`. When the page changes, get a new snapshot and examine the difference in the golden file. The difference tells you exactly what changed. ## The cancellation [Section titled “The cancellation”](#the-cancellation) The runtime examines the context before it parses the HTML in the memory, and also before each output member and each collection row. A cancellation gives `E_EXECUTION_CANCELED` and keeps the cause. A field policy or a row policy cannot recover it. The TypeScript runtime uses an `AbortSignal` at the same boundaries. ## Next step [Section titled “Next step”](#next-step) * [Diagnostics](./diagnostics/) — how to read the result of a failure. * [Go](../golang/compile-and-extract/) or [TypeScript and Bun](../npm/compile-and-extract/) — how to call a snapshot from a program.
# Patterns
> How to extract more than one page with Scraping KDL — the list-to-detail pair, the stop conditions of the pagination, and the pacing and the retry that stay in your application.
One extraction gets one document. The language has no loop. It does not follow a link, it does not repeat a request, and it does not decide when a crawl is complete. Your application controls that logic and calls a small extractor for each page. This boundary is intentional. It keeps the acquisition policy, the pacing, the retry, the removal of the duplicates, and the checkpoint in your code, where you can see them. It also permits an independent development and an independent test of a list extractor and a detail extractor. The full document, with the loops for Go and TypeScript, is in [patterns.md](https://github.com/hsblabs/scrape-kdl/blob/main/docs/patterns.md). ## The list and the detail [Section titled “The list and the detail”](#the-list-and-the-detail) The list program gives two values for each item: an absolute URL, and the stable identifier that the detail program needs. Two values are necessary. The URL makes the target visible for an examination. The identifier prevents a wrong operation: a full URL in a template placeholder becomes percent-encoded and then does not identify the same page. Save this document as `list.kdl`:
```kdl
extractor "catalog-list" version="2026-07-15" language-version="2026-07-15" {
source "html" {
fetch mode="http" url="https://example.invalid/catalog?page={page}"
}
input "page" type="int" required=#true
field "next_url" type="string?" required=#false {
select "a.next" match="first"
value "attr" name="href"
apply "url-resolve" base="https://example.invalid/catalog"
}
collection "items" min-items=0 {
select "article.item"
field "detail_url" type="string" required=#true {
select "a.detail" match="one"
value "attr" name="href"
apply "url-resolve" base="https://example.invalid/catalog"
}
field "detail_id" type="string" required=#true {
select "a.detail" match="one"
value "attr" name="href"
apply "path-segment" index=-1
apply "coalesce" value=""
apply "assert-matches" pattern=".+"
}
}
}
```
Save this document as `detail.kdl`:
```kdl
extractor "catalog-detail" version="2026-07-15" language-version="2026-07-15" {
source "html" {
fetch mode="http" url="https://example.invalid/items/{item_id}"
}
input "item_id" type="string" required=#true
field "title" type="string" required=#true {
select "h1" match="one"
value "text"
apply "normalize-whitespace"
}
}
```
This pair assumes that the last decoded path segment identifies the detail page, and that the template of the detail program makes the same target again. If the host, the query, or more than one path segment is significant, declare those parts as separate inputs. Never write one URL in a log and then get a different URL. ## The loop [Section titled “The loop”](#the-loop) Keep the loop outside of the two programs. The option `--json` gives a stable envelope for `jq`. Put the full argument of `--input` in quotation marks, because a decoded identifier can contain a space or a metacharacter of the shell.
```bash
#!/usr/bin/env bash
set -euo pipefail
main() {
local max_pages=100
local page page_json next_url
for ((page = 1; page <= max_pages; page++)); do
sleep 1
page_json="$(scrape-kdl extract ./list.kdl --input "page=$page" --json)"
jq -e '.ok == true and (.result.value.items | type == "array")' >/dev/null <<<"$page_json"
while IFS=$'\t' read -r detail_url detail_id; do
printf 'extracting %s\n' "$detail_url" >&2
sleep 1
scrape-kdl extract ./detail.kdl --input "item_id=$detail_id" --json
done < <(jq -r '.result.value.items[]? | [.detail_url, .detail_id] | @tsv' <<<"$page_json")
next_url="$(jq -r '.result.value.next_url // empty' <<<"$page_json")"
if [[ -z "$next_url" ]]; then
return 0
fi
done
printf 'pagination exceeded %d pages\n' "$max_pages" >&2
return 1
}
main "$@"
```
This example writes one JSON document for each detail result. A production caller writes each document to a durable storage and records the page and the `detail_url` before it continues. ## The stop condition [Section titled “The stop condition”](#the-stop-condition) Select a stop contract that agrees with the target: * Keep `min-items=0` and stop when the collection is empty. Use this contract when an empty page is a normal end marker. * Give an optional `next_url` and stop when its value is `null`. The loop above uses this contract. You can still use a numeric input `page` when the target has a next link and also stable page numbers. * Set `min-items=1` when an empty page is not normal. The last page then fails with `E_COLLECTION_CARDINALITY` at the path `output.items`. Your caller can use that exact code and that exact path as the end marker. For the third contract, examine the code before you stop. Do not treat each failure as the end of the pagination:
```bash
if ! page_json="$(scrape-kdl extract ./list.kdl --input "page=$page" --json)"; then
if jq -e '.error.code == "E_COLLECTION_CARDINALITY" and .error.path == "output.items"' \
>/dev/null <<<"$page_json"; then
break
fi
printf '%s\n' "$page_json" >&2
return 1
fi
```
Always set a maximum number of the pages or of the items. A site that changes can give the same next link without an end. Each loop in this document stops after 100 pages. It does not assume that a repetition is a successful completion. ## The pacing and the retry [Section titled “The pacing and the retry”](#the-pacing-and-the-retry) Scraping KDL adds no delay, does not repeat a page that failed, does not remove a duplicate URL, and does not keep the progress of a crawl. Your application must supply these policies. It must also keep sufficient state to continue without a repetition of unsafe work. Repeat only an error that your contract with the target classifies as temporary. Use a bounded backoff and keep the cancellation. The delay of one second in the example only shows you the position of the pacing. A production policy must cover each request, obey the guidance of the server, and adapt the concurrency and the delay to the target. Refer to [Security and Responsible Use](./security-and-responsible-use/) before you use these patterns on a service that you do not operate. ## Next step [Section titled “Next step”](#next-step) * [Offline Snapshots](./offline-snapshots/) — how to test the two programs without a network. * [Go](../golang/compile-and-extract/) or [TypeScript and Bun](../npm/compile-and-extract/) — the same loop in a library.
# Security and Responsible Use
> The trust model of Scraping KDL, the responsibilities of a host, the protections of the runtime, and the operational rules before you extract from a service that you do not operate.
Scraping KDL is an extraction tool. It does not give you permission to access or to re-use the content of a different person. You decide if each use is authorized and correct. This page is operational guidance. It is not legal advice. The normative documents are [security-model.md](https://github.com/hsblabs/scrape-kdl/blob/main/docs/security-model.md) and [responsible-use.md](https://github.com/hsblabs/scrape-kdl/blob/main/docs/responsible-use.md). ## A specification is a trusted asset [Section titled “A specification is a trusted asset”](#a-specification-is-a-trusted-asset) A KDL document is executable configuration. There are three levels of authority: | The document contains | The authority | | ------------------------------------- | ----------------------------------------------------- | | An HTTP source, no external transform | It controls the outbound URL and the selectors. | | A browser source | It also controls the interactions with the page. | | An `evaluate-js` node | It is executable code inside the context of the page. | Thus the runtime assumes that a specification is a trusted asset of your application. It does not supply a secure sandbox for untrusted KDL. Do not compile a document that a user of your service wrote. ## What the runtime supplies [Section titled “What the runtime supplies”](#what-the-runtime-supplies) * JavaScript is off until you give an explicit opt-in. * The capability validation completes before each network operation and each browser operation. * The response size and the operations have limits, and the limits obey a cancellation. * A hook for a URL policy examines the initial target and each HTTP redirect. `PublicInternetURLPolicy` and `NewPublicInternetHTTPClient` are prepared for the public internet, and the Go CLI applies them by default. * The runtime validates that a JavaScript result is JSON-compatible. * Each regular expression uses RE2 and has a linear execution time. * An adapter can hold a lease for the full extraction. Then the operations of two extractions cannot interleave on one page. * A structured error code does not contain a session value. The TypeScript HTTP runtime has the same sequence. Its preflight of the program, the selectors, the inputs, the session, the capabilities, and the external transforms completes before it calls `fetch`. It makes each redirect itself, thus the URL policy executes before each redirected request. It removes an authorization header and a host-only cookie header between the origins, and it limits the streamed body before the decode. ## What you must supply [Section titled “What you must supply”](#what-you-must-supply) The runtime is one layer. Your host must add: * an allowlist of the outbound network, or an isolated network; * a timeout for a request and for a browser; * a limit for the size of a response; * a browser process with low privileges and a separate context for each tenant; * a redaction of the secrets in your logs; * an explicit review before you enable JavaScript or an external transform. An injected source loader is your authority boundary. It gets a lexically resolved path of an import. Limit those paths to the intended set of the sources, obey a cancellation, and do not put the content of a source or a credential in an error. The compiler gives a loader no access to the file system, the network, or a subprocess. The functions `CompileFS` and `ValidateFS` limit the root and the names of the imports to the paths of `io/fs` and reject a lexical escape to a parent. Your `fs.FS` still defines the true authority. `os.DirFS` can follow a symbolic link outside of its directory. Use `os.Root.FS` when you need containment. ## The secrets [Section titled “The secrets”](#the-secrets) The CLI accepts a header and a cookie only from `--session-file FILE` or from `--session-file -`. It rejects a flag that carries a secret directly. A command argument can go into the history of a shell or become visible in a list of the processes.
```json
{
"headers": {"Authorization": ["Bearer example"]},
"cookies": [{"name": "session", "value": "example"}]
}
```
Make the file readable by the intended user only. Remove it or change it in agreement with your policy for the secrets. The policy `session policy="none"` stops the explicit session only. It does not clear the cookie jar of your `http.Client` or the state of an existing browser context. For an execution without a credential, supply an isolated client or an isolated context. Never put a live credential, a session cookie, an authorization header, or private extracted content in an issue, a log, a fixture, or an example. ## Before you use a live service [Section titled “Before you use a live service”](#before-you-use-a-live-service) Do these operations before you make a service the target: * Read its current terms of service and its policy for the automation. * Examine its instructions in `robots.txt`. * Get permission when the service or the applicable rules require it. * Use a documented API when that is the supported method to get the data. The [Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) gives an owner a standard method to declare a preference. A permission in `robots.txt` does not give you permission, does not have priority over the terms of the site, and does not decide if a use is legal. ## Limit the load [Section titled “Limit the load”](#limit-the-load) Your application controls the schedule, the concurrency, the retries, and the cache. It must: * send the requests slowly and add a variation of the delay where that is correct; * keep the concurrency inside a limit that the target accepts; * keep a response in a cache and not get the same unchanged content again; * stop or wait after a status `429` or `503`, after a timeout, and after a different sign of an overload; * limit the browser sessions, the sizes of the responses, the retries, and the total time. Scraping KDL has no global rate limiter. The absence of one is not permission to send unlimited traffic. ## Identify your client [Section titled “Identify your client”](#identify-your-client) When the automation is permitted, set a User-Agent that identifies your client. Add a useful contact or a URL of the project where that is correct. Do not imitate a different crawler or a browser to escape a policy of the target. The two CLIs have the option `--user-agent`, and a library host can set the equivalent option. ## The extracted content [Section titled “The extracted content”](#the-extracted-content) Collect only the data that your application needs. Examine the copyright, the database rights, the privacy, the confidentiality, the contractual limits, the retention, the access control, and the secure deletion before you keep or distribute the content. The requirements are different in each jurisdiction. A user in Japan must read the current Copyright Act and the Act on the Protection of Personal Information, and also each other applicable rule, and must get professional advice when that is necessary. ## No circumvention [Section titled “No circumvention”](#no-circumvention) This project does not supply and does not accept a function whose purpose is to bypass a CAPTCHA, an access control, a paywall, a rate limit, a bot detection, or a limit of an account. It does not supply credential stuffing, a false browser fingerprint, or stealth automation that hides a violation of a policy. Security research and interoperability work must use a system that you have permission to test. ## Next step [Section titled “Next step”](#next-step) * [HTTP Execution](./http-execution/) — the URL policy and the limits. * [Patterns](./patterns/) — the position of the pacing and the retry in your code.
# Selectors
> The portable CSS selector profile of Scraping KDL — what the compiler accepts, what it rejects, and why the same selector gives the same elements in HTTP mode and in browser mode.
Scraping KDL accepts a subset of CSS. The subset operates in the same manner in the internal DOM and in a live browser. The compiler rejects a selector outside of the subset. Thus you do not find the difference at execution time on one runtime only. The full profile is in [selectors-v0.1.md](https://github.com/hsblabs/scrape-kdl/blob/main/docs/spec/selectors-v0.1.md). ## What you can use [Section titled “What you can use”](#what-you-can-use) The profile contains the parts of CSS that each engine implements in the same manner: * the universal selector `*`, a type selector such as `div`, an ID selector such as `#main`, and a class selector such as `.entry`; * the attribute selectors: the presence form `[href]` and the operators `=`, `~=`, `|=`, `^=`, `$=`, and `*=`; * the combinators: descendant, child `>`, adjacent sibling `+`, and general sibling `~`; * a selector list with commas; * the structural pseudo-classes: `:first-child`, `:last-child`, `:only-child`, `:empty`, `:first-of-type`, `:last-of-type`, `:only-of-type`, `:nth-child(An+B)`, `:nth-last-child(An+B)`, `:nth-of-type(An+B)`, `:nth-last-of-type(An+B)`, and `:not(compound-selector)`.
```kdl
select "table.entries tbody tr:not(.header)"
```
## What the compiler rejects [Section titled “What the compiler rejects”](#what-the-compiler-rejects) The compiler rejects these constructions with the diagnostic `E_SELECTOR_UNSUPPORTED`: * each pseudo-element; * `:has()`, `:is()`, `:where()`, and `:scope`; * a shadow DOM selector and a vendor pseudo-selector; * a namespace selector; * a CSS escape sequence in an identifier or a string token; * the case-sensitivity flags `i` and `s` in an attribute selector; * a pseudo-class of the user interface state, such as `:hover`, `:focus`, `:visited`, and `:checked`. A selector with bad syntax gets the different diagnostic `E_SELECTOR_INVALID`. The rejection occurs at compile time, with the position of the character in the selector:
```text
extractor.kdl:9:5: error E_SELECTOR_UNSUPPORTED: selector byte 9: unsupported pseudo-class "has" [output.title.selection]
```
An implementation can use a larger selector engine internally. It must still reject each selector outside of the profile under the language version `2026-07-15`. Thus a program that compiles on one runtime also compiles on the other runtime. ## Why the profile is small [Section titled “Why the profile is small”](#why-the-profile-is-small) A pseudo-class of the user interface state, such as `:hover`, has a value only in a live browser. A structural pseudo-class gives the same result in each engine. If the two classes were in one profile, a program could operate correctly in browser mode and fail in HTTP mode. The profile prevents this condition. Each accepted selector has the same meaning in the two modes. ## Alternatives to `:has()` [Section titled “Alternatives to :has()”](#alternatives-to-has) The pseudo-class `:has()` is the most frequent absent function. Use one of these methods: * Select the parent element with a collection, then use a field in the row to find the child. A row without the child gives a missing value, and you control that condition with `required`. * Select the child element directly when you need only its value. The parent element is frequently not necessary. * In browser mode only, use `evaluate-js` with `scope="current"`. This is an intentional escape from the portable profile. It needs the capability `browser.evaluate-js` and an explicit opt-in for JavaScript. ## Cardinality [Section titled “Cardinality”](#cardinality) The property `match` of the node `select` controls the number of the elements: | Value | Behavior | | --------------- | ------------------------------------------------------------------------------------------------------------ | | `one` (default) | Exactly one element. Zero elements is a missing value. Two or more elements causes `E_SELECTOR_CARDINALITY`. | | `first` | The first element in document order. Zero elements is a missing value. | A collection is different. Its `select` gives each element that agrees with the selector, in document order. One row comes from each element. Use `match="one"` when the page must have exactly one element. The runtime then tells you when the structure of the page changed. Use `match="first"` only when more than one element is correct and you want the first element. ## Semantics [Section titled “Semantics”](#semantics) * The matches follow the tree order of the DOM. * A selector list gives the elements in document order and has no duplicates. * The names of the elements and the attributes follow the ASCII case-insensitive rules of HTML. ## Next step [Section titled “Next step”](#next-step) * [Transforms](./transforms/) — how to make a value from the selected text. * [Browser Mode](./browser-mode/) — the selector behavior with a live page.
# Transforms
> How a value pipeline operates in Scraping KDL — the built-in registry, the declared transforms, the match tables, the external host functions, and the RE2 regular expression profile.
A value source gives you a string. A transform makes that string into the value that you declared. Each `apply` node executes in source order. The compiler checks the types of the full sequence before the execution. The normative registry is in [builtins-v0.1.md](https://github.com/hsblabs/scrape-kdl/blob/main/docs/spec/builtins-v0.1.md). Compiled examples are in the [transform cookbook](https://github.com/hsblabs/scrape-kdl/blob/main/docs/cookbook.md). ## The pipeline [Section titled “The pipeline”](#the-pipeline)
```kdl
field "price" type="u32" required=#true {
select ".price" match="one"
value "text"
apply "trim"
apply "replace" old="," new=""
apply "parse-int" as="u32"
}
```
Each call gets the output of the previous call. The output type of one call must agree with the input type of the next call. A bad sequence gives `E_TRANSFORM_TYPE_MISMATCH` at compile time, not at execution time. The final output must be assignable to the `type` of the field. There is no implicit conversion. A `string` does not become a `u32` without `parse-int`. ## The built-in registry [Section titled “The built-in registry”](#the-built-in-registry) The built-in transforms have four groups: | Group | Function | Examples | | ---------- | ---------------------------------------- | --------------------------------------------------------------------- | | String | Change the shape of the text. | `trim`, `normalize-whitespace`, `replace`, `regex-capture`, `split` | | Conversion | Make a typed value from the text. | `parse-int`, `parse-float`, `parse-bool`, `empty-to-null`, `coalesce` | | URL | Read or resolve a URL with RFC 3986. | `url-resolve`, `url-query`, `url-path`, `path-segment` | | Validation | Give the input again, or cause an error. | `assert-matches`, `assert-enum`, `assert-min`, `assert-max` | Refer to the registry for the full list and for each signature. These rules apply to each built-in: * A built-in name is reserved. You cannot shadow it. * An unknown, a duplicate, or a type-incompatible call property is an error. * A string index is an index of the Unicode scalar values. It is not an offset of the UTF-8 bytes or of the UTF-16 code units. * Each numeric output must be finite and inside the range of the target type. The transform `parse-int` does not remove the unwanted spaces. Apply `trim` first. It must also consume the full input. Thus the text `12 kg` causes an extraction error and does not give the value `12`. This behavior is intentional. A silent partial parse hides a change of the page. ## The regular expressions [Section titled “The regular expressions”](#the-regular-expressions) Each regular expression uses the RE2 syntax. RE2 does not have lookaround, backreference, named capture group, or conditional expression. These absent functions are the cost of a predictable execution time. You can give the flags `i`, `m`, and `s` with the property `flags`. Each flag can be present one time only. In a replacement string, `$0` is the full match and `$1` to `$99` are the numbered captures. Write `$$` for a literal dollar sign. The syntax of JavaScript for a regular expression is not the syntax of the language. A pattern that operates in `RegExp` can still be an error with `E_REGEX_INVALID`. ## The declared transforms [Section titled “The declared transforms”](#the-declared-transforms) Give a name to a sequence that you use more than one time:
```kdl
transform "extract_horse_id" input="string" output="string?" {
pipeline {
apply "regex-capture" pattern=#"/horse/([^/?#]+)"# group=1
}
}
```
A declared transform has exactly one body: `pipeline`, `match`, or `external`. In version 0.1 a declared transform takes no call argument and no call property. Make a second transform when you need different parameters. ## The match tables [Section titled “The match tables”](#the-match-tables) Use `match` for a table of the scalar values:
```kdl
transform "normalize_sex" input="string" output="string" {
match {
case "牡" "male"
case "牝" "female"
case "セ" "gelding"
default "unknown"
}
}
```
The runtime compares the cases in source order with exact equality. Exactly one `default` is necessary. Two cases with the same input value are an error. The input type and the output type must be scalar or nullable scalar. ## The external transforms [Section titled “The external transforms”](#the-external-transforms) An external transform is a function of the host. Use it when the logic cannot be in the language, for example a decryption or a call to an internal service.
```kdl
transform "decrypt_payload" input="string" output="object" {
external symbol="decrypt_payload"
}
```
The host supplies the symbol from a registry. The compiler adds the capability `transform.external:decrypt_payload` to the IR. If the registry does not have the symbol, the validation fails before the fetch. Thus you do not find the absent function at the middle of an extraction. After the function gives a result, the runtime immediately checks the result against the declared output type. A mismatch fails with `E_EXTERNAL_TRANSFORM_RESULT_TYPE`, before each subsequent transform. An external transform is an intentional escape from the portable behavior. The same program then needs the same registry on each runtime. ## How a name is resolved [Section titled “How a name is resolved”](#how-a-name-is-resolved) The compiler resolves the argument of `apply` in this sequence: 1. an exact built-in name; 2. an exact local transform name; 3. a qualified imported name, in the form `alias.name`. You must write an imported transform with its alias. An unqualified reference to an imported transform is an error. A built-in name cannot be shadowed. Thus the meaning of `apply "trim"` never changes. ## Next step [Section titled “Next step”](#next-step) * [HTTP Execution](./http-execution/) — how the runtime gets the document. * [Diagnostics](./diagnostics/) — how to read a compile error. * [Patterns](./patterns/) — the usual combinations.
# Write an Extractor
> The structure of an extractor document — the source, the inputs, the fields, the collections, the transforms, and the difference between a missing value and an error.
An extractor document declares what to get and what shape the result has. This page shows you the parts of the document and the rules that control them. The full grammar is in [language-v0.1.md](https://github.com/hsblabs/scrape-kdl/blob/main/docs/spec/language-v0.1.md). ## The document [Section titled “The document”](#the-document) A file contains one `extractor` node or one `module` node. It cannot contain both.
```kdl
extractor "race-detail" version="2026-07-15" language-version="2026-07-15" {
source "html" {
fetch mode="http" url="https://example.com/race/{race_id}"
}
input "race_id" type="string" required=#true
field "title" type="string" required=#true {
select "h1" match="one"
value "text"
apply "normalize-whitespace"
}
}
```
Both properties of the root node are necessary: * `version` is your revision identifier. It must be a real calendar date in the form `YYYY-MM-DD`. * `language-version` selects the language contract. Its value must be `2026-07-15`. The children `input`, `transform`, `field`, and `collection` can be in any sequence. The extractor needs exactly one `source`. ## The source [Section titled “The source”](#the-source)
```kdl
source "html" {
fetch mode="http" url="https://example.com/items/{item_id}"
session policy="optional"
}
```
The node `source` accepts only the argument `"html"`. It needs exactly one `fetch` child. It can have one `session` child. It can have one `workflow` child, but only when the fetch mode is `browser`. The property `mode` is `http` or `browser`. In HTTP mode the nodes `workflow` and `evaluate-js` are forbidden. The compiler rejects them with `E_BROWSER_CAPABILITY_REQUIRED`. Refer to [Browser Mode](./browser-mode/). The property `policy` of the node `session` is `none`, `optional`, or `required`. The default is `none`. With `required`, the runtime stops before the fetch when the host supplies no session. ## The inputs [Section titled “The inputs”](#the-inputs)
```kdl
input "race_id" type="string" required=#true
input "lang" type="string" required=#false default="ja"
```
An input has one of these types: `string`, `bool`, `int`, or `float`. The property `required` has the default `#true`. A necessary input cannot have a default value. The URL template uses an input with the syntax `{input_name}`. The runtime expands the template before the fetch. It percent-encodes a string and keeps only the unreserved characters of RFC 3986. To write a literal brace, use `{{` or `}}`. If a required input is absent, the extraction stops before the fetch. ## The fields [Section titled “The fields”](#the-fields)
```kdl
field "horse_name" type="string" required=#true {
select ".horse-name a" match="one"
value "text"
apply "normalize-whitespace"
}
```
A field has one type and one value source. The children are: * zero or one `select`; * exactly one value source, either `value` or `evaluate-js`; * zero or more `apply` nodes, in source order; * zero or one `on-error`. The property `match` of the node `select` is `one` or `first`. The default is `one`. With `one`, two or more matches cause `E_SELECTOR_CARDINALITY`. With `first`, the runtime uses the first match in document order. Refer to [Selectors](./selectors/). There are three value sources for a static DOM: | Source | Result | | -------------------------- | ------------------------------------------------------------------------------------------------ | | `value "text"` | The text of the descendant nodes, in DOM order. The runtime does not remove the unwanted spaces. | | `value "html"` | The inner HTML of the selected element. | | `value "attr" name="href"` | The attribute of the DOM, not the resolved property of a browser. | For an absolute link, apply the transform `url-resolve` to the attribute. The attribute itself keeps its relative form. ## A missing value is not an error [Section titled “A missing value is not an error”](#a-missing-value-is-not-an-error) This distinction is important. The two conditions have different controls. A value is **missing** when the selector found zero elements, or when the attribute does not exist. The property `required` controls this condition: | Declaration | Result of a missing value | | ------------------------------------- | ------------------------------------------------------- | | `required=#true` | The error `E_REQUIRED_VALUE_MISSING`. | | `required=#false` with a `default` | The default value. No warning. `partial` stays `false`. | | `required=#false` without a `default` | The value `null`. No warning. `partial` stays `false`. | An **error** is a different condition. A transform failure, a type mismatch, a JavaScript error, or an adapter failure is an error. The node `on-error` controls this condition:
```kdl
on-error "warn"
```
| Policy | Result | | --------- | ------------------------------------------------------------------------ | | `fail` | The runtime propagates the error. | | `null` | The runtime gives `null` and sets `partial` to `true`. | | `warn` | The runtime gives `null`, adds a warning, and sets `partial` to `true`. | | `default` | The runtime gives the default of the field and sets `partial` to `true`. | The default policy is `fail` for a necessary field and `null` for an optional field. The policies `null` and `warn` need an output type that permits null. The policy `default` needs a default value. The node `on-error` does not control a missing selector or a missing attribute. Use `required` for that condition. ## The collections [Section titled “The collections”](#the-collections)
```kdl
collection "entries" min-items=1 on-row-error="skip" {
select "table.entries tbody tr"
field "number" type="u8" required=#true {
select ".number"
value "text"
apply "parse-int" as="u8"
}
}
```
A collection needs exactly one `select` and a minimum of one child field or collection. Each element that agrees with the selector becomes a row, in document order. A collection can contain another collection. The properties `min-items` and `max-items` limit the number of the rows. The value of `max-items` must be equal to or larger than `min-items`. The property `required=#true` gives an effective minimum of one row. The property `on-row-error` is `fail` or `skip`. The default is `fail`. With `skip`, the runtime drops a row when a child has an error that no policy recovered. Each dropped row adds a warning and sets `partial` to `true`. The runtime examines the limits after it drops the rows. ## The types [Section titled “The types”](#the-types) The primitive types are `string`, `bool`, `int`, the unsigned integers from `u8` to `u64`, the signed integers from `i8` to `i64`, `float`, `f32`, `f64`, `object`, and `unknown`. Add `[]` for an array and `?` for a nullable type. The operators bind from the left to the right. Thus `string?[]` is an array of nullable strings, and `string[]?` is a nullable array. Use parentheses to make the intent clear. There is no implicit conversion. A string does not become a number without the transform `parse-int` or `parse-float`. An integer overflow is an extraction error, not a truncation. Each float must be finite. ## The transforms [Section titled “The transforms”](#the-transforms) Each `apply` node executes in source order. The output type of one call must agree with the input type of the next call. The compiler examines the full pipeline and rejects a bad sequence with `E_TRANSFORM_TYPE_MISMATCH`. You can declare your own transform in the extractor or in a module:
```kdl
transform "extract_horse_id" input="string" output="string?" {
pipeline {
apply "regex-capture" pattern=#"/horse/([^/?#]+)"# group=1
}
}
```
A declared transform has exactly one body: `pipeline`, `match`, or `external`. Refer to [Transforms](./transforms/). ## The modules [Section titled “The modules”](#the-modules) Put the shared transforms in a module document and import it:
```kdl
import "./modules/common.kdl" as="common"
extractor "race-detail" version="2026-07-15" language-version="2026-07-15" {
// ...
field "horse_id" type="string?" {
select "a.horse" match="first"
value "attr" name="href"
apply "common.extract_horse_id"
}
}
```
The rules of an import are strict: * the path must be relative. A remote URL is an error. * the property `as` is necessary and each alias must be unique. * the target must be a module document. * a cycle in the import graph is an error. * you must write an imported transform with its alias, in the form `alias.name`. A module exports each transform that it declares directly. Version 0.1 has no re-export. ## The result [Section titled “The result”](#the-result) An extraction gives three parts:
```text
value the object from the fields and the collections
warnings the warnings, in the sequence of the execution
partial true when the runtime recovered an error or dropped a row
```
The flag `partial` becomes `true` only after a recovery or a dropped row. An expected optional value that is absent does not set the flag. Thus you can trust a result that has `partial: false`. ## Next step [Section titled “Next step”](#next-step) * [Selectors](./selectors/) — the portable subset of CSS. * [Transforms](./transforms/) — the pipeline, the match, and the external transforms. * [Patterns](./patterns/) — the usual document shapes.
# TypeScript and Bun
> The npm packages of Scraping KDL — the three entry points, the reason the core package has no access to the file system, and the position of the Playwright adapter.
The package `@hsblabs/scrape-kdl` gives you the compiler, the diagnostics, the IR, the HTTP runtime, the offline snapshot runtime, and the types of the browser adapter. It supports Node.js 22 or later and Bun 1.3 or later. It supports only ESM.
```bash
npm install @hsblabs/scrape-kdl@1.0.4
```
## The three entry points [Section titled “The three entry points”](#the-three-entry-points) | Entry point | Content | | ------------------------------- | ------------------------------------------------------------------------------------------------------------ | | `@hsblabs/scrape-kdl` | `compile`, `validate`, the `Program` interface, the execution options, and the types of the browser adapter. | | `@hsblabs/scrape-kdl/node` | `compileFile` and `validateFile`. It also exports each part of the core entry point again. | | `@hsblabs/scrape-kdl/authoring` | The bounded authoring model and the catalog of the built-in transforms. | The division is intentional. The core package has no automatic access to the file system and no automatic network loader. It resolves a relative path lexically and asks a `SourceLoader` for the bytes.
```ts
export interface SourceLoader {
load(path: string, context: SourceLoadContext): Promise;
}
```
Thus you decide which files the compiler can read. The entry point `/node` supplies the loader for the file system. Use `/node` when the compilation of a local file is correct for you. Use the core entry point with your own loader when you must limit the sources. Your loader is an authority boundary. Limit the paths to the intended set, obey the `AbortSignal`, and do not put the content of a source or a credential in an error. ## The first compilation [Section titled “The first compilation”](#the-first-compilation)
```ts
import { compileFile } from "@hsblabs/scrape-kdl/node";
const { program, diagnostics } = await compileFile("./extractor.kdl");
if (!program) {
for (const diagnostic of diagnostics) {
console.error(`${diagnostic.code}: ${diagnostic.message}`);
}
process.exit(1);
}
const result = await program.extract({ id: "123" });
console.log(result.value);
```
The function `compile` gives a `CompileResult`. The field `program` is absent when the diagnostics have an error. Examine `program` before you use it. Refer to [Compile and Extract in TypeScript](./compile-and-extract/). ## The metadata of a program [Section titled “The metadata of a program”](#the-metadata-of-a-program) A compiled program tells you what it needs, before you execute it:
```ts
program.metadata.capabilities; // readonly string[]
program.metadata.languageVersion; // "2026-07-15"
program.metadata.irVersion; // "2026-07-15"
program.metadata.files; // each source file, with its SHA-256
program.descriptor.source.fetchMode; // "http" or "browser"
program.descriptor.source.sessionPolicy; // "none", "optional", or "required"
```
Use `metadata.capabilities` to permit or to refuse a program in your host. Use `metadata.files` to make a record of the exact sources that you compiled. ## The browser mode [Section titled “The browser mode”](#the-browser-mode) The core package does not contain a browser. It declares the interface `BrowserAdapter` and executes a browser-mode program with the adapter that you give in `options.browser`. The official adapter is a separate package:
```bash
npm install @hsblabs/scrape-kdl-playwright@1.0.4 playwright
```
Refer to [Playwright Adapter](./playwright/) and to [Browser Mode](../guides/browser-mode/). ## The authoring model [Section titled “The authoring model”](#the-authoring-model) The entry point `/authoring` makes a KDL document from a structure of data. Use it in an editor, a generator, or a tool that makes an extractor from a selection of the user.
```ts
import { builtinCatalog, write } from "@hsblabs/scrape-kdl/authoring";
const catalog = builtinCatalog("2026-07-15");
```
The catalog is versioned. Select the exact language version. Do not use a version `latest`. The function `write` makes the KDL text. Then compile that text with the ordinary compiler and examine the diagnostics. ## Bun [Section titled “Bun”](#bun) Bun 1.3 or later supports the core package:
```bash
bun add @hsblabs/scrape-kdl@1.0.4
```
The tests of the Playwright adapter use Node.js 22 or later. ## Next step [Section titled “Next step”](#next-step) * [Compile and Extract in TypeScript](./compile-and-extract/) — the full API of the execution. * [Playwright Adapter](./playwright/) — browser mode with an official adapter.
# Compile and Extract in TypeScript
> The TypeScript execution API — compile with an injected loader, the execution options, the extraction result, the external transforms, the URL policy, and the cancellation.
This page shows you the API of the compilation and the execution. Each type here comes from the package `@hsblabs/scrape-kdl`. ## Compile [Section titled “Compile”](#compile)
```ts
import { readFile } from "node:fs/promises";
import { compile } from "@hsblabs/scrape-kdl";
const compiled = await compile({
path: "extractor.kdl",
data: await readFile("extractor.kdl", "utf8"),
});
if (!compiled.program) {
throw new Error(JSON.stringify(compiled.diagnostics));
}
const result = await compiled.program.extract({ id: "123" });
console.log(result.value);
```
A `Source` has a `path` and a `data`. The `path` is a logical identity for the diagnostics, the source locations, the resolution of the imports, and the file identities in the IR. It does not have to name a file of the operating system. The result of a compilation has a `program` and a `diagnostics`. The field `program` is absent when the diagnostics have an error. Always examine `program` before you use it. From the entry point `/node` you can also use `compileFile(path)` and `validateFile(path)`. The function `validate` gives only the diagnostics and no program. ## The imports [Section titled “The imports”](#the-imports) A source with an import needs a loader. Without a loader, the compilation fails before it gives an IR.
```ts
import { compile, type SourceLoader } from "@hsblabs/scrape-kdl";
const loader: SourceLoader = {
async load(path, context) {
if (!allowedPaths.has(path)) {
throw new Error(`refused: ${path}`);
}
return await readFile(path, "utf8");
},
};
const compiled = await compile(source, { loader });
```
The compiler resolves each import lexically, relative to the source that imports it, before it calls your loader. Your loader gives only the bytes. The compiler makes the parse, the validation, the detection of the cycles, the hash, and the deterministic order. A failure of the loader is an operational error and not a diagnostic of the document. The promise rejects with the reason of the abort or with a `SourceLoadError`. The field `SourceLoadError.cause` keeps the original failure. ## Execute [Section titled “Execute”](#execute)
```ts
const result = await program.extract(
{ id: "123" },
{ requestTimeoutMs: 15_000 },
);
```
The first argument has the runtime inputs. The second argument has the execution options: | Option | Function | | -------------------- | ----------------------------------------------------- | | `browser` | The `BrowserAdapter` for a browser-mode program. | | `allowJavaScript` | Permits `evaluate-js`. The default is off. | | `fetch` | Your own implementation of `fetch`. | | `session` | The headers and the cookies for the initial request. | | `externalTransforms` | The registry of the host functions. | | `requestTimeoutMs` | The timeout of the HTTP request. | | `maxResponseBytes` | The limit of the body of the response. | | `userAgent` | The User-Agent of the HTTP request. | | `urlPolicy` | A function that examines each URL before the request. | | `signal` | An `AbortSignal` for the cancellation. | ## The result [Section titled “The result”](#the-result)
```ts
interface ExtractionResult {
readonly value: Readonly>;
readonly warnings: readonly Warning[];
readonly partial: boolean;
}
```
A `Warning` has a `code`, a `message`, an optional `path`, and an optional `row`. The flag `partial` is `true` only after the runtime recovered an error or dropped a row. A failure gives an `ExecutionError`. It has a `code`, an optional `path`, and an optional `cause`. Examine the `code`. Refer to [Diagnostics](../guides/diagnostics/). The field `value` is a dynamic JSON value. The extractor validates its declared output types, but your application still owns the correspondence between the names of the fields and your TypeScript model. Validate the value with your own schema at that boundary. Do not use an unchecked type assertion. ## The offline snapshot [Section titled “The offline snapshot”](#the-offline-snapshot)
```ts
const html = await readFile("./page.html", "utf8");
const result = await program.extractSnapshot(html);
```
The method `extractSnapshot` does no acquisition. It accepts an HTTP-mode program and also a browser-mode program, but the program must be eligible. A program with a workflow or with an `evaluate-js` field value source fails with `E_SNAPSHOT_UNSUPPORTED`. Refer to [Offline Snapshots](../guides/offline-snapshots/). ## The external transforms [Section titled “The external transforms”](#the-external-transforms)
```ts
const result = await program.extract(inputs, {
externalTransforms: {
decrypt_payload: async (context, input) => decrypt(input as string),
},
});
```
A function gets a context with an optional `signal`, and the input. It gives a JSON value or a promise of a JSON value. If the registry does not have a symbol that the program needs, the validation fails before the fetch. After your function gives a result, the runtime immediately examines the result against the declared output type. ## The URL policy [Section titled “The URL policy”](#the-url-policy)
```ts
const result = await program.extract(inputs, {
urlPolicy: (context, url) => {
if (url.hostname !== "example.com") {
throw new Error(`refused host: ${url.hostname}`);
}
},
});
```
The policy executes before the initial request and before each redirect. The TypeScript runtime makes each redirect itself, thus the policy sees each hop. An error from the policy stops the extraction with `E_URL_POLICY`. The runtime also removes an authorization header and a host-only cookie header between the origins, and it limits the streamed body before it decodes it. The TypeScript package has no prepared policy for the public internet. Write your own policy. The Go equivalent is `PublicInternetURLPolicy`. ## The cancellation [Section titled “The cancellation”](#the-cancellation)
```ts
const controller = new AbortController();
setTimeout(() => controller.abort(), 5_000);
const result = await program.extract(inputs, { signal: controller.signal });
```
The runtime propagates the cancellation of the parent separately from the timeout of the request. It examines the signal before the parse of the HTML and also before each output member and each collection row. A cancellation gives `E_EXECUTION_CANCELED`. A field policy or a row policy cannot recover it. ## The compatibility [Section titled “The compatibility”](#the-compatibility)
```ts
import { supportedLanguageVersions, supportedIRVersions } from "@hsblabs/scrape-kdl";
```
The two functions give the exact versions that this build accepts. Select an exact version. There is no moving alias `latest`. ## Next step [Section titled “Next step”](#next-step) * [Playwright Adapter](./playwright/) — how to execute a browser-mode program. * [Patterns](../guides/patterns/) — a loop over more than one page.
# Playwright Adapter
> The official Playwright browser adapter for TypeScript — the ownership of the browser, the isolation of each context, the cleanup after a timeout, and the supported browsers.
The package `@hsblabs/scrape-kdl-playwright` implements the contracts `BrowserAdapter` and `BrowserAdapterLease`. It is a separate package. Thus the core package does not get Playwright in its dependency graph.
```bash
npm install @hsblabs/scrape-kdl-playwright@1.0.4 playwright
npx playwright install chromium
```
## The use [Section titled “The use”](#the-use)
```ts
import { chromium } from "playwright";
import { PlaywrightAdapter } from "@hsblabs/scrape-kdl-playwright";
const browser = await chromium.launch({ headless: true });
const adapter = new PlaywrightAdapter(browser);
try {
const result = await compiled.program.extract(
{ id: "123" },
{ browser: adapter, allowJavaScript: true },
);
console.log(result.value);
} finally {
await adapter.close();
await browser.close();
}
```
The option `allowJavaScript: true` is necessary only when your program has an `evaluate-js` node. Do not set it for a program that does not need it. ## The ownership [Section titled “The ownership”](#the-ownership) You own the `Browser`. The adapter owns only the isolated contexts that it makes. The method `adapter.close()` closes the contexts of the adapter. It never closes your browser. Close the browser yourself, as the example shows. Thus you can use one browser for more than one adapter, or keep one browser during the full life of your process. ## The isolation [Section titled “The isolation”](#the-isolation) Each call of `navigate` does these operations in this sequence: 1. It closes the previous context of the adapter. 2. It makes a new context. 3. It installs the explicit headers of the session, the cookies, and the User-Agent. 4. It navigates a new page. Thus a cookie, a storage value, a mutation of a page, or a failed operation from one extraction cannot go into the next extraction. ## The mapping of the operations [Section titled “The mapping of the operations”](#the-mapping-of-the-operations) * A portable selector becomes a Playwright locator, in the scope of the document or of the current element. * A read of the text uses the `textContent` of the descendants. A read of the HTML uses `innerHTML`. An attribute gives the value of the attribute of the DOM. * The workflow steps `wait`, `click`, `fill`, `press`, `scroll`, and the configured network-idle operation execute in source order. * With `scope="document"`, the JavaScript function gets no argument. With `scope="current"`, it gets the current DOM element. * A JavaScript result crosses the boundary of the adapter. The core runtime then examines it for the JSON compatibility and for the declared type `returns`. The adapter also limits a query for `match="first"` and `match="one"`. It does not make the full set of the matches. ## The timeout and the cancellation [Section titled “The timeout and the cancellation”](#the-timeout-and-the-cancellation) The adapter puts each operation in a race against the public timeout and the `AbortSignal`. For an operation that can continue inside the browser, a timeout or a cancellation first closes the isolated context. The runtime holds the lease of the adapter until this cleanup is complete. Thus no operation continues after the release of the lease. A later extraction makes a new context and operates correctly. ## The supported browsers [Section titled “The supported browsers”](#the-supported-browsers) Chromium is the blocking target of version 1. The scheduled workflow of the browser also reports the results of Firefox and WebKit, with a non-blocking status. Use Chromium for a production extraction. A promotion of the support of a different browser needs a separate compatibility decision. ## Next step [Section titled “Next step”](#next-step) * [Browser Mode](../guides/browser-mode/) — the workflow steps and the rules of the JavaScript. * [Security and Responsible Use](../guides/security-and-responsible-use/) — the isolation of the contexts and the control of the network.
# References
> The canonical documents of Scraping KDL — the normative language specifications, the IR schema, the type declarations, the compatibility policy, and the external standards.
The pages of this site tell you how to use Scraping KDL. The documents below are the canon. When a page here and a document below do not agree, the document below is correct. ## The normative specification [Section titled “The normative specification”](#the-normative-specification) | Document | Content | | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | [Language v0.1](https://github.com/hsblabs/scrape-kdl/blob/main/docs/spec/language-v0.1.md) | The full grammar, the semantics, and the validation rules. | | [Built-ins v0.1](https://github.com/hsblabs/scrape-kdl/blob/main/docs/spec/builtins-v0.1.md) | Each built-in transform, with its signature and its behavior. | | [Selectors v0.1](https://github.com/hsblabs/scrape-kdl/blob/main/docs/spec/selectors-v0.1.md) | The portable subset of CSS and the rejected constructions. | | [Diagnostics](https://github.com/hsblabs/scrape-kdl/blob/main/docs/spec/diagnostics.md) | Each diagnostic code, its severity, and its condition. | | [Grammar summary](https://github.com/hsblabs/scrape-kdl/blob/main/docs/spec/grammar-summary.ebnf) | The EBNF summary of the syntax. | Machine-readable data for a tool: * [`builtins-v0.1.contract.json`](https://github.com/hsblabs/scrape-kdl/blob/main/docs/spec/builtins-v0.1.contract.json) — the signatures of the transforms. * [`builtins-v0.1.authoring.json`](https://github.com/hsblabs/scrape-kdl/blob/main/docs/spec/builtins-v0.1.authoring.json) — the data for an editor and for a completion. * [`conformance-coverage.json`](https://github.com/hsblabs/scrape-kdl/blob/main/docs/spec/conformance-coverage.json) — the coverage of the fixtures against the specification. ## The Validated IR [Section titled “The Validated IR”](#the-validated-ir) | Document | Content | | ------------------------------------------------------------------------------------- | --------------------------------------------------- | | [IR schema](https://github.com/hsblabs/scrape-kdl/blob/main/docs/ir/schema.json) | The JSON Schema of the Validated IR. | | [IR README](https://github.com/hsblabs/scrape-kdl/blob/main/docs/ir/README.md) | The version policy and the layout of the directory. | | [Example IR](https://github.com/hsblabs/scrape-kdl/blob/main/docs/ir/example.ir.json) | A complete IR document. | The IR is the boundary between the compiler and a runtime. Use the schema when you write a tool that reads a program or that makes one. ## The API declarations [Section titled “The API declarations”](#the-api-declarations) * [TypeScript `index.d.ts`](https://github.com/hsblabs/scrape-kdl/blob/main/docs/api/typescript/index.d.ts) — the portable entry point. * [TypeScript `node.d.ts`](https://github.com/hsblabs/scrape-kdl/blob/main/docs/api/typescript/node.d.ts) — the entry point for the file system. * [TypeScript `authoring.d.ts`](https://github.com/hsblabs/scrape-kdl/blob/main/docs/api/typescript/authoring.d.ts) — the data for an editor tool. * [Public API v1](https://github.com/hsblabs/scrape-kdl/blob/main/docs/public-api-v1.md) — the stable surface of the two runtimes. For the Go API, use `go doc`:
```bash
go doc github.com/hsblabs/scrape-kdl
```
## The runtime documents [Section titled “The runtime documents”](#the-runtime-documents) | Document | Content | | ------------------------------------------------------------------------------------------------ | --------------------------------------------------------- | | [Compiler pipeline](https://github.com/hsblabs/scrape-kdl/blob/main/docs/compiler-pipeline.md) | The seven stages of the validation. | | [HTTP runtime](https://github.com/hsblabs/scrape-kdl/blob/main/docs/http-runtime.md) | The behavior of the request and of the recovery. | | [Browser runtime](https://github.com/hsblabs/scrape-kdl/blob/main/docs/browser-runtime.md) | The contract of the adapter and the workflow. | | [Playwright adapter](https://github.com/hsblabs/scrape-kdl/blob/main/docs/playwright-adapter.md) | The details of the TypeScript adapter. | | [go-rod adapter](https://github.com/hsblabs/scrape-kdl/blob/main/docs/rod-adapter.md) | The details of the Go adapter. | | [HTML compatibility](https://github.com/hsblabs/scrape-kdl/blob/main/docs/html-compatibility.md) | The parse of the HTML and the differences from a browser. | | [Performance](https://github.com/hsblabs/scrape-kdl/blob/main/docs/performance.md) | The measured behavior and the limits. | ## The compatibility and the security [Section titled “The compatibility and the security”](#the-compatibility-and-the-security) | Document | Content | | ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------- | | [Compatibility](https://github.com/hsblabs/scrape-kdl/blob/main/docs/compatibility.md) | The supported versions of Go, of Node.js, and of Bun. | | [Versioning](https://github.com/hsblabs/scrape-kdl/blob/main/docs/versioning.md) | The version of the language, the version of the IR, and the version of the release. | | [Migrate to v1](https://github.com/hsblabs/scrape-kdl/blob/main/docs/migrating-to-v1.md) | The changes from an earlier version. | | [Changelog](https://github.com/hsblabs/scrape-kdl/blob/main/CHANGELOG.md) | The history of the releases. | | [Security model](https://github.com/hsblabs/scrape-kdl/blob/main/docs/security-model.md) | The trust levels and the protections of the runtime. | | [Security policy](https://github.com/hsblabs/scrape-kdl/blob/main/SECURITY.md) | How to report a vulnerability. | | [Responsible use](https://github.com/hsblabs/scrape-kdl/blob/main/docs/responsible-use.md) | The obligations of an operator. | ## The project [Section titled “The project”](#the-project) * [Repository](https://github.com/hsblabs/scrape-kdl) — the source, the fixtures, and the tests. * [Support](https://github.com/hsblabs/scrape-kdl/blob/main/SUPPORT.md) — where to ask a question. * [Contributing](https://github.com/hsblabs/scrape-kdl/blob/main/CONTRIBUTING.md) — how to make a change. * [Code of conduct](https://github.com/hsblabs/scrape-kdl/blob/main/CODE_OF_CONDUCT.md) * [Decision records](https://github.com/hsblabs/scrape-kdl/tree/main/docs/adr) — the reason for each architectural decision. ## The external standards [Section titled “The external standards”](#the-external-standards) * [KDL](https://kdl.dev/) — the document language of the syntax. Scraping KDL accepts a documented subset. * [RE2 syntax](https://github.com/google/re2/wiki/Syntax) — the profile of the regular expressions. A lookaround, a backreference, and a named group are not available. * [WHATWG HTML](https://html.spec.whatwg.org/multipage/parsing.html) — the standard of the parse of the HTML. * [CSS Selectors Level 4](https://www.w3.org/TR/selectors-4/) — the full language. The portable profile is a small subset of it. * [JSON Schema](https://json-schema.org/) — the schema language of the IR document.