Lectito

Lectito is a Rust library and CLI tool for extracting readable article content from HTML.

Most web pages contain way more than the text a reader came for, like ads, navigation, related links, comment areas, tracking markup, hidden elements, and presentation wrappers. Lectito tries to identify the main content root and return a smaller document that is useful for reading, storage, search, and conversion.

It returns:

cleaned article HTML
Markdown
plain text
page metadata
extraction diagnostics

The CLI can also generate PDF files when installed with its optional pdf feature.

Lectito is parser-first. The core API accepts HTML and an optional base URL. URL fetching exists in the CLI for convenience, but the library does not require network access.

This keeps the library usable in environments that already have HTML available: crawlers, browser extensions, desktop apps, mobile apps, tests, and offline archives.

Main APIs

#![allow(unused)]
fn main() {
use lectito::{extract, ReadabilityOptions};

let html = r#"<article><h1>Title</h1><p>Article text.</p></article>"#;
let article = extract(html, Some("https://example.com/post"), &ReadabilityOptions::default())?;

if let Some(article) = article {
    println!("{}", article.markdown);
}
Ok::<(), lectito::Error>(())
}

Use extract_with_diagnostics when tuning extraction or debugging a bad page.

Use is_probably_readable before extraction when you only need a quick yes/no answer.

Installation

Lectito is split into a core library and a CLI. Use the library when your application already has HTML. Use the CLI for local inspection, shell scripts, and quick conversions.

Library

Add lectito to your Rust project:

[dependencies]
lectito = "0.1"

For local development against this workspace:

[dependencies]
lectito = { path = "crates/core" }

The Rust crate name is lectito.

The core crate has no runtime service requirement. It parses the string you pass in and returns an article result.

CLI

Install the CLI from crates.io:

cargo install lectito-cli

Install with PDF output enabled:

cargo install lectito-cli --features pdf

For local development against this workspace:

cargo install --path crates/cli

Use the same feature flag for local PDF testing:

cargo install --path crates/cli --features pdf

The binary is named lectito.

lectito --help

The CLI can read from a file, stdin, or a URL. URL support is a command-line convenience; it is not part of the core library contract.

Fixture helpers are workspace-only and are not part of the published CLI package.

For local fixture inspection, run the unpublished workspace helper:

cargo run -p lectito-fixtures --bin lectito-fixture -- sample-name

MCP Server

Install the stdio MCP server from crates.io:

cargo install lectito-mcp

The binary is named lectito-mcp. Use it from MCP clients that support local stdio servers.

License

Lectito is licensed under MPL-2.0.

Quick Start

Extract From HTML

Start with extract for normal use. It takes the source HTML, an optional base URL, and ReadabilityOptions. The base URL lets Lectito resolve relative links, images, and metadata URLs in the extracted output.

use lectito::{extract, ReadabilityOptions};

fn main() -> Result<(), lectito::Error> {
    let html = r#"
        <html>
          <head><title>Example</title></head>
          <body>
            <article>
              <h1>Example</h1>
              <p>This is the article body.</p>
            </article>
          </body>
        </html>
    "#;

    let article = extract(html, Some("https://example.com/article"), &ReadabilityOptions::default())?;

    if let Some(article) = article {
        println!("{:?}", article.title);
        println!("{}", article.markdown);
    }

    Ok(())
}

extract returns Ok(None) when no useful article content is found. That is different from an error. An empty or navigation-only page can be parsed successfully and still have no article.

Check Readability

Use is_probably_readable when you only need to decide whether a page is worth running through full extraction. It is faster and returns a boolean.

#![allow(unused)]
fn main() {
use lectito::{is_probably_readable, ReadableOptions};

let readable = is_probably_readable(html, &ReadableOptions::default())?;
Ok::<(), lectito::Error>(())
}

CLI

The CLI mirrors the library. The root command extracts content, and readable performs the quick readability check.

lectito article.html
lectito https://example.com/article --format json --pretty
lectito readable article.html

CLI Usage

The CLI is designed for inspecting extraction behavior and converting documents from the terminal.

The root command extracts article content. The CLI also has these subcommands:

readable: check whether a document looks readable
inspect: print extraction metadata and scoring details
tropes: report AI-like writing tropes in text or extracted article content
llms: fetch, parse, and expand llms.txt files

Extract

Pass a URL, an AT URI, a file path, or - for stdin. Markdown with TOML frontmatter is the default output.

lectito article.html
lectito https://example.com/article
lectito at://did:plc:abc123/site.standard.document/xyz
lectito - < article.html

When a fetched page advertises rel="site.standard.document", the CLI resolves the ATProto record and uses the record content when it can render it. Direct at:// inputs are supported for renderable site.standard.document records. If a normal web URL cannot be resolved through Standard.site, the CLI extracts from the fetched HTML.

Output formats:

Use HTML, text, or JSON when Markdown is not the right output for the next tool. Use PDF when you need a simple file for reading or sharing and you have installed the CLI with --features pdf.

lectito article.html --format html
lectito article.html --format text
lectito article.html --format json --pretty
lectito article.html --format pdf --output article.pdf
lectito article.html --frontmatter=false
lectito article.html --output article.md

PDF output is an optional CLI feature:

cargo install lectito-cli --features pdf
cargo run -p lectito-cli --features pdf -- article.html \
  --format pdf \
  --output article.pdf

The PDF renderer converts the extracted article Markdown into a readable PDF with built-in fonts. It covers common Markdown blocks such as headings, lists, code blocks, tables, blockquotes, and footnotes.

PDF output always writes a file and prints the path. If you omit --output, Lectito creates {hash}.pdf in the current directory.

Useful options:

The defaults work for most article pages. Tune these flags when a page is too short, too broad, or has a known content container.

lectito article.html --char-threshold 800
lectito article.html --nb-top-candidates 8
lectito article.html --content-selector article
lectito article.html --base-url https://example.com/post --site-profile example.com.toml
lectito article.html --max-elems-to-parse 10000
lectito article.html --media article
lectito article.html --media none
lectito article.html --keep-classes --preserve-class language-rust

--content-selector is the strongest extraction hint. Use it when you know the article root for a page or fixture. Without that flag, the CLI still tries common article-body containers before falling back to generic scoring.

--media accepts none, conservative, article, or all. The default is article, which keeps figures/images that appear to be part of the article body.

--site-profile can be repeated. Each file must be a TOML site profile. User profiles take precedence over bundled profiles for the same host.

--disable-json-ld turns off JSON-LD metadata extraction and the JSON-LD article-body fast path. Use it when structured data is stale or misleading.

Diagnostics are written to stderr after the main output to keep keep stdout usable for the extracted article while still showing debug information in the terminal.

lectito article.html --diagnostic-format pretty
lectito article.html --diagnostic-format json

--inspect prints a compact extraction summary to stderr while keeping article output on stdout:

lectito article.html --inspect

Full extraction has a timeout so unusually large or hostile pages do not hang the command:

lectito article.html --timeout 10

Readable

readable checks whether the document appears to contain enough article-like text. It does not return extracted content.

lectito readable article.html
lectito readable --stdin < article.html
lectito readable https://example.com/article
lectito readable article.html --json --pretty
lectito readable article.html --timeout 10

Thresholds:

lectito readable article.html --min-content-length 140 --min-score 20

Inspect

inspect prints extraction metadata and scoring details without printing the article body.

lectito inspect article.html
lectito inspect https://example.com/article
lectito inspect article.html --json --pretty

Tropes

Use tropes to scan plain text, a text file, or input read from stdin. The default output is a compact human-readable report.

Add --json for the same report shape returned by the Rust, HTTP, and WASM APIs.

Add --pretty to format JSON or to include the no-findings message in readable output.

Choose the scanner preset with --preset balanced, --preset lenient, or --preset strict. Balanced is the default. Lenient omits low-severity findings and requires stronger repeated evidence; strict includes lower-evidence structural and document findings.

lectito tropes notes.txt
lectito tropes --stdin --pretty < notes.txt
lectito tropes --stdin --json < notes.txt

Pass --extract when the input is HTML, a URL, or an AT URI and you want Lectito to extract the article text before scanning it:

lectito tropes article.html --extract --pretty
lectito tropes https://example.com/article --extract --json
lectito tropes --stdin --extract --json < article.html

These public pages are useful live smoke-test examples. They cover release notes, API documentation, a reference article, and a prose essay:

lectito tropes https://blog.rust-lang.org/2024/11/28/Rust-1.83.0.html \
  --extract --json
lectito tropes https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy \
  --extract --json
lectito tropes https://en.wikipedia.org/wiki/Mozilla --extract --json
lectito tropes https://www.paulgraham.com/makersschedule.html --extract --json

Without --extract, a URL or file is scanned as the input document text. With --extract, the analyzer receives article.text_content.

Extraction uses its normal timeout; set --timeout when a remote page needs a different limit.

The report includes a 0–100 heuristic score, a coarse signal, exact findings, and summary counts. Findings include source metadata for the bundled rule in tropes.md, including its source heading.

Trope output is evidence for an editing or review pass. It is not an authorship detector and cannot establish that a person or model wrote the input.

The CLI uses the bundled catalog. It currently has no option for adding custom rules or changing the severity floor. Use min_severity in Rust, minSeverity in HTTP, or minSeverity in WASM when you need to limit the findings.

llms.txt

Use the llms subcommands when a site publishes an llms.txt file or when you want to bundle its linked resources into one Markdown context file.

lectito llms fetch https://example.com
lectito llms parse https://example.com/llms.txt --pretty
lectito llms expand https://example.com/llms.txt --output llms-full.txt
lectito llms generate https://example.com/docs/ --output llms.txt
lectito llms generate https://example.com/docs/ --output llms.txt --full llms-full.txt
lectito llms generate --sitemap https://example.com/sitemap.xml --output llms.txt
lectito llms generate https://example.com --discover --output llms.txt

fetch resolves a bare site URL to /llms.txt. parse prints structured JSON. expand reads the linked resources, keeps Markdown resources as-is, and runs HTML resources through Lectito before adding them to the bundle. generate crawls same-origin links from a seed page and writes a new llms.txt index. It uses canonical links for generated entries when pages publish them, includes HTTP Last-Modified or sitemap lastmod values in notes, and ranks accepted pages so likely entry points appear first. Pass --full (or --full-output) to write the expanded Markdown context while generating the index.

Links in the special Optional section are skipped unless you pass --include-optional:

lectito llms expand https://example.com/llms.txt --include-optional

Keep generated files small by limiting crawl depth and page count:

lectito llms generate https://example.com/docs/ --max-depth 1 --max-pages 10
lectito llms generate --sitemap https://example.com/sitemap.xml --max-pages 50

Filter generated entries and add a delay between page fetches:

lectito llms generate --sitemap https://example.com/sitemap.xml \
  --filter /docs/ \
  --filter '!/docs/archive/' \
  --filter '!*/drafts/*' \
  --delay 250

Remote generation checks robots.txt before fetching page URLs. It evaluates rules as Lectito by default:

lectito llms generate https://example.com/docs/ --robots-agent Lectito
lectito llms generate https://example.com/docs/ --ignore-robots

See the llms.txt guide for the expected file shape and the tradeoffs.

Exit Codes

0: article extracted, or readability check returned true
1: no article was extracted, or readability check returned false
2: input, file, or network error
3: extraction, readability, configuration, or timeout error

llms.txt

llms.txt is a Markdown file that gives language models and agent tools a curated entry point for a site. Sites usually publish it at /llms.txt.

Lectito supports the practical parts of the convention:

fetching a site's llms.txt
parsing its sections and links
expanding linked pages into one Markdown context file
crawling a bounded set of pages to generate an llms.txt index

It does not treat llms.txt as access control. Use robots.txt, HTTP authorization, and normal server controls for that.

File Shape

A small file looks like this:

# Example Docs

> Documentation for Example's public API.

Use the current API reference when generated examples disagree with older blog
posts.

## Docs

- [Quick start](https://example.com/docs/quick-start.md): First integration
  steps.
- [API reference](https://example.com/docs/api.md): Endpoint and object
  reference.

## Optional

- [Changelog](https://example.com/docs/changelog.md)

Lectito expects:

one H1 title
an optional blockquote summary
optional notes before the first H2
H2 sections containing Markdown links

The Optional section has special handling. lectito llms expand skips those links by default so the generated context stays smaller.

Fetch

Fetch a site's llms.txt:

lectito llms fetch https://example.com

For bare site URLs, Lectito requests /llms.txt. Explicit URLs are used as given:

lectito llms fetch https://example.com/docs/llms.txt

You can write the result to a file:

lectito llms fetch https://example.com --output llms.txt

Parse

Parse an llms.txt file into JSON:

lectito llms parse llms.txt --pretty

This is useful for checking whether section names, optional links, and notes are being read as expected.

Expand

Expand linked resources into one Markdown file:

lectito llms expand llms.txt --output llms-full.txt

Lectito keeps Markdown resources unchanged. When a linked resource looks like HTML, Lectito extracts the readable article and inserts the extracted Markdown. For remote links, Lectito checks the HTTP Content-Type header before falling back to URL suffixes and simple Markdown markers.

Each resource is separated and labeled:

---
# Source: Quick start
URL: https://example.com/docs/quick-start.md
Notes: First integration steps.
...

Use --include-optional to include the Optional section:

lectito llms expand llms.txt --include-optional --output llms-full.txt

Use --max-links when you want a smaller bundle:

lectito llms expand llms.txt --max-links 10

Generate

Generate an llms.txt file from a seed page:

lectito llms generate https://example.com/docs/ --output llms.txt

The crawler is intentionally bounded. For URL seeds, Lectito follows same-origin links only. For local HTML files, it follows relative local links. Assets such as images, stylesheets, scripts, PDFs, archives, and feeds are skipped.

To write the expanded context at the same time, pass --full:

lectito llms generate https://example.com/docs/ \
  --output llms.txt \
  --full llms-full.txt

--full-output is the same option with a more explicit name.

You can also generate from a sitemap:

lectito llms generate --sitemap https://example.com/sitemap.xml \
  --output llms.txt

Or discover sitemaps from a URL seed:

lectito llms generate https://example.com --discover \
  --output llms.txt

Discovery reads Sitemap: lines from robots.txt. When no sitemap is listed there, Lectito tries /sitemap.xml.

Sitemap indexes are supported. Lectito reads child sitemaps up to --max-sitemaps, then fetches page URLs up to --max-pages:

lectito llms generate --sitemap https://example.com/sitemap.xml \
  --max-sitemaps 10 \
  --max-pages 100 \
  --output llms.txt

Remote sitemap generation keeps sitemap and page URLs on the same origin as the sitemap input. Local sitemap files may list any absolute page URL.

By default, generation fetches up to 25 pages and follows links up to depth 2:

lectito llms generate https://example.com/docs/ \
  --max-pages 10 \
  --max-depth 1

Use --filter for the common path and glob cases. Prefix a pattern with ! to exclude it:

lectito llms generate --sitemap https://example.com/sitemap.xml \
  --filter /docs/ \
  --filter '!/docs/archive/' \
  --filter '!*/drafts/*'

Patterns that start with / match URL paths. Plain path values are prefixes. Path patterns with * or ? are globs. Other glob patterns match the full URL.

Use --delay to wait between page fetches:

lectito llms generate https://example.com/docs/ --delay 250

Remote generation checks robots.txt before fetching page URLs. Lectito keeps the existing browser-like user agent for HTTP requests, but evaluates robots rules as Lectito unless you pass another token:

lectito llms generate https://example.com/docs/ \
  --robots-agent LectitoDocsBot

Use --ignore-robots only when you explicitly want to bypass those checks:

lectito llms generate https://example.com/docs/ --ignore-robots

Only pages that produce readable article content are included. Each accepted page becomes one link in the generated file. Lectito uses the extracted title as the link label, switches to a page's canonical URL when one is available, and uses the extracted excerpt as the link note.

Remote generation also reads Last-Modified response headers. Sitemap generation reads lastmod values. When either value is present, Lectito adds it to the generated note and uses it as a small ranking signal. Ranking favors likely entry points such as docs roots, guides, API references, and pages with useful notes. Archive-like URLs are pushed down.

Set the generated title, summary, or section name when the defaults are too generic:

lectito llms generate https://example.com/docs/ \
  --title "Example Docs" \
  --summary "Public documentation for Example." \
  --section "Guides" \
  --output llms.txt

When To Use It

Use llms.txt when you want agents to start from a small, curated list of important pages. It works well for docs, public APIs, policy pages, and small knowledge bases.

Do not expect every model provider or search engine to read it. The reliable use case is explicit: a developer, tool, or agent asks Lectito to fetch or expand the file.

Basic Usage

Use extract when you want article content.

The function does not fetch the page. Pass it the HTML you want parsed. This is usually cleaner in applications because networking, caching, cookies, and browser rendering are application concerns.

#![allow(unused)]
fn main() {
use lectito::{extract, ReadabilityOptions};

let options = ReadabilityOptions::default();
let article = extract(html, Some("https://example.com/post"), &options)?;

match article {
    Some(article) => println!("{}", article.text_content),
    None => eprintln!("no article content found"),
}
Ok::<(), lectito::Error>(())
}

The base URL is optional. Pass it when the document contains relative links, images, or metadata URLs.

Raw HTML Limits

Lectito parses the HTML string you pass in. It does not run JavaScript, keep a browser session, submit forms, attach cookies, or fetch authenticated resources. For pages that build their article body on the client, capture rendered HTML in your crawler or browser automation layer before calling extract.

The CLI fetches URLs as a convenience, but it has the same raw-HTML boundary. If a site needs login state, consent flows, or browser-specific state, fetch that page in your own application (or a browser) and pass the resulting HTML through stdin or the Rust API.

When extraction succeeds, Lectito returns Some(Article). When the page parses but does not contain a useful article, it returns None. Reserve error handling for invalid base URLs, configured size limits, and serialization failures.

Article Output

Article contains the extracted content in several forms:

#![allow(unused)]
fn main() {
if let Some(article) = article {
    println!("{}", article.content);
    println!("{}", article.markdown);
    println!("{}", article.text_content);
}
}

Use extract_with_diagnostics when you need to see how extraction chose a root. Diagnostics are meant for development and regression work. Most application code should call extract.

#![allow(unused)]
fn main() {
use lectito::{extract_with_diagnostics, ReadabilityOptions};

let report = extract_with_diagnostics(html, base_url, &ReadabilityOptions::default())?;

if let Some(article) = report.article {
    println!("{}", article.markdown);
}

eprintln!("{:?}", report.diagnostics.outcome);

Ok::<(), lectito::Error>(())
}

Configuration

ReadabilityOptions control extraction.

The defaults are conservative. They favor article pages with enough text to be useful and avoid exposing internal scoring knobs unless they affect common integration cases.

#![allow(unused)]
fn main() {
use lectito::{MediaRetention, ReadabilityOptions};

let options = ReadabilityOptions {
    char_threshold: 800,
    nb_top_candidates: 8,
    content_selector: Some("article".to_string()),
    site_profiles: Vec::new(),
    media_retention: MediaRetention::Article,
    ..ReadabilityOptions::default()
};
}

Fields:

Field	Default	Meaning
`max_elems_to_parse`	`None`	Reject documents above this element count.
`nb_top_candidates`	`5`	Number of high-scoring candidates to consider.
`char_threshold`	`500`	Minimum extracted text length for an accepted attempt.
`content_selector`	`None`	CSS selector to force as the content root.
`site_profiles`	`[]`	TOML site profiles for host-scoped extraction hints.
`mobile_viewport_width`	`Some(480)`	Width used by recovery rules for mobile snapshots.
`classes_to_preserve`	`[]`	Class names kept during cleanup.
`keep_classes`	`false`	Keep all class attributes.
`disable_json_ld`	`false`	Skip JSON-LD metadata extraction.
`link_density_modifier`	`0.0`	Adjust link-density cleanup tolerance.
`media_retention`	`Article`	Control figure/image/media retention.

Prefer content_selector when you already know the page shape. It bypasses root scoring for that document, then runs the normal cleanup pipeline.

When content_selector is not set, Lectito still tries a small list of common article-body containers such as #article-body and .entry-content before generic scoring. That catches many large publisher pages without site-specific profiles.

Use site_profiles when you want URL-scoped extraction hints, removal selectors, and metadata hints. Profiles are attempted before generic scoring, but weak profile output falls back to the generic extractor.

Use max_elems_to_parse as a guardrail for untrusted input. It rejects very large documents before extraction work continues.

Use media_retention when output fidelity matters. Article keeps body figures and images by default; None removes media; Conservative is text-first; All keeps media that remains in the selected article subtree.

ReadableOptions controls is_probably_readable.

Lower min_content_length for short posts or documentation pages. Raise min_score when you want the quick check to reject borderline pages.

#![allow(unused)]
fn main() {
use lectito::ReadableOptions;

let options = ReadableOptions {
    min_content_length: 140,
    min_score: 20.0,
};
}

Output Formats

Lectito produces all output formats during extraction.

The formats come from the same cleaned article root. That means callers can store HTML for fidelity, use Markdown for display or editing, and use plain text for search without running extraction multiple times.

#![allow(unused)]
fn main() {
let article = extract(html, base_url, &ReadabilityOptions::default())?.unwrap();

let html = article.content;
let markdown = article.markdown;
let text = article.text_content;
}

The core crate returns HTML, Markdown, plain text, and metadata. The CLI can also write the extracted Markdown as a PDF when installed with its optional pdf feature.

HTML

content is cleaned article HTML. Scripts, styles, navigation, sidebars, and other page chrome are removed where possible. Relative URLs are resolved when a base URL is provided.

Use HTML when you need the closest representation of the extracted article. It keeps images, links, tables, inline markup, and other structure that can be lost in plain text.

Markdown

markdown is generated from the cleaned article HTML. It preserves common reader content:

headings
paragraphs
links and images
lists
blockquotes
code blocks
tables
math
footnotes

Markdown cleanup also strips zero-width break hints, drops empty links, keeps images intact, and removes duplicate title headings before rendering.

The CLI Markdown output includes TOML frontmatter:

lectito article.html

Markdown is useful when the next step is a reader view, note-taking system, static archive, or editor. It is also easier to diff in tests than HTML.

Plain Text

text_content is normalized article text. Use it for indexing, previews, and readability checks.

Plain text should not be treated as a rendering format. It discards links, images, and most document structure.

JSON

The CLI can serialize the article:

lectito article.html --format json --pretty

JSON is the best CLI format when another program needs metadata and content together.

PDF

PDF output is available in the CLI when the pdf feature is enabled:

cargo install lectito-cli --features pdf
lectito article.html --format pdf --output article.pdf

The renderer starts from the extracted Markdown and writes a readable PDF with built-in fonts. It keeps common article structure such as headings, paragraphs, lists, blockquotes, code blocks, tables, definition lists, images as text, horizontal rules, and footnotes.

PDF output always writes a file and prints the path. If you omit --output, Lectito creates {hash}.pdf in the current directory.

Use PDF when you need a portable reading copy.

Use HTML or Markdown when the next step needs richer structure or editable text.

Quality Expectations

Output	Best use	Expect	Do not expect
Markdown	Reader views, notes, archives, editing	Good preservation of headings, paragraphs, links, images, lists, blockquotes, code, tables, math, and footnotes.	Byte-for-byte source fidelity or every custom widget.
HTML	Rendering or post-processing extracted articles	The closest structural view of the cleaned article root, with links and media kept according to options.	A complete sanitizer policy or the original page layout.
Text	Search, previews, indexing, basic summaries	Normalized article text with block boundaries for headings, paragraphs, lists, code, and definition lists.	A rich rendering format with links, images, or full table structure.
JSON	Programmatic CLI integrations	Metadata plus HTML, Markdown, text, length, and source-related fields in one object.	Stable values for publisher metadata when source pages disagree or omit fields.
PDF	Portable reading copies from the CLI	A generated PDF built from extracted Markdown, with common block structure preserved.	Existing-PDF editing, exact source layout, custom fonts, or print-grade design.
`inspect`	Debugging extraction choices	Selected root, candidate scores, cleanup counts, recovery data, and site-rule information.	A user-facing article format.
`readable`	Cheap filtering before full extraction	A boolean estimate using text length, visibility, class/id hints, and link density.	The same answer full extraction would produce on every borderline page.

How It Works

Lectito follows the same broad approach as Mozilla Readability, with a few fast paths for common article snapshots.

The extractor starts with a full HTML document and tries to find the subtree that behaves like an article. It uses signals that tend to survive across sites: text length, paragraph density, semantic tags, class and id names, and the ratio of links to readable text.

Recover useful content from raw HTML snapshots, including declarative shadow DOM.
Parse the document.
Recover useful content from parsed snapshots, including selected mobile and shadow-root cases.
Extract metadata, including JSON-LD before scripts are stripped.
Accept long JSON-LD article text when structured data contains the body.
Try known article containers such as #article-body before broad scoring.
Try a matching site profile or code extractor when one applies.
Remove scripts, styles, hidden nodes, and unlikely content.
Score candidate content roots by text length, tag type, class/id hints, and link density.
Select the best root and include useful siblings.
Clean the selected content.
Apply schema text fallback when structured data is clearly better.
Return HTML, Markdown, text, and diagnostics.

Extraction runs several attempts. Later attempts relax cleanup rules when the first pass produces too little text. The first attempt that reaches char_threshold is accepted. If no attempt reaches the threshold, Lectito may return the best non-empty attempt.

This retry model matters because pages fail in different ways. Some pages hide the useful content behind classes that look like chrome. Others include enough related links or widgets to pull the score away from the main text. Relaxed attempts give Lectito another chance without making the first pass too loose.

content_selector can short-circuit root selection for known documents:

#![allow(unused)]
fn main() {
let options = ReadabilityOptions {
    content_selector: Some("main article".to_string()),
    ..ReadabilityOptions::default()
};
}

Lectito also has a small built-in list of known content containers, including #article-body, [itemprop='articleBody'], .article-body, and .entry-content. These are attempted before generic scoring. They still go through cleanup, media handling, URL rewriting, and diagnostics.

Site profiles provide URL-scoped hints without disabling generic extraction:

#![allow(unused)]
fn main() {
let options = ReadabilityOptions {
    site_profiles: vec![r#"
        name = "example"
        hosts = ["example.com"]
        content_roots = ["article"]
        remove = [".ad", "nav"]
    "#.to_string()],
    ..ReadabilityOptions::default()
};
}

If a profile produces content below char_threshold, Lectito records the profile decision in diagnostics and continues with generic readability attempts.

After the root is selected, cleanup removes empty nodes, normalizes links and media, preserves selected classes, and prepares the HTML for Markdown and text conversion.

Diagnostics

Use diagnostics to inspect extraction decisions.

Diagnostics are for development, fixture work, and bug reports. They explain which candidates were considered, which root was selected, and why an extraction was accepted or downgraded to a best attempt.

#![allow(unused)]
fn main() {
use lectito::{extract_with_diagnostics, ReadabilityOptions};

let report = extract_with_diagnostics(html, base_url, &ReadabilityOptions::default())?;
println!("{:?}", report.diagnostics.outcome);
}

ExtractionReport contains:

article: the extracted article, if found
diagnostics: details about attempts and candidate selection

Outcomes:

Outcome	Meaning
`Accepted`	An attempt met `char_threshold`.
`BestAttempt`	No attempt met the threshold, but non-empty content was found.
`NoContent`	No useful content was found.

Each attempt records:

cleanup flags
candidate count
top candidates
entry points
selected root
cleanup counts
recovery counts
extracted text length

Fast paths such as JSON-LD article text or a known content container may record an accepted attempt with candidate_count = 0. That means Lectito accepted a specific root before generic candidate scoring ran.

When a site profile or code extractor matches, diagnostics include site_rule. That record reports the matched profile or extractor, whether it was bundled, which roots were selected, how many removals ran, whether the result met char_threshold, and any fallback reason.

Start with outcome, selected_root, and text_len. If the selected root is wrong, inspect the candidate list. If the root is right but output is noisy, inspect cleanup counts and preserved classes.

CLI diagnostics:

lectito article.html --diagnostic-format pretty
lectito article.html --diagnostic-format json
lectito inspect article.html

MCP Server

lectito-mcp is a local stdio MCP server for article search and reading. MCP clients start it as a subprocess and exchange JSON-RPC messages over stdin and stdout.

The server exposes two tools:

Tool	Description
`search_articles`	search DuckDuckGo HTML results and return titles, URLs, and snippets.
`read_article`	fetch a public HTTP(S) URL and extract readable article content with Lectito.

Because this is a stdio server, keep stdout reserved for MCP messages.

Write logs to stderr if you wrap or supervise the binary.

For agent workflow instructions, see the Article Reader Skill. The skill points agents at search_articles and read_article when those MCP tools are available.

Build

Install the published MCP server with Cargo:

cargo install lectito-mcp

The binary will be available as:

lectito-mcp

Build the release binary from the repository root:

cargo build -p lectito-mcp --release

The binary will be at:

target/release/lectito-mcp

For local development, you can build the debug binary:

cargo build -p lectito-mcp

Then run a minimal tools-list smoke test:

printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' \
  | target/debug/lectito-mcp

Configuration

Set these environment variables from your MCP client config when you need to change the defaults:

Variable	Default	Purpose
`LECTITO_MCP_DEFAULT_SEARCH_RESULTS`	`5`	Default `search_articles` result count.
`LECTITO_MCP_MAX_SEARCH_RESULTS`	`10`	Upper bound for `search_articles.limit`.
`LECTITO_MCP_MAX_FETCH_BYTES`	`2097152`	Maximum bytes read while fetching a page.
`LECTITO_MCP_REDIRECT_LIMIT`	`5`	Maximum redirect hops for article fetches.
`LECTITO_MCP_REQUEST_TIMEOUT_SECS`	`20`	HTTP request timeout in seconds.
`LECTITO_MCP_MAX_ARTICLE_CHARS`	`30000`	Upper bound for `read_article.maxChars`.
`LECTITO_MCP_ALLOW_PRIVATE_NETWORK`	`false`	Allow private or loopback fetch targets.

Leave LECTITO_MCP_ALLOW_PRIVATE_NETWORK=false unless you intentionally want the server to read URLs from local or private networks.

Codex

Codex supports stdio MCP servers in the CLI and IDE extension. The durable configuration lives in ~/.codex/config.toml, or in a trusted repo-local .codex/config.toml.

The simplest install is:

codex mcp add lectito -- lectito-mcp

If you are using a local checkout instead of the Cargo-installed binary, point Codex at the built executable:

codex mcp add lectito -- /path/to/lectito/target/release/lectito-mcp

For a checked-in project config with the Cargo-installed binary, add a table like this:

[mcp_servers.lectito]
command = "lectito-mcp"
startup_timeout_sec = 10
tool_timeout_sec = 60

[mcp_servers.lectito.env]
LECTITO_MCP_MAX_SEARCH_RESULTS = "10"
LECTITO_MCP_DEFAULT_SEARCH_RESULTS = "5"

During development, run through Cargo only after the crate has been built at least once:

[mcp_servers.lectito]
command = "cargo"
args = ["run", "-p", "lectito-mcp", "--quiet"]
cwd = "/path/to/lectito"

Restart Codex after editing config.toml. In the Codex TUI, use /mcp to check whether the server connected.

Claude Code

Claude Code can add local stdio MCP servers with claude mcp add. Put -- before the server command so Claude does not parse server flags as Claude flags.

claude mcp add --transport stdio lectito \
  -- lectito-mcp

To share the setup with a project, create .mcp.json in the repo:

{
  "mcpServers": {
    "lectito": {
      "type": "stdio",
      "command": "lectito-mcp",
      "env": {
        "LECTITO_MCP_MAX_SEARCH_RESULTS": "10",
        "LECTITO_MCP_DEFAULT_SEARCH_RESULTS": "5"
      }
    }
  }
}

If you prefer to run through Cargo while developing:

claude mcp add --transport stdio lectito \
  -- cargo run -p lectito-mcp --quiet

Run that command from the repository root, or use a project .mcp.json with an absolute binary path.

Pi Coding Agent

Pi does not include built-in MCP support. Its documented extension model is skills, prompt templates, TypeScript extensions, and packages. For Pi, install the article-reading workflow as a skill, then use the Lectito CLI or a custom Pi extension for the backend.

Pi can load skill directories with --skill. Use the checked-in article-reader skill to teach Pi when to search for articles, read URLs, and cite sources:

pi --skill /path/to/lectito/skills/article-reader

For a one-shot run:

pi --skill /path/to/lectito/skills/article-reader \
  "Read and summarize https://example.com/article"

For persistent discovery, place the skill under one of Pi's skill locations, such as .pi/skills/, .agents/skills/, ~/.pi/agent/skills/, or ~/.agents/skills/.

If you are building a Pi extension that bridges to MCP, build lectito-mcp from the same checkout and launch this stdio command from the extension:

cargo build -p lectito-mcp --release

/path/to/lectito/target/release/lectito-mcp

The local Pi CLI used while writing this page exposes pi install <source> for packages and --skill <path> for skill directories, but no MCP subcommand.

Sources

Codex MCP documentation
Claude Code MCP documentation
Model Context Protocol
Pi CLI help and installed docs: pi --help, pi install --help, docs/skills.md, and docs/usage.md from @earendil-works/pi-coding-agent

Article Reader Skill

Lectito includes an article-reader skill for agents that understand SKILL.md directories (implement agent skills).

The skill tells an agent when to search for articles, when to read a URL, how to cite article-specific claims, and when to fall back to the Lectito CLI.

The source lives in the repository:

The skill is useful on its own, but it works best with the lectito-mcp server because the MCP server exposes the search_articles and read_article tools named by the skill.

Install From Source

Clone the repository and point your agent at the skill directory:

git clone https://github.com/stormlightlabs/lectito.git
cd lectito

Use this path in agents that accept a skill directory:

/path/to/lectito/skills/article-reader

For Pi, load the skill explicitly with --skill:

pi --skill /path/to/lectito/skills/article-reader

For a one-shot Pi run:

pi --skill /path/to/lectito/skills/article-reader \
  "Find and summarize articles about Rust parser libraries"

You can also simply point pi to the skill's source and it'll handle creating it for you.

If the MCP tools are available, the skill should use them. If they are not, the skill reference tells the agent how to use the Lectito CLI for a known URL and when to use another approved search mechanism.

Pair With MCP

Install the MCP server:

cargo install lectito-mcp

Then configure your MCP-capable agent with:

lectito-mcp

For local development, build the MCP server from the same checkout:

cargo build -p lectito-mcp --release

and point your agent at:

/path/to/lectito/target/release/lectito-mcp

Keep the skill and the MCP server on the same revision when possible. The skill names the tool contract, while the MCP crate implements it.

API Overview

Lectito exposes the same extraction and trope-analysis engine through four surfaces:

Rust Crate API for native applications and server integrations.
CLI Usage for files, stdin, URLs, and AT URIs.
The HTTP service for JSON requests at /v1/extract, /v1/evaluate, /v1/transform, and /v1/tropes/analyze.
WASM API for browser, web worker, bundler, and Node.js integrations.

Rust owns the implementation. The CLI, HTTP service, and WASM binding call the same core functions, so trope findings use the same rule IDs, ranges, scores, and provenance on every surface.

HTTP trope analysis

Send text to POST /api/v1/tropes/analyze:

curl -X POST https://lectito.stormlightlabs.org/api/v1/tropes/analyze \
  -H 'content-type: application/json' \
  -d '{
    "text":"It is worth noting that this serves as an example.",
    "options":{"minSeverity":"medium"}
  }'

The response contains report and an elapsedMs measurement.

The report uses the core JSON shape: score, signal, findings, and summary. Request option names use camelCase; report fields retain their Rust snake_case names.

The endpoint returns heuristic writing diagnostics, not an authorship score. A finding is evidence that a phrase or structure matches a bundled rule; it does not show who wrote the text.

POST /api/v1/extract can include the same report by setting "tropeDiagnostics": true.

The analyzer scans the extracted article's text_content, not the source HTML.

The field is omitted from the response when the flag is false.

The built-in rules come from the vendored tropes.md catalog and each finding includes its source file and heading. The HTTP API does not accept custom rules; the options object selects a preset, severity floor, and whether to include suggestions. balanced is the default. lenient removes low-severity findings and requires stronger repeated evidence; strict includes lower-evidence structural and document findings.

Rust Crate API

Public exports from lectito:

The crate exposes the extraction API, output structs, trope diagnostics, errors, and Markdown helpers.

#![allow(unused)]
fn main() {
pub use config::{Article, MarkdownOptions, MediaRetention, ReadabilityOptions, ReadableOptions};
pub use diagnostics::{
    AttemptDiagnostic, CandidateDiagnostic, CandidateSelection,
    CleanupDiagnostic, ContentSelectorDiagnostic, ExtractionDiagnostics,
    ExtractionOutcome, ExtractionReport, FlagDiagnostic, NodeDiagnostic,
    RecoveryDiagnostic,
};
pub use error::Error;
pub use extract::{clean_article_html, extract, extract_with_diagnostics};
pub use markdown::{html_to_markdown, markdown_to_html, markdown_with_toml_frontmatter};
pub use readable::is_probably_readable;
pub use tropes::analyze_tropes;
pub use tropes::{
    RuleSource, Severity, TextRange, TropeCategory, TropeFinding, TropeOptions,
    TropePreset, TropeReport, TropeSignal, TropeSummary,
};
}

Extraction

Use extract for normal application code.

#![allow(unused)]
fn main() {
pub fn extract(
    html: &str,
    base_url: Option<&str>,
    options: &ReadabilityOptions,
) -> Result<Option<Article>, Error>
}

Returns Ok(Some(article)) when content is found, Ok(None) when the document has no useful article content, and Err for invalid input or processing failures.

Extraction tries JSON-LD article text and common article-body containers before generic readability scoring.

Set content_selector when you already know the article root.

Set disable_json_ld when structured data is wrong for the page.

Use extract_with_diagnostics when you need extraction details in addition to the article.

#![allow(unused)]
fn main() {
pub fn extract_with_diagnostics(
    html: &str,
    base_url: Option<&str>,
    options: &ReadabilityOptions,
) -> Result<ExtractionReport, Error>
}

Returns the same article result with extraction diagnostics.

Use clean_article_html when you only need the cleaned article HTML.

#![allow(unused)]
fn main() {
pub fn clean_article_html(
    html: &str,
    base_url: Option<&str>,
    options: &ReadabilityOptions,
) -> Result<Option<String>, Error>
}

Readability Check

Use is_probably_readable before full extraction when you are filtering many documents.

#![allow(unused)]
fn main() {
pub fn is_probably_readable(
    html: &str,
    options: &ReadableOptions,
) -> Result<bool, Error>
}

Returns a quick readability estimate without full extraction.

Markdown

The Markdown helpers are available separately for callers that already have a clean HTML fragment, want to render Markdown as HTML, or want CLI-style frontmatter.

#![allow(unused)]
fn main() {
pub fn html_to_markdown(html: &str) -> String
}

Converts HTML fragments to Markdown.

#![allow(unused)]
fn main() {
pub fn markdown_to_html(markdown: &str, options: &MarkdownOptions) -> String
}

Converts Markdown to HTML using CommonMark/GFM options.

#![allow(unused)]
fn main() {
pub fn markdown_with_toml_frontmatter(
    article: &Article,
    source: Option<&str>,
) -> Result<String, Error>
}

Formats an article as Markdown with TOML frontmatter.

Trope Diagnostics

Use analyze_tropes to inspect text for AI-like writing tropes and other formulaic patterns.

The function is local and deterministic so it does not fetch content, call a model, or make an authorship judgment.

#![allow(unused)]
fn main() {
use lectito::{analyze_tropes, TropeOptions};

let text = "It is worth noting that this serves as an example.";
let report = analyze_tropes(text, &TropeOptions::default());

println!("{:?}: {}", report.signal, report.summary.finding_count);
for finding in report.findings {
    println!("{}: {}", finding.rule_id, finding.matched_text);
}
}

A trope report is a heuristic writing diagnostic. It reports matched phrases and structural patterns; it does not determine authorship or prove that a person or model wrote the text.

TropeOptions accepts a preset, a min_severity filter, and include_suggestions. TropePreset::Balanced is the default. Lenient omits low-severity findings and uses higher evidence thresholds for repeated patterns. Strict keeps low-severity findings and lowers some structural and document thresholds, so it can be noisier. Findings include the matched text, byte and UTF-16 ranges, line and column positions, category, severity, and an optional editing suggestion.

TropeReport::score is a 0–100 density heuristic, not a probability. The signal field groups that score into low, medium, high, or very_high. summary includes word and finding counts, findings per 1,000 words, category and severity counts, top categories, and repeated rule IDs.

The built-in catalog is vendored at crates/core/src/tropes/tropes.md and normalized into crates/core/src/tropes/rules.json. It is based on the AI Writing Tropes to Avoid catalog from tropes.fyi. Each built-in finding carries the source file and heading that produced it.

Custom rules are not supported by the current public API. Callers can filter the bundled rules by severity, but cannot add patterns to the analyzer.

Errors

Extraction and conversion functions return Result<_, lectito::Error>. An empty or non-article document is represented by Ok(None) from extract; it is not an error.

The most relevant extraction errors are:

Error::InvalidBaseUrl when the optional base URL cannot be parsed.
Error::MaxElemsExceeded when the document exceeds ReadabilityOptions::max_elems_to_parse.
Error::InvalidSiteProfile when a TOML site profile cannot be parsed or converted into selectors.
Error::HtmlParse when HTML parsing fails before extraction starts.
Error::Serialization when the cleaned article cannot be serialized.

The public constructors Error::invalid_site_profile and Error::max_elems_exceeded create the corresponding structured variants when an integration needs to preserve Lectito's error shape.

WASM API

The npm package @stormlightlabs/lectito exposes Lectito to JavaScript through wasm-bindgen.

It supports browser, web worker, bundler, and Node.js use.

npm install @stormlightlabs/lectito

The Rust crate is still named lectito-wasm.

Build Targets

wasm-pack build crates/wasm --target bundler
wasm-pack build crates/wasm --target web
wasm-pack build crates/wasm --target nodejs

wasm-pack writes lectito_wasm.d.ts with the public TypeScript API.

Initialization

Bundler builds initialize when imported:

import { extract } from "@stormlightlabs/lectito";

const article = extract(html, "https://example.com/post");

The web target needs the async initializer:

import init, { extract } from "./lectito_wasm.js";

await init();

const article = extract(html, "https://example.com/post");

The nodejs target initializes when loaded:

const { extract } = require("./lectito_wasm.js");

const article = extract(html, "https://example.com/post");

Functions

export function extract(
  html: string,
  baseUrl?: string | null,
  options?: ReadabilityOptions | null,
): Article | null;

export function extractWithDiagnostics(
  html: string,
  baseUrl?: string | null,
  options?: ReadabilityOptions | null,
): ExtractionReport;

export function isProbablyReadable(html: string, options?: ReadableOptions | null): boolean;

export function cleanHtml(
  html: string,
  baseUrl?: string | null,
  options?: CleanHtmlOptions | null,
): string | null;

export function htmlToMarkdown(html: string): string;

export function markdownToHtml(markdown: string, options?: MarkdownOptions | null): string;

export function analyzeTropes(text: string, options?: TropeOptions | null): TropeReport;

Types

Option fields use camelCase. Returned article fields keep the core Rust snake_case names.

export type MediaRetention = "none" | "conservative" | "article" | "all";

export interface ReadabilityOptions {
  maxElemsToParse?: number | null;
  nbTopCandidates?: number;
  charThreshold?: number;
  contentSelector?: string | null;
  siteProfiles?: string[];
  mobileViewportWidth?: number | null;
  classesToPreserve?: string[];
  keepClasses?: boolean;
  disableJsonLd?: boolean;
  linkDensityModifier?: number;
  mediaRetention?: MediaRetention;
}

export interface ReadableOptions {
  minContentLength?: number;
  minScore?: number;
}

export interface MarkdownOptions {
  gfm?: boolean;
  footnotes?: boolean;
  math?: boolean;
  allowRawHtml?: boolean;
}

export type TropePreset = "lenient" | "balanced" | "strict";

export type TropeCategory =
  | "word_choice"
  | "sentence_structure"
  | "paragraph_structure"
  | "tone"
  | "formatting"
  | "composition";

export type TropeSignal = "low" | "medium" | "high" | "very_high";

export type Severity = "low" | "medium" | "high";

export interface TropeOptions {
  preset?: TropePreset;
  minSeverity?: Severity;
  includeSuggestions?: boolean;
}

export interface TextRange {
  start_byte: number;
  end_byte: number;
  start_utf16: number;
  end_utf16: number;
  start_line: number;
  start_column: number;
  end_line: number;
  end_column: number;
}

export interface RuleSource {
  file: string;
  heading: string;
}

export interface TropeFinding {
  rule_id: string;
  category: TropeCategory;
  severity: Severity;
  range: TextRange;
  matched_text: string;
  message: string;
  suggestion: string | null;
  source: RuleSource | null;
}

export interface TropeSummary {
  word_count: number;
  finding_count: number;
  findings_per_1000_words: number;
  category_counts: Partial<Record<TropeCategory, number>>;
  severity_counts: Partial<Record<Severity, number>>;
  top_categories: TropeCategory[];
  repeated_rules: string[];
}

export interface TropeReport {
  score: number;
  signal: TropeSignal;
  findings: TropeFinding[];
  summary: TropeSummary;
}

export type CleanHtmlOptions = ReadabilityOptions;

export interface Article {
  title?: string | null;
  byline?: string | null;
  dir?: string | null;
  lang?: string | null;
  content: string;
  markdown: string;
  text_content: string;
  length: number;
  excerpt?: string | null;
  site_name?: string | null;
  published_time?: string | null;
  image?: string | null;
  domain?: string | null;
  favicon?: string | null;
}

export interface ExtractionReport {
  article: Article | null;
  diagnostics: unknown;
}

mediaRetention accepts "none", "conservative", "article", or "all".

Trope diagnostics

import { analyzeTropes } from "@stormlightlabs/lectito";

const report = analyzeTropes("It is worth noting that this serves as an example.");
console.log(report.signal, report.findings);

analyzeTropes returns heuristic writing diagnostics. It reports matched phrases and structures; it does not determine authorship or prove that a person or model wrote the text.

minSeverity filters findings at the requested severity or higher. includeSuggestions controls the nullable suggestion field. The default preset is "balanced". The "lenient" preset removes low-severity findings and requires stronger repeated evidence. The "strict" preset includes lower-evidence structural and document findings. Report ranges include both byte and UTF-16 offsets so callers can decorate JavaScript strings without converting the input first.

The source field points to the vendored tropes.md catalog and its source heading. Custom rules are not supported by the current WASM API; it uses the bundled rule catalog only.

Errors

Functions throw JavaScript Error objects for invalid base URLs, oversized documents, option conversion failures, and serialization failures.

Sanitization

cleanHtml performs Lectito article cleanup. It is not a complete untrusted-HTML security policy.

Browser integrations that accept arbitrary HTML should run a dedicated sanitizer such as DOMPurify before passing content into Lectito. Sanitize again before rendering returned HTML when the original input is untrusted.

Release Checks

Run the WASM tests and build all supported package targets:

pnpm --dir packages/web exec wasm-pack test --node ../../crates/wasm
pnpm --dir packages/web exec wasm-pack build ../../crates/wasm --target bundler --out-dir ../../target/wasm-pack/bundler
pnpm --dir packages/web exec wasm-pack build ../../crates/wasm --target web --out-dir ../../target/wasm-pack/web
pnpm --dir packages/web exec wasm-pack build ../../crates/wasm --target nodejs --out-dir ../../target/wasm-pack/nodejs

The build commands run wasm-opt; restricted sandboxes may need permission to execute it.

Article

Article is the extraction result.

The struct is serializable and contains both content and metadata. The content fields are generated from the selected article root; metadata can come from document metadata, JSON-LD, Open Graph tags, or the extracted content itself.

#![allow(unused)]
fn main() {
pub struct Article {
    pub title: Option<String>,
    pub byline: Option<String>,
    pub dir: Option<String>,
    pub lang: Option<String>,
    pub content: String,
    pub markdown: String,
    pub text_content: String,
    pub length: usize,
    pub excerpt: Option<String>,
    pub site_name: Option<String>,
    pub published_time: Option<String>,
    pub image: Option<String>,
    pub domain: Option<String>,
    pub favicon: Option<String>,
}
}

Fields:

Field	Meaning
`title`	Best title from metadata or document content.
`byline`	Author/byline when detected.
`dir`	Text direction, such as `ltr` or `rtl`.
`lang`	Document language when detected.
`content`	Cleaned article HTML.
`markdown`	Markdown generated from `content`.
`text_content`	Plain text generated from `content`.
`length`	UTF-16 length of extracted text, matching Mozilla Readability.
`excerpt`	Short summary or first useful paragraph.
`site_name`	Publisher or site name.
`published_time`	Publication timestamp when detected.
`image`	Lead image URL when detected.
`domain`	Source domain when available.
`favicon`	Favicon URL when detected.

content, markdown, and text_content are different views of the same extracted article. Prefer content when structure matters, markdown when the article will be displayed or edited as text, and text_content when indexing or summarizing.

length follows Mozilla Readability's UTF-16 convention. It can differ from a Rust chars().count() value for text outside the Basic Multilingual Plane.

Options

ReadabilityOptions

ReadabilityOptions changes extraction behavior. Most callers should start with ReadabilityOptions::default() and only set fields that solve a specific problem.

#![allow(unused)]
fn main() {
pub struct ReadabilityOptions {
    pub max_elems_to_parse: Option<usize>,
    pub nb_top_candidates: usize,
    pub char_threshold: usize,
    pub content_selector: Option<String>,
    pub site_profiles: Vec<String>,
    pub mobile_viewport_width: Option<usize>,
    pub classes_to_preserve: Vec<String>,
    pub keep_classes: bool,
    pub disable_json_ld: bool,
    pub link_density_modifier: f32,
    pub media_retention: MediaRetention,
}

pub enum MediaRetention {
    None,
    Conservative,
    Article,
    All,
}
}

Defaults:

#![allow(unused)]
fn main() {
ReadabilityOptions {
    max_elems_to_parse: None,
    nb_top_candidates: 5,
    char_threshold: 500,
    content_selector: None,
    site_profiles: Vec::new(),
    mobile_viewport_width: Some(480),
    classes_to_preserve: Vec::new(),
    keep_classes: false,
    disable_json_ld: false,
    link_density_modifier: 0.0,
    media_retention: MediaRetention::Article,
}
}

content_selector is the most direct override. Use it when the caller knows where the article lives in the document. When it is unset, Lectito still tries a small built-in list of common article-body containers before generic scoring.

site_profiles accepts TOML profile strings that provide host-scoped content roots, removal selectors, metadata hints, cleanup settings, and fallback behavior. Profiles run before generic scoring, after the JSON-LD and known container fast paths.

char_threshold controls when an attempt is accepted. nb_top_candidates controls how many candidates remain in play during generic scoring.

disable_json_ld skips JSON-LD metadata extraction and the JSON-LD article-body fast path. It does not disable Open Graph, Twitter card, or DOM metadata.

media_retention controls image and media preservation in the extracted article:

None: remove figures, images, and embedded media from content.
Conservative: text-first cleanup; media survives only if the generic extractor keeps it.
Article: keep figures/images that look like article body content. This is the default.
All: keep media that remains in the selected article subtree, subject to unsafe/embed cleanup.

ReadableOptions

ReadableOptions only affects is_probably_readable. It does not change full article extraction.

#![allow(unused)]
fn main() {
pub struct ReadableOptions {
    pub min_content_length: usize,
    pub min_score: f32,
}
}

Use lower thresholds for short-form content. Use higher thresholds when false positives are more expensive than missed articles.

Defaults:

#![allow(unused)]
fn main() {
ReadableOptions {
    min_content_length: 140,
    min_score: 20.0,
}
}

Site Profiles

Site profiles are TOML extraction hints scoped by URL host. They are useful when a site has a stable content container or predictable clutter, but still returns ordinary article-shaped HTML.

Profiles run before generic readability scoring. If a profile produces text below char_threshold, Lectito records the profile decision in diagnostics and continues with generic extraction.

Example

name = "example"
hosts = ["example.com"]
subdomains = true
path_prefixes = ["/blog"]
exclude_path_prefixes = ["/blog/comments"]
content_roots = ["article", "#content"]
remove = [".ad", "nav", "footer"]
remove_id_or_class = ["sidebar"]

[metadata]
title = ["h1"]
author = [".byline"]
date = ["time/@datetime"]
image = ["meta[property='og:image']/@content"]
site_name = "Example"
title_suffixes = [" - Example"]

[cleanup]
enabled = true
prune = true

[fallback]
generic_on_empty = true

Fields

Field	Meaning
`name`	Human-readable profile name used in diagnostics.
`hosts`	Hosts matched by the profile. `www.` is ignored during matching.
`subdomains`	When true, subdomains of each host also match.
`path_prefixes`	Optional path prefixes. Omit to match every path on the host.
`exclude_path_prefixes`	Optional path prefixes that suppress the profile after host matching.
`content_roots`	CSS selectors or supported XPath selectors for article roots.
`remove`	CSS selectors or supported XPath selectors to remove before extraction.
`remove_id_or_class`	Exact id or class tokens to remove.

Metadata fields are optional selector lists, except site_name, which is a constant. Selectors may target attributes with the supported XPath .../@attr form.

Cleanup defaults to enabled. prune controls conditional cleanup. Disabling cleanup should be reserved for sites where the profile root is already clean and generic cleanup removes useful structure.

Selector Support

Profiles accept CSS selectors directly. They also accept a focused XPath subset for compatibility with rule corpuses and older bundled rules:

//tag
//*[@id='value']
//tag[@class='a b']
//tag[contains(@class, 'value')]
/text() suffixes
/@attribute suffixes for metadata selectors

Unsupported XPath expressions are ignored by selector matching, so bundled profiles should have tests that prove their roots match representative pages.

User Profiles

Rust callers pass profile TOML strings through ReadabilityOptions:

#![allow(unused)]
fn main() {
let options = ReadabilityOptions {
    site_profiles: vec![std::fs::read_to_string("example.com.toml")?],
    ..ReadabilityOptions::default()
};
}

The CLI accepts repeatable profile paths:

lectito article.html --base-url https://example.com/post --site-profile example.com.toml

User profiles take precedence over bundled profiles. More specific host and path matches win within each source group.

Lectito.rs