Metadata-Version: 2.5
Name: mdread
Version: 0.1.0
Summary: Fetch any public URL as clean Markdown — agent-friendly HTML→MD
Project-URL: Bug Tracker, https://github.com/Alg0rix/mdread/issues
Project-URL: Documentation, https://github.com/Alg0rix/mdread#readme
Project-URL: Homepage, https://github.com/Alg0rix/mdread
Project-URL: Repository, https://github.com/Alg0rix/mdread
Author: Alg0rix
License: MIT
License-File: LICENSE
Keywords: agent,fetch,html,html-to-markdown,llm,markdown,scrape,web
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Topic :: Text Processing :: Markup :: Markdown
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: beautifulsoup4>=4.12.0
Requires-Dist: httpx>=0.27.0
Provides-Extra: dev
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Description-Content-Type: text/markdown

# mdread

**Fetch any public URL as clean Markdown.**

Agent-friendly HTML→Markdown with redirect re-validation and size limits. Extracted from [tomo](https://github.com/Alg0rix/tomo)'s `web_fetch` tool.

```bash
pip install mdread
# or with uv:
uv add mdread
# or as a CLI tool:
uv tool install mdread
```

## Why mdread?

| Feature | What you get |
|--------|----------------|
| **One call** | `mdread.fetch(url)` → Markdown string |
| **Readable** | Prefers `<article>` / `<main>`; strips scripts, nav, chrome |
| **Bounded** | Large/script-heavy pages fall back to plain text; output is truncated |
| **Library + CLI** | Import it, or run `mdread https://…` |

## Quick start

```python
from mdread import fetch, html_to_markdown, HtmlToMarkdown

# Fetch a page as Markdown
md = fetch("https://example.com")
print(md)

# Convert HTML you already have
print(html_to_markdown("<h1>Hi</h1><p>Hello <strong>world</strong>.</p>"))
# → # Hi
#   Hello **world**.

# Full CommonMark converter (options: ATX headings, fenced code, …)
converter = HtmlToMarkdown({"headingStyle": "atx", "codeBlockStyle": "fenced"})
print(converter.convert("<h2>Title</h2><ul><li>One</li></ul>"))
```

### Structured result

```python
from mdread import fetch_result, FetchError

result = fetch_result("https://example.com", raise_on_error=True)
print(result.url, result.truncated, len(result.text))
```

### CLI

```bash
mdread https://example.com
mdread --timeout 30 --max-chars 50000 https://docs.python.org/3/
python -m mdread https://example.com
```

## API

| Symbol | Description |
|--------|-------------|
| `fetch(url, *, timeout=15, max_chars=100_000, …)` | GET URL → Markdown/text string |
| `fetch_result(…)` | Same, returns `FetchResult` on success |
| `html_to_markdown(html)` | HTML string → Markdown (chrome stripped) |
| `HtmlToMarkdown` | Configurable CommonMark converter |
| `check_url` | URL validation helper |
| `FetchError` | Raised when `raise_on_error=True` |

On soft failure (default), `fetch` returns a string starting with `Error:` so agent tool loops never crash.

## Safety defaults

- Only `http` / `https`
- Manual redirect following (each hop re-checked), max 5 redirects
- Default 15s timeout, 100k char output cap

## Install / develop

```bash
git clone https://github.com/Alg0rix/mdread
cd mdread
uv sync --group dev   # or: pip install -e ".[dev]"
pytest
```

## License

MIT
