name,summary,classifiers,description,author,author_email,description_content_type,home_page,keywords,license,maintainer,maintainer_email,package_url,platform,project_url,project_urls,release_url,requires_dist,requires_python,version,yanked,yanked_reason datasette-sitemap,Generate sitemap.xml for Datasette sites,"[""Framework :: Datasette"", ""License :: OSI Approved :: Apache Software License""]","# datasette-sitemap [![PyPI](https://img.shields.io/pypi/v/datasette-sitemap.svg)](https://pypi.org/project/datasette-sitemap/) [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-sitemap?include_prereleases&label=changelog)](https://github.com/simonw/datasette-sitemap/releases) [![Tests](https://github.com/simonw/datasette-sitemap/workflows/Test/badge.svg)](https://github.com/simonw/datasette-sitemap/actions?query=workflow%3ATest) [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-sitemap/blob/main/LICENSE) Generate sitemap.xml for Datasette sites ## Installation Install this plugin in the same environment as Datasette. datasette install datasette-sitemap ## Demo This plugin is used for the sitemap on [til.simonwillison.net](https://til.simonwillison.net/): - https://til.simonwillison.net/sitemap.xml Here's [the configuration](https://github.com/simonw/til/blob/d4f67743a90a67100b46145986b2dec6f8d96583/metadata.yaml#L14-L16) used for that sitemap. ## Usage Once configured, this plugin adds a sitemap at `/sitemap.xml` with a list of URLs. This list is defined using a SQL query in `metadata.json` (or `.yml`) that looks like this: ```json { ""plugins"": { ""datasette-sitemap"": { ""query"": ""select '/' || id as path from my_table"" } } } ``` Using `metadata.yml` allows for multi-line SQL queries which can be easier to maintain: ```yaml plugins: datasette-sitemap: query: | select '/' || id as path from my_table ``` The SQL query must return a column called `path`. The values in this column must begin with a `/`. They will be used to generate a sitemap that looks like this: ```xml https://example.com/1 https://example.com/2 ``` You can use ``UNION`` in your SQL query to combine results from multiple tables, or include literal paths that you want to include in the index: ```sql select '/data/table1/' || id as path from table1 union select '/data/table2/' || id as path from table2 union select '/about' as path ``` If your Datasette instance has multiple databases you can configure the database to query using the `database` configuration property. By default the domain name for the genearted URLs in the sitemap will be detected from the incoming request. You can set `base_url` instead to override this. This should not include a trailing slash. This example shows both of those settings, running the query against the `content` database and setting a custom base URL: ```yaml plugins: datasette-sitemap: query: | select '/plugins/' || name as path from plugins union select '/tools/' || name as path from tools union select '/news' as path database: content base_url: https://datasette.io ``` [Try that query](https://datasette.io/content?sql=select+%27%2Fplugins%2F%27+||+name+as+path+from+plugins%0D%0Aunion%0D%0Aselect+%27%2Ftools%2F%27+||+name+as+path+from+tools%0D%0Aunion%0D%0Aselect+%27%2Fnews%27+as+path%0D%0A). ## robots.txt This plugin adds a `robots.txt` file pointing to the sitemap: ``` Sitemap: http://example.com/sitemap.xml ``` You can take full control of the sitemap by installing and configuring the [datasette-block-robots](https://datasette.io/plugins/datasette-block-robots) plugin. This plugin will add the `Sitemap:` line even if you are using `datasette-block-robots` for the rest of your `robots.txt` file. ## Adding paths to the sitemap from other plugins This plugin adds a new [plugin hook](https://docs.datasette.io/en/stable/plugin_hooks.html) to Datasete called `sitemap_extra_paths()` which can be used by other plugins to add their own additional lines to the `sitemap.xml` file. The hook accepts these optional parameters: - `datasette`: The current [Datasette instance](https://docs.datasette.io/en/stable/internals.html#datasette-class). You can use this to execute SQL queries or read plugin configuration settings. - `request`: The [Request object](https://docs.datasette.io/en/stable/internals.html#request-object) representing the incoming request to `/sitemap.xml`. The hook should return a list of strings, each representing a path to be added to the sitemap. Each path must begin with a `/`. It can also return an `async def` function, which will be awaited and used to generate a list of lines. Use this option if you need to make `await` calls inside you hook implementation. This example uses the hook to add two extra paths, one of which came from a SQL query: ```python from datasette import hookimpl @hookimpl def sitemap_extra_paths(datasette): async def inner(): db = datasette.get_database() path_from_db = (await db.execute(""select '/example'"")).single_value() return [""/about"", path_from_db] return inner ``` ## Development To set up this plugin locally, first checkout the code. Then create a new virtual environment: cd datasette-sitemap python3 -m venv venv source venv/bin/activate Now install the dependencies and test dependencies: pip install -e '.[test]' To run the tests: pytest ",Simon Willison,,text/markdown,https://github.com/simonw/datasette-sitemap,,"Apache License, Version 2.0",,,https://pypi.org/project/datasette-sitemap/,,https://pypi.org/project/datasette-sitemap/,"{""CI"": ""https://github.com/simonw/datasette-sitemap/actions"", ""Changelog"": ""https://github.com/simonw/datasette-sitemap/releases"", ""Homepage"": ""https://github.com/simonw/datasette-sitemap"", ""Issues"": ""https://github.com/simonw/datasette-sitemap/issues""}",https://pypi.org/project/datasette-sitemap/1.0/,"[""datasette"", ""pytest ; extra == 'test'"", ""pytest-asyncio ; extra == 'test'"", ""datasette-block-robots ; extra == 'test'""]",>=3.7,1.0,0, markdown-to-sqlite,CLI tool for loading markdown files into a SQLite database,"[""Intended Audience :: Developers"", ""Intended Audience :: End Users/Desktop"", ""Intended Audience :: Science/Research"", ""License :: OSI Approved :: Apache Software License"", ""Programming Language :: Python :: 3.6"", ""Programming Language :: Python :: 3.7"", ""Topic :: Database""]","# markdown-to-sqlite [![PyPI](https://img.shields.io/pypi/v/markdown-to-sqlite.svg)](https://pypi.python.org/pypi/markdown-to-sqlite) [![Changelog](https://img.shields.io/github/v/release/simonw/markdown-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/markdown-to-sqlite/releases) [![Tests](https://github.com/simonw/markdown-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/markdown-to-sqlite/actions?query=workflow%3ATest) [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/markdown-to-sqlite/blob/main/LICENSE) CLI tool for loading markdown files into a SQLite database. YAML embedded in the markdown files will be used to populate additional columns. Usage: markdown-to-sqlite [OPTIONS] DBNAME TABLE PATHS... For example: $ markdown-to-sqlite docs.db documents file1.md file2.md ## Breaking change Prior to version 1.0 this argument order was different - markdown files were listed before the database and table. ",Simon Willison,,text/markdown,https://github.com/simonw/markdown-to-sqlite,,"Apache License, Version 2.0",,,https://pypi.org/project/markdown-to-sqlite/,,https://pypi.org/project/markdown-to-sqlite/,"{""CI"": ""https://github.com/simonw/markdown-to-sqlite/actions"", ""Changelog"": ""https://github.com/simonw/markdown-to-sqlite/releases"", ""Homepage"": ""https://github.com/simonw/markdown-to-sqlite"", ""Issues"": ""https://github.com/simonw/markdown-to-sqlite/issues""}",https://pypi.org/project/markdown-to-sqlite/1.0/,"[""yamldown"", ""markdown"", ""sqlite-utils"", ""click"", ""pytest ; extra == 'test'""]",>=3.6,1.0,0, yaml-to-sqlite,Utility for converting YAML files to SQLite,"[""Development Status :: 3 - Alpha"", ""Intended Audience :: Developers"", ""Intended Audience :: End Users/Desktop"", ""Intended Audience :: Science/Research"", ""License :: OSI Approved :: Apache Software License"", ""Programming Language :: Python :: 3.6"", ""Programming Language :: Python :: 3.7""]","# yaml-to-sqlite [![PyPI](https://img.shields.io/pypi/v/yaml-to-sqlite.svg)](https://pypi.org/project/yaml-to-sqlite/) [![Changelog](https://img.shields.io/github/v/release/simonw/yaml-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/yaml-to-sqlite/releases) [![Tests](https://github.com/simonw/yaml-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/yaml-to-sqlite/actions?query=workflow%3ATest) [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/yaml-to-sqlite/blob/main/LICENSE) Load the contents of a YAML file into a SQLite database table. ``` $ yaml-to-sqlite --help Usage: yaml-to-sqlite [OPTIONS] DB_PATH TABLE YAML_FILE Convert YAML files to SQLite Options: --version Show the version and exit. --pk TEXT Column to use as a primary key --single-column TEXT If YAML file is a list of values, populate this column --help Show this message and exit. ``` ## Usage Given a `news.yml` file containing the following: ```yaml - date: 2021-06-05 body: |- [Datasette 0.57](https://docs.datasette.io/en/stable/changelog.html#v0-57) is out with an important security patch. - date: 2021-05-10 body: |- [Django SQL Dashboard](https://simonwillison.net/2021/May/10/django-sql-dashboard/) is a new tool that brings a useful authenticated subset of Datasette to Django projects that are built on top of PostgreSQL. ``` Running this command: ```bash $ yaml-to-sqlite news.db stories news.yml ``` Will create a database file with this schema: ```bash $ sqlite-utils schema news.db CREATE TABLE [stories] ( [date] TEXT, [body] TEXT ); ``` The `--pk` option can be used to set a column as the primary key for the table: ```bash $ yaml-to-sqlite news.db stories news.yml --pk date $ sqlite-utils schema news.db CREATE TABLE [stories] ( [date] TEXT PRIMARY KEY, [body] TEXT ); ``` ## Single column YAML lists The `--single-column` option can be used when the YAML file is a list of values, for example a file called `dogs.yml` containing the following: ```yaml - Cleo - Pancakes - Nixie ``` Running this command: ```bash $ yaml-to-sqlite dogs.db dogs.yaml --single-column=name ``` Will create a single `dogs` table with a single `name` column that is the primary key: ```bash $ sqlite-utils schema dogs.db CREATE TABLE [dogs] ( [name] TEXT PRIMARY KEY ); $ sqlite-utils dogs.db 'select * from dogs' -t name -------- Cleo Pancakes Nixie ``` ",Simon Willison,,text/markdown,https://github.com/simonw/yaml-to-sqlite,,"Apache License, Version 2.0",,,https://pypi.org/project/yaml-to-sqlite/,,https://pypi.org/project/yaml-to-sqlite/,"{""Homepage"": ""https://github.com/simonw/yaml-to-sqlite""}",https://pypi.org/project/yaml-to-sqlite/1.0/,"[""click"", ""PyYAML"", ""sqlite-utils (>=3.9.1)"", ""pytest ; extra == 'test'""]",,1.0,0,