# repos

node_id: R_kgDOHcwLAg
name: datasette-query-files
full_name: eyeseast/datasette-query-files
html_url: https://github.com/eyeseast/datasette-query-files
description: Write Datasette canned queries as plain SQL files
created_at: 2022-06-04T18:52:07Z
updated_at: 2022-07-02T19:46:52Z
pushed_at: 2022-07-02T20:40:51Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin", "python", "sql", "sqlite"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-query-files

  [![PyPI](https://img.shields.io/pypi/v/datasette-query-files.svg)](https://pypi.org/project/datasette-query-files/)
  [![Changelog](https://img.shields.io/github/v/release/eyeseast/datasette-query-files?include_prereleases&label=changelog)](https://github.com/eyeseast/datasette-query-files/releases)
  [![Tests](https://github.com/eyeseast/datasette-query-files/workflows/Test/badge.svg)](https://github.com/eyeseast/datasette-query-files/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/eyeseast/datasette-query-files/blob/main/LICENSE)

  Write Datasette canned queries as plain SQL files.

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-query-files

  Or using `pip` or `pipenv`:

      pip install datasette-query-files
      pipenv install datasette-query-files

  ## Usage

  This plugin will look for [canned queries](https://docs.datasette.io/en/stable/sql_queries.html#canned-queries) in the filesystem, in addition any defined in metadata.

  Let's say you're working in a directory called `project-directory`, with a database file called `my-project.db`. Start by creating a `queries` directory with a `my-project` directory inside it. Any SQL file inside that `my-project` folder will become a canned query that can be run on the `my-project` database. If you have a `query-name.sql` file and a `query-name.json` (or `query-name.yml`) file in the same directory, the JSON file will be used as query metadata.

  ```
  project-directory/
    my-project.db
    queries/
      my-project/
        query-name.sql # a query
        query-name.yml # query metadata
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-query-files
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-query-files" class="anchor" aria-hidden="true" href="#user-content-datasette-query-files"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-query-files</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-query-files/" rel="nofollow"><img src="https://camo.githubusercontent.com/c204f01390d188a0f01eeafcd5f9f369b653b2cfaaef56534a36497a575cc317/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d71756572792d66696c65732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-query-files.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/eyeseast/datasette-query-files/releases"><img src="https://camo.githubusercontent.com/d614cd96ac7137aced94769f27f45a90dfc8ddd9bbff39b15d6c276704b7e703/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f65796573656173742f6461746173657474652d71756572792d66696c65733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/eyeseast/datasette-query-files?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/eyeseast/datasette-query-files/actions?query=workflow%3ATest"><img src="https://github.com/eyeseast/datasette-query-files/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/eyeseast/datasette-query-files/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Write Datasette canned queries as plain SQL files.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-query-files"><pre class="notranslate"><code>datasette install datasette-query-files
  </code></pre></div>
  <p dir="auto">Or using <code>pip</code> or <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install datasette-query-files
  pipenv install datasette-query-files"><pre class="notranslate"><code>pip install datasette-query-files
  pipenv install datasette-query-files
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">This plugin will look for <a href="https://docs.datasette.io/en/stable/sql_queries.html#canned-queries" rel="nofollow">canned queries</a> in the filesystem, in addition any defined in metadata.</p>
  <p dir="auto">Let's say you're working in a directory called <code>project-directory</code>, with a database file called <code>my-project.db</code>. Start by creating a <code>queries</code> directory with a <code>my-project</code> directory inside it. Any SQL file inside that <code>my-project</code> folder will become a canned query that can be run on the <code>my-project</code> database. If you have a <code>query-name.sql</code> file and a <code>query-name.json</code> (or <code>query-name.yml</code>) file in the same directory, the JSON file will be used as query metadata.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="project-directory/
    my-project.db
    queries/
      my-project/
        query-name.sql # a query
        query-name.yml # query metadata"><pre class="notranslate"><code>project-directory/
    my-project.db
    queries/
      my-project/
        query-name.sql # a query
        query-name.yml # query metadata
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-query-files
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-query-files
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>

template_repository:
  {"id": 400878073, "node_id": "MDEwOlJlcG9zaXRvcnk0MDA4NzgwNzM=", "name": "datasette-plugin-template-repository", "full_name": "simonw/datasette-plugin-template-repository", "private": false, "owner": {"login": "simonw", "id": 9599, "node_id": "MDQ6VXNlcjk1OTk=", "avatar_url": "https://avatars.githubusercontent.com/u/9599?v=4", "gravatar_id": "", "url": "https://api.github.com/users/simonw", "html_url": "https://github.com/simonw", "followers_url": "https://api.github.com/users/simonw/followers", "following_url": "https://api.github.com/users/simonw/following{/other_user}", "gists_url": "https://api.github.com/users/simonw/gists{/gist_id}", "starred_url": "https://api.github.com/users/simonw/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/simonw/subscriptions", "organizations_url": "https://api.github.com/users/simonw/orgs", "repos_url": "https://api.github.com/users/simonw/repos", "events_url": "https://api.github.com/users/simonw/events{/privacy}", "received_events_url": "https://api.github.com/users/simonw/received_events", "type": "User", "site_admin": false}, "html_url": "https://github.com/simonw/datasette-plugin-template-repository", "description": "GitHub template repository for creating new Datasette plugins, using the simonw/datasette-plugin cookiecutter template", "fork": false, "url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository", "forks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/forks", "keys_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/keys{/key_id}", "collaborators_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/collaborators{/collaborator}", "teams_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/teams", "hooks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/hooks", "issue_events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/events{/number}", "events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/events", "assignees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/assignees{/user}", "branches_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/branches{/branch}", "tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/tags", "blobs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/blobs{/sha}", "git_tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/tags{/sha}", "git_refs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/refs{/sha}", "trees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/trees{/sha}", "statuses_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/statuses/{sha}", "languages_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/languages", "stargazers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/stargazers", "contributors_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contributors", "subscribers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscribers", "subscription_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscription", "commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/commits{/sha}", "git_commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/commits{/sha}", "comments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/comments{/number}", "issue_comment_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/comments{/number}", "contents_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contents/{+path}", "compare_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/compare/{base}...{head}", "merges_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/merges", "archive_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/{archive_format}{/ref}", "downloads_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/downloads", "issues_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues{/number}", "pulls_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/pulls{/number}", "milestones_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/milestones{/number}", "notifications_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/notifications{?since,all,participating}", "labels_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/labels{/name}", "releases_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/releases{/id}", "deployments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/deployments", "created_at": "2021-08-28T19:50:28Z", "updated_at": "2022-06-10T13:28:46Z", "pushed_at": "2022-03-16T23:42:16Z", "git_url": "git://github.com/simonw/datasette-plugin-template-repository.git", "ssh_url": "git@github.com:simonw/datasette-plugin-template-repository.git", "clone_url": "https://github.com/simonw/datasette-plugin-template-repository.git", "svn_url": "https://github.com/simonw/datasette-plugin-template-repository", "homepage": "", "size": 9, "stargazers_count": 15, "watchers_count": 15, "language": null, "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": true, "has_pages": false, "forks_count": 0, "mirror_url": null, "archived": false, "disabled": false, "open_issues_count": 0, "license": null, "allow_forking": true, "is_template": true, "web_commit_signoff_required": false, "topics": [], "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 15, "default_branch": "main", "permissions": {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}, "temp_clone_token": ""}


node_id: R_kgDOHeFHyg
name: datasette-socrata
full_name: simonw/datasette-socrata
html_url: https://github.com/simonw/datasette-socrata
description: Import data from Socrata into Datasette
created_at: 2022-06-08T15:13:44Z
updated_at: 2022-06-09T21:13:34Z
pushed_at: 2022-06-17T20:29:53Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-socrata

  [![PyPI](https://img.shields.io/pypi/v/datasette-socrata.svg)](https://pypi.org/project/datasette-socrata/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-socrata?include_prereleases&label=changelog)](https://github.com/simonw/datasette-socrata/releases)
  [![Tests](https://github.com/simonw/datasette-socrata/workflows/Test/badge.svg)](https://github.com/simonw/datasette-socrata/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-socrata/blob/main/LICENSE)

  Import data from Socrata into Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-socrata

  ## Usage

  Make sure you have [enabled WAL mode](https://til.simonwillison.net/sqlite/enabling-wal-mode) on your database files before using this plugin.

  Once installed, an interface for importing data from Socrata will become available at this URL:

      /-/import-socrata

  Users will be able to paste in a URL to a dataset on Socrata in order to initialize an import.

  You can also pre-fill the form by passing a `?url=` parameter, for example:

      /-/import-socrata?url=https://data.sfgov.org/City-Infrastructure/Street-Tree-List/tkzw-k3nq

  Any database that is attached to Datasette, is NOT loaded as immutable (with the `-i` option) and that has WAL mode enabled will be available for users to import data into.

  The `import-socrata` permission governs access. By default the `root` actor (accessible using `datasette --root` to start Datasette) is granted that permission.

  You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to other users.

  ## Configuration

  If you only want Socrata imports to be allowed to a specific database, you can configure that using plugin configration in `metadata.yml`:

  ```yaml
  plugins:
    datasette-socrata:
      database: socrata
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-socrata
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-socrata" class="anchor" aria-hidden="true" href="#user-content-datasette-socrata"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-socrata</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-socrata/" rel="nofollow"><img src="https://camo.githubusercontent.com/db40380ae3f84b80e55f3156dd6039851796a73ae5eb79e839c54dc78bc28c28/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d736f63726174612e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-socrata.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-socrata/releases"><img src="https://camo.githubusercontent.com/d093beb5cfaa4caa322dc585e63996f01f1fd16cbe6cbc6f05cb9db54c7b88ec/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d736f63726174613f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-socrata?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-socrata/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-socrata/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-socrata/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Import data from Socrata into Datasette</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-socrata"><pre class="notranslate"><code>datasette install datasette-socrata
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Make sure you have <a href="https://til.simonwillison.net/sqlite/enabling-wal-mode" rel="nofollow">enabled WAL mode</a> on your database files before using this plugin.</p>
  <p dir="auto">Once installed, an interface for importing data from Socrata will become available at this URL:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/-/import-socrata"><pre class="notranslate"><code>/-/import-socrata
  </code></pre></div>
  <p dir="auto">Users will be able to paste in a URL to a dataset on Socrata in order to initialize an import.</p>
  <p dir="auto">You can also pre-fill the form by passing a <code>?url=</code> parameter, for example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/-/import-socrata?url=https://data.sfgov.org/City-Infrastructure/Street-Tree-List/tkzw-k3nq"><pre class="notranslate"><code>/-/import-socrata?url=https://data.sfgov.org/City-Infrastructure/Street-Tree-List/tkzw-k3nq
  </code></pre></div>
  <p dir="auto">Any database that is attached to Datasette, is NOT loaded as immutable (with the <code>-i</code> option) and that has WAL mode enabled will be available for users to import data into.</p>
  <p dir="auto">The <code>import-socrata</code> permission governs access. By default the <code>root</code> actor (accessible using <code>datasette --root</code> to start Datasette) is granted that permission.</p>
  <p dir="auto">You can use permission plugins such as <a href="https://github.com/simonw/datasette-permissions-sql">datasette-permissions-sql</a> to grant additional access to other users.</p>
  <h2 dir="auto"><a id="user-content-configuration" class="anchor" aria-hidden="true" href="#user-content-configuration"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Configuration</h2>
  <p dir="auto">If you only want Socrata imports to be allowed to a specific database, you can configure that using plugin configration in <code>metadata.yml</code>:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="plugins:
    datasette-socrata:
      database: socrata"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-socrata</span>:
      <span class="pl-ent">database</span>: <span class="pl-s">socrata</span></pre></div>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-socrata
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-socrata
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHilbpw
name: datasette-scale-to-zero
full_name: simonw/datasette-scale-to-zero
html_url: https://github.com/simonw/datasette-scale-to-zero
description: Quit Datasette if it has not received traffic for a specified time period
created_at: 2022-06-21T22:49:04Z
updated_at: 2022-07-13T14:02:31Z
pushed_at: 2022-08-05T22:29:17Z
homepage: https://datasette.io/plugins/datasette-scale-to-zero
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-scale-to-zero

  [![PyPI](https://img.shields.io/pypi/v/datasette-scale-to-zero.svg)](https://pypi.org/project/datasette-scale-to-zero/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-scale-to-zero?include_prereleases&label=changelog)](https://github.com/simonw/datasette-scale-to-zero/releases)
  [![Tests](https://github.com/simonw/datasette-scale-to-zero/workflows/Test/badge.svg)](https://github.com/simonw/datasette-scale-to-zero/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-scale-to-zero/blob/main/LICENSE)

  Quit Datasette if it has not received traffic for a specified time period

  Some hosting providers such as [Fly](https://fly.io/) offer a scale to zero mechanism, where servers can shut down and will be automatically started when new traffic arrives.

  This plugin can be used to configure Datasette to quit X minutes (or seconds, or hours) after the last request it received. It can also cause the Datasette server to exit after a configured maximum time whether or not it is receiving traffic.

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-scale-to-zero

  ## Configuration

  This plugin will only take effect if it has been configured.

  Add the following to your ``metadata.json`` or ``metadata.yml`` configuration file:

  ```json
  {
      "plugins": {
          "datasette-scale-to-zero": {
              "duration": "10m"
          }
      }
  }
  ```
  This will cause Datasette to quit if it has not received any HTTP traffic for 10 minutes.

  You can set this value using a suffix of `m` for minutes, `h` for hours or `s` for seconds.

  To cause Datasette to exit if the server has been running for longer than a specific time, use `"max-age"`:

  ```json
  {
      "plugins": {
          "datasette-scale-to-zero": {
              "max-age": "10h"
          }
      }
  }
  ```
  This example will exit the Datasette server if it has been running for more than ten hours.

  You can use `"duration"` and `"max-age"` together in the same configuration file:

  ```json
  {
      "plugins": {
          "datasette-scale-to-zero": {
              "max-age": "10h",
              "duration": "5m"
          }
      }
  }
  ```
  This example will quit if no traffic has been received in five minutes, or if the server has been running for ten hours.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-scale-to-zero
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-scale-to-zero" class="anchor" href="#user-content-datasette-scale-to-zero" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-scale-to-zero</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-scale-to-zero/" rel="nofollow"><img src="https://camo.githubusercontent.com/ca2386dea8d6d4b851a3603de67aefac24923128b004780fa51d9c4b8d0ff1c6/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d7363616c652d746f2d7a65726f2e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-scale-to-zero.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-scale-to-zero/releases"><img src="https://camo.githubusercontent.com/096b12a9ae4fda1fae3c316e4b4625210e2a51100ed34dd764c7baf7f15fd266/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d7363616c652d746f2d7a65726f3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-scale-to-zero?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-scale-to-zero/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-scale-to-zero/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-scale-to-zero/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Quit Datasette if it has not received traffic for a specified time period</p>
  <p dir="auto">Some hosting providers such as <a href="https://fly.io/" rel="nofollow">Fly</a> offer a scale to zero mechanism, where servers can shut down and will be automatically started when new traffic arrives.</p>
  <p dir="auto">This plugin can be used to configure Datasette to quit X minutes (or seconds, or hours) after the last request it received. It can also cause the Datasette server to exit after a configured maximum time whether or not it is receiving traffic.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" href="#user-content-installation" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-scale-to-zero"><pre class="notranslate"><code>datasette install datasette-scale-to-zero
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-configuration" class="anchor" href="#user-content-configuration" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Configuration</h2>
  <p dir="auto">This plugin will only take effect if it has been configured.</p>
  <p dir="auto">Add the following to your <code>metadata.json</code> or <code>metadata.yml</code> configuration file:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-scale-to-zero&quot;: {
              &quot;duration&quot;: &quot;10m&quot;
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-scale-to-zero"</span>: {
              <span class="pl-ent">"duration"</span>: <span class="pl-s"><span class="pl-pds">"</span>10m<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <p dir="auto">This will cause Datasette to quit if it has not received any HTTP traffic for 10 minutes.</p>
  <p dir="auto">You can set this value using a suffix of <code>m</code> for minutes, <code>h</code> for hours or <code>s</code> for seconds.</p>
  <p dir="auto">To cause Datasette to exit if the server has been running for longer than a specific time, use <code>"max-age"</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-scale-to-zero&quot;: {
              &quot;max-age&quot;: &quot;10h&quot;
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-scale-to-zero"</span>: {
              <span class="pl-ent">"max-age"</span>: <span class="pl-s"><span class="pl-pds">"</span>10h<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <p dir="auto">This example will exit the Datasette server if it has been running for more than ten hours.</p>
  <p dir="auto">You can use <code>"duration"</code> and <code>"max-age"</code> together in the same configuration file:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-scale-to-zero&quot;: {
              &quot;max-age&quot;: &quot;10h&quot;,
              &quot;duration&quot;: &quot;5m&quot;
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-scale-to-zero"</span>: {
              <span class="pl-ent">"max-age"</span>: <span class="pl-s"><span class="pl-pds">"</span>10h<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"duration"</span>: <span class="pl-s"><span class="pl-pds">"</span>5m<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <p dir="auto">This example will quit if no traffic has been received in five minutes, or if the server has been running for ten hours.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" href="#user-content-development" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-scale-to-zero
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-scale-to-zero
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHk6Aqw
name: s3-ocr
full_name: simonw/s3-ocr
html_url: https://github.com/simonw/s3-ocr
description: Tools for running OCR against files stored in S3
created_at: 2022-06-28T21:33:09Z
updated_at: 2022-08-10T21:24:45Z
pushed_at: 2022-08-10T04:43:17Z
language: Python
license: apache-2.0
topics: ["ocr", "s3", "textract"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # s3-ocr

  [![PyPI](https://img.shields.io/pypi/v/s3-ocr.svg)](https://pypi.org/project/s3-ocr/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/s3-ocr?include_prereleases&label=changelog)](https://github.com/simonw/s3-ocr/releases)
  [![Tests](https://github.com/simonw/s3-ocr/workflows/Test/badge.svg)](https://github.com/simonw/s3-ocr/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/s3-ocr/blob/master/LICENSE)

  Tools for running OCR against files stored in S3

  Background on this project: [s3-ocr: Extract text from PDF files stored in an S3 bucket](https://simonwillison.net/2022/Jun/30/s3-ocr/)

  ## Installation

  Install this tool using `pip`:

      pip install s3-ocr

  ## Demo

  You can see the results of running this tool against three PDFs from the Internet Archive ([one](https://archive.org/details/unmaskingrobert00houdgoog), [two](https://archive.org/details/practicalmagicia00harr), [three](https://archive.org/details/latestmagicbeing00hoff)) in [this example table](https://s3-ocr-demo.datasette.io/pages/pages?_facet=path#facet-path) hosted using [Datasette](https://datasette.io/).

  ## Starting OCR against PDFs in a bucket

  The `start` command takes a list of keys and submits them to [Textract](https://aws.amazon.com/textract/) for OCR processing.

  You need to have AWS configured using environment variables, credentials file in your home directory or a JSON or INI file generated using [s3-credentials](https://datasette.io/tools/s3-credentials).

  You can start the process running like this:

      s3-ocr start name-of-your-bucket my-pdf-file.pdf

  The paths you specify should be paths within the bucket. If you stored your PDF files in folders inside the bucket it should look like this:

      s3-ocr start name-of-your-bucket path/to/one.pdf path/to/two.pdf

  OCR can take some time. The results of the OCR will be stored in `textract-output` in your bucket.

  To process every file in the bucket with a `.pdf` extension use `--all`:

      s3-ocr start name-of-bucket --all

  To process every file with a `.pdf` extension within a specific folder, use `--prefix`:

      s3-ocr start name-of-bucket --prefix path/to/folder

  ### s3-ocr start --help

  <!-- [[[cog
  import cog
  from s3_ocr import cli
  from click.testing import CliRunner
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["start", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help)
  )
  ]]] -->
  ```
  Usage: s3-ocr start [OPTIONS] BUCKET [KEYS]...

    Start OCR tasks for PDF files in an S3 bucket

        s3-ocr start name-of-bucket path/to/one.pdf path/to/two.pdf

    To process every file with a .pdf extension:

        s3-ocr start name-of-bucket --all

    To process every .pdf in the PUBLIC/ folder:

        s3-ocr start name-of-bucket --prefix PUBLIC/

  Options:
    --all                 Process all PDF files in the bucket
    --prefix TEXT         Process all PDF files within this prefix
    --dry-run             Show what this would do, but don't actually do it
    --no-retry            Don't retry failed requests
    --access-key TEXT     AWS access key ID
    --secret-key TEXT     AWS secret access key
    --session-token TEXT  AWS session token
    --endpoint-url TEXT   Custom endpoint URL
    -a, --auth FILENAME   Path to JSON/INI file containing credentials
    --help                Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## Checking status

  The `s3-ocr status <bucket-name>` command shows a rough indication of progress through the tasks:

  ```
  % s3-ocr status sfms-history
  153 complete out of 532 jobs
  ```
  It compares the jobs that have been submitted, based on `.s3-ocr.json` files, to the jobs that have their results written to the `textract-output/` folder.

  ### s3-ocr status --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["status", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr status [OPTIONS] BUCKET

    Show status of OCR jobs for a bucket

  Options:
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Inspecting a job

  The `s3-ocr inspect-job <job_id>` command can be used to check the status of a specific job ID:
  ```
  % s3-ocr inspect-job b267282745685226339b7e0d4366c4ff6887b7e293ed4b304dc8bb8b991c7864
  {
    "DocumentMetadata": {
      "Pages": 583
    },
    "JobStatus": "SUCCEEDED",
    "DetectDocumentTextModelVersion": "1.0"
  }
  ```

  ### s3-ocr inspect-job --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["inspect-job", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr inspect-job [OPTIONS] JOB_ID

    Show the current status of an OCR job

        s3-ocr inspect-job <job_id>

  Options:
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Fetching the results

  Once an OCR job has completed you can download the resulting JSON using the `fetch` command:

      s3-ocr fetch name-of-bucket path/to/file.pdf

  This will save files in the current directory with names like this:

  - `4d9b5cf580e761fdb16fd24edce14737ebc562972526ef6617554adfa11d6038-1.json`
  - `4d9b5cf580e761fdb16fd24edce14737ebc562972526ef6617554adfa11d6038-2.json`

  The number of files will vary depending on the length of the document.

  If you don't want separate files you can combine them together using the `-c/--combine` option:

      s3-ocr fetch name-of-bucket path/to/file.pdf --combine output.json

  The `output.json` file will then contain data that looks something like this:

  ```
  {
    "Blocks": [
      {
        "BlockType": "PAGE",
        "Geometry": {...}
        "Page": 1,
        ...
      },
      {
        "BlockType": "LINE",
        "Page": 1,
        ...
        "Text": "Barry",
      },
  ```
  ### s3-ocr fetch --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["fetch", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr fetch [OPTIONS] BUCKET KEY

    Fetch the OCR results for a specified file

        s3-ocr fetch name-of-bucket path/to/key.pdf

    This will save files in the current directory called things like

        a806e67e504fc15f...48314e-1.json     a806e67e504fc15f...48314e-2.json

    To combine these together into a single JSON file with a specified name, use:

        s3-ocr fetch name-of-bucket path/to/key.pdf --combine output.json

    Use "--output -" to print the combined JSON to standard output instead.

  Options:
    -c, --combine FILENAME  Write combined JSON to file
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Fetching just the text of a page

  If you don't want to deal with the JSON directly, you can use the `text` command to retrieve just the text extracted from a PDF:

      s3-ocr text name-of-bucket path/to/file.pdf

  This will output plain text to standard output.

  To save that to a file, use this:

      s3-ocr text name-of-bucket path/to/file.pdf > text.txt

  Separate pages will be separated by three newlines. To separate them using a `----` horizontal divider instead add `--divider`:

      s3-ocr text name-of-bucket path/to/file.pdf --divider

  ### s3-ocr text --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["text", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr text [OPTIONS] BUCKET KEY

    Retrieve the text from an OCRd PDF file

        s3-ocr text name-of-bucket path/to/key.pdf

  Options:
    --divider             Add ---- between pages
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Avoiding processing duplicates

  If you move files around within your S3 bucket `s3-ocr` can lose track of which files have already been processed. This can lead to additional Textract charges for processing should you run `s3-ocr start` against those new files.

  The `s3-ocr dedupe` command addresses this by scanning your bucket for files that have a new name but have previously been processed. It does this by looking at the `ETag` for each file, which represents the MD5 hash of the file contents.

  The command will write out new `.s3ocr.json` files for each detected duplicate. This will avoid those duplicates being run those duplicates through OCR a second time should yo run `s3-ocr start`.

      s3-ocr dedupe name-of-bucket

  Add `--dry-run` for a preview of the changes that will be made to your bucket.

  ### s3-ocr dedupe --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["dedupe", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr dedupe [OPTIONS] BUCKET

    Scan every file in the bucket checking for duplicates - files that have not
    yet been OCRd but that have the same contents (based on ETag) as a file that
    HAS been OCRd.

        s3-ocr dedupe name-of-bucket

  Options:
    --dry-run             Show output without writing anything to S3
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Changes made to your bucket

  To keep track of which files have been submitted for processing, `s3-ocr` will create a JSON file for every file that it adds to the OCR queue.

  This file will be called:

      path-to-file/name-of-file.pdf.s3-ocr.json

  Each of these JSON files contains data that looks like this:

  ```json
  {
    "job_id": "a34eb4e8dc7e70aa9668f7272aa403e85997364199a654422340bc5ada43affe",
    "etag": "\"b0c77472e15500347ebf46032a454e8e\""
  }
  ```
  The recorded `job_id` can be used later to associate the file with the results of the OCR task in `textract-output/`.

  The `etag` is the ETag of the S3 object at the time it was submitted. This can be used later to determine if a file has changed since it last had OCR run against it.

  This design for the tool, with the `.s3-ocr.json` files tracking jobs that have been submitted, means that it is safe to run `s3-ocr start` against the same bucket multiple times without the risk of starting duplicate OCR jobs.

  ## Creating a SQLite index of your OCR results

  The `s3-ocr index <bucket> <database_file>` command creates a SQLite database containing the results of the OCR, and configures SQLite full-text search against the text:

  ```
  % s3-ocr index sfms-history index.db
  Fetching job details  [####################################]  100%
  Populating pages table  [####################----------------]   55%  00:03:18
  ```
  The schema of the resulting database looks like this (excluding the FTS tables):
  ```sql
  CREATE TABLE [pages] (
     [path] TEXT,
     [page] INTEGER,
     [folder] TEXT,
     [text] TEXT,
     PRIMARY KEY ([path], [page])
  );
  CREATE TABLE [ocr_jobs] (
     [key] TEXT PRIMARY KEY,
     [job_id] TEXT,
     [etag] TEXT,
     [s3_ocr_etag] TEXT
  );
  CREATE TABLE [fetched_jobs] (
     [job_id] TEXT PRIMARY KEY
  );
  ```
  The database is designed to be used with [Datasette](https://datasette.io).

  ### s3-ocr index --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["index", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr index [OPTIONS] BUCKET DATABASE

    Create a SQLite database with OCR results for files in a bucket

  Options:
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd s3-ocr
      python -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

  To regenerate the README file with the latest `--help`:

      cog -r README.md

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-s3-ocr" class="anchor" href="#user-content-s3-ocr" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>s3-ocr</h1>
  <p dir="auto"><a href="https://pypi.org/project/s3-ocr/" rel="nofollow"><img src="https://camo.githubusercontent.com/697cb78ed5f8b2955201be9925084bf6c5603a1cebc448d174f5684ae1453d68/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73332d6f63722e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/s3-ocr.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/s3-ocr/releases"><img src="https://camo.githubusercontent.com/a5fa08d6edb96e5b9f55b10efcb2820fdb2168be4a766b9a38a87da938251c66/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f73332d6f63723f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/s3-ocr?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/s3-ocr/actions?query=workflow%3ATest"><img src="https://github.com/simonw/s3-ocr/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/s3-ocr/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Tools for running OCR against files stored in S3</p>
  <p dir="auto">Background on this project: <a href="https://simonwillison.net/2022/Jun/30/s3-ocr/" rel="nofollow">s3-ocr: Extract text from PDF files stored in an S3 bucket</a></p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" href="#user-content-installation" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this tool using <code>pip</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install s3-ocr"><pre class="notranslate"><code>pip install s3-ocr
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-demo" class="anchor" href="#user-content-demo" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p dir="auto">You can see the results of running this tool against three PDFs from the Internet Archive (<a href="https://archive.org/details/unmaskingrobert00houdgoog" rel="nofollow">one</a>, <a href="https://archive.org/details/practicalmagicia00harr" rel="nofollow">two</a>, <a href="https://archive.org/details/latestmagicbeing00hoff" rel="nofollow">three</a>) in <a href="https://s3-ocr-demo.datasette.io/pages/pages?_facet=path#facet-path" rel="nofollow">this example table</a> hosted using <a href="https://datasette.io/" rel="nofollow">Datasette</a>.</p>
  <h2 dir="auto"><a id="user-content-starting-ocr-against-pdfs-in-a-bucket" class="anchor" href="#user-content-starting-ocr-against-pdfs-in-a-bucket" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Starting OCR against PDFs in a bucket</h2>
  <p dir="auto">The <code>start</code> command takes a list of keys and submits them to <a href="https://aws.amazon.com/textract/" rel="nofollow">Textract</a> for OCR processing.</p>
  <p dir="auto">You need to have AWS configured using environment variables, credentials file in your home directory or a JSON or INI file generated using <a href="https://datasette.io/tools/s3-credentials" rel="nofollow">s3-credentials</a>.</p>
  <p dir="auto">You can start the process running like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr start name-of-your-bucket my-pdf-file.pdf"><pre class="notranslate"><code>s3-ocr start name-of-your-bucket my-pdf-file.pdf
  </code></pre></div>
  <p dir="auto">The paths you specify should be paths within the bucket. If you stored your PDF files in folders inside the bucket it should look like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr start name-of-your-bucket path/to/one.pdf path/to/two.pdf"><pre class="notranslate"><code>s3-ocr start name-of-your-bucket path/to/one.pdf path/to/two.pdf
  </code></pre></div>
  <p dir="auto">OCR can take some time. The results of the OCR will be stored in <code>textract-output</code> in your bucket.</p>
  <p dir="auto">To process every file in the bucket with a <code>.pdf</code> extension use <code>--all</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr start name-of-bucket --all"><pre class="notranslate"><code>s3-ocr start name-of-bucket --all
  </code></pre></div>
  <p dir="auto">To process every file with a <code>.pdf</code> extension within a specific folder, use <code>--prefix</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr start name-of-bucket --prefix path/to/folder"><pre class="notranslate"><code>s3-ocr start name-of-bucket --prefix path/to/folder
  </code></pre></div>
  <h3 dir="auto"><a id="user-content-s3-ocr-start---help" class="anchor" href="#user-content-s3-ocr-start---help" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>s3-ocr start --help</h3>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: s3-ocr start [OPTIONS] BUCKET [KEYS]...

    Start OCR tasks for PDF files in an S3 bucket

        s3-ocr start name-of-bucket path/to/one.pdf path/to/two.pdf

    To process every file with a .pdf extension:

        s3-ocr start name-of-bucket --all

    To process every .pdf in the PUBLIC/ folder:

        s3-ocr start name-of-bucket --prefix PUBLIC/

  Options:
    --all                 Process all PDF files in the bucket
    --prefix TEXT         Process all PDF files within this prefix
    --dry-run             Show what this would do, but don't actually do it
    --no-retry            Don't retry failed requests
    --access-key TEXT     AWS access key ID
    --secret-key TEXT     AWS secret access key
    --session-token TEXT  AWS session token
    --endpoint-url TEXT   Custom endpoint URL
    -a, --auth FILENAME   Path to JSON/INI file containing credentials
    --help                Show this message and exit.
  "><pre class="notranslate"><code>Usage: s3-ocr start [OPTIONS] BUCKET [KEYS]...

    Start OCR tasks for PDF files in an S3 bucket

        s3-ocr start name-of-bucket path/to/one.pdf path/to/two.pdf

    To process every file with a .pdf extension:

        s3-ocr start name-of-bucket --all

    To process every .pdf in the PUBLIC/ folder:

        s3-ocr start name-of-bucket --prefix PUBLIC/

  Options:
    --all                 Process all PDF files in the bucket
    --prefix TEXT         Process all PDF files within this prefix
    --dry-run             Show what this would do, but don't actually do it
    --no-retry            Don't retry failed requests
    --access-key TEXT     AWS access key ID
    --secret-key TEXT     AWS secret access key
    --session-token TEXT  AWS session token
    --endpoint-url TEXT   Custom endpoint URL
    -a, --auth FILENAME   Path to JSON/INI file containing credentials
    --help                Show this message and exit.

  </code></pre></div>

  <h2 dir="auto"><a id="user-content-checking-status" class="anchor" href="#user-content-checking-status" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Checking status</h2>
  <p dir="auto">The <code>s3-ocr status &lt;bucket-name&gt;</code> command shows a rough indication of progress through the tasks:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="% s3-ocr status sfms-history
  153 complete out of 532 jobs"><pre class="notranslate"><code>% s3-ocr status sfms-history
  153 complete out of 532 jobs
  </code></pre></div>
  <p dir="auto">It compares the jobs that have been submitted, based on <code>.s3-ocr.json</code> files, to the jobs that have their results written to the <code>textract-output/</code> folder.</p>
  <h3 dir="auto"><a id="user-content-s3-ocr-status---help" class="anchor" href="#user-content-s3-ocr-status---help" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>s3-ocr status --help</h3>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: s3-ocr status [OPTIONS] BUCKET

    Show status of OCR jobs for a bucket

  Options:
    --access-key ..."><pre class="notranslate"><code>Usage: s3-ocr status [OPTIONS] BUCKET

    Show status of OCR jobs for a bucket

  Options:
    --access-key ...
  </code></pre></div>

  <h2 dir="auto"><a id="user-content-inspecting-a-job" class="anchor" href="#user-content-inspecting-a-job" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Inspecting a job</h2>
  <p dir="auto">The <code>s3-ocr inspect-job &lt;job_id&gt;</code> command can be used to check the status of a specific job ID:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="% s3-ocr inspect-job b267282745685226339b7e0d4366c4ff6887b7e293ed4b304dc8bb8b991c7864
  {
    &quot;DocumentMetadata&quot;: {
      &quot;Pages&quot;: 583
    },
    &quot;JobStatus&quot;: &quot;SUCCEEDED&quot;,
    &quot;DetectDocumentTextModelVersion&quot;: &quot;1.0&quot;
  }"><pre class="notranslate"><code>% s3-ocr inspect-job b267282745685226339b7e0d4366c4ff6887b7e293ed4b304dc8bb8b991c7864
  {
    "DocumentMetadata": {
      "Pages": 583
    },
    "JobStatus": "SUCCEEDED",
    "DetectDocumentTextModelVersion": "1.0"
  }
  </code></pre></div>
  <h3 dir="auto"><a id="user-content-s3-ocr-inspect-job---help" class="anchor" href="#user-content-s3-ocr-inspect-job---help" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>s3-ocr inspect-job --help</h3>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: s3-ocr inspect-job [OPTIONS] JOB_ID

    Show the current status of an OCR job

        s3-ocr inspect-job &lt;job_id&gt;

  Options:
    --access-key ..."><pre class="notranslate"><code>Usage: s3-ocr inspect-job [OPTIONS] JOB_ID

    Show the current status of an OCR job

        s3-ocr inspect-job &lt;job_id&gt;

  Options:
    --access-key ...
  </code></pre></div>

  <h2 dir="auto"><a id="user-content-fetching-the-results" class="anchor" href="#user-content-fetching-the-results" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Fetching the results</h2>
  <p dir="auto">Once an OCR job has completed you can download the resulting JSON using the <code>fetch</code> command:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr fetch name-of-bucket path/to/file.pdf"><pre class="notranslate"><code>s3-ocr fetch name-of-bucket path/to/file.pdf
  </code></pre></div>
  <p dir="auto">This will save files in the current directory with names like this:</p>
  <ul dir="auto">
  <li><code>4d9b5cf580e761fdb16fd24edce14737ebc562972526ef6617554adfa11d6038-1.json</code></li>
  <li><code>4d9b5cf580e761fdb16fd24edce14737ebc562972526ef6617554adfa11d6038-2.json</code></li>
  </ul>
  <p dir="auto">The number of files will vary depending on the length of the document.</p>
  <p dir="auto">If you don't want separate files you can combine them together using the <code>-c/--combine</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr fetch name-of-bucket path/to/file.pdf --combine output.json"><pre class="notranslate"><code>s3-ocr fetch name-of-bucket path/to/file.pdf --combine output.json
  </code></pre></div>
  <p dir="auto">The <code>output.json</code> file will then contain data that looks something like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
    &quot;Blocks&quot;: [
      {
        &quot;BlockType&quot;: &quot;PAGE&quot;,
        &quot;Geometry&quot;: {...}
        &quot;Page&quot;: 1,
        ...
      },
      {
        &quot;BlockType&quot;: &quot;LINE&quot;,
        &quot;Page&quot;: 1,
        ...
        &quot;Text&quot;: &quot;Barry&quot;,
      },"><pre class="notranslate"><code>{
    "Blocks": [
      {
        "BlockType": "PAGE",
        "Geometry": {...}
        "Page": 1,
        ...
      },
      {
        "BlockType": "LINE",
        "Page": 1,
        ...
        "Text": "Barry",
      },
  </code></pre></div>
  <h3 dir="auto"><a id="user-content-s3-ocr-fetch---help" class="anchor" href="#user-content-s3-ocr-fetch---help" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>s3-ocr fetch --help</h3>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: s3-ocr fetch [OPTIONS] BUCKET KEY

    Fetch the OCR results for a specified file

        s3-ocr fetch name-of-bucket path/to/key.pdf

    This will save files in the current directory called things like

        a806e67e504fc15f...48314e-1.json     a806e67e504fc15f...48314e-2.json

    To combine these together into a single JSON file with a specified name, use:

        s3-ocr fetch name-of-bucket path/to/key.pdf --combine output.json

    Use &quot;--output -&quot; to print the combined JSON to standard output instead.

  Options:
    -c, --combine FILENAME  Write combined JSON to file
    --access-key ..."><pre class="notranslate"><code>Usage: s3-ocr fetch [OPTIONS] BUCKET KEY

    Fetch the OCR results for a specified file

        s3-ocr fetch name-of-bucket path/to/key.pdf

    This will save files in the current directory called things like

        a806e67e504fc15f...48314e-1.json     a806e67e504fc15f...48314e-2.json

    To combine these together into a single JSON file with a specified name, use:

        s3-ocr fetch name-of-bucket path/to/key.pdf --combine output.json

    Use "--output -" to print the combined JSON to standard output instead.

  Options:
    -c, --combine FILENAME  Write combined JSON to file
    --access-key ...
  </code></pre></div>

  <h2 dir="auto"><a id="user-content-fetching-just-the-text-of-a-page" class="anchor" href="#user-content-fetching-just-the-text-of-a-page" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Fetching just the text of a page</h2>
  <p dir="auto">If you don't want to deal with the JSON directly, you can use the <code>text</code> command to retrieve just the text extracted from a PDF:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr text name-of-bucket path/to/file.pdf"><pre class="notranslate"><code>s3-ocr text name-of-bucket path/to/file.pdf
  </code></pre></div>
  <p dir="auto">This will output plain text to standard output.</p>
  <p dir="auto">To save that to a file, use this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr text name-of-bucket path/to/file.pdf &gt; text.txt"><pre class="notranslate"><code>s3-ocr text name-of-bucket path/to/file.pdf &gt; text.txt
  </code></pre></div>
  <p dir="auto">Separate pages will be separated by three newlines. To separate them using a <code>----</code> horizontal divider instead add <code>--divider</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr text name-of-bucket path/to/file.pdf --divider"><pre class="notranslate"><code>s3-ocr text name-of-bucket path/to/file.pdf --divider
  </code></pre></div>
  <h3 dir="auto"><a id="user-content-s3-ocr-text---help" class="anchor" href="#user-content-s3-ocr-text---help" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>s3-ocr text --help</h3>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: s3-ocr text [OPTIONS] BUCKET KEY

    Retrieve the text from an OCRd PDF file

        s3-ocr text name-of-bucket path/to/key.pdf

  Options:
    --divider             Add ---- between pages
    --access-key ..."><pre class="notranslate"><code>Usage: s3-ocr text [OPTIONS] BUCKET KEY

    Retrieve the text from an OCRd PDF file

        s3-ocr text name-of-bucket path/to/key.pdf

  Options:
    --divider             Add ---- between pages
    --access-key ...
  </code></pre></div>

  <h2 dir="auto"><a id="user-content-avoiding-processing-duplicates" class="anchor" href="#user-content-avoiding-processing-duplicates" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Avoiding processing duplicates</h2>
  <p dir="auto">If you move files around within your S3 bucket <code>s3-ocr</code> can lose track of which files have already been processed. This can lead to additional Textract charges for processing should you run <code>s3-ocr start</code> against those new files.</p>
  <p dir="auto">The <code>s3-ocr dedupe</code> command addresses this by scanning your bucket for files that have a new name but have previously been processed. It does this by looking at the <code>ETag</code> for each file, which represents the MD5 hash of the file contents.</p>
  <p dir="auto">The command will write out new <code>.s3ocr.json</code> files for each detected duplicate. This will avoid those duplicates being run those duplicates through OCR a second time should yo run <code>s3-ocr start</code>.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="s3-ocr dedupe name-of-bucket"><pre class="notranslate"><code>s3-ocr dedupe name-of-bucket
  </code></pre></div>
  <p dir="auto">Add <code>--dry-run</code> for a preview of the changes that will be made to your bucket.</p>
  <h3 dir="auto"><a id="user-content-s3-ocr-dedupe---help" class="anchor" href="#user-content-s3-ocr-dedupe---help" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>s3-ocr dedupe --help</h3>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: s3-ocr dedupe [OPTIONS] BUCKET

    Scan every file in the bucket checking for duplicates - files that have not
    yet been OCRd but that have the same contents (based on ETag) as a file that
    HAS been OCRd.

        s3-ocr dedupe name-of-bucket

  Options:
    --dry-run             Show output without writing anything to S3
    --access-key ..."><pre class="notranslate"><code>Usage: s3-ocr dedupe [OPTIONS] BUCKET

    Scan every file in the bucket checking for duplicates - files that have not
    yet been OCRd but that have the same contents (based on ETag) as a file that
    HAS been OCRd.

        s3-ocr dedupe name-of-bucket

  Options:
    --dry-run             Show output without writing anything to S3
    --access-key ...
  </code></pre></div>

  <h2 dir="auto"><a id="user-content-changes-made-to-your-bucket" class="anchor" href="#user-content-changes-made-to-your-bucket" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Changes made to your bucket</h2>
  <p dir="auto">To keep track of which files have been submitted for processing, <code>s3-ocr</code> will create a JSON file for every file that it adds to the OCR queue.</p>
  <p dir="auto">This file will be called:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="path-to-file/name-of-file.pdf.s3-ocr.json"><pre class="notranslate"><code>path-to-file/name-of-file.pdf.s3-ocr.json
  </code></pre></div>
  <p dir="auto">Each of these JSON files contains data that looks like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
    &quot;job_id&quot;: &quot;a34eb4e8dc7e70aa9668f7272aa403e85997364199a654422340bc5ada43affe&quot;,
    &quot;etag&quot;: &quot;\&quot;b0c77472e15500347ebf46032a454e8e\&quot;&quot;
  }"><pre>{
    <span class="pl-ent">"job_id"</span>: <span class="pl-s"><span class="pl-pds">"</span>a34eb4e8dc7e70aa9668f7272aa403e85997364199a654422340bc5ada43affe<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"etag"</span>: <span class="pl-s"><span class="pl-pds">"</span><span class="pl-cce">\"</span>b0c77472e15500347ebf46032a454e8e<span class="pl-cce">\"</span><span class="pl-pds">"</span></span>
  }</pre></div>
  <p dir="auto">The recorded <code>job_id</code> can be used later to associate the file with the results of the OCR task in <code>textract-output/</code>.</p>
  <p dir="auto">The <code>etag</code> is the ETag of the S3 object at the time it was submitted. This can be used later to determine if a file has changed since it last had OCR run against it.</p>
  <p dir="auto">This design for the tool, with the <code>.s3-ocr.json</code> files tracking jobs that have been submitted, means that it is safe to run <code>s3-ocr start</code> against the same bucket multiple times without the risk of starting duplicate OCR jobs.</p>
  <h2 dir="auto"><a id="user-content-creating-a-sqlite-index-of-your-ocr-results" class="anchor" href="#user-content-creating-a-sqlite-index-of-your-ocr-results" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Creating a SQLite index of your OCR results</h2>
  <p dir="auto">The <code>s3-ocr index &lt;bucket&gt; &lt;database_file&gt;</code> command creates a SQLite database containing the results of the OCR, and configures SQLite full-text search against the text:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="% s3-ocr index sfms-history index.db
  Fetching job details  [####################################]  100%
  Populating pages table  [####################----------------]   55%  00:03:18"><pre class="notranslate"><code>% s3-ocr index sfms-history index.db
  Fetching job details  [####################################]  100%
  Populating pages table  [####################----------------]   55%  00:03:18
  </code></pre></div>
  <p dir="auto">The schema of the resulting database looks like this (excluding the FTS tables):</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="CREATE TABLE [pages] (
     [path] TEXT,
     [page] INTEGER,
     [folder] TEXT,
     [text] TEXT,
     PRIMARY KEY ([path], [page])
  );
  CREATE TABLE [ocr_jobs] (
     [key] TEXT PRIMARY KEY,
     [job_id] TEXT,
     [etag] TEXT,
     [s3_ocr_etag] TEXT
  );
  CREATE TABLE [fetched_jobs] (
     [job_id] TEXT PRIMARY KEY
  );"><pre>CREATE TABLE [pages] (
     [<span class="pl-k">path</span>] <span class="pl-k">TEXT</span>,
     [page] <span class="pl-k">INTEGER</span>,
     [folder] <span class="pl-k">TEXT</span>,
     [<span class="pl-k">text</span>] <span class="pl-k">TEXT</span>,
     <span class="pl-k">PRIMARY KEY</span> ([<span class="pl-k">path</span>], [page])
  );
  CREATE TABLE [ocr_jobs] (
     [key] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>,
     [job_id] <span class="pl-k">TEXT</span>,
     [etag] <span class="pl-k">TEXT</span>,
     [s3_ocr_etag] <span class="pl-k">TEXT</span>
  );
  CREATE TABLE [fetched_jobs] (
     [job_id] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>
  );</pre></div>
  <p dir="auto">The database is designed to be used with <a href="https://datasette.io" rel="nofollow">Datasette</a>.</p>
  <h3 dir="auto"><a id="user-content-s3-ocr-index---help" class="anchor" href="#user-content-s3-ocr-index---help" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>s3-ocr index --help</h3>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: s3-ocr index [OPTIONS] BUCKET DATABASE

    Create a SQLite database with OCR results for files in a bucket

  Options:
    --access-key ..."><pre class="notranslate"><code>Usage: s3-ocr index [OPTIONS] BUCKET DATABASE

    Create a SQLite database with OCR results for files in a bucket

  Options:
    --access-key ...
  </code></pre></div>

  <h2 dir="auto"><a id="user-content-development" class="anchor" href="#user-content-development" aria-hidden="true"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd s3-ocr
  python -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd s3-ocr
  python -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  <p dir="auto">To regenerate the README file with the latest <code>--help</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cog -r README.md"><pre class="notranslate"><code>cog -r README.md
  </code></pre></div>
  </article></div>


node_id: R_kgDOHmgSAw
name: datasette-expose-env
full_name: simonw/datasette-expose-env
html_url: https://github.com/simonw/datasette-expose-env
description: Datasette plugin to expose selected environment variables at /-/env for debugging
created_at: 2022-07-03T21:14:29Z
updated_at: 2022-07-03T21:27:12Z
pushed_at: 2022-07-03T21:28:55Z
homepage: https://datasette.io/plugins/datasette-expose-env
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-expose-env

  [![PyPI](https://img.shields.io/pypi/v/datasette-expose-env.svg)](https://pypi.org/project/datasette-expose-env/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-expose-env?include_prereleases&label=changelog)](https://github.com/simonw/datasette-expose-env/releases)
  [![Tests](https://github.com/simonw/datasette-expose-env/workflows/Test/badge.svg)](https://github.com/simonw/datasette-expose-env/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-expose-env/blob/main/LICENSE)

  Datasette plugin to expose selected environment variables at `/-/env` for debugging

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-expose-env

  ## Configuration

  Decide on a list of environment variables you would like to expose, then add the following to your `metadata.yml` configuration:

  ```yaml
  plugins:
      datasette-expose-env:
      - ENV_VAR_1
      - ENV_VAR_2
      - ENV_VAR_3
  ```

  If you are using JSON in a `metadata.json` file use the following:

  ```json
  {
      "plugins": {
          "datasette-expose-env": [
              "ENV_VAR_1",
              "ENV_VAR_2",
              "ENV_VAR_3"
          ]
      }
  }
  ```

  Visit `/-/env` on your Datasette instance to see the values of the environment variables.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-expose-env
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-expose-env" class="anchor" aria-hidden="true" href="#user-content-datasette-expose-env"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-expose-env</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-expose-env/" rel="nofollow"><img src="https://camo.githubusercontent.com/517c6344ee172d57f8ff5e4fcd9342e7ecfec45aababd88e8e85f53c9ba43ece/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6578706f73652d656e762e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-expose-env.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-expose-env/releases"><img src="https://camo.githubusercontent.com/070f8badacd8aece25028f405f9622f5a9e066d0252971b3cda58801c04c3105/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6578706f73652d656e763f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-expose-env?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-expose-env/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-expose-env/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-expose-env/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Datasette plugin to expose selected environment variables at <code>/-/env</code> for debugging</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-expose-env"><pre class="notranslate"><code>datasette install datasette-expose-env
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-configuration" class="anchor" aria-hidden="true" href="#user-content-configuration"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Configuration</h2>
  <p dir="auto">Decide on a list of environment variables you would like to expose, then add the following to your <code>metadata.yml</code> configuration:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="plugins:
      datasette-expose-env:
      - ENV_VAR_1
      - ENV_VAR_2
      - ENV_VAR_3"><pre><span class="pl-ent">plugins</span>:
      <span class="pl-ent">datasette-expose-env</span>:
      - <span class="pl-s">ENV_VAR_1</span>
      - <span class="pl-s">ENV_VAR_2</span>
      - <span class="pl-s">ENV_VAR_3</span></pre></div>
  <p dir="auto">If you are using JSON in a <code>metadata.json</code> file use the following:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-expose-env&quot;: [
              &quot;ENV_VAR_1&quot;,
              &quot;ENV_VAR_2&quot;,
              &quot;ENV_VAR_3&quot;
          ]
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-expose-env"</span>: [
              <span class="pl-s"><span class="pl-pds">"</span>ENV_VAR_1<span class="pl-pds">"</span></span>,
              <span class="pl-s"><span class="pl-pds">"</span>ENV_VAR_2<span class="pl-pds">"</span></span>,
              <span class="pl-s"><span class="pl-pds">"</span>ENV_VAR_3<span class="pl-pds">"</span></span>
          ]
      }
  }</pre></div>
  <p dir="auto">Visit <code>/-/env</code> on your Datasette instance to see the values of the environment variables.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-expose-env
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-expose-env
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>

template_repository:
  {"id": 400878073, "node_id": "MDEwOlJlcG9zaXRvcnk0MDA4NzgwNzM=", "name": "datasette-plugin-template-repository", "full_name": "simonw/datasette-plugin-template-repository", "private": false, "owner": {"login": "simonw", "id": 9599, "node_id": "MDQ6VXNlcjk1OTk=", "avatar_url": "https://avatars.githubusercontent.com/u/9599?v=4", "gravatar_id": "", "url": "https://api.github.com/users/simonw", "html_url": "https://github.com/simonw", "followers_url": "https://api.github.com/users/simonw/followers", "following_url": "https://api.github.com/users/simonw/following{/other_user}", "gists_url": "https://api.github.com/users/simonw/gists{/gist_id}", "starred_url": "https://api.github.com/users/simonw/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/simonw/subscriptions", "organizations_url": "https://api.github.com/users/simonw/orgs", "repos_url": "https://api.github.com/users/simonw/repos", "events_url": "https://api.github.com/users/simonw/events{/privacy}", "received_events_url": "https://api.github.com/users/simonw/received_events", "type": "User", "site_admin": false}, "html_url": "https://github.com/simonw/datasette-plugin-template-repository", "description": "GitHub template repository for creating new Datasette plugins, using the simonw/datasette-plugin cookiecutter template", "fork": false, "url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository", "forks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/forks", "keys_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/keys{/key_id}", "collaborators_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/collaborators{/collaborator}", "teams_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/teams", "hooks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/hooks", "issue_events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/events{/number}", "events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/events", "assignees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/assignees{/user}", "branches_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/branches{/branch}", "tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/tags", "blobs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/blobs{/sha}", "git_tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/tags{/sha}", "git_refs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/refs{/sha}", "trees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/trees{/sha}", "statuses_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/statuses/{sha}", "languages_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/languages", "stargazers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/stargazers", "contributors_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contributors", "subscribers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscribers", "subscription_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscription", "commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/commits{/sha}", "git_commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/commits{/sha}", "comments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/comments{/number}", "issue_comment_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/comments{/number}", "contents_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contents/{+path}", "compare_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/compare/{base}...{head}", "merges_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/merges", "archive_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/{archive_format}{/ref}", "downloads_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/downloads", "issues_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues{/number}", "pulls_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/pulls{/number}", "milestones_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/milestones{/number}", "notifications_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/notifications{?since,all,participating}", "labels_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/labels{/name}", "releases_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/releases{/id}", "deployments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/deployments", "created_at": "2021-08-28T19:50:28Z", "updated_at": "2022-06-10T13:28:46Z", "pushed_at": "2022-03-16T23:42:16Z", "git_url": "git://github.com/simonw/datasette-plugin-template-repository.git", "ssh_url": "git@github.com:simonw/datasette-plugin-template-repository.git", "clone_url": "https://github.com/simonw/datasette-plugin-template-repository.git", "svn_url": "https://github.com/simonw/datasette-plugin-template-repository", "homepage": "", "size": 9, "stargazers_count": 15, "watchers_count": 15, "language": null, "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": true, "has_pages": false, "forks_count": 0, "mirror_url": null, "archived": false, "disabled": false, "open_issues_count": 0, "license": null, "allow_forking": true, "is_template": true, "web_commit_signoff_required": false, "topics": [], "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 15, "default_branch": "main", "permissions": {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}, "temp_clone_token": ""}


node_id: R_kgDOHoFAng
name: sqlite-comprehend
full_name: simonw/sqlite-comprehend
html_url: https://github.com/simonw/sqlite-comprehend
description: Tools for running data in a SQLite database through AWS Comprehend
created_at: 2022-07-08T06:26:15Z
updated_at: 2022-07-11T21:44:34Z
pushed_at: 2022-07-12T14:21:42Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # sqlite-comprehend

  [![PyPI](https://img.shields.io/pypi/v/sqlite-comprehend.svg)](https://pypi.org/project/sqlite-comprehend/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-comprehend?include_prereleases&label=changelog)](https://github.com/simonw/sqlite-comprehend/releases)
  [![Tests](https://github.com/simonw/sqlite-comprehend/workflows/Test/badge.svg)](https://github.com/simonw/sqlite-comprehend/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sqlite-comprehend/blob/master/LICENSE)

  Tools for running data in a SQLite database through [AWS Comprehend](https://aws.amazon.com/comprehend/)

  See [sqlite-comprehend: run AWS entity extraction against content in a SQLite database](https://simonwillison.net/2022/Jul/11/sqlite-comprehend/) for background on this project.

  ## Installation

  Install this tool using `pip`:

      pip install sqlite-comprehend

  ## Demo

  You can see examples of tables generated using this command here:

  - [comprehend_entities](https://datasette.simonwillison.net/simonwillisonblog/comprehend_entities) - the extracted entities, classified by type
  - [blog_entry_comprehend_entities](https://datasette.simonwillison.net/simonwillisonblog/blog_entry_comprehend_entities) - a table relating entities to the entries that they appear in
  - [comprehend_entity_types](https://datasette.simonwillison.net/simonwillisonblog/comprehend_entity_types) - a small lookup table of entity types

  ## Configuration

  You will need AWS credentials with the `comprehend:BatchDetectEntities` [IAM permission](https://docs.aws.amazon.com/comprehend/latest/dg/access-control-managing-permissions.html).

  You can configure credentials [using these instructions](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html). You can also save them to a JSON or INI configuration file and pass them to the command using `-a credentials.ini`, or pass them using the `--access-key` and `--secret-key` options.

  ## Entity extraction

  The `sqlite-comprehend entities` command runs entity extraction against every row in the specified table and saves the results to your database.

  Specify the database, the table and one or more columns containing text in that table. The following runs against the `text` column in the `pages` table of the `sfms.db` SQLite database:

      sqlite-comprehend sfms.db pages text

  Results will be written into a `pages_comprehend_entities` table. Change the name of the output table by passing `-o other_table_name`.

  You can run against a subset of rows by adding a `--where` clause:

      sqlite-comprehend sfms.db pages text --where 'id < 10'

  You can also used named parameters in your `--where` clause:

      sqlite-comprehend sfms.db pages text --where 'id < :maxid' -p maxid 10

  Only the first 5,000 characters for each row will be considered. Be sure to review [Comprehend's pricing](https://aws.amazon.com/comprehend/pricing/) - which starts at $0.0001 per hundred characters.

  If your context includes HTML tags, you can strip them out before extracting entities by adding `--strip-tags`:

      sqlite-comprehend sfms.db pages text --strip-tags

  Rows that have been processed are recorded in the `pages_comprehend_entities_done` table. If you run the command more than once it will only process rows that have been newly added.

  You can delete records from that `_done` table to run them again.

  ### sqlite-comprehend entities --help

  <!-- [[[cog
  from click.testing import CliRunner
  from sqlite_comprehend import cli
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["entities", "--help"])
  help = result.output.replace("Usage: cli", "Usage: sqlite-comprehend")
  cog.out(
      "```\n{}\n```".format(help)
  )
  ]]] -->
  ```
  Usage: sqlite-comprehend entities [OPTIONS] DATABASE TABLE COLUMNS...

    Detect entities in columns in a table

    To extract entities from columns text1 and text2 in mytable:

        sqlite-comprehend entities my.db mytable text1 text2

    To run against just a subset of the rows in the table, add:

        --where "id < :max_id" -p max_id 50

    Results will be written to a table called mytable_comprehend_entities

    To specify a different output table, use -o custom_table_name

  Options:
    --where TEXT                WHERE clause to filter table
    -p, --param <TEXT TEXT>...  Named :parameters for SQL query
    -o, --output TEXT           Custom output table
    -r, --reset                 Start from scratch, deleting previous results
    --strip-tags                Strip HTML tags before extracting entities
    --access-key TEXT           AWS access key ID
    --secret-key TEXT           AWS secret access key
    --session-token TEXT        AWS session token
    --endpoint-url TEXT         Custom endpoint URL
    -a, --auth FILENAME         Path to JSON/INI file containing credentials
    --help                      Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## Schema

  Assuming an input table called `pages` the tables created by this tool will have the following schema:

  <!-- [[[cog
  import cog, json
  from sqlite_comprehend import cli
  from unittest.mock import patch
  from click.testing import CliRunner
  import sqlite_utils
  import tempfile, pathlib
  tmpdir = pathlib.Path(tempfile.mkdtemp())
  db_path = str(tmpdir / "data.db")
  db = sqlite_utils.Database(db_path)
  db["pages"].insert_all(
      [
          {
              "id": 1,
              "text": "John Bob",
          },
          {
              "id": 2,
              "text": "Sandra X",
          },
      ],
      pk="id",
  )
  with patch('boto3.client') as client:
      client.return_value.batch_detect_entities.return_value = {
          "ResultList": [
              {
                  "Index": 0,
                  "Entities": [
                      {
                          "Score": 0.8,
                          "Type": "PERSON",
                          "Text": "John Bob",
                          "BeginOffset": 0,
                          "EndOffset": 5,
                      },
                  ],
              },
              {
                  "Index": 1,
                  "Entities": [
                      {
                          "Score": 0.8,
                          "Type": "PERSON",
                          "Text": "Sandra X",
                          "BeginOffset": 0,
                          "EndOffset": 5,
                      },
                  ],
              },
          ],
          "ErrorList": [],
      }
      runner = CliRunner()
      result = runner.invoke(cli.cli, [
          "entities", db_path, "pages", "text"
      ])
  cog.out("```sql\n")
  cog.out(db.schema)
  cog.out("\n```")
  ]]] -->
  ```sql
  CREATE TABLE [pages] (
     [id] INTEGER PRIMARY KEY,
     [text] TEXT
  );
  CREATE TABLE [comprehend_entity_types] (
     [id] INTEGER PRIMARY KEY,
     [value] TEXT
  );
  CREATE TABLE [comprehend_entities] (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT,
     [type] INTEGER REFERENCES [comprehend_entity_types]([id])
  );
  CREATE TABLE [pages_comprehend_entities] (
     [id] INTEGER REFERENCES [pages]([id]),
     [score] FLOAT,
     [entity] INTEGER REFERENCES [comprehend_entities]([id]),
     [begin_offset] INTEGER,
     [end_offset] INTEGER
  );
  CREATE UNIQUE INDEX [idx_comprehend_entity_types_value]
      ON [comprehend_entity_types] ([value]);
  CREATE UNIQUE INDEX [idx_comprehend_entities_type_name]
      ON [comprehend_entities] ([type], [name]);
  CREATE TABLE [pages_comprehend_entities_done] (
     [id] INTEGER PRIMARY KEY REFERENCES [pages]([id])
  );
  ```
  <!-- [[[end]]] -->

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd sqlite-comprehend
      python -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-sqlite-comprehend" class="anchor" aria-hidden="true" href="#user-content-sqlite-comprehend"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>sqlite-comprehend</h1>
  <p dir="auto"><a href="https://pypi.org/project/sqlite-comprehend/" rel="nofollow"><img src="https://camo.githubusercontent.com/469666373ceff91e6d61dd34291b32805e10d84b0499d1d671f5379190a6358f/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d636f6d70726568656e642e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/sqlite-comprehend.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/sqlite-comprehend/releases"><img src="https://camo.githubusercontent.com/3af3baa88364e1d1baa0619b4bb45a483a453734d0b69c5aaf50f3e18e86193d/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f73716c6974652d636f6d70726568656e643f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/sqlite-comprehend?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/sqlite-comprehend/actions?query=workflow%3ATest"><img src="https://github.com/simonw/sqlite-comprehend/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/sqlite-comprehend/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Tools for running data in a SQLite database through <a href="https://aws.amazon.com/comprehend/" rel="nofollow">AWS Comprehend</a></p>
  <p dir="auto">See <a href="https://simonwillison.net/2022/Jul/11/sqlite-comprehend/" rel="nofollow">sqlite-comprehend: run AWS entity extraction against content in a SQLite database</a> for background on this project.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this tool using <code>pip</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install sqlite-comprehend"><pre class="notranslate"><code>pip install sqlite-comprehend
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p dir="auto">You can see examples of tables generated using this command here:</p>
  <ul dir="auto">
  <li><a href="https://datasette.simonwillison.net/simonwillisonblog/comprehend_entities" rel="nofollow">comprehend_entities</a> - the extracted entities, classified by type</li>
  <li><a href="https://datasette.simonwillison.net/simonwillisonblog/blog_entry_comprehend_entities" rel="nofollow">blog_entry_comprehend_entities</a> - a table relating entities to the entries that they appear in</li>
  <li><a href="https://datasette.simonwillison.net/simonwillisonblog/comprehend_entity_types" rel="nofollow">comprehend_entity_types</a> - a small lookup table of entity types</li>
  </ul>
  <h2 dir="auto"><a id="user-content-configuration" class="anchor" aria-hidden="true" href="#user-content-configuration"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Configuration</h2>
  <p dir="auto">You will need AWS credentials with the <code>comprehend:BatchDetectEntities</code> <a href="https://docs.aws.amazon.com/comprehend/latest/dg/access-control-managing-permissions.html" rel="nofollow">IAM permission</a>.</p>
  <p dir="auto">You can configure credentials <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html" rel="nofollow">using these instructions</a>. You can also save them to a JSON or INI configuration file and pass them to the command using <code>-a credentials.ini</code>, or pass them using the <code>--access-key</code> and <code>--secret-key</code> options.</p>
  <h2 dir="auto"><a id="user-content-entity-extraction" class="anchor" aria-hidden="true" href="#user-content-entity-extraction"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Entity extraction</h2>
  <p dir="auto">The <code>sqlite-comprehend entities</code> command runs entity extraction against every row in the specified table and saves the results to your database.</p>
  <p dir="auto">Specify the database, the table and one or more columns containing text in that table. The following runs against the <code>text</code> column in the <code>pages</code> table of the <code>sfms.db</code> SQLite database:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-comprehend sfms.db pages text"><pre class="notranslate"><code>sqlite-comprehend sfms.db pages text
  </code></pre></div>
  <p dir="auto">Results will be written into a <code>pages_comprehend_entities</code> table. Change the name of the output table by passing <code>-o other_table_name</code>.</p>
  <p dir="auto">You can run against a subset of rows by adding a <code>--where</code> clause:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-comprehend sfms.db pages text --where 'id &lt; 10'"><pre class="notranslate"><code>sqlite-comprehend sfms.db pages text --where 'id &lt; 10'
  </code></pre></div>
  <p dir="auto">You can also used named parameters in your <code>--where</code> clause:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-comprehend sfms.db pages text --where 'id &lt; :maxid' -p maxid 10"><pre class="notranslate"><code>sqlite-comprehend sfms.db pages text --where 'id &lt; :maxid' -p maxid 10
  </code></pre></div>
  <p dir="auto">Only the first 5,000 characters for each row will be considered. Be sure to review <a href="https://aws.amazon.com/comprehend/pricing/" rel="nofollow">Comprehend's pricing</a> - which starts at $0.0001 per hundred characters.</p>
  <p dir="auto">If your context includes HTML tags, you can strip them out before extracting entities by adding <code>--strip-tags</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-comprehend sfms.db pages text --strip-tags"><pre class="notranslate"><code>sqlite-comprehend sfms.db pages text --strip-tags
  </code></pre></div>
  <p dir="auto">Rows that have been processed are recorded in the <code>pages_comprehend_entities_done</code> table. If you run the command more than once it will only process rows that have been newly added.</p>
  <p dir="auto">You can delete records from that <code>_done</code> table to run them again.</p>
  <h3 dir="auto"><a id="user-content-sqlite-comprehend-entities---help" class="anchor" aria-hidden="true" href="#user-content-sqlite-comprehend-entities---help"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>sqlite-comprehend entities --help</h3>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: sqlite-comprehend entities [OPTIONS] DATABASE TABLE COLUMNS...

    Detect entities in columns in a table

    To extract entities from columns text1 and text2 in mytable:

        sqlite-comprehend entities my.db mytable text1 text2

    To run against just a subset of the rows in the table, add:

        --where &quot;id &lt; :max_id&quot; -p max_id 50

    Results will be written to a table called mytable_comprehend_entities

    To specify a different output table, use -o custom_table_name

  Options:
    --where TEXT                WHERE clause to filter table
    -p, --param &lt;TEXT TEXT&gt;...  Named :parameters for SQL query
    -o, --output TEXT           Custom output table
    -r, --reset                 Start from scratch, deleting previous results
    --strip-tags                Strip HTML tags before extracting entities
    --access-key TEXT           AWS access key ID
    --secret-key TEXT           AWS secret access key
    --session-token TEXT        AWS session token
    --endpoint-url TEXT         Custom endpoint URL
    -a, --auth FILENAME         Path to JSON/INI file containing credentials
    --help                      Show this message and exit.
  "><pre class="notranslate"><code>Usage: sqlite-comprehend entities [OPTIONS] DATABASE TABLE COLUMNS...

    Detect entities in columns in a table

    To extract entities from columns text1 and text2 in mytable:

        sqlite-comprehend entities my.db mytable text1 text2

    To run against just a subset of the rows in the table, add:

        --where "id &lt; :max_id" -p max_id 50

    Results will be written to a table called mytable_comprehend_entities

    To specify a different output table, use -o custom_table_name

  Options:
    --where TEXT                WHERE clause to filter table
    -p, --param &lt;TEXT TEXT&gt;...  Named :parameters for SQL query
    -o, --output TEXT           Custom output table
    -r, --reset                 Start from scratch, deleting previous results
    --strip-tags                Strip HTML tags before extracting entities
    --access-key TEXT           AWS access key ID
    --secret-key TEXT           AWS secret access key
    --session-token TEXT        AWS session token
    --endpoint-url TEXT         Custom endpoint URL
    -a, --auth FILENAME         Path to JSON/INI file containing credentials
    --help                      Show this message and exit.

  </code></pre></div>

  <h2 dir="auto"><a id="user-content-schema" class="anchor" aria-hidden="true" href="#user-content-schema"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Schema</h2>
  <p dir="auto">Assuming an input table called <code>pages</code> the tables created by this tool will have the following schema:</p>

  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="CREATE TABLE [pages] (
     [id] INTEGER PRIMARY KEY,
     [text] TEXT
  );
  CREATE TABLE [comprehend_entity_types] (
     [id] INTEGER PRIMARY KEY,
     [value] TEXT
  );
  CREATE TABLE [comprehend_entities] (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT,
     [type] INTEGER REFERENCES [comprehend_entity_types]([id])
  );
  CREATE TABLE [pages_comprehend_entities] (
     [id] INTEGER REFERENCES [pages]([id]),
     [score] FLOAT,
     [entity] INTEGER REFERENCES [comprehend_entities]([id]),
     [begin_offset] INTEGER,
     [end_offset] INTEGER
  );
  CREATE UNIQUE INDEX [idx_comprehend_entity_types_value]
      ON [comprehend_entity_types] ([value]);
  CREATE UNIQUE INDEX [idx_comprehend_entities_type_name]
      ON [comprehend_entities] ([type], [name]);
  CREATE TABLE [pages_comprehend_entities_done] (
     [id] INTEGER PRIMARY KEY REFERENCES [pages]([id])
  );"><pre>CREATE TABLE [pages] (
     [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
     [<span class="pl-k">text</span>] <span class="pl-k">TEXT</span>
  );
  CREATE TABLE [comprehend_entity_types] (
     [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
     [value] <span class="pl-k">TEXT</span>
  );
  CREATE TABLE [comprehend_entities] (
     [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
     [name] <span class="pl-k">TEXT</span>,
     [type] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [comprehend_entity_types]([id])
  );
  CREATE TABLE [pages_comprehend_entities] (
     [id] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [pages]([id]),
     [score] FLOAT,
     [entity] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [comprehend_entities]([id]),
     [begin_offset] <span class="pl-k">INTEGER</span>,
     [end_offset] <span class="pl-k">INTEGER</span>
  );
  CREATE UNIQUE INDEX [idx_comprehend_entity_types_value]
      <span class="pl-k">ON</span> [comprehend_entity_types] ([value]);
  CREATE UNIQUE INDEX [idx_comprehend_entities_type_name]
      <span class="pl-k">ON</span> [comprehend_entities] ([type], [name]);
  CREATE TABLE [pages_comprehend_entities_done] (
     [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span> <span class="pl-k">REFERENCES</span> [pages]([id])
  );</pre></div>

  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd sqlite-comprehend
  python -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd sqlite-comprehend
  python -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOH2kMZA
name: datasette-mp3-audio
full_name: simonw/datasette-mp3-audio
html_url: https://github.com/simonw/datasette-mp3-audio
description: Turn .mp3 URLs into an audio player in the Datasette interface
created_at: 2022-08-20T16:20:06Z
updated_at: 2022-08-21T04:49:01Z
pushed_at: 2022-08-21T06:43:00Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-mp3-audio

  [![PyPI](https://img.shields.io/pypi/v/datasette-mp3-audio.svg)](https://pypi.org/project/datasette-mp3-audio/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-mp3-audio?include_prereleases&label=changelog)](https://github.com/simonw/datasette-mp3-audio/releases)
  [![Tests](https://github.com/simonw/datasette-mp3-audio/workflows/Test/badge.svg)](https://github.com/simonw/datasette-mp3-audio/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-mp3-audio/blob/main/LICENSE)

  Turn .mp3 URLs into an audio player in the Datasette interface

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-mp3-audio

  ## Demo

  Try this plugin at [https://scotrail.datasette.io/scotrail/announcements](https://scotrail.datasette.io/scotrail/announcements)

  The demo uses ScotRail train announcements from [matteason/scotrail-announcements-june-2022](https://github.com/matteason/scotrail-announcements-june-2022).

  ## Usage

  Once installed, any cells with a value that ends in `.mp3` and starts with either `http://` or `/` or `https://` will be turned into an embedded HTML audio element like this:

  ```html
  <audio controls src="... value ..."><a href="...">Download MP3</a></audio>
  ```

  A "Play X MP3s on this page" button will be added to athe top of any table page listing more than one MP3.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-mp3-audio
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-mp3-audio" class="anchor" aria-hidden="true" href="#user-content-datasette-mp3-audio"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-mp3-audio</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-mp3-audio/" rel="nofollow"><img src="https://camo.githubusercontent.com/be406dad915277ef158f3ac64acd7eaa1935dadb232dba65be9ac0d88488aaf6/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6d70332d617564696f2e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-mp3-audio.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-mp3-audio/releases"><img src="https://camo.githubusercontent.com/7efb947563daf991f6c300a68a7a14f92ec115507ae8e4771a32c2199dd0260d/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6d70332d617564696f3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-mp3-audio?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-mp3-audio/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-mp3-audio/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-mp3-audio/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Turn .mp3 URLs into an audio player in the Datasette interface</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-mp3-audio"><pre class="notranslate"><code>datasette install datasette-mp3-audio
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p dir="auto">Try this plugin at <a href="https://scotrail.datasette.io/scotrail/announcements" rel="nofollow">https://scotrail.datasette.io/scotrail/announcements</a></p>
  <p dir="auto">The demo uses ScotRail train announcements from <a href="https://github.com/matteason/scotrail-announcements-june-2022">matteason/scotrail-announcements-june-2022</a>.</p>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Once installed, any cells with a value that ends in <code>.mp3</code> and starts with either <code>http://</code> or <code>/</code> or <code>https://</code> will be turned into an embedded HTML audio element like this:</p>
  <div class="highlight highlight-text-html-basic notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;audio controls src=&quot;... value ...&quot;&gt;&lt;a href=&quot;...&quot;&gt;Download MP3&lt;/a&gt;&lt;/audio&gt;"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">audio</span> <span class="pl-c1">controls</span> <span class="pl-c1">src</span>="<span class="pl-s">... value ...</span>"<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;</span><span class="pl-ent">a</span> <span class="pl-c1">href</span>="<span class="pl-s">...</span>"<span class="pl-kos">&gt;</span>Download MP3<span class="pl-kos">&lt;/</span><span class="pl-ent">a</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">audio</span><span class="pl-kos">&gt;</span></pre></div>
  <p dir="auto">A "Play X MP3s on this page" button will be added to athe top of any table page listing more than one MP3.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-mp3-audio
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-mp3-audio
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>

template_repository:
[cell repeats an earlier one in this file: 6347 characters]


node_id: R_kgDOH4H41Q
name: datasette-multiline-links
full_name: simonw/datasette-multiline-links
html_url: https://github.com/simonw/datasette-multiline-links
description: Make multiple newline separated URLs clickable in Datasette
created_at: 2022-08-24T22:16:30Z
updated_at: 2022-08-24T22:16:52Z
pushed_at: 2022-08-24T22:56:16Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-multiline-links

  [![PyPI](https://img.shields.io/pypi/v/datasette-multiline-links.svg)](https://pypi.org/project/datasette-multiline-links/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-multiline-links?include_prereleases&label=changelog)](https://github.com/simonw/datasette-multiline-links/releases)
  [![Tests](https://github.com/simonw/datasette-multiline-links/workflows/Test/badge.svg)](https://github.com/simonw/datasette-multiline-links/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-multiline-links/blob/main/LICENSE)

  Make multiple newline separated URLs clickable in Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-multiline-links

  ## Demo

  Try this plugin out against a [Google Sheets spreadsheet](https://docs.google.com/spreadsheets/d/1wZhPLMCHKJvwOkP4juclhjFgqIY8fQFMemwKL2c64vk) of previously featured datasets from [Data is Plural](https://www.data-is-plural.com/) using [Datasette Lite](https://lite.datasette.io/) here:

  * <a href="https://lite.datasette.io/?install=datasette-multiline-links&csv=https://docs.google.com/spreadsheets/d/1wZhPLMCHKJvwOkP4juclhjFgqIY8fQFMemwKL2c64vk/export?format=csv#/data?sql=select+edition%2C+headline%2C+text%2C+links%2C+hattips+from+export+where%0Atext+like+'%25'+||+%3Aq+||+'%25'+or+headline+like+'%25'+||+%3Aq+||+'%25'+order+by+edition+desc&q=loans">Demo this plugin in Datasette Lite</a>

  ## Usage

  Once installed, if a cell has contents like this:
  ```
  https://example.com
  Not a link
  https://google.com
  ```
  It will be rendered as:
  ```html
  <a href="https://example.com">https://example.com</a>
  Not a link
  <a href="https://google.com">https://google.com</a>
  ```
  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-multiline-links
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-multiline-links" class="anchor" aria-hidden="true" href="#user-content-datasette-multiline-links"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-multiline-links</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-multiline-links/" rel="nofollow"><img src="https://camo.githubusercontent.com/059c99528ca3f757a565a1430c1310891efcace69d475fbdb914d16f879f8592/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6d756c74696c696e652d6c696e6b732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-multiline-links.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-multiline-links/releases"><img src="https://camo.githubusercontent.com/5c3c310bb49e90c51aa56403feccb12f948067586b18dc3f1e32833756259b7e/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6d756c74696c696e652d6c696e6b733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-multiline-links?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-multiline-links/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-multiline-links/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-multiline-links/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Make multiple newline separated URLs clickable in Datasette</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-multiline-links"><pre class="notranslate"><code>datasette install datasette-multiline-links
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p dir="auto">Try this plugin out against a <a href="https://docs.google.com/spreadsheets/d/1wZhPLMCHKJvwOkP4juclhjFgqIY8fQFMemwKL2c64vk" rel="nofollow">Google Sheets spreadsheet</a> of previously featured datasets from <a href="https://www.data-is-plural.com/" rel="nofollow">Data is Plural</a> using <a href="https://lite.datasette.io/" rel="nofollow">Datasette Lite</a> here:</p>
  <ul dir="auto">
  <li><a href="https://lite.datasette.io/?install=datasette-multiline-links&amp;csv=https://docs.google.com/spreadsheets/d/1wZhPLMCHKJvwOkP4juclhjFgqIY8fQFMemwKL2c64vk/export?format=csv#/data?sql=select+edition%2C+headline%2C+text%2C+links%2C+hattips+from+export+where%0Atext+like+'%25'+||+%3Aq+||+'%25'+or+headline+like+'%25'+||+%3Aq+||+'%25'+order+by+edition+desc&amp;q=loans" rel="nofollow">Demo this plugin in Datasette Lite</a></li>
  </ul>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Once installed, if a cell has contents like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="https://example.com
  Not a link
  https://google.com"><pre class="notranslate"><code>https://example.com
  Not a link
  https://google.com
  </code></pre></div>
  <p dir="auto">It will be rendered as:</p>
  <div class="highlight highlight-text-html-basic notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;a href=&quot;https://example.com&quot;&gt;https://example.com&lt;/a&gt;
  Not a link
  &lt;a href=&quot;https://google.com&quot;&gt;https://google.com&lt;/a&gt;"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">a</span> <span class="pl-c1">href</span>="<span class="pl-s">https://example.com</span>"<span class="pl-kos">&gt;</span>https://example.com<span class="pl-kos">&lt;/</span><span class="pl-ent">a</span><span class="pl-kos">&gt;</span>
  Not a link
  <span class="pl-kos">&lt;</span><span class="pl-ent">a</span> <span class="pl-c1">href</span>="<span class="pl-s">https://google.com</span>"<span class="pl-kos">&gt;</span>https://google.com<span class="pl-kos">&lt;/</span><span class="pl-ent">a</span><span class="pl-kos">&gt;</span></pre></div>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-multiline-links
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-multiline-links
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>

template_repository:
[cell repeats an earlier one in this file: 6347 characters]


node_id: R_kgDOH50_7g
name: datasette-sitemap
full_name: simonw/datasette-sitemap
html_url: https://github.com/simonw/datasette-sitemap
description: Generate sitemap.xml for Datasette sites
created_at: 2022-08-29T21:22:46Z
updated_at: 2022-08-29T23:04:23Z
pushed_at: 2022-08-30T17:59:53Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-sitemap

  [![PyPI](https://img.shields.io/pypi/v/datasette-sitemap.svg)](https://pypi.org/project/datasette-sitemap/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-sitemap?include_prereleases&label=changelog)](https://github.com/simonw/datasette-sitemap/releases)
  [![Tests](https://github.com/simonw/datasette-sitemap/workflows/Test/badge.svg)](https://github.com/simonw/datasette-sitemap/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-sitemap/blob/main/LICENSE)

  Generate sitemap.xml for Datasette sites

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-sitemap

  ## Demo

  This plugin is used for the sitemap on [til.simonwillison.net](https://til.simonwillison.net/):

  - https://til.simonwillison.net/sitemap.xml

  Here's [the configuration](https://github.com/simonw/til/blob/d4f67743a90a67100b46145986b2dec6f8d96583/metadata.yaml#L14-L16) used for that sitemap.

  ## Usage

  Once configured, this plugin adds a sitemap at `/sitemap.xml` with a list of URLs.

  This list is defined using a SQL query in `metadata.json` (or `.yml`) that looks like this:

  ```json
  {
    "plugins": {
      "datasette-sitemap": {
        "query": "select '/' || id as path from my_table"
      }
    }
  }
  ```

  Using `metadata.yml` allows for multi-line SQL queries which can be easier to maintain:

  ```yaml
  plugins:
    datasette-sitemap:
      query: |
        select
          '/' || id as path
        from
          my_table
  ```
  The SQL query must return a column called `path`. The values in this column must begin with a `/`. They will be used to generate a sitemap that looks like this:

  ```xml
  <?xml version="1.0" encoding="UTF-8"?>
  <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <url><loc>https://example.com/1</loc></url>
    <url><loc>https://example.com/2</loc></url>
  </urlset>
  ```
  You can use ``UNION`` in your SQL query to combine results from multiple tables, or include literal paths that you want to include in the index:

  ```sql
  select
    '/data/table1/' || id as path
    from table1
  union
  select
    '/data/table2/' || id as path
    from table2
  union
  select
    '/about' as path
  ```
  If your Datasette instance has multiple databases you can configure the database to query using the `database` configuration property.

  By default the domain name for the genearted URLs in the sitemap will be detected from the incoming request.

  You can set `base_url` instead to override this. This should not include a trailing slash.

  This example shows both of those settings, running the query against the `content` database and setting a custom base URL:

  ```yaml
  plugins:
    datasette-sitemap:
      query: |
        select '/plugins/' || name as path from plugins
        union
        select '/tools/' || name as path from tools
        union
        select '/news' as path
      database: content
      base_url: https://datasette.io
  ```
  [Try that query](https://datasette.io/content?sql=select+%27%2Fplugins%2F%27+||+name+as+path+from+plugins%0D%0Aunion%0D%0Aselect+%27%2Ftools%2F%27+||+name+as+path+from+tools%0D%0Aunion%0D%0Aselect+%27%2Fnews%27+as+path%0D%0A).

  ## robots.txt

  This plugin adds a `robots.txt` file pointing to the sitemap:

  ```
  Sitemap: http://example.com/sitemap.xml
  ```

  You can take full control of the sitemap by installing and configuring the [datasette-block-robots](https://datasette.io/plugins/datasette-block-robots) plugin.

  This plugin will add the `Sitemap:` line even if you are using `datasette-block-robots` for the rest of your `robots.txt` file.

  ## Adding paths to the sitemap from other plugins

  This plugin adds a new [plugin hook](https://docs.datasette.io/en/stable/plugin_hooks.html) to Datasete called `sitemap_extra_paths()` which can be used by other plugins to add their own additional lines to the `sitemap.xml` file.

  The hook accepts these optional parameters:

  - `datasette`: The current [Datasette instance](https://docs.datasette.io/en/stable/internals.html#datasette-class). You can use this to execute SQL queries or read plugin configuration settings.
  - `request`: The [Request object](https://docs.datasette.io/en/stable/internals.html#request-object) representing the incoming request to `/sitemap.xml`.

  The hook should return a list of strings, each representing a path to be added to the sitemap. Each path must begin with a `/`.

  It can also return an `async def` function, which will be awaited and used to generate a list of lines. Use this option if you need to make `await` calls inside you hook implementation.

  This example uses the hook to add two extra paths, one of which came from a SQL query:

  ```python
  from datasette import hookimpl

  @hookimpl
  def sitemap_extra_paths(datasette):
      async def inner():
          db = datasette.get_database()
          path_from_db = (await db.execute("select '/example'")).single_value()
          return ["/about", path_from_db]
      return inner
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-sitemap
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-sitemap" class="anchor" aria-hidden="true" href="#user-content-datasette-sitemap"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-sitemap</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-sitemap/" rel="nofollow"><img src="https://camo.githubusercontent.com/6771365acdee2a47414d423ce45cba60d99dbc6a20fa98637c0617476f8053da/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d736974656d61702e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-sitemap.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-sitemap/releases"><img src="https://camo.githubusercontent.com/4ceed1473baea92d47265f3ec83d36d6218a335777f158e5785a7033a2affba6/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d736974656d61703f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-sitemap?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-sitemap/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-sitemap/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-sitemap/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Generate sitemap.xml for Datasette sites</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-sitemap"><pre class="notranslate"><code>datasette install datasette-sitemap
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p dir="auto">This plugin is used for the sitemap on <a href="https://til.simonwillison.net/" rel="nofollow">til.simonwillison.net</a>:</p>
  <ul dir="auto">
  <li><a href="https://til.simonwillison.net/sitemap.xml" rel="nofollow">https://til.simonwillison.net/sitemap.xml</a></li>
  </ul>
  <p dir="auto">Here's <a href="https://github.com/simonw/til/blob/d4f67743a90a67100b46145986b2dec6f8d96583/metadata.yaml#L14-L16">the configuration</a> used for that sitemap.</p>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Once configured, this plugin adds a sitemap at <code>/sitemap.xml</code> with a list of URLs.</p>
  <p dir="auto">This list is defined using a SQL query in <code>metadata.json</code> (or <code>.yml</code>) that looks like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    &quot;plugins&quot;: {
      &quot;datasette-sitemap&quot;: {
        &quot;query&quot;: &quot;select '/' || id as path from my_table&quot;
      }
    }
  }"><pre>{
    <span class="pl-ent">"plugins"</span>: {
      <span class="pl-ent">"datasette-sitemap"</span>: {
        <span class="pl-ent">"query"</span>: <span class="pl-s"><span class="pl-pds">"</span>select '/' || id as path from my_table<span class="pl-pds">"</span></span>
      }
    }
  }</pre></div>
  <p dir="auto">Using <code>metadata.yml</code> allows for multi-line SQL queries which can be easier to maintain:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-sitemap:
      query: |
        select
          '/' || id as path
        from
          my_table"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-sitemap</span>:
      <span class="pl-ent">query</span>: <span class="pl-s">|</span>
  <span class="pl-s">      select</span>
  <span class="pl-s">        '/' || id as path</span>
  <span class="pl-s">      from</span>
  <span class="pl-s">        my_table</span></pre></div>
  <p dir="auto">The SQL query must return a column called <code>path</code>. The values in this column must begin with a <code>/</code>. They will be used to generate a sitemap that looks like this:</p>
  <div class="highlight highlight-text-xml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;
  &lt;urlset xmlns=&quot;http://www.sitemaps.org/schemas/sitemap/0.9&quot;&gt;
    &lt;url&gt;&lt;loc&gt;https://example.com/1&lt;/loc&gt;&lt;/url&gt;
    &lt;url&gt;&lt;loc&gt;https://example.com/2&lt;/loc&gt;&lt;/url&gt;
  &lt;/urlset&gt;"><pre>&lt;?<span class="pl-ent">xml</span><span class="pl-e"> version</span>=<span class="pl-s"><span class="pl-pds">"</span>1.0<span class="pl-pds">"</span></span><span class="pl-e"> encoding</span>=<span class="pl-s"><span class="pl-pds">"</span>UTF-8<span class="pl-pds">"</span></span>?&gt;
  &lt;<span class="pl-ent">urlset</span> <span class="pl-e">xmlns</span>=<span class="pl-s"><span class="pl-pds">"</span>http://www.sitemaps.org/schemas/sitemap/0.9<span class="pl-pds">"</span></span>&gt;
    &lt;<span class="pl-ent">url</span>&gt;&lt;<span class="pl-ent">loc</span>&gt;https://example.com/1&lt;/<span class="pl-ent">loc</span>&gt;&lt;/<span class="pl-ent">url</span>&gt;
    &lt;<span class="pl-ent">url</span>&gt;&lt;<span class="pl-ent">loc</span>&gt;https://example.com/2&lt;/<span class="pl-ent">loc</span>&gt;&lt;/<span class="pl-ent">url</span>&gt;
  &lt;/<span class="pl-ent">urlset</span>&gt;</pre></div>
  <p dir="auto">You can use <code>UNION</code> in your SQL query to combine results from multiple tables, or include literal paths that you want to include in the index:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="select
    '/data/table1/' || id as path
    from table1
  union
  select
    '/data/table2/' || id as path
    from table2
  union
  select
    '/about' as path"><pre><span class="pl-k">select</span>
    <span class="pl-s"><span class="pl-pds">'</span>/data/table1/<span class="pl-pds">'</span></span> <span class="pl-k">||</span> id <span class="pl-k">as</span> <span class="pl-k">path</span>
    <span class="pl-k">from</span> table1
  <span class="pl-k">union</span>
  <span class="pl-k">select</span>
    <span class="pl-s"><span class="pl-pds">'</span>/data/table2/<span class="pl-pds">'</span></span> <span class="pl-k">||</span> id <span class="pl-k">as</span> <span class="pl-k">path</span>
    <span class="pl-k">from</span> table2
  <span class="pl-k">union</span>
  <span class="pl-k">select</span>
    <span class="pl-s"><span class="pl-pds">'</span>/about<span class="pl-pds">'</span></span> <span class="pl-k">as</span> <span class="pl-k">path</span></pre></div>
  <p dir="auto">If your Datasette instance has multiple databases you can configure the database to query using the <code>database</code> configuration property.</p>
  <p dir="auto">By default the domain name for the genearted URLs in the sitemap will be detected from the incoming request.</p>
  <p dir="auto">You can set <code>base_url</code> instead to override this. This should not include a trailing slash.</p>
  <p dir="auto">This example shows both of those settings, running the query against the <code>content</code> database and setting a custom base URL:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-sitemap:
      query: |
        select '/plugins/' || name as path from plugins
        union
        select '/tools/' || name as path from tools
        union
        select '/news' as path
      database: content
      base_url: https://datasette.io"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-sitemap</span>:
      <span class="pl-ent">query</span>: <span class="pl-s">|</span>
  <span class="pl-s">      select '/plugins/' || name as path from plugins</span>
  <span class="pl-s">      union</span>
  <span class="pl-s">      select '/tools/' || name as path from tools</span>
  <span class="pl-s">      union</span>
  <span class="pl-s">      select '/news' as path</span>
  <span class="pl-s"></span>    <span class="pl-ent">database</span>: <span class="pl-s">content</span>
      <span class="pl-ent">base_url</span>: <span class="pl-s">https://datasette.io</span></pre></div>
  <p dir="auto"><a href="https://datasette.io/content?sql=select+%27%2Fplugins%2F%27+%7C%7C+name+as+path+from+plugins%0D%0Aunion%0D%0Aselect+%27%2Ftools%2F%27+%7C%7C+name+as+path+from+tools%0D%0Aunion%0D%0Aselect+%27%2Fnews%27+as+path%0D%0A" rel="nofollow">Try that query</a>.</p>
  <h2 dir="auto"><a id="user-content-robotstxt" class="anchor" aria-hidden="true" href="#user-content-robotstxt"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>robots.txt</h2>
  <p dir="auto">This plugin adds a <code>robots.txt</code> file pointing to the sitemap:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Sitemap: http://example.com/sitemap.xml"><pre class="notranslate"><code>Sitemap: http://example.com/sitemap.xml
  </code></pre></div>
  <p dir="auto">You can take full control of the sitemap by installing and configuring the <a href="https://datasette.io/plugins/datasette-block-robots" rel="nofollow">datasette-block-robots</a> plugin.</p>
  <p dir="auto">This plugin will add the <code>Sitemap:</code> line even if you are using <code>datasette-block-robots</code> for the rest of your <code>robots.txt</code> file.</p>
  <h2 dir="auto"><a id="user-content-adding-paths-to-the-sitemap-from-other-plugins" class="anchor" aria-hidden="true" href="#user-content-adding-paths-to-the-sitemap-from-other-plugins"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Adding paths to the sitemap from other plugins</h2>
  <p dir="auto">This plugin adds a new <a href="https://docs.datasette.io/en/stable/plugin_hooks.html" rel="nofollow">plugin hook</a> to Datasete called <code>sitemap_extra_paths()</code> which can be used by other plugins to add their own additional lines to the <code>sitemap.xml</code> file.</p>
  <p dir="auto">The hook accepts these optional parameters:</p>
  <ul dir="auto">
  <li><code>datasette</code>: The current <a href="https://docs.datasette.io/en/stable/internals.html#datasette-class" rel="nofollow">Datasette instance</a>. You can use this to execute SQL queries or read plugin configuration settings.</li>
  <li><code>request</code>: The <a href="https://docs.datasette.io/en/stable/internals.html#request-object" rel="nofollow">Request object</a> representing the incoming request to <code>/sitemap.xml</code>.</li>
  </ul>
  <p dir="auto">The hook should return a list of strings, each representing a path to be added to the sitemap. Each path must begin with a <code>/</code>.</p>
  <p dir="auto">It can also return an <code>async def</code> function, which will be awaited and used to generate a list of lines. Use this option if you need to make <code>await</code> calls inside you hook implementation.</p>
  <p dir="auto">This example uses the hook to add two extra paths, one of which came from a SQL query:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from datasette import hookimpl

  @hookimpl
  def sitemap_extra_paths(datasette):
      async def inner():
          db = datasette.get_database()
          path_from_db = (await db.execute(&quot;select '/example'&quot;)).single_value()
          return [&quot;/about&quot;, path_from_db]
      return inner"><pre><span class="pl-k">from</span> <span class="pl-s1">datasette</span> <span class="pl-k">import</span> <span class="pl-s1">hookimpl</span>

  <span class="pl-en">@<span class="pl-s1">hookimpl</span></span>
  <span class="pl-k">def</span> <span class="pl-en">sitemap_extra_paths</span>(<span class="pl-s1">datasette</span>):
      <span class="pl-k">async</span> <span class="pl-k">def</span> <span class="pl-en">inner</span>():
          <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-s1">datasette</span>.<span class="pl-en">get_database</span>()
          <span class="pl-s1">path_from_db</span> <span class="pl-c1">=</span> (<span class="pl-k">await</span> <span class="pl-s1">db</span>.<span class="pl-en">execute</span>(<span class="pl-s">"select '/example'"</span>)).<span class="pl-en">single_value</span>()
          <span class="pl-k">return</span> [<span class="pl-s">"/about"</span>, <span class="pl-s1">path_from_db</span>]
      <span class="pl-k">return</span> <span class="pl-s1">inner</span></pre></div>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-sitemap
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-sitemap
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>

template_repository:
  {"id": 400878073, "node_id": "MDEwOlJlcG9zaXRvcnk0MDA4NzgwNzM=", "name": "datasette-plugin-template-repository", "full_name": "simonw/datasette-plugin-template-repository", "private": false, "owner": {"login": "simonw", "id": 9599, "node_id": "MDQ6VXNlcjk1OTk=", "avatar_url": "https://avatars.githubusercontent.com/u/9599?v=4", "gravatar_id": "", "url": "https://api.github.com/users/simonw", "html_url": "https://github.com/simonw", "followers_url": "https://api.github.com/users/simonw/followers", "following_url": "https://api.github.com/users/simonw/following{/other_user}", "gists_url": "https://api.github.com/users/simonw/gists{/gist_id}", "starred_url": "https://api.github.com/users/simonw/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/simonw/subscriptions", "organizations_url": "https://api.github.com/users/simonw/orgs", "repos_url": "https://api.github.com/users/simonw/repos", "events_url": "https://api.github.com/users/simonw/events{/privacy}", "received_events_url": "https://api.github.com/users/simonw/received_events", "type": "User", "site_admin": false}, "html_url": "https://github.com/simonw/datasette-plugin-template-repository", "description": "GitHub template repository for creating new Datasette plugins, using the simonw/datasette-plugin cookiecutter template", "fork": false, "url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository", "forks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/forks", "keys_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/keys{/key_id}", "collaborators_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/collaborators{/collaborator}", "teams_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/teams", "hooks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/hooks", "issue_events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/events{/number}", "events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/events", "assignees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/assignees{/user}", "branches_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/branches{/branch}", "tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/tags", "blobs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/blobs{/sha}", "git_tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/tags{/sha}", "git_refs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/refs{/sha}", "trees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/trees{/sha}", "statuses_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/statuses/{sha}", "languages_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/languages", "stargazers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/stargazers", "contributors_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contributors", "subscribers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscribers", "subscription_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscription", "commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/commits{/sha}", "git_commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/commits{/sha}", "comments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/comments{/number}", "issue_comment_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/comments{/number}", "contents_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contents/{+path}", "compare_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/compare/{base}...{head}", "merges_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/merges", "archive_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/{archive_format}{/ref}", "downloads_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/downloads", "issues_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues{/number}", "pulls_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/pulls{/number}", "milestones_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/milestones{/number}", "notifications_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/notifications{?since,all,participating}", "labels_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/labels{/name}", "releases_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/releases{/id}", "deployments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/deployments", "created_at": "2021-08-28T19:50:28Z", "updated_at": "2022-08-27T17:22:27Z", "pushed_at": "2022-03-16T23:42:16Z", "git_url": "git://github.com/simonw/datasette-plugin-template-repository.git", "ssh_url": "git@github.com:simonw/datasette-plugin-template-repository.git", "clone_url": "https://github.com/simonw/datasette-plugin-template-repository.git", "svn_url": "https://github.com/simonw/datasette-plugin-template-repository", "homepage": "", "size": 9, "stargazers_count": 16, "watchers_count": 16, "language": null, "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": true, "has_pages": false, "forks_count": 0, "mirror_url": null, "archived": false, "disabled": false, "open_issues_count": 0, "license": null, "allow_forking": true, "is_template": true, "web_commit_signoff_required": false, "topics": [], "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 16, "default_branch": "main", "permissions": {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}, "temp_clone_token": ""}


node_id: R_kgDOH7vZQw
name: datasette-render-image-tags
full_name: simonw/datasette-render-image-tags
html_url: https://github.com/simonw/datasette-render-image-tags
description: Turn any URLs ending in .jpg/.png/.gif into img tags with width 200
created_at: 2022-09-04T00:36:44Z
updated_at: 2022-09-04T00:37:13Z
pushed_at: 2022-09-04T00:48:32Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-render-image-tags

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-image-tags.svg)](https://pypi.org/project/datasette-render-image-tags/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-render-image-tags?include_prereleases&label=changelog)](https://github.com/simonw/datasette-render-image-tags/releases)
  [![Tests](https://github.com/simonw/datasette-render-image-tags/workflows/Test/badge.svg)](https://github.com/simonw/datasette-render-image-tags/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-image-tags/blob/main/LICENSE)

  Turn any URLs ending in .jpg/.png/.gif into img tags with width 200

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-render-image-tags

  ## Usage

  Once installed, any cells contaning a URL that ends with `.png` or `.jpg` or `.jpeg` or `.gif` will be rendered using an image tag, with a width of 200px.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-render-image-tags
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-render-image-tags" class="anchor" aria-hidden="true" href="#user-content-datasette-render-image-tags"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-render-image-tags</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-render-image-tags/" rel="nofollow"><img src="https://camo.githubusercontent.com/57f7abc8fd4b0dd9d66333a0711ecae79dd3fec5e13057e175624b6ceded2cf3/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d72656e6465722d696d6167652d746167732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-render-image-tags.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-render-image-tags/releases"><img src="https://camo.githubusercontent.com/41d793901ad12364d0a8e2f28a0e5f972129ebd4816b0db29500eb2eaf2d6f9d/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d72656e6465722d696d6167652d746167733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-render-image-tags?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-render-image-tags/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-render-image-tags/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-render-image-tags/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Turn any URLs ending in .jpg/.png/.gif into img tags with width 200</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-render-image-tags"><pre class="notranslate"><code>datasette install datasette-render-image-tags
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Once installed, any cells contaning a URL that ends with <code>.png</code> or <code>.jpg</code> or <code>.jpeg</code> or <code>.gif</code> will be rendered using an image tag, with a width of 200px.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-render-image-tags
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-render-image-tags
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>

template_repository:
  {"id": 400878073, "node_id": "MDEwOlJlcG9zaXRvcnk0MDA4NzgwNzM=", "name": "datasette-plugin-template-repository", "full_name": "simonw/datasette-plugin-template-repository", "private": false, "owner": {"login": "simonw", "id": 9599, "node_id": "MDQ6VXNlcjk1OTk=", "avatar_url": "https://avatars.githubusercontent.com/u/9599?v=4", "gravatar_id": "", "url": "https://api.github.com/users/simonw", "html_url": "https://github.com/simonw", "followers_url": "https://api.github.com/users/simonw/followers", "following_url": "https://api.github.com/users/simonw/following{/other_user}", "gists_url": "https://api.github.com/users/simonw/gists{/gist_id}", "starred_url": "https://api.github.com/users/simonw/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/simonw/subscriptions", "organizations_url": "https://api.github.com/users/simonw/orgs", "repos_url": "https://api.github.com/users/simonw/repos", "events_url": "https://api.github.com/users/simonw/events{/privacy}", "received_events_url": "https://api.github.com/users/simonw/received_events", "type": "User", "site_admin": false}, "html_url": "https://github.com/simonw/datasette-plugin-template-repository", "description": "GitHub template repository for creating new Datasette plugins, using the simonw/datasette-plugin cookiecutter template", "fork": false, "url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository", "forks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/forks", "keys_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/keys{/key_id}", "collaborators_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/collaborators{/collaborator}", "teams_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/teams", "hooks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/hooks", "issue_events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/events{/number}", "events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/events", "assignees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/assignees{/user}", "branches_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/branches{/branch}", "tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/tags", "blobs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/blobs{/sha}", "git_tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/tags{/sha}", "git_refs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/refs{/sha}", "trees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/trees{/sha}", "statuses_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/statuses/{sha}", "languages_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/languages", "stargazers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/stargazers", "contributors_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contributors", "subscribers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscribers", "subscription_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscription", "commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/commits{/sha}", "git_commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/commits{/sha}", "comments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/comments{/number}", "issue_comment_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/comments{/number}", "contents_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contents/{+path}", "compare_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/compare/{base}...{head}", "merges_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/merges", "archive_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/{archive_format}{/ref}", "downloads_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/downloads", "issues_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues{/number}", "pulls_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/pulls{/number}", "milestones_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/milestones{/number}", "notifications_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/notifications{?since,all,participating}", "labels_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/labels{/name}", "releases_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/releases{/id}", "deployments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/deployments", "created_at": "2021-08-28T19:50:28Z", "updated_at": "2022-08-27T17:22:27Z", "pushed_at": "2022-03-16T23:42:16Z", "git_url": "git://github.com/simonw/datasette-plugin-template-repository.git", "ssh_url": "git@github.com:simonw/datasette-plugin-template-repository.git", "clone_url": "https://github.com/simonw/datasette-plugin-template-repository.git", "svn_url": "https://github.com/simonw/datasette-plugin-template-repository", "homepage": "", "size": 9, "stargazers_count": 16, "watchers_count": 16, "language": null, "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": true, "has_pages": false, "forks_count": 0, "mirror_url": null, "archived": false, "disabled": false, "open_issues_count": 0, "license": null, "allow_forking": true, "is_template": true, "web_commit_signoff_required": false, "topics": [], "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 16, "default_branch": "main", "permissions": {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}, "temp_clone_token": ""}


node_id: R_kgDOH99rFw
name: datasette-sandstorm-support
full_name: simonw/datasette-sandstorm-support
html_url: https://github.com/simonw/datasette-sandstorm-support
description: Authentication and permissions for Datasette on Sandstorm
created_at: 2022-09-09T17:11:04Z
updated_at: 2022-09-09T17:36:46Z
pushed_at: 2022-09-16T17:14:12Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-sandstorm-support

  [![PyPI](https://img.shields.io/pypi/v/datasette-sandstorm-support.svg)](https://pypi.org/project/datasette-sandstorm-support/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-sandstorm-support?include_prereleases&label=changelog)](https://github.com/simonw/datasette-sandstorm-support/releases)
  [![Tests](https://github.com/simonw/datasette-sandstorm-support/workflows/Test/badge.svg)](https://github.com/simonw/datasette-sandstorm-support/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-sandstorm-support/blob/main/LICENSE)

  Authentication and permissions for Datasette on Sandstorm

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-sandstorm-support

  ## Usage

  This plugin is part of [datasette-sandstorm](https://github.com/ocdtrekkie/datasette-sandstorm).

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-sandstorm-support
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-sandstorm-support" class="anchor" aria-hidden="true" href="#user-content-datasette-sandstorm-support"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-sandstorm-support</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-sandstorm-support/" rel="nofollow"><img src="https://camo.githubusercontent.com/6e32b5bcf5e4e48dcb4a2ea4c18cd41da494afb1c48d2458ff9b01f0ec485495/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73616e6473746f726d2d737570706f72742e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-sandstorm-support.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-sandstorm-support/releases"><img src="https://camo.githubusercontent.com/ed98d1cc299400313c2b6f5f65227b2c41429724e4b09b44581cd0653ce654c1/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d73616e6473746f726d2d737570706f72743f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-sandstorm-support?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-sandstorm-support/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-sandstorm-support/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-sandstorm-support/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Authentication and permissions for Datasette on Sandstorm</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-sandstorm-support"><pre class="notranslate"><code>datasette install datasette-sandstorm-support
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">This plugin is part of <a href="https://github.com/ocdtrekkie/datasette-sandstorm">datasette-sandstorm</a>.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-sandstorm-support
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-sandstorm-support
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>

template_repository:
  {"id": 400878073, "node_id": "MDEwOlJlcG9zaXRvcnk0MDA4NzgwNzM=", "name": "datasette-plugin-template-repository", "full_name": "simonw/datasette-plugin-template-repository", "private": false, "owner": {"login": "simonw", "id": 9599, "node_id": "MDQ6VXNlcjk1OTk=", "avatar_url": "https://avatars.githubusercontent.com/u/9599?v=4", "gravatar_id": "", "url": "https://api.github.com/users/simonw", "html_url": "https://github.com/simonw", "followers_url": "https://api.github.com/users/simonw/followers", "following_url": "https://api.github.com/users/simonw/following{/other_user}", "gists_url": "https://api.github.com/users/simonw/gists{/gist_id}", "starred_url": "https://api.github.com/users/simonw/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/simonw/subscriptions", "organizations_url": "https://api.github.com/users/simonw/orgs", "repos_url": "https://api.github.com/users/simonw/repos", "events_url": "https://api.github.com/users/simonw/events{/privacy}", "received_events_url": "https://api.github.com/users/simonw/received_events", "type": "User", "site_admin": false}, "html_url": "https://github.com/simonw/datasette-plugin-template-repository", "description": "GitHub template repository for creating new Datasette plugins, using the simonw/datasette-plugin cookiecutter template", "fork": false, "url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository", "forks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/forks", "keys_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/keys{/key_id}", "collaborators_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/collaborators{/collaborator}", "teams_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/teams", "hooks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/hooks", "issue_events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/events{/number}", "events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/events", "assignees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/assignees{/user}", "branches_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/branches{/branch}", "tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/tags", "blobs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/blobs{/sha}", "git_tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/tags{/sha}", "git_refs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/refs{/sha}", "trees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/trees{/sha}", "statuses_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/statuses/{sha}", "languages_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/languages", "stargazers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/stargazers", "contributors_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contributors", "subscribers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscribers", "subscription_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscription", "commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/commits{/sha}", "git_commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/commits{/sha}", "comments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/comments{/number}", "issue_comment_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/comments{/number}", "contents_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contents/{+path}", "compare_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/compare/{base}...{head}", "merges_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/merges", "archive_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/{archive_format}{/ref}", "downloads_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/downloads", "issues_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues{/number}", "pulls_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/pulls{/number}", "milestones_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/milestones{/number}", "notifications_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/notifications{?since,all,participating}", "labels_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/labels{/name}", "releases_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/releases{/id}", "deployments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/deployments", "created_at": "2021-08-28T19:50:28Z", "updated_at": "2022-08-27T17:22:27Z", "pushed_at": "2022-03-16T23:42:16Z", "git_url": "git://github.com/simonw/datasette-plugin-template-repository.git", "ssh_url": "git@github.com:simonw/datasette-plugin-template-repository.git", "clone_url": "https://github.com/simonw/datasette-plugin-template-repository.git", "svn_url": "https://github.com/simonw/datasette-plugin-template-repository", "homepage": "", "size": 9, "stargazers_count": 16, "watchers_count": 16, "language": null, "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": true, "has_pages": false, "forks_count": 0, "mirror_url": null, "archived": false, "disabled": false, "open_issues_count": 0, "license": null, "allow_forking": true, "is_template": true, "web_commit_signoff_required": false, "topics": [], "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 16, "default_branch": "main", "permissions": {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}, "temp_clone_token": ""}


node_id: R_kgDOIIe2Hg
name: datasette-public
full_name: simonw/datasette-public
html_url: https://github.com/simonw/datasette-public
description: Make specific Datasette tables visible to the public
created_at: 2022-10-05T00:03:28Z
updated_at: 2022-10-05T00:03:40Z
pushed_at: 2022-10-07T22:34:48Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-public

  [![PyPI](https://img.shields.io/pypi/v/datasette-public.svg)](https://pypi.org/project/datasette-public/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-public?include_prereleases&label=changelog)](https://github.com/simonw/datasette-public/releases)
  [![Tests](https://github.com/simonw/datasette-public/workflows/Test/badge.svg)](https://github.com/simonw/datasette-public/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-public/blob/main/LICENSE)

  Make specific Datasette tables visible to the public

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-public

  ## Usage

  Any tables listed in the `_public_tables` table will be visible to the public, even if the rest of the Datasette instance does not allow anonymous access.

  The root user (and any user with the new `public-tables` permission) will get a new option in the table action menu allowing them to toggle a table between public and private.

  Installing this plugin also causes `allow-sql` permission checks to fall back to checking if the user has access to the entire database. This is to avoid users with access to a single public table being able to access data from other tables using the `?_where=` query string parameter.

  ## Configuration

  This plugin creates a new table in one of your databases called `_public_tables`.

  This table defaults to being created in the first database passed to Datasette.

  To create it in a different named database, use this plugin configuration:

  ```json
  {
    "plugins": {
      "datasette-public": {
        "database": "database_to_create_table_in"
      }
    }
  }
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-public
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-public" class="anchor" aria-hidden="true" href="#user-content-datasette-public"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-public</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-public/" rel="nofollow"><img src="https://camo.githubusercontent.com/1798aea770a4aa04657940b1f93ba819d45dca65244ab80f88ef88b148d3eb38/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d7075626c69632e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-public.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-public/releases"><img src="https://camo.githubusercontent.com/57a70736767e5fae2b3260d8f712e4727075426f1a0dcf7bf2acb8041a1c37a6/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d7075626c69633f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-public?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-public/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-public/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-public/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Make specific Datasette tables visible to the public</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-public"><pre class="notranslate"><code>datasette install datasette-public
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Any tables listed in the <code>_public_tables</code> table will be visible to the public, even if the rest of the Datasette instance does not allow anonymous access.</p>
  <p dir="auto">The root user (and any user with the new <code>public-tables</code> permission) will get a new option in the table action menu allowing them to toggle a table between public and private.</p>
  <p dir="auto">Installing this plugin also causes <code>allow-sql</code> permission checks to fall back to checking if the user has access to the entire database. This is to avoid users with access to a single public table being able to access data from other tables using the <code>?_where=</code> query string parameter.</p>
  <h2 dir="auto"><a id="user-content-configuration" class="anchor" aria-hidden="true" href="#user-content-configuration"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Configuration</h2>
  <p dir="auto">This plugin creates a new table in one of your databases called <code>_public_tables</code>.</p>
  <p dir="auto">This table defaults to being created in the first database passed to Datasette.</p>
  <p dir="auto">To create it in a different named database, use this plugin configuration:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    &quot;plugins&quot;: {
      &quot;datasette-public&quot;: {
        &quot;database&quot;: &quot;database_to_create_table_in&quot;
      }
    }
  }"><pre>{
    <span class="pl-ent">"plugins"</span>: {
      <span class="pl-ent">"datasette-public"</span>: {
        <span class="pl-ent">"database"</span>: <span class="pl-s"><span class="pl-pds">"</span>database_to_create_table_in<span class="pl-pds">"</span></span>
      }
    }
  }</pre></div>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-public
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-public
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOIJsqeg
name: datasette-external-links-new-tabs
full_name: ocdtrekkie/datasette-external-links-new-tabs
html_url: https://github.com/ocdtrekkie/datasette-external-links-new-tabs
description: Datasette plugin to open external links in new tabs
created_at: 2022-10-07T03:34:09Z
updated_at: 2022-10-07T05:25:44Z
pushed_at: 2022-10-07T04:51:51Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-external-links-new-tabs

  [![PyPI](https://img.shields.io/pypi/v/datasette-external-links-new-tabs.svg)](https://pypi.org/project/datasette-external-links-new-tabs/)
  [![Changelog](https://img.shields.io/github/v/release/ocdtrekkie/datasette-external-links-new-tabs?include_prereleases&label=changelog)](https://github.com/ocdtrekkie/datasette-external-links-new-tabs/releases)
  [![Tests](https://github.com/ocdtrekkie/datasette-external-links-new-tabs/workflows/Test/badge.svg)](https://github.com/ocdtrekkie/datasette-external-links-new-tabs/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/ocdtrekkie/datasette-external-links-new-tabs/blob/main/LICENSE)

  Datasette plugin to open external links in new tabs

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-external-links-new-tabs

  ## Usage

  There are no usage instructions, it simply opens external links in a new tab.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-external-links-new-tabs
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-external-links-new-tabs" class="anchor" aria-hidden="true" href="#user-content-datasette-external-links-new-tabs"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-external-links-new-tabs</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-external-links-new-tabs/" rel="nofollow"><img src="https://camo.githubusercontent.com/c56d7346bebc740a458bbe809279e0e3b6aa7617ff0b867ba136b3911fb49020/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d65787465726e616c2d6c696e6b732d6e65772d746162732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-external-links-new-tabs.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/ocdtrekkie/datasette-external-links-new-tabs/releases"><img src="https://camo.githubusercontent.com/8206047a3f15fac97ac2c43d88c5c191ad06255610266a60b6afc492ee009ad3/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f6f63647472656b6b69652f6461746173657474652d65787465726e616c2d6c696e6b732d6e65772d746162733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/ocdtrekkie/datasette-external-links-new-tabs?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/ocdtrekkie/datasette-external-links-new-tabs/actions?query=workflow%3ATest"><img src="https://github.com/ocdtrekkie/datasette-external-links-new-tabs/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/ocdtrekkie/datasette-external-links-new-tabs/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Datasette plugin to open external links in new tabs</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-external-links-new-tabs"><pre class="notranslate"><code>datasette install datasette-external-links-new-tabs
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">There are no usage instructions, it simply opens external links in a new tab.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-external-links-new-tabs
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-external-links-new-tabs
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>

template_repository:
[cell repeats an earlier one in this file: 6347 characters]


node_id: R_kgDOISPElA
name: datasette-gunicorn
full_name: simonw/datasette-gunicorn
html_url: https://github.com/simonw/datasette-gunicorn
description: Plugin for running Datasette using Gunicorn
created_at: 2022-10-22T20:23:57Z
updated_at: 2022-10-22T21:00:11Z
pushed_at: 2022-10-23T16:00:49Z
homepage: https://datasette.io/plugins/datasette-gunicorn
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-gunicorn

  [![PyPI](https://img.shields.io/pypi/v/datasette-gunicorn.svg)](https://pypi.org/project/datasette-gunicorn/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-gunicorn?include_prereleases&label=changelog)](https://github.com/simonw/datasette-gunicorn/releases)
  [![Tests](https://github.com/simonw/datasette-gunicorn/workflows/Test/badge.svg)](https://github.com/simonw/datasette-gunicorn/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-gunicorn/blob/main/LICENSE)

  Run a [Datasette](https://datasette.io/) server using [Gunicorn](https://gunicorn.org/)

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-gunicorn

  ## Usage

  The plugin adds a new `datasette gunicorn` command. This takes most of the same options as `datasette serve`, plus one more option for setting the number of Gunicorn workers to start:

  `-w/--workers X` - set the number of workers. Defaults to 1.

  To start serving a database using 4 workers, run the following:

      datasette gunicorn fixtures.db -w 4

  It is advisable to switch your datasette [into WAL mode](https://til.simonwillison.net/sqlite/enabling-wal-mode) to get the best performance out of this configuration:

      sqlite3 fixtures.db 'PRAGMA journal_mode=WAL;'

  Run `datasette gunicorn --help` for a full list of options (which are the same as `datasette serve --help`, with the addition of the new `-w` option).

  ## datasette gunicorn --help

  Not all of the options to `datasette serve` are supported. Here's the full list of available options:

  <!-- [[[cog
  import cog
  from datasette import cli
  from click.testing import CliRunner
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["gunicorn", "--help"])
  help = result.output.replace("Usage: cli", "Usage: datasette")
  cog.out(
      "```\n{}\n```".format(help)
  )
  ]]] -->
  ```
  Usage: datasette gunicorn [OPTIONS] [FILES]...

    Start a Gunicorn server running to serve Datasette

  Options:
    -i, --immutable PATH      Database files to open in immutable mode
    -h, --host TEXT           Host for server. Defaults to 127.0.0.1 which means
                              only connections from the local machine will be
                              allowed. Use 0.0.0.0 to listen to all IPs and allow
                              access from other machines.
    -p, --port INTEGER RANGE  Port for server, defaults to 8001. Use -p 0 to
                              automatically assign an available port.
                              [0<=x<=65535]
    --cors                    Enable CORS by serving Access-Control-Allow-Origin:
                              *
    --load-extension TEXT     Path to a SQLite extension to load
    --inspect-file TEXT       Path to JSON file created using "datasette inspect"
    -m, --metadata FILENAME   Path to JSON/YAML file containing license/source
                              metadata
    --template-dir DIRECTORY  Path to directory containing custom templates
    --plugins-dir DIRECTORY   Path to directory containing custom plugins
    --static MOUNT:DIRECTORY  Serve static files from this directory at /MOUNT/...
    --memory                  Make /_memory database available
    --config CONFIG           Deprecated: set config option using
                              configname:value. Use --setting instead.
    --setting SETTING...      Setting, see
                              docs.datasette.io/en/stable/settings.html
    --secret TEXT             Secret used for signing secure values, such as
                              signed cookies
    --version-note TEXT       Additional note to show on /-/versions
    --help-settings           Show available settings
    --create                  Create database files if they do not exist
    --crossdb                 Enable cross-database joins using the /_memory
                              database
    --nolock                  Ignore locking, open locked files in read-only mode
    -w, --workers INTEGER     Number of Gunicorn workers  [default: 1]
    --help                    Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-gunicorn
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-gunicorn" class="anchor" aria-hidden="true" href="#user-content-datasette-gunicorn"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-gunicorn</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-gunicorn/" rel="nofollow"><img src="https://camo.githubusercontent.com/d6e321e48a6e8dbbe37e1b98456244562198307c261edd8b14e54b0a193ef48e/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d67756e69636f726e2e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-gunicorn.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-gunicorn/releases"><img src="https://camo.githubusercontent.com/2ca4032ada5e556dec300abbc0be96c53ffebb002f75ef02e0e9b56f08569b47/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d67756e69636f726e3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-gunicorn?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-gunicorn/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-gunicorn/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-gunicorn/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Run a <a href="https://datasette.io/" rel="nofollow">Datasette</a> server using <a href="https://gunicorn.org/" rel="nofollow">Gunicorn</a></p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-gunicorn"><pre class="notranslate"><code>datasette install datasette-gunicorn
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">The plugin adds a new <code>datasette gunicorn</code> command. This takes most of the same options as <code>datasette serve</code>, plus one more option for setting the number of Gunicorn workers to start:</p>
  <p dir="auto"><code>-w/--workers X</code> - set the number of workers. Defaults to 1.</p>
  <p dir="auto">To start serving a database using 4 workers, run the following:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette gunicorn fixtures.db -w 4"><pre class="notranslate"><code>datasette gunicorn fixtures.db -w 4
  </code></pre></div>
  <p dir="auto">It is advisable to switch your datasette <a href="https://til.simonwillison.net/sqlite/enabling-wal-mode" rel="nofollow">into WAL mode</a> to get the best performance out of this configuration:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite3 fixtures.db 'PRAGMA journal_mode=WAL;'"><pre class="notranslate"><code>sqlite3 fixtures.db 'PRAGMA journal_mode=WAL;'
  </code></pre></div>
  <p dir="auto">Run <code>datasette gunicorn --help</code> for a full list of options (which are the same as <code>datasette serve --help</code>, with the addition of the new <code>-w</code> option).</p>
  <h2 dir="auto"><a id="user-content-datasette-gunicorn---help" class="anchor" aria-hidden="true" href="#user-content-datasette-gunicorn---help"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette gunicorn --help</h2>
  <p dir="auto">Not all of the options to <code>datasette serve</code> are supported. Here's the full list of available options:</p>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: datasette gunicorn [OPTIONS] [FILES]...

    Start a Gunicorn server running to serve Datasette

  Options:
    -i, --immutable PATH      Database files to open in immutable mode
    -h, --host TEXT           Host for server. Defaults to 127.0.0.1 which means
                              only connections from the local machine will be
                              allowed. Use 0.0.0.0 to listen to all IPs and allow
                              access from other machines.
    -p, --port INTEGER RANGE  Port for server, defaults to 8001. Use -p 0 to
                              automatically assign an available port.
                              [0&lt;=x&lt;=65535]
    --cors                    Enable CORS by serving Access-Control-Allow-Origin:
                              *
    --load-extension TEXT     Path to a SQLite extension to load
    --inspect-file TEXT       Path to JSON file created using &quot;datasette inspect&quot;
    -m, --metadata FILENAME   Path to JSON/YAML file containing license/source
                              metadata
    --template-dir DIRECTORY  Path to directory containing custom templates
    --plugins-dir DIRECTORY   Path to directory containing custom plugins
    --static MOUNT:DIRECTORY  Serve static files from this directory at /MOUNT/...
    --memory                  Make /_memory database available
    --config CONFIG           Deprecated: set config option using
                              configname:value. Use --setting instead.
    --setting SETTING...      Setting, see
                              docs.datasette.io/en/stable/settings.html
    --secret TEXT             Secret used for signing secure values, such as
                              signed cookies
    --version-note TEXT       Additional note to show on /-/versions
    --help-settings           Show available settings
    --create                  Create database files if they do not exist
    --crossdb                 Enable cross-database joins using the /_memory
                              database
    --nolock                  Ignore locking, open locked files in read-only mode
    -w, --workers INTEGER     Number of Gunicorn workers  [default: 1]
    --help                    Show this message and exit.
  "><pre class="notranslate"><code>Usage: datasette gunicorn [OPTIONS] [FILES]...

    Start a Gunicorn server running to serve Datasette

  Options:
    -i, --immutable PATH      Database files to open in immutable mode
    -h, --host TEXT           Host for server. Defaults to 127.0.0.1 which means
                              only connections from the local machine will be
                              allowed. Use 0.0.0.0 to listen to all IPs and allow
                              access from other machines.
    -p, --port INTEGER RANGE  Port for server, defaults to 8001. Use -p 0 to
                              automatically assign an available port.
                              [0&lt;=x&lt;=65535]
    --cors                    Enable CORS by serving Access-Control-Allow-Origin:
                              *
    --load-extension TEXT     Path to a SQLite extension to load
    --inspect-file TEXT       Path to JSON file created using "datasette inspect"
    -m, --metadata FILENAME   Path to JSON/YAML file containing license/source
                              metadata
    --template-dir DIRECTORY  Path to directory containing custom templates
    --plugins-dir DIRECTORY   Path to directory containing custom plugins
    --static MOUNT:DIRECTORY  Serve static files from this directory at /MOUNT/...
    --memory                  Make /_memory database available
    --config CONFIG           Deprecated: set config option using
                              configname:value. Use --setting instead.
    --setting SETTING...      Setting, see
                              docs.datasette.io/en/stable/settings.html
    --secret TEXT             Secret used for signing secure values, such as
                              signed cookies
    --version-note TEXT       Additional note to show on /-/versions
    --help-settings           Show available settings
    --create                  Create database files if they do not exist
    --crossdb                 Enable cross-database joins using the /_memory
                              database
    --nolock                  Ignore locking, open locked files in read-only mode
    -w, --workers INTEGER     Number of Gunicorn workers  [default: 1]
    --help                    Show this message and exit.

  </code></pre></div>

  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-gunicorn
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-gunicorn
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


