# repos

node_id: MDEwOlJlcG9zaXRvcnkxOTc4ODIzODI=
name: healthkit-to-sqlite
full_name: dogsheep/healthkit-to-sqlite
html_url: https://github.com/dogsheep/healthkit-to-sqlite
description: Convert an Apple Healthkit export zip to a SQLite database
created_at: 2019-07-20T05:03:12Z
updated_at: 2021-08-20T00:55:34Z
pushed_at: 2021-08-20T00:56:17Z
homepage: https://datasette.io/tools/healthkit-to-sqlite
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-tool", "dogsheep", "healthkit", "sqlite"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
readme:
  # healthkit-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/healthkit-to-sqlite.svg)](https://pypi.org/project/healthkit-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/healthkit-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/healthkit-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/healthkit-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/healthkit-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/healthkit-to-sqlite/blob/main/LICENSE)

  Convert an Apple Healthkit export zip to a SQLite database

  ## How to install

      $ pip install healthkit-to-sqlite

  ## How to use

  First you need to export your Apple HealthKit data.

  1. On your iPhone, open the "Health" app
  2. Click the profile icon in the top right
  3. Click "Export Health Data" at the bottom of that page
  4. Save the resulting file somewhere you can access it, or AirDrop it directly to your laptop.

  Now you can convert the resulting `export.zip` file to SQLite like so:

      $ healthkit-to-sqlite export.zip healthkit.db

  A progress bar will be displayed. You can disable this using `--silent`.

  ```
  Importing from HealthKit  [#-------------]    5%  00:01:33
  ```

  You can explore the resulting data using [Datasette](https://datasette.readthedocs.io/) like this:

      $ datasette healthkit.db

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-healthkit-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-healthkit-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>healthkit-to-sqlite</h1>
  <p><a href="https://pypi.org/project/healthkit-to-sqlite/" rel="nofollow"><img src="https://camo.githubusercontent.com/d8fd72edb8183afd279306fabff220e4c1670906c58c256e9e3bd7fbdea8c76f/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6865616c74686b69742d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/healthkit-to-sqlite.svg" style="max-width:100%;"></a>
  <a href="https://github.com/dogsheep/healthkit-to-sqlite/releases"><img src="https://camo.githubusercontent.com/9092a0b3a68a53e13d8470fdf4909dda3cabe09e9e74b53960ad2063c79ae8e4/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f646f6773686565702f6865616c74686b69742d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/dogsheep/healthkit-to-sqlite?include_prereleases&amp;label=changelog" style="max-width:100%;"></a>
  <a href="https://github.com/dogsheep/healthkit-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/dogsheep/healthkit-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width:100%;"></a>
  <a href="https://github.com/dogsheep/healthkit-to-sqlite/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Convert an Apple Healthkit export zip to a SQLite database</p>
  <h2><a id="user-content-how-to-install" class="anchor" aria-hidden="true" href="#user-content-how-to-install"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How to install</h2>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ pip install healthkit-to-sqlite
  "><pre><code>$ pip install healthkit-to-sqlite
  </code></pre></div>
  <h2><a id="user-content-how-to-use" class="anchor" aria-hidden="true" href="#user-content-how-to-use"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How to use</h2>
  <p>First you need to export your Apple HealthKit data.</p>
  <ol>
  <li>On your iPhone, open the "Health" app</li>
  <li>Click the profile icon in the top right</li>
  <li>Click "Export Health Data" at the bottom of that page</li>
  <li>Save the resulting file somewhere you can access it, or AirDrop it directly to your laptop.</li>
  </ol>
  <p>Now you can convert the resulting <code>export.zip</code> file to SQLite like so:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ healthkit-to-sqlite export.zip healthkit.db
  "><pre><code>$ healthkit-to-sqlite export.zip healthkit.db
  </code></pre></div>
  <p>A progress bar will be displayed. You can disable this using <code>--silent</code>.</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="Importing from HealthKit  [#-------------]    5%  00:01:33
  "><pre><code>Importing from HealthKit  [#-------------]    5%  00:01:33
  </code></pre></div>
  <p>You can explore the resulting data using <a href="https://datasette.readthedocs.io/" rel="nofollow">Datasette</a> like this:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ datasette healthkit.db
  "><pre><code>$ datasette healthkit.db
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMDU0MjkzNzU=
name: swarm-to-sqlite
full_name: dogsheep/swarm-to-sqlite
html_url: https://github.com/dogsheep/swarm-to-sqlite
description: Create a SQLite database containing your checkin history from Foursquare Swarm
created_at: 2019-08-30T17:37:29Z
updated_at: 2021-02-22T07:58:39Z
pushed_at: 2021-01-18T04:36:03Z
language: Python
license: apache-2.0
default_branch: main
permissions: {"admin": false, "push": false, "pull": false}
topics:
  ["sqlite", "foursquare", "swarm", "foursquare-api", "datasette", "dogsheep", "datasette-io", "datasette-tool"]

readme:
  # swarm-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/swarm-to-sqlite.svg)](https://pypi.org/project/swarm-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/swarm-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/swarm-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/swarm-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/swarm-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/swarm-to-sqlite/blob/main/LICENSE)

  Create a SQLite database containing your checkin history from Foursquare Swarm.

  ## How to install

      $ pip install swarm-to-sqlite

  ## Usage

  You will need to first obtain a valid OAuth token for your Foursquare account. You can do so using this tool: https://your-foursquare-oauth-token.glitch.me/

  Simplest usage is to simply provide the name of the database file you wish to write to. The tool will prompt you to paste in your token, and will then download your checkins and store them in the specified database file.

      $ swarm-to-sqlite checkins.db
      Please provide your Foursquare OAuth token:
      Importing 3699 checkins  [#########-----------------------] 27% 00:02:31

  You can also pass the token as a command-line option:

      $ swarm-to-sqlite checkins.db --token=XXX

  Or as an environment variable:

      $ export FOURSQUARE_TOKEN=XXX
      $ swarm-to-sqlite checkins.db

  To retrieve just checkins within the past X hours, days or weeks, use the `--since=` option. For example, to pull only checkins that happened within the last 10 days use:

      $ swarm-to-sqlite checkins.db --token=XXX --since=10d

  Use `2w` for two weeks, `10h` for ten hours, `3d` for three days.

  In addition to saving the checkins to a database, you can also write them to a JSON file using the `--save` option:

      $ swarm-to-sqlite checkins.db --save=checkins.json

  Having done this, you can re-import checkins directly from that file (rather than making API calls to fetch data from Foursquare) like this:

      $ swarm-to-sqlite checkins.db --load=checkins.json

  ## Using with Datasette

  The SQLite database produced by this tool is designed to be browsed using [Datasette](https://datasette.io/).

  You can install the [datasette-cluster-map](https://datasette.io/plugins/datasette-cluster-map) plugin to view your checkins on a map.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-swarm-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-swarm-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>swarm-to-sqlite</h1>
  <p><a href="https://pypi.org/project/swarm-to-sqlite/" rel="nofollow"><img src="https://camo.githubusercontent.com/336537cfcc544f29699c00a2aa4b5a199c9a21f53e43aff833e268e664735ed9/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f737761726d2d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/swarm-to-sqlite.svg" style="max-width:100%;"></a>
  <a href="https://github.com/dogsheep/swarm-to-sqlite/releases"><img src="https://camo.githubusercontent.com/3a244282c89501bdea48af0387ebed5c0a03d41ef8f87331ed3fe63421f42c4f/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f646f6773686565702f737761726d2d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/dogsheep/swarm-to-sqlite?include_prereleases&amp;label=changelog" style="max-width:100%;"></a>
  <a href="https://github.com/dogsheep/swarm-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/dogsheep/swarm-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width:100%;"></a>
  <a href="https://github.com/dogsheep/swarm-to-sqlite/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Create a SQLite database containing your checkin history from Foursquare Swarm.</p>
  <h2><a id="user-content-how-to-install" class="anchor" aria-hidden="true" href="#user-content-how-to-install"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How to install</h2>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ pip install swarm-to-sqlite
  "><pre><code>$ pip install swarm-to-sqlite
  </code></pre></div>
  <h2><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p>You will need to first obtain a valid OAuth token for your Foursquare account. You can do so using this tool: <a href="https://your-foursquare-oauth-token.glitch.me/" rel="nofollow">https://your-foursquare-oauth-token.glitch.me/</a></p>
  <p>Simplest usage is to simply provide the name of the database file you wish to write to. The tool will prompt you to paste in your token, and will then download your checkins and store them in the specified database file.</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ swarm-to-sqlite checkins.db
  Please provide your Foursquare OAuth token:
  Importing 3699 checkins  [#########-----------------------] 27% 00:02:31
  "><pre><code>$ swarm-to-sqlite checkins.db
  Please provide your Foursquare OAuth token:
  Importing 3699 checkins  [#########-----------------------] 27% 00:02:31
  </code></pre></div>
  <p>You can also pass the token as a command-line option:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ swarm-to-sqlite checkins.db --token=XXX
  "><pre><code>$ swarm-to-sqlite checkins.db --token=XXX
  </code></pre></div>
  <p>Or as an environment variable:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ export FOURSQUARE_TOKEN=XXX
  $ swarm-to-sqlite checkins.db
  "><pre><code>$ export FOURSQUARE_TOKEN=XXX
  $ swarm-to-sqlite checkins.db
  </code></pre></div>
  <p>To retrieve just checkins within the past X hours, days or weeks, use the <code>--since=</code> option. For example, to pull only checkins that happened within the last 10 days use:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ swarm-to-sqlite checkins.db --token=XXX --since=10d
  "><pre><code>$ swarm-to-sqlite checkins.db --token=XXX --since=10d
  </code></pre></div>
  <p>Use <code>2w</code> for two weeks, <code>10h</code> for ten hours, <code>3d</code> for three days.</p>
  <p>In addition to saving the checkins to a database, you can also write them to a JSON file using the <code>--save</code> option:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ swarm-to-sqlite checkins.db --save=checkins.json
  "><pre><code>$ swarm-to-sqlite checkins.db --save=checkins.json
  </code></pre></div>
  <p>Having done this, you can re-import checkins directly from that file (rather than making API calls to fetch data from Foursquare) like this:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ swarm-to-sqlite checkins.db --load=checkins.json
  "><pre><code>$ swarm-to-sqlite checkins.db --load=checkins.json
  </code></pre></div>
  <h2><a id="user-content-using-with-datasette" class="anchor" aria-hidden="true" href="#user-content-using-with-datasette"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Using with Datasette</h2>
  <p>The SQLite database produced by this tool is designed to be browsed using <a href="https://datasette.io/" rel="nofollow">Datasette</a>.</p>
  <p>You can install the <a href="https://datasette.io/plugins/datasette-cluster-map" rel="nofollow">datasette-cluster-map</a> plugin to view your checkins on a map.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMDYxNTY4NjY=
name: twitter-to-sqlite
full_name: dogsheep/twitter-to-sqlite
html_url: https://github.com/dogsheep/twitter-to-sqlite
description: Save data from Twitter to a SQLite database
created_at: 2019-09-03T19:30:08Z
updated_at: 2021-12-26T18:08:43Z
pushed_at: 2021-12-26T18:08:40Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-tool", "dogsheep", "sqlite", "twitter", "twitter-api"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # twitter-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/twitter-to-sqlite.svg)](https://pypi.org/project/twitter-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/twitter-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/twitter-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/twitter-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/twitter-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/twitter-to-sqlite/blob/main/LICENSE)

  Save data from Twitter to a SQLite database.

  **This tool currently uses Twitter API v1**. You may be unable to use it if you do not have an API key for that version of the API.

  <!-- toc -->

  - [How to install](#how-to-install)
  - [Authentication](#authentication)
  - [Retrieving tweets by specific accounts](#retrieving-tweets-by-specific-accounts)
  - [Retrieve user profiles in bulk](#retrieve-user-profiles-in-bulk)
  - [Retrieve tweets in bulk](#retrieve-tweets-in-bulk)
  - [Retrieving Twitter followers](#retrieving-twitter-followers)
  - [Retrieving friends](#retrieving-friends)
  - [Retrieving favorited tweets](#retrieving-favorited-tweets)
  - [Retrieving Twitter lists](#retrieving-twitter-lists)
  - [Retrieving Twitter list memberships](#retrieving-twitter-list-memberships)
  - [Retrieving just follower and friend IDs](#retrieving-just-follower-and-friend-ids)
  - [Retrieving tweets from your home timeline](#retrieving-tweets-from-your-home-timeline)
  - [Retrieving your mentions](#retrieving-your-mentions)
  - [Providing input from a SQL query with --sql and --attach](#providing-input-from-a-sql-query-with---sql-and---attach)
  - [Running searches](#running-searches)
  - [Capturing tweets in real-time with track and follow](#capturing-tweets-in-real-time-with-track-and-follow)
    * [track](#track)
    * [follow](#follow)
  - [Importing data from your Twitter archive](#importing-data-from-your-twitter-archive)
  - [Design notes](#design-notes)

  <!-- tocstop -->

  ## How to install

      $ pip install twitter-to-sqlite

  ## Authentication

  First, you will need to create a Twitter application at https://developer.twitter.com/en/apps. You may need to apply for a Twitter developer account - if so, you may find this [example of an email application](https://raw.githubusercontent.com/dogsheep/twitter-to-sqlite/main/email.png) useful that has been approved in the past.

  Once you have created your application, navigate to the "Keys and tokens" page and make note of the following:

  * Your API key
  * Your API secret key
  * Your access token
  * Your access token secret

  You will need to save all four of these values to a JSON file in order to use this tool.

  You can create that JSON file by running the following command and pasting in the values at the prompts:

      $ twitter-to-sqlite auth
      Create an app here: https://developer.twitter.com/en/apps
      Then navigate to 'Keys and tokens' and paste in the following:

      API key: xxx
      API secret key: xxx
      Access token: xxx
      Access token secret: xxx

  This will create a file called `auth.json` in your current directory containing the required values. To save the file at a different path or filename, use the `--auth=myauth.json` option.

  ## Retrieving tweets by specific accounts

  The `user-timeline` command retrieves all of the tweets posted by the specified user accounts. It defaults to the account belonging to the authenticated user:

      $ twitter-to-sqlite user-timeline twitter.db
      Importing tweets  [#####-------------------------------]  2799/17780  00:01:39

  All of these commands assume that there is an `auth.json` file in the current directory. You can provide the path to your `auth.json` file using `-a`:

      $ twitter-to-sqlite user-timeline twitter.db -a /path/to/auth.json

  To load tweets for other users, pass their screen names as arguments:

      $ twitter-to-sqlite user-timeline twitter.db cleopaws nichemuseums

  Twitter's API only returns up to around 3,200 tweets for most user accounts, but you may find that it returns all available tweets for your own user account.

  You can pass numeric Twitter user IDs instead of screen names using the `--ids` parameter.

  You can use `--since` to retrieve every tweet since the last time you imported for that user, or `--since_id=xxx` to retrieve every tweet since a specific tweet ID.

  This command also accepts `--sql` and `--attach` options, documented below.

  ## Retrieve user profiles in bulk

  If you have a list of Twitter screen names (or user IDs) you can bulk fetch their fully inflated Twitter profiles using the `users-lookup` command:

      $ twitter-to-sqlite users-lookup users.db simonw cleopaws

  You can pass user IDs instead using the `--ids` option:

      $ twitter-to-sqlite users-lookup users.db 12497 3166449535 --ids

  This command also accepts `--sql` and `--attach` options, documented below.

  ## Retrieve tweets in bulk

  If you have a list of tweet IDS you can bulk fetch them using the `statuses-lookup` command:

      $ twitter-to-sqlite statuses-lookup tweets.db 1122154819815239680 1122154178493575169

  The `--sql` and `--attach` options are supported.

  Here's a recipe to retrieve any tweets that existing tweets are in-reply-to which have not yet been stored in your database:

      $ twitter-to-sqlite statuses-lookup tweets.db \
          --sql='
              select in_reply_to_status_id
              from tweets
              where in_reply_to_status_id is not null' \
          --skip-existing

  The `--skip-existing` option means that tweets that have already been stored in the database will not be fetched again.

  ## Retrieving Twitter followers

  The `followers` command retrieves details of every follower of the specified accounts. You can use it to retrieve your own followers, or you can pass one or more screen names to pull the followers for other accounts.

  The following command pulls your followers and saves them in a SQLite database file called `twitter.db`:

      $ twitter-to-sqlite followers twitter.db

  This command is **extremely slow**, because Twitter impose a rate limit of no more than one request per minute to this endpoint! If you are running it against an account with thousands of followers you should expect this to take several hours.

  To retrieve followers for another account, use:

      $ twitter-to-sqlite followers twitter.db cleopaws

  This command also accepts the `--ids`, `--sql` and `--attach` options.

  See [Analyzing my Twitter followers with Datasette](https://simonwillison.net/2018/Jan/28/analyzing-my-twitter-followers/) for the original inspiration for this command.

  ## Retrieving friends

  The `friends` command works like the `followers` command, but retrieves the specified (or currently authenticated) user's friends - defined as accounts that the user is following.

      $ twitter-to-sqlite friends twitter.db

  It takes the same options as the `followers` command.

  ## Retrieving favorited tweets

  The `favorites` command retrieves tweets that have been favorited by a specified user. Called without any extra arguments it retrieves tweets favorited by the currently authenticated user:

      $ twitter-to-sqlite favorites faves.db

  You can also use the `--screen_name` or `--user_id` arguments to retrieve favorite tweets for another user:

      $ twitter-to-sqlite favorites faves-obama.db --screen_name=BarackObama

  Use the `--stop_after=xxx` argument to retrieve only the most recent number of favorites, e.g. to get the authenticated user's 50 most recent favorites:

      $ twitter-to-sqlite favorites faves.db --stop_after=50

  ## Retrieving Twitter lists

  The `lists` command retrieves all of the lists belonging to one or more users.

      $ twitter-to-sqlite lists lists.db simonw dogsheep

  This command also accepts the `--sql` and `--attach` and `--ids` options.

  To additionally fetch the list of members for each list, use `--members`.

  ## Retrieving Twitter list memberships

  The `list-members` command can be used to retrieve details of one or more Twitter lists, including all of their members.

      $ twitter-to-sqlite list-members members.db simonw/the-good-place

  You can pass multiple `screen_name/list_slug` identifiers.

  If you know the numeric IDs of the lists instead, you can use `--ids`:

      $ twitter-to-sqlite list-members members.db 927913322841653248 --ids

  ## Retrieving just follower and friend IDs

  It's also possible to retrieve just the numeric Twitter IDs of the accounts that specific users are following ("friends" in Twitter's API terminology) or followed-by:

      $ twitter-to-sqlite followers-ids members.db simonw cleopaws

  This will populate the `following` table with `followed_id`/`follower_id` pairs for the two specified accounts, listing every account ID that is following either of those two accounts.

      $ twitter-to-sqlite friends-ids members.db simonw cleopaws

  This will do the same thing but pull the IDs that those accounts are following.

  Both of these commands also support `--sql` and `--attach` as an alternative to passing screen names as direct command-line arguments. You can use `--ids` to process the inputs as user IDs rather than screen names.

  The underlying Twitter APIs have a rate limit of 15 requests every 15 minutes - though they do return up to 5,000 IDs in each call. By default both of these subcommands will wait for 61 seconds between API calls in order to stay within the rate limit - you can adjust this behaviour down to just one second delay if you know you will not be making many calls using `--sleep=1`.

  ## Retrieving tweets from your home timeline

  The `home-timeline` command retrieves up to 800 tweets from the home timeline of the authenticated user - generally this means tweets from people you follow.

      $ twitter-to-sqlite home-timeline twitter.db
      Importing timeline  [#################--------]  591/800  00:01:14

  The tweets are stored in the `tweets` table, and a record is added to the `timeline_tweets` table noting that this tweet came in due to being spotted in the timeline of your user.

  You can use `--since` to retrieve just tweets that have been posted since the last time this command was run, or `--since_id=xxx` to explicitly pass in a tweet ID to use as the last position.

  You can then view your timeline in Datasette using the following URL:

  `/tweets/tweets?_where=id+in+(select+tweet+from+[timeline_tweets])&_sort_desc=id&_facet=user`

  This will filter your tweets table to just tweets that appear in your timeline, ordered by most recent first and use faceting to show you which users are responsible for the most tweets.

  ## Retrieving your mentions

  The `mentions-timeline` command works like `home-timeline` except it retrieves tweets that mention the authenticated user's account. It records the user account that was mentioned in a `mentions_tweets` table.

  It supports `--since` and `--since_id` in the same was as `home-timeline` does.

  ## Providing input from a SQL query with --sql and --attach

  This option is available for some subcommands - run `twitter-to-sqlite command-name --help` to check.

  You can provide Twitter screen names (or user IDs or tweet IDs) directly as command-line arguments, or you can provide those screen names or IDs by executing a SQL query.

  For example: consider a SQLite database with an `attendees` table listing names and Twitter accounts - something like this:

  | First   | Last       | Twitter      |
  |---------|------------|--------------|
  | Simon   | Willison   | simonw       |
  | Avril   | Lavigne    | AvrilLavigne |

  You can run the `users-lookup` command to pull the Twitter profile of every user listed in that database by loading the screen names using a `--sql` query:

      $ twitter-to-sqlite users-lookup my.db --sql="select Twitter from attendees"

  If your database table contains Twitter IDs, you can select those IDs and pass the `--ids` argument. For example, to fetch the profiles of users who have had their user IDs inserted into the `following` table using the `twitter-to-sqlite friends-ids` command:

      $ twitter-to-sqlite users-lookup my.db --sql="select follower_id from following" --ids

  Or to avoid re-fetching users that have already been fetched:

      $ twitter-to-sqlite users-lookup my.db \
          --sql="select followed_id from following where followed_id not in (
              select id from users)" --ids

  If your data lives in a separate database file you can attach it using `--attach`. For example, consider the attendees example above but the data lives in an `attendees.db` file, and you want to fetch the user profiles into a `tweets.db` file. You could do that like this:

      $ twitter-to-sqlite users-lookup tweets.db \
          --attach=attendees.db \
          --sql="select Twitter from attendees.attendees"

  The filename (without the extension) will be used as the database alias within SQLite. If you want a different alias for some reason you can specify that with a colon like this:

      $ twitter-to-sqlite users-lookup tweets.db \
          --attach=foo:attendees.db \
          --sql="select Twitter from foo.attendees"

  ## Running searches

  The `search` command runs a search against the Twitter [standard search API](https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets).

      $ twitter-to-sqlite search tweets.db "dogsheep"

  This will import up to around 320 tweets that match that search term into the `tweets` table. It will also create a record in the `search_runs` table recording that the search took place, and many-to-many records in the `search_runs_tweets` table recording which tweets were seen for that search at that time.

  You can use the `--since` parameter to check for previous search runs with the same arguments and only retrieve tweets that were posted since the last retrieved matching tweet.

  The following additional options for `search` are supported:

  * `--geocode`: `latitude,longitude,radius` where radius is a number followed by mi or km
  * `--lang`: ISO 639-1 language code e.g. `en` or `es`
  * `--locale`: Locale: only `ja` is currently effective
  * `--result_type`: `mixed`, `recent` or `popular`. Defaults to `mixed`
  * `--count`: Number of results per page, defaults to the maximum of 100
  * `--stop_after`: Stop after this many results
  * `--since_id`: Pull tweets since this Tweet ID. You probably want to use `--since` instead of this.

  ## Capturing tweets in real-time with track and follow

  This functionality is **experimental**. Please [file bug reports](https://github.com/dogsheep/twitter-to-sqlite/issues) if you find any!

  Twitter provides a real-time API which can be used to subscribe to tweets as they happen. `twitter-to-sqlite` can use this API to continually update a SQLite database with tweets matching certain keywords, or referencing specific users.

  ### track

  To track keywords, use the `track` command:

      $ twitter-to-sqlite track tweets.db kakapo

  This command will continue to run until you hit Ctrl+C. It will capture any tweets mentioning the keyword [kakapo](https://en.wikipedia.org/wiki/Kakapo) and store them in the `tweets.db` database file.

  You can pass multiple keywords as a space separated list. This will capture tweets matching either of those keywords:

      $ twitter-to-sqlite track tweets.db kakapo raccoon

  You can enclose phrases in quotes to search for tweets matching both of those keywords:

      $ twitter-to-sqlite track tweets.db 'trash panda'

  See [the Twitter track documentation](https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/basic-stream-parameters#track) for advanced tips on using this command.

  Add the `--verbose` option to see matching tweets (in their verbose JSON form) displayed to the terminal as they are captured:

      $ twitter-to-sqlite track tweets.db raccoon --verbose

  ### follow

  The `follow` command will capture all tweets that are relevant to one or more specific Twitter users.

      $ twitter-to-sqlite follow tweets.db nytimes

  This includes tweets by those users, tweets that reply to or quote those users and retweets by that user. See [the Twitter follow documentation](https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/basic-stream-parameters#follow) for full details.

  The command accepts one or more screen names.

  You can feed it numeric Twitter user IDs instead of screen names by using the `--ids` flag.

  The command also supports the `--sql` and `--attach` options, and the `--verbose` option for displaying tweets as they are captured.

  Here's how to start following tweets from every user ID currently represented as being followed in the `following` table (populated using the `friends-ids` command):

      $ twitter-to-sqlite follow tweets.db \
          --sql="select distinct followed_id from following" \
          --ids

  ## Importing data from your Twitter archive

  You can request an archive of your Twitter data by [following these instructions](https://help.twitter.com/en/managing-your-account/how-to-download-your-twitter-archive).

  Twitter will send you a link to download a `.zip` file. You can import the contents of that file into a set of tables in a new database file called `archive.db` (each table beginning with the `archive_` prefix) using the `import` command:

      $ twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2.zip

  This command does not populate any of the regular tables, since Twitter's export data does not exactly match the schema returned by the Twitter API.

  It will delete and recreate the corresponding `archive_*` tables every time you run it. If this is not what you want, run the command against a new SQLite database file name rather than running it against one that already exists.

  If you have already decompressed your archive, you can run this against the directory that you decompressed it to:

      $ twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2/

  You can also run it against one or more specific files within that folder. For example, to import just the follower.js and following.js files:

      $ twitter-to-sqlite import archive.db \
          ~/Downloads/twitter-2019-06-25-b31f2/follower.js \
          ~/Downloads/twitter-2019-06-25-b31f2/following.js

  You may want to use other commands to populate tables based on data from the archive. For example, to retrieve full API versions of each of the tweets you have favourited in your archive, you could run the following:

      $ twitter-to-sqlite statuses-lookup archive.db \
          --sql='select tweetId from archive_like' \
          --skip-existing

  If you want these imported tweets to then be reflected in the `favorited_by` table, you can do so by applying the following SQL query:

      $ sqlite3 archive.db
      SQLite version 3.22.0 2018-01-22 18:45:57
      Enter ".help" for usage hints.
      sqlite> INSERT OR IGNORE INTO favorited_by (tweet, user)
         ...>     SELECT tweetId, 'YOUR_TWITTER_ID' FROM archive_like;
      <Ctrl+D>

  Replace YOUR_TWITTER_ID with your numeric Twitter ID. If you don't know that ID you can find it out by running the following:

      $ twitter-to-sqlite fetch \
          "https://api.twitter.com/1.1/account/verify_credentials.json" \
          | grep '"id"' | head -n 1

  ## Design notes

  * Tweet IDs are stored as integers, to afford sorting by ID in a sensible way
  * While we configure foreign key relationships between tables, we do not ask SQLite to enforce them. This is used by the `following` table to allow the `followers-ids` and `friends-ids` commands to populate it with user IDs even if the user accounts themselves are not yet present in the `users` table.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-twitter-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-twitter-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>twitter-to-sqlite</h1>
  <p dir="auto"><a href="https://pypi.org/project/twitter-to-sqlite/" rel="nofollow"><img src="https://camo.githubusercontent.com/fce3d286daf4a0ac037476f32a7f1885dd7f90329ec1df80f7fe6b9322c72f5c/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f747769747465722d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/twitter-to-sqlite.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/twitter-to-sqlite/releases"><img src="https://camo.githubusercontent.com/e1228185d86e3eb446efcb27ff81748cdad3b90ac14ae58c61dab61a90bb992d/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f646f6773686565702f747769747465722d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/dogsheep/twitter-to-sqlite?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/twitter-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/dogsheep/twitter-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/twitter-to-sqlite/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Save data from Twitter to a SQLite database.</p>
  <p dir="auto"><strong>This tool currently uses Twitter API v1</strong>. You may be unable to use it if you do not have an API key for that version of the API.</p>

  <ul dir="auto">
  <li><a href="#user-content-how-to-install">How to install</a></li>
  <li><a href="#user-content-authentication">Authentication</a></li>
  <li><a href="#user-content-retrieving-tweets-by-specific-accounts">Retrieving tweets by specific accounts</a></li>
  <li><a href="#user-content-retrieve-user-profiles-in-bulk">Retrieve user profiles in bulk</a></li>
  <li><a href="#user-content-retrieve-tweets-in-bulk">Retrieve tweets in bulk</a></li>
  <li><a href="#user-content-retrieving-twitter-followers">Retrieving Twitter followers</a></li>
  <li><a href="#user-content-retrieving-friends">Retrieving friends</a></li>
  <li><a href="#user-content-retrieving-favorited-tweets">Retrieving favorited tweets</a></li>
  <li><a href="#user-content-retrieving-twitter-lists">Retrieving Twitter lists</a></li>
  <li><a href="#user-content-retrieving-twitter-list-memberships">Retrieving Twitter list memberships</a></li>
  <li><a href="#user-content-retrieving-just-follower-and-friend-ids">Retrieving just follower and friend IDs</a></li>
  <li><a href="#user-content-retrieving-tweets-from-your-home-timeline">Retrieving tweets from your home timeline</a></li>
  <li><a href="#user-content-retrieving-your-mentions">Retrieving your mentions</a></li>
  <li><a href="#user-content-providing-input-from-a-sql-query-with---sql-and---attach">Providing input from a SQL query with --sql and --attach</a></li>
  <li><a href="#user-content-running-searches">Running searches</a></li>
  <li><a href="#user-content-capturing-tweets-in-real-time-with-track-and-follow">Capturing tweets in real-time with track and follow</a>
  <ul dir="auto">
  <li><a href="#user-content-track">track</a></li>
  <li><a href="#user-content-follow">follow</a></li>
  </ul>
  </li>
  <li><a href="#user-content-importing-data-from-your-twitter-archive">Importing data from your Twitter archive</a></li>
  <li><a href="#user-content-design-notes">Design notes</a></li>
  </ul>

  <h2 dir="auto"><a id="user-content-how-to-install" class="anchor" aria-hidden="true" href="#user-content-how-to-install"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How to install</h2>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install twitter-to-sqlite"><pre><code>$ pip install twitter-to-sqlite
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-authentication" class="anchor" aria-hidden="true" href="#user-content-authentication"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Authentication</h2>
  <p dir="auto">First, you will need to create a Twitter application at <a href="https://developer.twitter.com/en/apps" rel="nofollow">https://developer.twitter.com/en/apps</a>. You may need to apply for a Twitter developer account - if so, you may find this <a href="https://raw.githubusercontent.com/dogsheep/twitter-to-sqlite/main/email.png" rel="nofollow">example of an email application</a> useful that has been approved in the past.</p>
  <p dir="auto">Once you have created your application, navigate to the "Keys and tokens" page and make note of the following:</p>
  <ul dir="auto">
  <li>Your API key</li>
  <li>Your API secret key</li>
  <li>Your access token</li>
  <li>Your access token secret</li>
  </ul>
  <p dir="auto">You will need to save all four of these values to a JSON file in order to use this tool.</p>
  <p dir="auto">You can create that JSON file by running the following command and pasting in the values at the prompts:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite auth
  Create an app here: https://developer.twitter.com/en/apps
  Then navigate to 'Keys and tokens' and paste in the following:

  API key: xxx
  API secret key: xxx
  Access token: xxx
  Access token secret: xxx"><pre><code>$ twitter-to-sqlite auth
  Create an app here: https://developer.twitter.com/en/apps
  Then navigate to 'Keys and tokens' and paste in the following:

  API key: xxx
  API secret key: xxx
  Access token: xxx
  Access token secret: xxx
  </code></pre></div>
  <p dir="auto">This will create a file called <code>auth.json</code> in your current directory containing the required values. To save the file at a different path or filename, use the <code>--auth=myauth.json</code> option.</p>
  <h2 dir="auto"><a id="user-content-retrieving-tweets-by-specific-accounts" class="anchor" aria-hidden="true" href="#user-content-retrieving-tweets-by-specific-accounts"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Retrieving tweets by specific accounts</h2>
  <p dir="auto">The <code>user-timeline</code> command retrieves all of the tweets posted by the specified user accounts. It defaults to the account belonging to the authenticated user:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite user-timeline twitter.db
  Importing tweets  [#####-------------------------------]  2799/17780  00:01:39"><pre><code>$ twitter-to-sqlite user-timeline twitter.db
  Importing tweets  [#####-------------------------------]  2799/17780  00:01:39
  </code></pre></div>
  <p dir="auto">All of these commands assume that there is an <code>auth.json</code> file in the current directory. You can provide the path to your <code>auth.json</code> file using <code>-a</code>:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite user-timeline twitter.db -a /path/to/auth.json"><pre><code>$ twitter-to-sqlite user-timeline twitter.db -a /path/to/auth.json
  </code></pre></div>
  <p dir="auto">To load tweets for other users, pass their screen names as arguments:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite user-timeline twitter.db cleopaws nichemuseums"><pre><code>$ twitter-to-sqlite user-timeline twitter.db cleopaws nichemuseums
  </code></pre></div>
  <p dir="auto">Twitter's API only returns up to around 3,200 tweets for most user accounts, but you may find that it returns all available tweets for your own user account.</p>
  <p dir="auto">You can pass numeric Twitter user IDs instead of screen names using the <code>--ids</code> parameter.</p>
  <p dir="auto">You can use <code>--since</code> to retrieve every tweet since the last time you imported for that user, or <code>--since_id=xxx</code> to retrieve every tweet since a specific tweet ID.</p>
  <p dir="auto">This command also accepts <code>--sql</code> and <code>--attach</code> options, documented below.</p>
  <h2 dir="auto"><a id="user-content-retrieve-user-profiles-in-bulk" class="anchor" aria-hidden="true" href="#user-content-retrieve-user-profiles-in-bulk"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Retrieve user profiles in bulk</h2>
  <p dir="auto">If you have a list of Twitter screen names (or user IDs) you can bulk fetch their fully inflated Twitter profiles using the <code>users-lookup</code> command:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite users-lookup users.db simonw cleopaws"><pre><code>$ twitter-to-sqlite users-lookup users.db simonw cleopaws
  </code></pre></div>
  <p dir="auto">You can pass user IDs instead using the <code>--ids</code> option:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite users-lookup users.db 12497 3166449535 --ids"><pre><code>$ twitter-to-sqlite users-lookup users.db 12497 3166449535 --ids
  </code></pre></div>
  <p dir="auto">This command also accepts <code>--sql</code> and <code>--attach</code> options, documented below.</p>
  <h2 dir="auto"><a id="user-content-retrieve-tweets-in-bulk" class="anchor" aria-hidden="true" href="#user-content-retrieve-tweets-in-bulk"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Retrieve tweets in bulk</h2>
  <p dir="auto">If you have a list of tweet IDS you can bulk fetch them using the <code>statuses-lookup</code> command:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite statuses-lookup tweets.db 1122154819815239680 1122154178493575169"><pre><code>$ twitter-to-sqlite statuses-lookup tweets.db 1122154819815239680 1122154178493575169
  </code></pre></div>
  <p dir="auto">The <code>--sql</code> and <code>--attach</code> options are supported.</p>
  <p dir="auto">Here's a recipe to retrieve any tweets that existing tweets are in-reply-to which have not yet been stored in your database:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite statuses-lookup tweets.db \
      --sql='
          select in_reply_to_status_id
          from tweets
          where in_reply_to_status_id is not null' \
      --skip-existing"><pre><code>$ twitter-to-sqlite statuses-lookup tweets.db \
      --sql='
          select in_reply_to_status_id
          from tweets
          where in_reply_to_status_id is not null' \
      --skip-existing
  </code></pre></div>
  <p dir="auto">The <code>--skip-existing</code> option means that tweets that have already been stored in the database will not be fetched again.</p>
  <h2 dir="auto"><a id="user-content-retrieving-twitter-followers" class="anchor" aria-hidden="true" href="#user-content-retrieving-twitter-followers"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Retrieving Twitter followers</h2>
  <p dir="auto">The <code>followers</code> command retrieves details of every follower of the specified accounts. You can use it to retrieve your own followers, or you can pass one or more screen names to pull the followers for other accounts.</p>
  <p dir="auto">The following command pulls your followers and saves them in a SQLite database file called <code>twitter.db</code>:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite followers twitter.db"><pre><code>$ twitter-to-sqlite followers twitter.db
  </code></pre></div>
  <p dir="auto">This command is <strong>extremely slow</strong>, because Twitter impose a rate limit of no more than one request per minute to this endpoint! If you are running it against an account with thousands of followers you should expect this to take several hours.</p>
  <p dir="auto">To retrieve followers for another account, use:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite followers twitter.db cleopaws"><pre><code>$ twitter-to-sqlite followers twitter.db cleopaws
  </code></pre></div>
  <p dir="auto">This command also accepts the <code>--ids</code>, <code>--sql</code> and <code>--attach</code> options.</p>
  <p dir="auto">See <a href="https://simonwillison.net/2018/Jan/28/analyzing-my-twitter-followers/" rel="nofollow">Analyzing my Twitter followers with Datasette</a> for the original inspiration for this command.</p>
  <h2 dir="auto"><a id="user-content-retrieving-friends" class="anchor" aria-hidden="true" href="#user-content-retrieving-friends"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Retrieving friends</h2>
  <p dir="auto">The <code>friends</code> command works like the <code>followers</code> command, but retrieves the specified (or currently authenticated) user's friends - defined as accounts that the user is following.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite friends twitter.db"><pre><code>$ twitter-to-sqlite friends twitter.db
  </code></pre></div>
  <p dir="auto">It takes the same options as the <code>followers</code> command.</p>
  <h2 dir="auto"><a id="user-content-retrieving-favorited-tweets" class="anchor" aria-hidden="true" href="#user-content-retrieving-favorited-tweets"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Retrieving favorited tweets</h2>
  <p dir="auto">The <code>favorites</code> command retrieves tweets that have been favorited by a specified user. Called without any extra arguments it retrieves tweets favorited by the currently authenticated user:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite favorites faves.db"><pre><code>$ twitter-to-sqlite favorites faves.db
  </code></pre></div>
  <p dir="auto">You can also use the <code>--screen_name</code> or <code>--user_id</code> arguments to retrieve favorite tweets for another user:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite favorites faves-obama.db --screen_name=BarackObama"><pre><code>$ twitter-to-sqlite favorites faves-obama.db --screen_name=BarackObama
  </code></pre></div>
  <p dir="auto">Use the <code>--stop_after=xxx</code> argument to retrieve only the most recent number of favorites, e.g. to get the authenticated user's 50 most recent favorites:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite favorites faves.db --stop_after=50"><pre><code>$ twitter-to-sqlite favorites faves.db --stop_after=50
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-retrieving-twitter-lists" class="anchor" aria-hidden="true" href="#user-content-retrieving-twitter-lists"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Retrieving Twitter lists</h2>
  <p dir="auto">The <code>lists</code> command retrieves all of the lists belonging to one or more users.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite lists lists.db simonw dogsheep"><pre><code>$ twitter-to-sqlite lists lists.db simonw dogsheep
  </code></pre></div>
  <p dir="auto">This command also accepts the <code>--sql</code> and <code>--attach</code> and <code>--ids</code> options.</p>
  <p dir="auto">To additionally fetch the list of members for each list, use <code>--members</code>.</p>
  <h2 dir="auto"><a id="user-content-retrieving-twitter-list-memberships" class="anchor" aria-hidden="true" href="#user-content-retrieving-twitter-list-memberships"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Retrieving Twitter list memberships</h2>
  <p dir="auto">The <code>list-members</code> command can be used to retrieve details of one or more Twitter lists, including all of their members.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite list-members members.db simonw/the-good-place"><pre><code>$ twitter-to-sqlite list-members members.db simonw/the-good-place
  </code></pre></div>
  <p dir="auto">You can pass multiple <code>screen_name/list_slug</code> identifiers.</p>
  <p dir="auto">If you know the numeric IDs of the lists instead, you can use <code>--ids</code>:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite list-members members.db 927913322841653248 --ids"><pre><code>$ twitter-to-sqlite list-members members.db 927913322841653248 --ids
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-retrieving-just-follower-and-friend-ids" class="anchor" aria-hidden="true" href="#user-content-retrieving-just-follower-and-friend-ids"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Retrieving just follower and friend IDs</h2>
  <p dir="auto">It's also possible to retrieve just the numeric Twitter IDs of the accounts that specific users are following ("friends" in Twitter's API terminology) or followed-by:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite followers-ids members.db simonw cleopaws"><pre><code>$ twitter-to-sqlite followers-ids members.db simonw cleopaws
  </code></pre></div>
  <p dir="auto">This will populate the <code>following</code> table with <code>followed_id</code>/<code>follower_id</code> pairs for the two specified accounts, listing every account ID that is following either of those two accounts.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite friends-ids members.db simonw cleopaws"><pre><code>$ twitter-to-sqlite friends-ids members.db simonw cleopaws
  </code></pre></div>
  <p dir="auto">This will do the same thing but pull the IDs that those accounts are following.</p>
  <p dir="auto">Both of these commands also support <code>--sql</code> and <code>--attach</code> as an alternative to passing screen names as direct command-line arguments. You can use <code>--ids</code> to process the inputs as user IDs rather than screen names.</p>
  <p dir="auto">The underlying Twitter APIs have a rate limit of 15 requests every 15 minutes - though they do return up to 5,000 IDs in each call. By default both of these subcommands will wait for 61 seconds between API calls in order to stay within the rate limit - you can adjust this behaviour down to just one second delay if you know you will not be making many calls using <code>--sleep=1</code>.</p>
  <h2 dir="auto"><a id="user-content-retrieving-tweets-from-your-home-timeline" class="anchor" aria-hidden="true" href="#user-content-retrieving-tweets-from-your-home-timeline"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Retrieving tweets from your home timeline</h2>
  <p dir="auto">The <code>home-timeline</code> command retrieves up to 800 tweets from the home timeline of the authenticated user - generally this means tweets from people you follow.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite home-timeline twitter.db
  Importing timeline  [#################--------]  591/800  00:01:14"><pre><code>$ twitter-to-sqlite home-timeline twitter.db
  Importing timeline  [#################--------]  591/800  00:01:14
  </code></pre></div>
  <p dir="auto">The tweets are stored in the <code>tweets</code> table, and a record is added to the <code>timeline_tweets</code> table noting that this tweet came in due to being spotted in the timeline of your user.</p>
  <p dir="auto">You can use <code>--since</code> to retrieve just tweets that have been posted since the last time this command was run, or <code>--since_id=xxx</code> to explicitly pass in a tweet ID to use as the last position.</p>
  <p dir="auto">You can then view your timeline in Datasette using the following URL:</p>
  <p dir="auto"><code>/tweets/tweets?_where=id+in+(select+tweet+from+[timeline_tweets])&amp;_sort_desc=id&amp;_facet=user</code></p>
  <p dir="auto">This will filter your tweets table to just tweets that appear in your timeline, ordered by most recent first and use faceting to show you which users are responsible for the most tweets.</p>
  <h2 dir="auto"><a id="user-content-retrieving-your-mentions" class="anchor" aria-hidden="true" href="#user-content-retrieving-your-mentions"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Retrieving your mentions</h2>
  <p dir="auto">The <code>mentions-timeline</code> command works like <code>home-timeline</code> except it retrieves tweets that mention the authenticated user's account. It records the user account that was mentioned in a <code>mentions_tweets</code> table.</p>
  <p dir="auto">It supports <code>--since</code> and <code>--since_id</code> in the same was as <code>home-timeline</code> does.</p>
  <h2 dir="auto"><a id="user-content-providing-input-from-a-sql-query-with---sql-and---attach" class="anchor" aria-hidden="true" href="#user-content-providing-input-from-a-sql-query-with---sql-and---attach"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Providing input from a SQL query with --sql and --attach</h2>
  <p dir="auto">This option is available for some subcommands - run <code>twitter-to-sqlite command-name --help</code> to check.</p>
  <p dir="auto">You can provide Twitter screen names (or user IDs or tweet IDs) directly as command-line arguments, or you can provide those screen names or IDs by executing a SQL query.</p>
  <p dir="auto">For example: consider a SQLite database with an <code>attendees</code> table listing names and Twitter accounts - something like this:</p>
  <table>
  <thead>
  <tr>
  <th>First</th>
  <th>Last</th>
  <th>Twitter</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Simon</td>
  <td>Willison</td>
  <td>simonw</td>
  </tr>
  <tr>
  <td>Avril</td>
  <td>Lavigne</td>
  <td>AvrilLavigne</td>
  </tr>
  </tbody>
  </table>
  <p dir="auto">You can run the <code>users-lookup</code> command to pull the Twitter profile of every user listed in that database by loading the screen names using a <code>--sql</code> query:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite users-lookup my.db --sql=&quot;select Twitter from attendees&quot;"><pre><code>$ twitter-to-sqlite users-lookup my.db --sql="select Twitter from attendees"
  </code></pre></div>
  <p dir="auto">If your database table contains Twitter IDs, you can select those IDs and pass the <code>--ids</code> argument. For example, to fetch the profiles of users who have had their user IDs inserted into the <code>following</code> table using the <code>twitter-to-sqlite friends-ids</code> command:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite users-lookup my.db --sql=&quot;select follower_id from following&quot; --ids"><pre><code>$ twitter-to-sqlite users-lookup my.db --sql="select follower_id from following" --ids
  </code></pre></div>
  <p dir="auto">Or to avoid re-fetching users that have already been fetched:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite users-lookup my.db \
      --sql=&quot;select followed_id from following where followed_id not in (
          select id from users)&quot; --ids"><pre><code>$ twitter-to-sqlite users-lookup my.db \
      --sql="select followed_id from following where followed_id not in (
          select id from users)" --ids
  </code></pre></div>
  <p dir="auto">If your data lives in a separate database file you can attach it using <code>--attach</code>. For example, consider the attendees example above but the data lives in an <code>attendees.db</code> file, and you want to fetch the user profiles into a <code>tweets.db</code> file. You could do that like this:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite users-lookup tweets.db \
      --attach=attendees.db \
      --sql=&quot;select Twitter from attendees.attendees&quot;"><pre><code>$ twitter-to-sqlite users-lookup tweets.db \
      --attach=attendees.db \
      --sql="select Twitter from attendees.attendees"
  </code></pre></div>
  <p dir="auto">The filename (without the extension) will be used as the database alias within SQLite. If you want a different alias for some reason you can specify that with a colon like this:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite users-lookup tweets.db \
      --attach=foo:attendees.db \
      --sql=&quot;select Twitter from foo.attendees&quot;"><pre><code>$ twitter-to-sqlite users-lookup tweets.db \
      --attach=foo:attendees.db \
      --sql="select Twitter from foo.attendees"
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-running-searches" class="anchor" aria-hidden="true" href="#user-content-running-searches"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Running searches</h2>
  <p dir="auto">The <code>search</code> command runs a search against the Twitter <a href="https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets" rel="nofollow">standard search API</a>.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite search tweets.db &quot;dogsheep&quot;"><pre><code>$ twitter-to-sqlite search tweets.db "dogsheep"
  </code></pre></div>
  <p dir="auto">This will import up to around 320 tweets that match that search term into the <code>tweets</code> table. It will also create a record in the <code>search_runs</code> table recording that the search took place, and many-to-many records in the <code>search_runs_tweets</code> table recording which tweets were seen for that search at that time.</p>
  <p dir="auto">You can use the <code>--since</code> parameter to check for previous search runs with the same arguments and only retrieve tweets that were posted since the last retrieved matching tweet.</p>
  <p dir="auto">The following additional options for <code>search</code> are supported:</p>
  <ul dir="auto">
  <li><code>--geocode</code>: <code>latitude,longitude,radius</code> where radius is a number followed by mi or km</li>
  <li><code>--lang</code>: ISO 639-1 language code e.g. <code>en</code> or <code>es</code></li>
  <li><code>--locale</code>: Locale: only <code>ja</code> is currently effective</li>
  <li><code>--result_type</code>: <code>mixed</code>, <code>recent</code> or <code>popular</code>. Defaults to <code>mixed</code></li>
  <li><code>--count</code>: Number of results per page, defaults to the maximum of 100</li>
  <li><code>--stop_after</code>: Stop after this many results</li>
  <li><code>--since_id</code>: Pull tweets since this Tweet ID. You probably want to use <code>--since</code> instead of this.</li>
  </ul>
  <h2 dir="auto"><a id="user-content-capturing-tweets-in-real-time-with-track-and-follow" class="anchor" aria-hidden="true" href="#user-content-capturing-tweets-in-real-time-with-track-and-follow"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Capturing tweets in real-time with track and follow</h2>
  <p dir="auto">This functionality is <strong>experimental</strong>. Please <a href="https://github.com/dogsheep/twitter-to-sqlite/issues">file bug reports</a> if you find any!</p>
  <p dir="auto">Twitter provides a real-time API which can be used to subscribe to tweets as they happen. <code>twitter-to-sqlite</code> can use this API to continually update a SQLite database with tweets matching certain keywords, or referencing specific users.</p>
  <h3 dir="auto"><a id="user-content-track" class="anchor" aria-hidden="true" href="#user-content-track"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>track</h3>
  <p dir="auto">To track keywords, use the <code>track</code> command:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite track tweets.db kakapo"><pre><code>$ twitter-to-sqlite track tweets.db kakapo
  </code></pre></div>
  <p dir="auto">This command will continue to run until you hit Ctrl+C. It will capture any tweets mentioning the keyword <a href="https://en.wikipedia.org/wiki/Kakapo" rel="nofollow">kakapo</a> and store them in the <code>tweets.db</code> database file.</p>
  <p dir="auto">You can pass multiple keywords as a space separated list. This will capture tweets matching either of those keywords:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite track tweets.db kakapo raccoon"><pre><code>$ twitter-to-sqlite track tweets.db kakapo raccoon
  </code></pre></div>
  <p dir="auto">You can enclose phrases in quotes to search for tweets matching both of those keywords:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite track tweets.db 'trash panda'"><pre><code>$ twitter-to-sqlite track tweets.db 'trash panda'
  </code></pre></div>
  <p dir="auto">See <a href="https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/basic-stream-parameters#track" rel="nofollow">the Twitter track documentation</a> for advanced tips on using this command.</p>
  <p dir="auto">Add the <code>--verbose</code> option to see matching tweets (in their verbose JSON form) displayed to the terminal as they are captured:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite track tweets.db raccoon --verbose"><pre><code>$ twitter-to-sqlite track tweets.db raccoon --verbose
  </code></pre></div>
  <h3 dir="auto"><a id="user-content-follow" class="anchor" aria-hidden="true" href="#user-content-follow"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>follow</h3>
  <p dir="auto">The <code>follow</code> command will capture all tweets that are relevant to one or more specific Twitter users.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite follow tweets.db nytimes"><pre><code>$ twitter-to-sqlite follow tweets.db nytimes
  </code></pre></div>
  <p dir="auto">This includes tweets by those users, tweets that reply to or quote those users and retweets by that user. See <a href="https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/basic-stream-parameters#follow" rel="nofollow">the Twitter follow documentation</a> for full details.</p>
  <p dir="auto">The command accepts one or more screen names.</p>
  <p dir="auto">You can feed it numeric Twitter user IDs instead of screen names by using the <code>--ids</code> flag.</p>
  <p dir="auto">The command also supports the <code>--sql</code> and <code>--attach</code> options, and the <code>--verbose</code> option for displaying tweets as they are captured.</p>
  <p dir="auto">Here's how to start following tweets from every user ID currently represented as being followed in the <code>following</code> table (populated using the <code>friends-ids</code> command):</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite follow tweets.db \
      --sql=&quot;select distinct followed_id from following&quot; \
      --ids"><pre><code>$ twitter-to-sqlite follow tweets.db \
      --sql="select distinct followed_id from following" \
      --ids
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-importing-data-from-your-twitter-archive" class="anchor" aria-hidden="true" href="#user-content-importing-data-from-your-twitter-archive"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Importing data from your Twitter archive</h2>
  <p dir="auto">You can request an archive of your Twitter data by <a href="https://help.twitter.com/en/managing-your-account/how-to-download-your-twitter-archive" rel="nofollow">following these instructions</a>.</p>
  <p dir="auto">Twitter will send you a link to download a <code>.zip</code> file. You can import the contents of that file into a set of tables in a new database file called <code>archive.db</code> (each table beginning with the <code>archive_</code> prefix) using the <code>import</code> command:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2.zip"><pre><code>$ twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2.zip
  </code></pre></div>
  <p dir="auto">This command does not populate any of the regular tables, since Twitter's export data does not exactly match the schema returned by the Twitter API.</p>
  <p dir="auto">It will delete and recreate the corresponding <code>archive_*</code> tables every time you run it. If this is not what you want, run the command against a new SQLite database file name rather than running it against one that already exists.</p>
  <p dir="auto">If you have already decompressed your archive, you can run this against the directory that you decompressed it to:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2/"><pre><code>$ twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2/
  </code></pre></div>
  <p dir="auto">You can also run it against one or more specific files within that folder. For example, to import just the follower.js and following.js files:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite import archive.db \
      ~/Downloads/twitter-2019-06-25-b31f2/follower.js \
      ~/Downloads/twitter-2019-06-25-b31f2/following.js"><pre><code>$ twitter-to-sqlite import archive.db \
      ~/Downloads/twitter-2019-06-25-b31f2/follower.js \
      ~/Downloads/twitter-2019-06-25-b31f2/following.js
  </code></pre></div>
  <p dir="auto">You may want to use other commands to populate tables based on data from the archive. For example, to retrieve full API versions of each of the tweets you have favourited in your archive, you could run the following:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite statuses-lookup archive.db \
      --sql='select tweetId from archive_like' \
      --skip-existing"><pre><code>$ twitter-to-sqlite statuses-lookup archive.db \
      --sql='select tweetId from archive_like' \
      --skip-existing
  </code></pre></div>
  <p dir="auto">If you want these imported tweets to then be reflected in the <code>favorited_by</code> table, you can do so by applying the following SQL query:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ sqlite3 archive.db
  SQLite version 3.22.0 2018-01-22 18:45:57
  Enter &quot;.help&quot; for usage hints.
  sqlite&gt; INSERT OR IGNORE INTO favorited_by (tweet, user)
     ...&gt;     SELECT tweetId, 'YOUR_TWITTER_ID' FROM archive_like;
  &lt;Ctrl+D&gt;"><pre><code>$ sqlite3 archive.db
  SQLite version 3.22.0 2018-01-22 18:45:57
  Enter ".help" for usage hints.
  sqlite&gt; INSERT OR IGNORE INTO favorited_by (tweet, user)
     ...&gt;     SELECT tweetId, 'YOUR_TWITTER_ID' FROM archive_like;
  &lt;Ctrl+D&gt;
  </code></pre></div>
  <p dir="auto">Replace YOUR_TWITTER_ID with your numeric Twitter ID. If you don't know that ID you can find it out by running the following:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ twitter-to-sqlite fetch \
      &quot;https://api.twitter.com/1.1/account/verify_credentials.json&quot; \
      | grep '&quot;id&quot;' | head -n 1"><pre><code>$ twitter-to-sqlite fetch \
      "https://api.twitter.com/1.1/account/verify_credentials.json" \
      | grep '"id"' | head -n 1
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-design-notes" class="anchor" aria-hidden="true" href="#user-content-design-notes"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Design notes</h2>
  <ul dir="auto">
  <li>Tweet IDs are stored as integers, to afford sorting by ID in a sensible way</li>
  <li>While we configure foreign key relationships between tables, we do not ask SQLite to enforce them. This is used by the <code>following</code> table to allow the <code>followers-ids</code> and <code>friends-ids</code> commands to populate it with user IDs even if the user accounts themselves are not yet present in the <code>users</code> table.</li>
  </ul>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMDYyMDI4NjQ=
name: inaturalist-to-sqlite
full_name: dogsheep/inaturalist-to-sqlite
html_url: https://github.com/dogsheep/inaturalist-to-sqlite
description: Create a SQLite database containing your observation history from iNaturalist
created_at: 2019-09-04T01:21:21Z
updated_at: 2020-12-19T05:18:38Z
pushed_at: 2020-10-22T00:08:58Z
language: Python
license: apache-2.0
topics: ["sqlite", "inaturalist", "datasette", "dogsheep", "datasette-io", "datasette-tool"]
default_branch: master
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # inaturalist-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/inaturalist-to-sqlite.svg)](https://pypi.org/project/inaturalist-to-sqlite/)
  [![CircleCI](https://circleci.com/gh/dogsheep/inaturalist-to-sqlite.svg?style=svg)](https://circleci.com/gh/dogsheep/inaturalist-to-sqlite)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/inaturalist-to-sqlite/blob/master/LICENSE)

  Create a SQLite database containing your observation history from [iNaturalist](https://www.inaturalist.org/).

  ## How to install

      $ pip install inaturalist-to-sqlite

  ## Usage

      $ inaturalist-to-sqlite inaturalist.db yourusername

  (Or try `simonw` if you don't yet have an iNaturalist account)

  This will import all of your iNaturalist observations into a SQLite database called `inaturalist.db`.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-inaturalist-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-inaturalist-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>inaturalist-to-sqlite</h1>
  <p><a href="https://pypi.org/project/inaturalist-to-sqlite/" rel="nofollow"><img src="https://camo.githubusercontent.com/0b4aee6bb6f3aeb706c5195fce3537b66445f052f38021aefeb48672cf06cf74/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f696e61747572616c6973742d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/inaturalist-to-sqlite.svg" style="max-width:100%;"></a>
  <a href="https://circleci.com/gh/dogsheep/inaturalist-to-sqlite" rel="nofollow"><img src="https://camo.githubusercontent.com/96dcb5c0cfa03bf010686f57c71dfef278ab6aaa53eb39cb48d28bde427c55b7/68747470733a2f2f636972636c6563692e636f6d2f67682f646f6773686565702f696e61747572616c6973742d746f2d73716c6974652e7376673f7374796c653d737667" alt="CircleCI" data-canonical-src="https://circleci.com/gh/dogsheep/inaturalist-to-sqlite.svg?style=svg" style="max-width:100%;"></a>
  <a href="https://github.com/dogsheep/inaturalist-to-sqlite/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Create a SQLite database containing your observation history from <a href="https://www.inaturalist.org/" rel="nofollow">iNaturalist</a>.</p>
  <h2><a id="user-content-how-to-install" class="anchor" aria-hidden="true" href="#user-content-how-to-install"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How to install</h2>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ pip install inaturalist-to-sqlite
  "><pre><code>$ pip install inaturalist-to-sqlite
  </code></pre></div>
  <h2><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ inaturalist-to-sqlite inaturalist.db yourusername
  "><pre><code>$ inaturalist-to-sqlite inaturalist.db yourusername
  </code></pre></div>
  <p>(Or try <code>simonw</code> if you don't yet have an iNaturalist account)</p>
  <p>This will import all of your iNaturalist observations into a SQLite database called <code>inaturalist.db</code>.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMDY2NDk3NzA=
name: google-takeout-to-sqlite
full_name: dogsheep/google-takeout-to-sqlite
html_url: https://github.com/dogsheep/google-takeout-to-sqlite
description: Save data from Google Takeout to a SQLite database
created_at: 2019-09-05T20:15:15Z
updated_at: 2021-06-08T15:31:47Z
pushed_at: 2021-02-24T00:34:55Z
language: Python
license: apache-2.0
topics: ["google", "sqlite", "datasette", "dogsheep", "datasette-io", "datasette-tool"]
default_branch: master
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # google-takeout-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/google-takeout-to-sqlite.svg)](https://pypi.org/project/google-takeout-to-sqlite/)
  [![CircleCI](https://circleci.com/gh/dogsheep/google-takeout-to-sqlite.svg?style=svg)](https://circleci.com/gh/dogsheep/google-takeout-to-sqlite)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/google-takeout-to-sqlite/blob/master/LICENSE)

  Save data from google-takeout to a SQLite database.

  ## How to install

      $ pip install google-takeout-to-sqlite

  Request your Google data from https://takeout.google.com/ - wait for the email and download the zip file.

  This tool only supports a subset of the available options. More will be added over time.

  ## My Activity

  You can request the "My Activity" export and then import it with the following command:

      $ google-takeout-to-sqlite my-activity takeout.db ~/Downloads/takeout-20190530.zip

  This will create a database file called `takeout.db` if one does not already exist.

  ## Location History

  Your location history records latitude, longitude and timestame for where Google has tracked your location. You can import it using this command:

      $ google-takeout-to-sqlite location-history takeout.db ~/Downloads/takeout-20190530.zip

  ## Browsing your data with Datasette

  Once you have imported Google data into a SQLite database file you can browse your data using [Datasette](https://github.com/simonw/datasette). Install Datasette like so:

      $ pip install datasette

  Now browse your data by running this and then visiting `http://localhost:8001/`

      $ datasette takeout.db

  Install the [datasette-cluster-map](https://github.com/simonw/datasette-cluster-map) plugin to see your location history on a map:

      $ pip install datasette-cluster-map

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-google-takeout-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-google-takeout-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>google-takeout-to-sqlite</h1>
  <p><a href="https://pypi.org/project/google-takeout-to-sqlite/" rel="nofollow"><img src="https://camo.githubusercontent.com/fb9e0c4c7734b2904f2b1aaaa4ce05fa195167989240be80a7c8e56e2b2a000e/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f676f6f676c652d74616b656f75742d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/google-takeout-to-sqlite.svg" style="max-width:100%;"></a>
  <a href="https://circleci.com/gh/dogsheep/google-takeout-to-sqlite" rel="nofollow"><img src="https://camo.githubusercontent.com/41bc34953e618d9a884ec9a8bad7d759bf580fc6b7c27982d3c4af4531121246/68747470733a2f2f636972636c6563692e636f6d2f67682f646f6773686565702f676f6f676c652d74616b656f75742d746f2d73716c6974652e7376673f7374796c653d737667" alt="CircleCI" data-canonical-src="https://circleci.com/gh/dogsheep/google-takeout-to-sqlite.svg?style=svg" style="max-width:100%;"></a>
  <a href="https://github.com/dogsheep/google-takeout-to-sqlite/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Save data from google-takeout to a SQLite database.</p>
  <h2><a id="user-content-how-to-install" class="anchor" aria-hidden="true" href="#user-content-how-to-install"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How to install</h2>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ pip install google-takeout-to-sqlite
  "><pre><code>$ pip install google-takeout-to-sqlite
  </code></pre></div>
  <p>Request your Google data from <a href="https://takeout.google.com/" rel="nofollow">https://takeout.google.com/</a> - wait for the email and download the zip file.</p>
  <p>This tool only supports a subset of the available options. More will be added over time.</p>
  <h2><a id="user-content-my-activity" class="anchor" aria-hidden="true" href="#user-content-my-activity"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>My Activity</h2>
  <p>You can request the "My Activity" export and then import it with the following command:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ google-takeout-to-sqlite my-activity takeout.db ~/Downloads/takeout-20190530.zip
  "><pre><code>$ google-takeout-to-sqlite my-activity takeout.db ~/Downloads/takeout-20190530.zip
  </code></pre></div>
  <p>This will create a database file called <code>takeout.db</code> if one does not already exist.</p>
  <h2><a id="user-content-location-history" class="anchor" aria-hidden="true" href="#user-content-location-history"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Location History</h2>
  <p>Your location history records latitude, longitude and timestame for where Google has tracked your location. You can import it using this command:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ google-takeout-to-sqlite location-history takeout.db ~/Downloads/takeout-20190530.zip
  "><pre><code>$ google-takeout-to-sqlite location-history takeout.db ~/Downloads/takeout-20190530.zip
  </code></pre></div>
  <h2><a id="user-content-browsing-your-data-with-datasette" class="anchor" aria-hidden="true" href="#user-content-browsing-your-data-with-datasette"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Browsing your data with Datasette</h2>
  <p>Once you have imported Google data into a SQLite database file you can browse your data using <a href="https://github.com/simonw/datasette">Datasette</a>. Install Datasette like so:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ pip install datasette
  "><pre><code>$ pip install datasette
  </code></pre></div>
  <p>Now browse your data by running this and then visiting <code>http://localhost:8001/</code></p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ datasette takeout.db
  "><pre><code>$ datasette takeout.db
  </code></pre></div>
  <p>Install the <a href="https://github.com/simonw/datasette-cluster-map">datasette-cluster-map</a> plugin to see your location history on a map:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ pip install datasette-cluster-map
  "><pre><code>$ pip install datasette-cluster-map
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMDcwNTI4ODI=
name: github-to-sqlite
full_name: dogsheep/github-to-sqlite
html_url: https://github.com/dogsheep/github-to-sqlite
description: Save data from GitHub to a SQLite database
created_at: 2019-09-08T02:50:28Z
updated_at: 2022-09-20T04:36:37Z
pushed_at: 2022-09-28T21:07:54Z
homepage: https://github-to-sqlite.dogsheep.net/
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-tool", "dogsheep", "github-api", "sqlite"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # github-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/github-to-sqlite.svg)](https://pypi.org/project/github-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/github-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/github-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/github-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/github-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/github-to-sqlite/blob/main/LICENSE)

  Save data from GitHub to a SQLite database.

  <!-- toc -->

  - [Demo](#demo)
  - [How to install](#how-to-install)
  - [Authentication](#authentication)
  - [Fetching issues for a repository](#fetching-issues-for-a-repository)
  - [Fetching pull requests for a repository](#fetching-pull-requests-for-a-repository)
  - [Fetching issue comments for a repository](#fetching-issue-comments-for-a-repository)
  - [Fetching commits for a repository](#fetching-commits-for-a-repository)
  - [Fetching releases for a repository](#fetching-releases-for-a-repository)
  - [Fetching tags for a repository](#fetching-tags-for-a-repository)
  - [Fetching contributors to a repository](#fetching-contributors-to-a-repository)
  - [Fetching repos belonging to a user or organization](#fetching-repos-belonging-to-a-user-or-organization)
  - [Fetching specific repositories](#fetching-specific-repositories)
  - [Fetching repos that have been starred by a user](#fetching-repos-that-have-been-starred-by-a-user)
  - [Fetching users that have starred specific repos](#fetching-users-that-have-starred-specific-repos)
  - [Fetching GitHub Actions workflows](#fetching-github-actions-workflows)
  - [Scraping dependents for a repository](#scraping-dependents-for-a-repository)
  - [Fetching emojis](#fetching-emojis)
  - [Making authenticated API calls](#making-authenticated-api-calls)

  <!-- tocstop -->

  ## Demo

  https://github-to-sqlite.dogsheep.net/ hosts a [Datasette](https://datasette.io/) demo of a database created by [running this tool](https://github.com/dogsheep/github-to-sqlite/blob/main/.github/workflows/deploy-demo.yml#L40-L60) against all of the repositories in the [Dogsheep GitHub organization](https://github.com/dogsheep), plus the [datasette](https://github.com/simonw/datasette) and [sqlite-utils](https://github.com/simonw/sqlite-utils) repositories.

  ## How to install

      $ pip install github-to-sqlite

  ## Authentication

  Create a GitHub personal access token: https://github.com/settings/tokens

  Run this command and paste in your new token:

      $ github-to-sqlite auth

  This will create a file called `auth.json` in your current directory containing the required value. To save the file at a different path or filename, use the `--auth=myauth.json` option.

  As an alternative to using an `auth.json` file you can add your access token to an environment variable called `GITHUB_TOKEN`.

  ## Fetching issues for a repository

  The `issues` command retrieves all of the issues belonging to a specified repository.

      $ github-to-sqlite issues github.db simonw/datasette

  If an `auth.json` file is present it will use the token from that file. It works without authentication for public repositories but you should be aware that GitHub have strict IP-based rate limits for unauthenticated requests.

  You can point to a different location of `auth.json` using `-a`:

      $ github-to-sqlite issues github.db simonw/datasette -a /path/to/auth.json

  You can use the `--issue` option one or more times to load specific issues:

      $ github-to-sqlite issues github.db simonw/datasette --issue=1

  Example: [issues table](https://github-to-sqlite.dogsheep.net/github/issues)

  ## Fetching pull requests for a repository

  While pull requests are a type of issue, you will get more information on pull requests by pulling them separately. For example, whether a pull request has been merged and when.

  Following the API of issues, the `pull-requests` command retrieves all of the pull requests belonging to a specified repository.

      $ github-to-sqlite pull-requests github.db simonw/datasette

  You can use the `--pull-request` option one or more times to load specific pull request:

      $ github-to-sqlite pull-requests github.db simonw/datasette --pull-request=81

  Note that the `merged_by` column on the `pull_requests` table will only be populated for pull requests that are loaded using the `--pull-request` option - the GitHub API does not return this field for pull requests that are loaded in bulk.

  Example: [pull_requests table](https://github-to-sqlite.dogsheep.net/github/pull_requests)

  ## Fetching issue comments for a repository

  The `issue-comments` command retrieves all of the comments on all of the issues in a repository.

  It is recommended you run `issues` first, so that each imported comment can have a foreign key poining to its issue.

      $ github-to-sqlite issues github.db simonw/datasette
      $ github-to-sqlite issue-comments github.db simonw/datasette

  You can use the `--issue` option to only load comments for a specific issue within that repository, for example:

      $ github-to-sqlite issue-comments github.db simonw/datasette --issue=1

  Example: [issue_comments table](https://github-to-sqlite.dogsheep.net/github/issue_comments)

  ## Fetching commits for a repository

  The `commits` command retrieves details of all of the commits for one or more repositories. It currently fetches the sha, commit message and author and committer details - it does no retrieve the full commit body.

      $ github-to-sqlite commits github.db simonw/datasette simonw/sqlite-utils

  The command accepts one or more repositories.

  By default it will stop as soon as it sees a commit that has previously been retrieved. You can force it to retrieve all commits (including those that have been previously inserted) using `--all`.

  Example: [commits table](https://github-to-sqlite.dogsheep.net/github/commits)

  ## Fetching releases for a repository

  The `releases` command retrieves the releases for one or more repositories.

      $ github-to-sqlite releases github.db simonw/datasette simonw/sqlite-utils

  The command accepts one or more repositories.

  Example: [releases table](https://github-to-sqlite.dogsheep.net/github/releases)

  ## Fetching tags for a repository

  The `tags` command retrieves all of the tags for one or more repositories.

      $ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils

  Example: [tags table](https://github-to-sqlite.dogsheep.net/github/tags)

  ## Fetching contributors to a repository

  The `contributors` command retrieves details of all of the contributors for one or more repositories.

      $ github-to-sqlite contributors github.db simonw/datasette simonw/sqlite-utils

  The command accepts one or more repositories. It populates a `contributors` table, with foreign keys to `repos` and `users` and a `contributions` table listing the number of commits to that repository for each contributor.

  Example: [contributors table](https://github-to-sqlite.dogsheep.net/github/contributors)

  ## Fetching repos belonging to a user or organization

  The `repos` command fetches repos belonging to a user or organization.

  Without any other arguments, this command will fetch all repos that the currently authenticated user owns, collaborates on or can access via one of their organizations:

      $ github-to-sqlite repos github.db

  To fetch repos belonging to a specific user or organization, provide their username as an argument:

      $ github-to-sqlite repos github.db dogsheep # organization
      $ github-to-sqlite repos github.db simonw # user

  You can pass more than one username to fetch for multiple users or organizations at once:

      $ github-to-sqlite repos github.db simonw dogsheep

  Add the `--readme` option to save the README for the repo in a column called `readme`. Add `--readme-html` to save the HTML rendered version of the README into a collumn called `readme_html`.

  Example: [repos table](https://github-to-sqlite.dogsheep.net/github/repos)

  ## Fetching specific repositories

  You can use `-r` with the `repos` command one or more times to fetch just specific repositories.

      $ github-to-sqlite repos github.db -r simonw/datasette -r dogsheep/github-to-sqlite

  ## Fetching repos that have been starred by a user

  The `starred` command fetches the repos that have been starred by a user.

      $ github-to-sqlite starred github.db simonw

  If you are using an `auth.json` file you can omit the username to retrieve the starred repos for the authenticated user.

  Example: [stars table](https://github-to-sqlite.dogsheep.net/github/stars)

  ## Fetching users that have starred specific repos

  The `stargazers` command fetches the users that have starred the specified repos.

      $ github-to-sqlite stargazers github.db simonw/datasette dogsheep/github-to-sqlite

  You can specify one or more repository using `owner/repo` syntax.

  Users fetched using this command will be inserted into the `users` table. Many-to-many records showing which repository they starred will be added to the `stars` table.

  ## Fetching GitHub Actions workflows

  The `workflows` command fetches the YAML workflow configurations from each repository's `.github/workflows` directory and parses them to populate `workflows`, `jobs` and `steps` tables.

      $ github-to-sqlite workflows github.db simonw/datasette dogsheep/github-to-sqlite

  You can specify one or more repository using `owner/repo` syntax.

  Example: [workflows table](https://github-to-sqlite.dogsheep.net/github/workflows), [jobs table](https://github-to-sqlite.dogsheep.net/github/jobs), [steps table](https://github-to-sqlite.dogsheep.net/github/steps)

  ## Scraping dependents for a repository

  The GitHub dependency graph can show other GitHub projects that depend on a specific repo, for example [simonw/datasette/network/dependents](https://github.com/simonw/datasette/network/dependents).

  This data is not yet available through the GitHub API. The `scrape-dependents` command scrapes those pages and uses the GitHub API to load full versions of the dependent repositories.

      $ github-to-sqlite scrape-dependents github.db simonw/datasette

  The command accepts one or more repositories.

  Add `-v` for verbose output.

  Example: [dependents table](https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc)

  ## Fetching emojis

  You can fetch a list of every emoji supported by GitHub using the `emojis` command:

      $ github-to-sqlite emojis github.db

  This will create a table callad `emojis` with a primary key `name` and a `url` column.

  If you add the `--fetch` option the command will also fetch the binary content of the images and place them in an `image` column:

      $ github-to-sqlite emojis emojis.db -f
      [########----------------------------]  397/1799   22%  00:03:43

  You can then use the [datasette-render-images](https://github.com/simonw/datasette-render-images) plugin to browse them visually.

  Example: [emojis table](https://github-to-sqlite.dogsheep.net/github/emojis)

  ## Making authenticated API calls

  The `github-to-sqlite get` command provides a convenient shortcut for making authenticated calls to the API. Once you have created your `auth.json` file (or set a `GITHUB_TOKEN` environment variable) you can use it like this:

      $ github-to-sqlite get https://api.github.com/gists

  This will make an authenticated call to the URL you provide and pretty-print the resulting JSON to the console.

  You can ommit the `https://api.github.com/` prefix, for example:

      $ github-to-sqlite get /gists

  Many GitHub APIs are [paginated using the HTTP Link header](https://docs.github.com/en/rest/guides/traversing-with-pagination). You can follow this pagination and output a list of all of the resulting items using `--paginate`:

      $ github-to-sqlite get /users/simonw/repos --paginate

  You can outline newline-delimited JSON for each item using `--nl`. This can be useful for streaming items into another tool.

      $ github-to-sqlite get /users/simonw/repos --nl

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-github-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-github-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>github-to-sqlite</h1>
  <p dir="auto"><a href="https://pypi.org/project/github-to-sqlite/" rel="nofollow"><img src="https://camo.githubusercontent.com/515e6efa4aef15e83b08072e5490c9040420223b2355d1d715cb628f66d60dff/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6769746875622d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/github-to-sqlite.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/github-to-sqlite/releases"><img src="https://camo.githubusercontent.com/ae3fbf680cae0fca1e9126549e1fd0e14e756f6f77046e878db7b1c9cbd78911/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f646f6773686565702f6769746875622d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/dogsheep/github-to-sqlite?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/github-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/dogsheep/github-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/github-to-sqlite/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Save data from GitHub to a SQLite database.</p>

  <ul dir="auto">
  <li><a href="#user-content-demo">Demo</a></li>
  <li><a href="#user-content-how-to-install">How to install</a></li>
  <li><a href="#user-content-authentication">Authentication</a></li>
  <li><a href="#user-content-fetching-issues-for-a-repository">Fetching issues for a repository</a></li>
  <li><a href="#user-content-fetching-pull-requests-for-a-repository">Fetching pull requests for a repository</a></li>
  <li><a href="#user-content-fetching-issue-comments-for-a-repository">Fetching issue comments for a repository</a></li>
  <li><a href="#user-content-fetching-commits-for-a-repository">Fetching commits for a repository</a></li>
  <li><a href="#user-content-fetching-releases-for-a-repository">Fetching releases for a repository</a></li>
  <li><a href="#user-content-fetching-tags-for-a-repository">Fetching tags for a repository</a></li>
  <li><a href="#user-content-fetching-contributors-to-a-repository">Fetching contributors to a repository</a></li>
  <li><a href="#user-content-fetching-repos-belonging-to-a-user-or-organization">Fetching repos belonging to a user or organization</a></li>
  <li><a href="#user-content-fetching-specific-repositories">Fetching specific repositories</a></li>
  <li><a href="#user-content-fetching-repos-that-have-been-starred-by-a-user">Fetching repos that have been starred by a user</a></li>
  <li><a href="#user-content-fetching-users-that-have-starred-specific-repos">Fetching users that have starred specific repos</a></li>
  <li><a href="#user-content-fetching-github-actions-workflows">Fetching GitHub Actions workflows</a></li>
  <li><a href="#user-content-scraping-dependents-for-a-repository">Scraping dependents for a repository</a></li>
  <li><a href="#user-content-fetching-emojis">Fetching emojis</a></li>
  <li><a href="#user-content-making-authenticated-api-calls">Making authenticated API calls</a></li>
  </ul>

  <h2 dir="auto"><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p dir="auto"><a href="https://github-to-sqlite.dogsheep.net/" rel="nofollow">https://github-to-sqlite.dogsheep.net/</a> hosts a <a href="https://datasette.io/" rel="nofollow">Datasette</a> demo of a database created by <a href="https://github.com/dogsheep/github-to-sqlite/blob/main/.github/workflows/deploy-demo.yml#L40-L60">running this tool</a> against all of the repositories in the <a href="https://github.com/dogsheep">Dogsheep GitHub organization</a>, plus the <a href="https://github.com/simonw/datasette">datasette</a> and <a href="https://github.com/simonw/sqlite-utils">sqlite-utils</a> repositories.</p>
  <h2 dir="auto"><a id="user-content-how-to-install" class="anchor" aria-hidden="true" href="#user-content-how-to-install"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How to install</h2>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install github-to-sqlite"><pre class="notranslate"><code>$ pip install github-to-sqlite
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-authentication" class="anchor" aria-hidden="true" href="#user-content-authentication"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Authentication</h2>
  <p dir="auto">Create a GitHub personal access token: <a href="https://github.com/settings/tokens">https://github.com/settings/tokens</a></p>
  <p dir="auto">Run this command and paste in your new token:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite auth"><pre class="notranslate"><code>$ github-to-sqlite auth
  </code></pre></div>
  <p dir="auto">This will create a file called <code>auth.json</code> in your current directory containing the required value. To save the file at a different path or filename, use the <code>--auth=myauth.json</code> option.</p>
  <p dir="auto">As an alternative to using an <code>auth.json</code> file you can add your access token to an environment variable called <code>GITHUB_TOKEN</code>.</p>
  <h2 dir="auto"><a id="user-content-fetching-issues-for-a-repository" class="anchor" aria-hidden="true" href="#user-content-fetching-issues-for-a-repository"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Fetching issues for a repository</h2>
  <p dir="auto">The <code>issues</code> command retrieves all of the issues belonging to a specified repository.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite issues github.db simonw/datasette"><pre class="notranslate"><code>$ github-to-sqlite issues github.db simonw/datasette
  </code></pre></div>
  <p dir="auto">If an <code>auth.json</code> file is present it will use the token from that file. It works without authentication for public repositories but you should be aware that GitHub have strict IP-based rate limits for unauthenticated requests.</p>
  <p dir="auto">You can point to a different location of <code>auth.json</code> using <code>-a</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite issues github.db simonw/datasette -a /path/to/auth.json"><pre class="notranslate"><code>$ github-to-sqlite issues github.db simonw/datasette -a /path/to/auth.json
  </code></pre></div>
  <p dir="auto">You can use the <code>--issue</code> option one or more times to load specific issues:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite issues github.db simonw/datasette --issue=1"><pre class="notranslate"><code>$ github-to-sqlite issues github.db simonw/datasette --issue=1
  </code></pre></div>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/issues" rel="nofollow">issues table</a></p>
  <h2 dir="auto"><a id="user-content-fetching-pull-requests-for-a-repository" class="anchor" aria-hidden="true" href="#user-content-fetching-pull-requests-for-a-repository"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Fetching pull requests for a repository</h2>
  <p dir="auto">While pull requests are a type of issue, you will get more information on pull requests by pulling them separately. For example, whether a pull request has been merged and when.</p>
  <p dir="auto">Following the API of issues, the <code>pull-requests</code> command retrieves all of the pull requests belonging to a specified repository.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite pull-requests github.db simonw/datasette"><pre class="notranslate"><code>$ github-to-sqlite pull-requests github.db simonw/datasette
  </code></pre></div>
  <p dir="auto">You can use the <code>--pull-request</code> option one or more times to load specific pull request:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite pull-requests github.db simonw/datasette --pull-request=81"><pre class="notranslate"><code>$ github-to-sqlite pull-requests github.db simonw/datasette --pull-request=81
  </code></pre></div>
  <p dir="auto">Note that the <code>merged_by</code> column on the <code>pull_requests</code> table will only be populated for pull requests that are loaded using the <code>--pull-request</code> option - the GitHub API does not return this field for pull requests that are loaded in bulk.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/pull_requests" rel="nofollow">pull_requests table</a></p>
  <h2 dir="auto"><a id="user-content-fetching-issue-comments-for-a-repository" class="anchor" aria-hidden="true" href="#user-content-fetching-issue-comments-for-a-repository"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Fetching issue comments for a repository</h2>
  <p dir="auto">The <code>issue-comments</code> command retrieves all of the comments on all of the issues in a repository.</p>
  <p dir="auto">It is recommended you run <code>issues</code> first, so that each imported comment can have a foreign key poining to its issue.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite issues github.db simonw/datasette
  $ github-to-sqlite issue-comments github.db simonw/datasette"><pre class="notranslate"><code>$ github-to-sqlite issues github.db simonw/datasette
  $ github-to-sqlite issue-comments github.db simonw/datasette
  </code></pre></div>
  <p dir="auto">You can use the <code>--issue</code> option to only load comments for a specific issue within that repository, for example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite issue-comments github.db simonw/datasette --issue=1"><pre class="notranslate"><code>$ github-to-sqlite issue-comments github.db simonw/datasette --issue=1
  </code></pre></div>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/issue_comments" rel="nofollow">issue_comments table</a></p>
  <h2 dir="auto"><a id="user-content-fetching-commits-for-a-repository" class="anchor" aria-hidden="true" href="#user-content-fetching-commits-for-a-repository"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Fetching commits for a repository</h2>
  <p dir="auto">The <code>commits</code> command retrieves details of all of the commits for one or more repositories. It currently fetches the sha, commit message and author and committer details - it does no retrieve the full commit body.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite commits github.db simonw/datasette simonw/sqlite-utils"><pre class="notranslate"><code>$ github-to-sqlite commits github.db simonw/datasette simonw/sqlite-utils
  </code></pre></div>
  <p dir="auto">The command accepts one or more repositories.</p>
  <p dir="auto">By default it will stop as soon as it sees a commit that has previously been retrieved. You can force it to retrieve all commits (including those that have been previously inserted) using <code>--all</code>.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/commits" rel="nofollow">commits table</a></p>
  <h2 dir="auto"><a id="user-content-fetching-releases-for-a-repository" class="anchor" aria-hidden="true" href="#user-content-fetching-releases-for-a-repository"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Fetching releases for a repository</h2>
  <p dir="auto">The <code>releases</code> command retrieves the releases for one or more repositories.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite releases github.db simonw/datasette simonw/sqlite-utils"><pre class="notranslate"><code>$ github-to-sqlite releases github.db simonw/datasette simonw/sqlite-utils
  </code></pre></div>
  <p dir="auto">The command accepts one or more repositories.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/releases" rel="nofollow">releases table</a></p>
  <h2 dir="auto"><a id="user-content-fetching-tags-for-a-repository" class="anchor" aria-hidden="true" href="#user-content-fetching-tags-for-a-repository"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Fetching tags for a repository</h2>
  <p dir="auto">The <code>tags</code> command retrieves all of the tags for one or more repositories.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils"><pre class="notranslate"><code>$ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils
  </code></pre></div>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/tags" rel="nofollow">tags table</a></p>
  <h2 dir="auto"><a id="user-content-fetching-contributors-to-a-repository" class="anchor" aria-hidden="true" href="#user-content-fetching-contributors-to-a-repository"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Fetching contributors to a repository</h2>
  <p dir="auto">The <code>contributors</code> command retrieves details of all of the contributors for one or more repositories.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite contributors github.db simonw/datasette simonw/sqlite-utils"><pre class="notranslate"><code>$ github-to-sqlite contributors github.db simonw/datasette simonw/sqlite-utils
  </code></pre></div>
  <p dir="auto">The command accepts one or more repositories. It populates a <code>contributors</code> table, with foreign keys to <code>repos</code> and <code>users</code> and a <code>contributions</code> table listing the number of commits to that repository for each contributor.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/contributors" rel="nofollow">contributors table</a></p>
  <h2 dir="auto"><a id="user-content-fetching-repos-belonging-to-a-user-or-organization" class="anchor" aria-hidden="true" href="#user-content-fetching-repos-belonging-to-a-user-or-organization"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Fetching repos belonging to a user or organization</h2>
  <p dir="auto">The <code>repos</code> command fetches repos belonging to a user or organization.</p>
  <p dir="auto">Without any other arguments, this command will fetch all repos that the currently authenticated user owns, collaborates on or can access via one of their organizations:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite repos github.db"><pre class="notranslate"><code>$ github-to-sqlite repos github.db
  </code></pre></div>
  <p dir="auto">To fetch repos belonging to a specific user or organization, provide their username as an argument:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite repos github.db dogsheep # organization
  $ github-to-sqlite repos github.db simonw # user"><pre class="notranslate"><code>$ github-to-sqlite repos github.db dogsheep # organization
  $ github-to-sqlite repos github.db simonw # user
  </code></pre></div>
  <p dir="auto">You can pass more than one username to fetch for multiple users or organizations at once:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite repos github.db simonw dogsheep"><pre class="notranslate"><code>$ github-to-sqlite repos github.db simonw dogsheep
  </code></pre></div>
  <p dir="auto">Add the <code>--readme</code> option to save the README for the repo in a column called <code>readme</code>. Add <code>--readme-html</code> to save the HTML rendered version of the README into a collumn called <code>readme_html</code>.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/repos" rel="nofollow">repos table</a></p>
  <h2 dir="auto"><a id="user-content-fetching-specific-repositories" class="anchor" aria-hidden="true" href="#user-content-fetching-specific-repositories"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Fetching specific repositories</h2>
  <p dir="auto">You can use <code>-r</code> with the <code>repos</code> command one or more times to fetch just specific repositories.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite repos github.db -r simonw/datasette -r dogsheep/github-to-sqlite"><pre class="notranslate"><code>$ github-to-sqlite repos github.db -r simonw/datasette -r dogsheep/github-to-sqlite
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-fetching-repos-that-have-been-starred-by-a-user" class="anchor" aria-hidden="true" href="#user-content-fetching-repos-that-have-been-starred-by-a-user"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Fetching repos that have been starred by a user</h2>
  <p dir="auto">The <code>starred</code> command fetches the repos that have been starred by a user.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite starred github.db simonw"><pre class="notranslate"><code>$ github-to-sqlite starred github.db simonw
  </code></pre></div>
  <p dir="auto">If you are using an <code>auth.json</code> file you can omit the username to retrieve the starred repos for the authenticated user.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/stars" rel="nofollow">stars table</a></p>
  <h2 dir="auto"><a id="user-content-fetching-users-that-have-starred-specific-repos" class="anchor" aria-hidden="true" href="#user-content-fetching-users-that-have-starred-specific-repos"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Fetching users that have starred specific repos</h2>
  <p dir="auto">The <code>stargazers</code> command fetches the users that have starred the specified repos.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite stargazers github.db simonw/datasette dogsheep/github-to-sqlite"><pre class="notranslate"><code>$ github-to-sqlite stargazers github.db simonw/datasette dogsheep/github-to-sqlite
  </code></pre></div>
  <p dir="auto">You can specify one or more repository using <code>owner/repo</code> syntax.</p>
  <p dir="auto">Users fetched using this command will be inserted into the <code>users</code> table. Many-to-many records showing which repository they starred will be added to the <code>stars</code> table.</p>
  <h2 dir="auto"><a id="user-content-fetching-github-actions-workflows" class="anchor" aria-hidden="true" href="#user-content-fetching-github-actions-workflows"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Fetching GitHub Actions workflows</h2>
  <p dir="auto">The <code>workflows</code> command fetches the YAML workflow configurations from each repository's <code>.github/workflows</code> directory and parses them to populate <code>workflows</code>, <code>jobs</code> and <code>steps</code> tables.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite workflows github.db simonw/datasette dogsheep/github-to-sqlite"><pre class="notranslate"><code>$ github-to-sqlite workflows github.db simonw/datasette dogsheep/github-to-sqlite
  </code></pre></div>
  <p dir="auto">You can specify one or more repository using <code>owner/repo</code> syntax.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/workflows" rel="nofollow">workflows table</a>, <a href="https://github-to-sqlite.dogsheep.net/github/jobs" rel="nofollow">jobs table</a>, <a href="https://github-to-sqlite.dogsheep.net/github/steps" rel="nofollow">steps table</a></p>
  <h2 dir="auto"><a id="user-content-scraping-dependents-for-a-repository" class="anchor" aria-hidden="true" href="#user-content-scraping-dependents-for-a-repository"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Scraping dependents for a repository</h2>
  <p dir="auto">The GitHub dependency graph can show other GitHub projects that depend on a specific repo, for example <a href="https://github.com/simonw/datasette/network/dependents">simonw/datasette/network/dependents</a>.</p>
  <p dir="auto">This data is not yet available through the GitHub API. The <code>scrape-dependents</code> command scrapes those pages and uses the GitHub API to load full versions of the dependent repositories.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite scrape-dependents github.db simonw/datasette"><pre class="notranslate"><code>$ github-to-sqlite scrape-dependents github.db simonw/datasette
  </code></pre></div>
  <p dir="auto">The command accepts one or more repositories.</p>
  <p dir="auto">Add <code>-v</code> for verbose output.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc" rel="nofollow">dependents table</a></p>
  <h2 dir="auto"><a id="user-content-fetching-emojis" class="anchor" aria-hidden="true" href="#user-content-fetching-emojis"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Fetching emojis</h2>
  <p dir="auto">You can fetch a list of every emoji supported by GitHub using the <code>emojis</code> command:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite emojis github.db"><pre class="notranslate"><code>$ github-to-sqlite emojis github.db
  </code></pre></div>
  <p dir="auto">This will create a table callad <code>emojis</code> with a primary key <code>name</code> and a <code>url</code> column.</p>
  <p dir="auto">If you add the <code>--fetch</code> option the command will also fetch the binary content of the images and place them in an <code>image</code> column:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite emojis emojis.db -f
  [########----------------------------]  397/1799   22%  00:03:43"><pre class="notranslate"><code>$ github-to-sqlite emojis emojis.db -f
  [########----------------------------]  397/1799   22%  00:03:43
  </code></pre></div>
  <p dir="auto">You can then use the <a href="https://github.com/simonw/datasette-render-images">datasette-render-images</a> plugin to browse them visually.</p>
  <p dir="auto">Example: <a href="https://github-to-sqlite.dogsheep.net/github/emojis" rel="nofollow">emojis table</a></p>
  <h2 dir="auto"><a id="user-content-making-authenticated-api-calls" class="anchor" aria-hidden="true" href="#user-content-making-authenticated-api-calls"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Making authenticated API calls</h2>
  <p dir="auto">The <code>github-to-sqlite get</code> command provides a convenient shortcut for making authenticated calls to the API. Once you have created your <code>auth.json</code> file (or set a <code>GITHUB_TOKEN</code> environment variable) you can use it like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite get https://api.github.com/gists"><pre class="notranslate"><code>$ github-to-sqlite get https://api.github.com/gists
  </code></pre></div>
  <p dir="auto">This will make an authenticated call to the URL you provide and pretty-print the resulting JSON to the console.</p>
  <p dir="auto">You can ommit the <code>https://api.github.com/</code> prefix, for example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite get /gists"><pre class="notranslate"><code>$ github-to-sqlite get /gists
  </code></pre></div>
  <p dir="auto">Many GitHub APIs are <a href="https://docs.github.com/en/rest/guides/traversing-with-pagination">paginated using the HTTP Link header</a>. You can follow this pagination and output a list of all of the resulting items using <code>--paginate</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite get /users/simonw/repos --paginate"><pre class="notranslate"><code>$ github-to-sqlite get /users/simonw/repos --paginate
  </code></pre></div>
  <p dir="auto">You can outline newline-delimited JSON for each item using <code>--nl</code>. This can be useful for streaming items into another tool.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ github-to-sqlite get /users/simonw/repos --nl"><pre class="notranslate"><code>$ github-to-sqlite get /users/simonw/repos --nl
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMDc2MzAxNzQ=
name: datasette-rure
full_name: simonw/datasette-rure
html_url: https://github.com/simonw/datasette-rure
created_at: 2019-09-10T18:09:33Z
updated_at: 2020-12-04T04:26:53Z
pushed_at: 2019-09-11T22:59:38Z
language: Python
license: apache-2.0
topics: ["sqlite", "regular-expressions", "datasette", "datasette-plugin", "datasette-io"]
default_branch: master
permissions: {"admin": false, "push": false, "pull": false}
description:
  Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine

readme:
  # datasette-rure

  [![PyPI](https://img.shields.io/pypi/v/datasette-rure.svg)](https://pypi.org/project/datasette-rure/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-rure.svg?style=svg)](https://circleci.com/gh/simonw/datasette-rure)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-rure/blob/master/LICENSE)

  Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine

  Install this plugin in the same environment as Datasette to enable the `regexp()` SQL function.

      $ pip install datasette-rure

  The plugin is built on top of the [rure-python](https://github.com/davidblewett/rure-python) library by David Blewett.

  ## regexp() to test regular expressions

  You can test if a value matches a regular expression like this:

      select regexp('hi.*there', 'hi there')
      -- returns 1
      select regexp('not.*there', 'hi there')
      -- returns 0

  You can also use SQLite's custom syntax to run matches:

      select 'hi there' REGEXP 'hi.*there'
      -- returns 1

  This means you can select rows based on regular expression matches - for example, to select every article where the title begins with an E or an F:

      select * from articles where title REGEXP '^[EF]'

  Try this out: [REGEXP interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+*+from+articles+where+title+REGEXP+%27%5E%5BEF%5D%27)

  ## regexp_match() to extract groups

  You can extract captured subsets of a pattern using `regexp_match()`.

      select regexp_match('.*( and .*)', title) as n from articles where n is not null
      -- Returns the ' and X' component of any matching titles, e.g.
      --     and Recognition
      --     and Transitions Their Place
      -- etc

  This will return the first parenthesis match when called with two arguments. You can call it with three arguments to indicate which match you would like to extract:

      select regexp_match('.*(and)(.*)', title, 2) as n from articles where n is not null

  The function will return `null` for invalid inputs e.g. a pattern without capture groups.

  Try this out: [regexp_match() interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+%27WHY+%27+%7C%7C+regexp_match%28%27Why+%28.*%29%27%2C+title%29+as+t+from+articles+where+t+is+not+null)

  ## regexp_matches() to extract multiple matches at once

  The `regexp_matches()` function can be used to extract multiple patterns from a single string. The result is returned as a JSON array, which can then be further processed using SQLite's [JSON functions](https://www.sqlite.org/json1.html).

  The first argument is a regular expression with named capture groups. The second argument is the string to be matched.

      select regexp_matches(
          'hello (?P<name>\w+) the (?P<species>\w+)',
          'hello bob the dog, hello maggie the cat, hello tarquin the otter'
      )

  This will return a list of JSON objects, each one representing the named captures from the original regular expression:

      [
          {"name": "bob", "species": "dog"},
          {"name": "maggie", "species": "cat"},
          {"name": "tarquin", "species": "otter"}
      ]

  Try this out: [regexp_matches() interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+regexp_matches%28%0D%0A++++%27hello+%28%3FP%3Cname%3E%5Cw%2B%29+the+%28%3FP%3Cspecies%3E%5Cw%2B%29%27%2C%0D%0A++++%27hello+bob+the+dog%2C+hello+maggie+the+cat%2C+hello+tarquin+the+otter%27%0D%0A%29)

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-rure" class="anchor" aria-hidden="true" href="#user-content-datasette-rure"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-rure</h1>
  <p><a href="https://pypi.org/project/datasette-rure/" rel="nofollow"><img src="https://camo.githubusercontent.com/71217f9bd1b366cef0b2c8147af82aa94cac531801f86b277bb0bf3b1d13c1c1/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d727572652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-rure.svg" style="max-width:100%;"></a>
  <a href="https://circleci.com/gh/simonw/datasette-rure" rel="nofollow"><img src="https://camo.githubusercontent.com/bc58494132e2732aabfe5f64284366a785e96850c5b5b86962a9d94d06f61489/68747470733a2f2f636972636c6563692e636f6d2f67682f73696d6f6e772f6461746173657474652d727572652e7376673f7374796c653d737667" alt="CircleCI" data-canonical-src="https://circleci.com/gh/simonw/datasette-rure.svg?style=svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-rure/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine</p>
  <p>Install this plugin in the same environment as Datasette to enable the <code>regexp()</code> SQL function.</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ pip install datasette-rure
  "><pre><code>$ pip install datasette-rure
  </code></pre></div>
  <p>The plugin is built on top of the <a href="https://github.com/davidblewett/rure-python">rure-python</a> library by David Blewett.</p>
  <h2><a id="user-content-regexp-to-test-regular-expressions" class="anchor" aria-hidden="true" href="#user-content-regexp-to-test-regular-expressions"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>regexp() to test regular expressions</h2>
  <p>You can test if a value matches a regular expression like this:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="select regexp('hi.*there', 'hi there')
  -- returns 1
  select regexp('not.*there', 'hi there')
  -- returns 0
  "><pre><code>select regexp('hi.*there', 'hi there')
  -- returns 1
  select regexp('not.*there', 'hi there')
  -- returns 0
  </code></pre></div>
  <p>You can also use SQLite's custom syntax to run matches:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="select 'hi there' REGEXP 'hi.*there'
  -- returns 1
  "><pre><code>select 'hi there' REGEXP 'hi.*there'
  -- returns 1
  </code></pre></div>
  <p>This means you can select rows based on regular expression matches - for example, to select every article where the title begins with an E or an F:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="select * from articles where title REGEXP '^[EF]'
  "><pre><code>select * from articles where title REGEXP '^[EF]'
  </code></pre></div>
  <p>Try this out: <a href="https://datasette-rure-demo.datasette.io/24ways?sql=select+*+from+articles+where+title+REGEXP+%27%5E%5BEF%5D%27" rel="nofollow">REGEXP interactive demo</a></p>
  <h2><a id="user-content-regexp_match-to-extract-groups" class="anchor" aria-hidden="true" href="#user-content-regexp_match-to-extract-groups"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>regexp_match() to extract groups</h2>
  <p>You can extract captured subsets of a pattern using <code>regexp_match()</code>.</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="select regexp_match('.*( and .*)', title) as n from articles where n is not null
  -- Returns the ' and X' component of any matching titles, e.g.
  --     and Recognition
  --     and Transitions Their Place
  -- etc
  "><pre><code>select regexp_match('.*( and .*)', title) as n from articles where n is not null
  -- Returns the ' and X' component of any matching titles, e.g.
  --     and Recognition
  --     and Transitions Their Place
  -- etc
  </code></pre></div>
  <p>This will return the first parenthesis match when called with two arguments. You can call it with three arguments to indicate which match you would like to extract:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="select regexp_match('.*(and)(.*)', title, 2) as n from articles where n is not null
  "><pre><code>select regexp_match('.*(and)(.*)', title, 2) as n from articles where n is not null
  </code></pre></div>
  <p>The function will return <code>null</code> for invalid inputs e.g. a pattern without capture groups.</p>
  <p>Try this out: <a href="https://datasette-rure-demo.datasette.io/24ways?sql=select+%27WHY+%27+%7C%7C+regexp_match%28%27Why+%28.*%29%27%2C+title%29+as+t+from+articles+where+t+is+not+null" rel="nofollow">regexp_match() interactive demo</a></p>
  <h2><a id="user-content-regexp_matches-to-extract-multiple-matches-at-once" class="anchor" aria-hidden="true" href="#user-content-regexp_matches-to-extract-multiple-matches-at-once"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>regexp_matches() to extract multiple matches at once</h2>
  <p>The <code>regexp_matches()</code> function can be used to extract multiple patterns from a single string. The result is returned as a JSON array, which can then be further processed using SQLite's <a href="https://www.sqlite.org/json1.html" rel="nofollow">JSON functions</a>.</p>
  <p>The first argument is a regular expression with named capture groups. The second argument is the string to be matched.</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="select regexp_matches(
      'hello (?P&lt;name&gt;\w+) the (?P&lt;species&gt;\w+)',
      'hello bob the dog, hello maggie the cat, hello tarquin the otter'
  )
  "><pre><code>select regexp_matches(
      'hello (?P&lt;name&gt;\w+) the (?P&lt;species&gt;\w+)',
      'hello bob the dog, hello maggie the cat, hello tarquin the otter'
  )
  </code></pre></div>
  <p>This will return a list of JSON objects, each one representing the named captures from the original regular expression:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="[
      {&quot;name&quot;: &quot;bob&quot;, &quot;species&quot;: &quot;dog&quot;},
      {&quot;name&quot;: &quot;maggie&quot;, &quot;species&quot;: &quot;cat&quot;},
      {&quot;name&quot;: &quot;tarquin&quot;, &quot;species&quot;: &quot;otter&quot;}
  ]
  "><pre><code>[
      {"name": "bob", "species": "dog"},
      {"name": "maggie", "species": "cat"},
      {"name": "tarquin", "species": "otter"}
  ]
  </code></pre></div>
  <p>Try this out: <a href="https://datasette-rure-demo.datasette.io/24ways?sql=select+regexp_matches%28%0D%0A++++%27hello+%28%3FP%3Cname%3E%5Cw%2B%29+the+%28%3FP%3Cspecies%3E%5Cw%2B%29%27%2C%0D%0A++++%27hello+bob+the+dog%2C+hello+maggie+the+cat%2C+hello+tarquin+the+otter%27%0D%0A%29" rel="nofollow">regexp_matches() interactive demo</a></p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMDkwOTEyNTY=
name: datasette-atom
full_name: simonw/datasette-atom
html_url: https://github.com/simonw/datasette-atom
description: Datasette plugin that adds a .atom output format
created_at: 2019-09-17T15:31:01Z
updated_at: 2021-03-26T02:06:51Z
pushed_at: 2021-01-24T23:59:36Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin", "datasette-io"]
default_branch: main
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # datasette-atom

  [![PyPI](https://img.shields.io/pypi/v/datasette-atom.svg)](https://pypi.org/project/datasette-atom/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-atom?include_prereleases&label=changelog)](https://github.com/simonw/datasette-atom/releases)
  [![Tests](https://github.com/simonw/datasette-atom/workflows/Test/badge.svg)](https://github.com/simonw/datasette-atom/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-atom/blob/main/LICENSE)

  Datasette plugin that adds support for generating [Atom feeds](https://validator.w3.org/feed/docs/atom.html) with the results of a SQL query.

  ## Installation

  Install this plugin in the same environment as Datasette to enable the `.atom` output extension.

      $ pip install datasette-atom

  ## Usage

  To create an Atom feed you need to define a custom SQL query that returns a required set of columns:

  * `atom_id` - a unique ID for each row. [This article](https://web.archive.org/web/20080211143232/http://diveintomark.org/archives/2004/05/28/howto-atom-id) has suggestions about ways to create these IDs.
  * `atom_title` - a title for that row.
  * `atom_updated` - an [RFC 3339](http://www.faqs.org/rfcs/rfc3339.html) timestamp representing the last time the entry was modified in a significant way. This can usually be the time that the row was created.

  The following columns are optional:

  * `atom_content` - content that should be shown in the feed. This will be treated as a regular string, so any embedded HTML tags will be escaped when they are displayed.
  * `atom_content_html` - content that should be shown in the feed. This will be treated as an HTML string, and will be sanitized using [Bleach](https://github.com/mozilla/bleach) to ensure it does not have any malicious code in it before being returned as part of a `<content type="html">` Atom element. If both are provided, this will be used in place of `atom_content`.
  * `atom_link` - a URL that should be used as the link that the feed entry points to.
  * `atom_author_name` - the name of the author of the entry. If you provide this you can also provide `atom_author_uri` and `atom_author_email` with a URL and e-mail address for that author.

  A query that returns these columns can then be returned as an Atom feed by adding the `.atom` extension.

  ## Example

  Here is an example SQL query which generates an Atom feed for new entries on [www.niche-museums.com](https://www.niche-museums.com/):

  ```sql
  select
    'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id,
    name as atom_title,
    created as atom_updated,
    'https://www.niche-museums.com/browse/museums/' || id as atom_link,
    coalesce(
      '<img src="' || photo_url || '?w=800&amp;h=400&amp;fit=crop&amp;auto=compress">',
      ''
    ) || '<p>' || description || '</p>' as atom_content_html
  from
    museums
  order by
    created desc
  limit
    15
  ```

  You can try this query by [pasting it in here](https://www.niche-museums.com/browse) - then click the `.atom` link to see it as an Atom feed.

  ## Using a canned query

  Datasette's [canned query mechanism](https://docs.datasette.io/en/stable/sql_queries.html#canned-queries) is a useful way to configure feeds. If a canned query definition has a `title` that will be used as the title of the Atom feed.

  Here's an example, defined using a `metadata.yaml` file:

  ```yaml
  databases:
    browse:
      queries:
        feed:
          title: Niche Museums
          sql: |-
            select
              'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id,
              name as atom_title,
              created as atom_updated,
              'https://www.niche-museums.com/browse/museums/' || id as atom_link,
              coalesce(
                '<img src="' || photo_url || '?w=800&amp;h=400&amp;fit=crop&amp;auto=compress">',
                ''
              ) || '<p>' || description || '</p>' as atom_content_html
            from
              museums
            order by
              created desc
            limit
              15
  ```
  ## Disabling HTML filtering

  The HTML allow-list used by Bleach for the `atom_content_html` column can be found in the `clean(html)` function at the bottom of [datasette_atom/__init__.py](https://github.com/simonw/datasette-atom/blob/main/datasette_atom/__init__.py).

  You can disable Bleach entirely for Atom feeds generated using a canned query. You should only do this if you are certain that no user-provided HTML could be included in that value.

  Here's how to do that in `metadata.json`:

  ```json
  {
    "plugins": {
      "datasette-atom": {
        "allow_unsafe_html_in_canned_queries": true
      }
    }
  }
  ```
  Setting this to `true` will disable Bleach filtering for all canned queries across all databases.

  You can disable Bleach filtering just for a specific list of canned queries like so:

  ```json
  {
    "plugins": {
      "datasette-atom": {
        "allow_unsafe_html_in_canned_queries": {
          "museums": ["latest", "moderation"]
        }
      }
    }
  }
  ```
  This will disable Bleach just for the canned queries called `latest` and `moderation` in the `museums.db` database.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-atom" class="anchor" aria-hidden="true" href="#user-content-datasette-atom"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-atom</h1>
  <p><a href="https://pypi.org/project/datasette-atom/" rel="nofollow"><img src="https://camo.githubusercontent.com/7957f60f0a749eb9a6313a3c35574dad2554ed586ef3bd67c4f0b768edd28fb2/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d61746f6d2e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-atom.svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-atom/releases"><img src="https://camo.githubusercontent.com/42a02d342f39e3dc05195df216b9f89fa84de19d94547b4c68a02ec12f0eaf2d/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d61746f6d3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-atom?include_prereleases&amp;label=changelog" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-atom/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-atom/workflows/Test/badge.svg" alt="Tests" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-atom/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Datasette plugin that adds support for generating <a href="https://validator.w3.org/feed/docs/atom.html" rel="nofollow">Atom feeds</a> with the results of a SQL query.</p>
  <h2><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p>Install this plugin in the same environment as Datasette to enable the <code>.atom</code> output extension.</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ pip install datasette-atom
  "><pre><code>$ pip install datasette-atom
  </code></pre></div>
  <h2><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p>To create an Atom feed you need to define a custom SQL query that returns a required set of columns:</p>
  <ul>
  <li><code>atom_id</code> - a unique ID for each row. <a href="https://web.archive.org/web/20080211143232/http://diveintomark.org/archives/2004/05/28/howto-atom-id" rel="nofollow">This article</a> has suggestions about ways to create these IDs.</li>
  <li><code>atom_title</code> - a title for that row.</li>
  <li><code>atom_updated</code> - an <a href="http://www.faqs.org/rfcs/rfc3339.html" rel="nofollow">RFC 3339</a> timestamp representing the last time the entry was modified in a significant way. This can usually be the time that the row was created.</li>
  </ul>
  <p>The following columns are optional:</p>
  <ul>
  <li><code>atom_content</code> - content that should be shown in the feed. This will be treated as a regular string, so any embedded HTML tags will be escaped when they are displayed.</li>
  <li><code>atom_content_html</code> - content that should be shown in the feed. This will be treated as an HTML string, and will be sanitized using <a href="https://github.com/mozilla/bleach">Bleach</a> to ensure it does not have any malicious code in it before being returned as part of a <code>&lt;content type="html"&gt;</code> Atom element. If both are provided, this will be used in place of <code>atom_content</code>.</li>
  <li><code>atom_link</code> - a URL that should be used as the link that the feed entry points to.</li>
  <li><code>atom_author_name</code> - the name of the author of the entry. If you provide this you can also provide <code>atom_author_uri</code> and <code>atom_author_email</code> with a URL and e-mail address for that author.</li>
  </ul>
  <p>A query that returns these columns can then be returned as an Atom feed by adding the <code>.atom</code> extension.</p>
  <h2><a id="user-content-example" class="anchor" aria-hidden="true" href="#user-content-example"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Example</h2>
  <p>Here is an example SQL query which generates an Atom feed for new entries on <a href="https://www.niche-museums.com/" rel="nofollow">www.niche-museums.com</a>:</p>
  <div class="highlight highlight-source-sql position-relative" data-snippet-clipboard-copy-content="select
    'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id,
    name as atom_title,
    created as atom_updated,
    'https://www.niche-museums.com/browse/museums/' || id as atom_link,
    coalesce(
      '&lt;img src=&quot;' || photo_url || '?w=800&amp;amp;h=400&amp;amp;fit=crop&amp;amp;auto=compress&quot;&gt;',
      ''
    ) || '&lt;p&gt;' || description || '&lt;/p&gt;' as atom_content_html
  from
    museums
  order by
    created desc
  limit
    15
  "><pre><span class="pl-k">select</span>
    <span class="pl-s"><span class="pl-pds">'</span>tag:niche-museums.com,<span class="pl-pds">'</span></span> <span class="pl-k">||</span> substr(created, <span class="pl-c1">0</span>, <span class="pl-c1">11</span>) <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>:<span class="pl-pds">'</span></span> <span class="pl-k">||</span> id <span class="pl-k">as</span> atom_id,
    name <span class="pl-k">as</span> atom_title,
    created <span class="pl-k">as</span> atom_updated,
    <span class="pl-s"><span class="pl-pds">'</span>https://www.niche-museums.com/browse/museums/<span class="pl-pds">'</span></span> <span class="pl-k">||</span> id <span class="pl-k">as</span> atom_link,
    coalesce(
      <span class="pl-s"><span class="pl-pds">'</span>&lt;img src="<span class="pl-pds">'</span></span> <span class="pl-k">||</span> photo_url <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>?w=800&amp;amp;h=400&amp;amp;fit=crop&amp;amp;auto=compress"&gt;<span class="pl-pds">'</span></span>,
      <span class="pl-s"><span class="pl-pds">'</span><span class="pl-pds">'</span></span>
    ) <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>&lt;p&gt;<span class="pl-pds">'</span></span> <span class="pl-k">||</span> description <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>&lt;/p&gt;<span class="pl-pds">'</span></span> <span class="pl-k">as</span> atom_content_html
  <span class="pl-k">from</span>
    museums
  <span class="pl-k">order by</span>
    created <span class="pl-k">desc</span>
  <span class="pl-k">limit</span>
    <span class="pl-c1">15</span></pre></div>
  <p>You can try this query by <a href="https://www.niche-museums.com/browse" rel="nofollow">pasting it in here</a> - then click the <code>.atom</code> link to see it as an Atom feed.</p>
  <h2><a id="user-content-using-a-canned-query" class="anchor" aria-hidden="true" href="#user-content-using-a-canned-query"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Using a canned query</h2>
  <p>Datasette's <a href="https://docs.datasette.io/en/stable/sql_queries.html#canned-queries" rel="nofollow">canned query mechanism</a> is a useful way to configure feeds. If a canned query definition has a <code>title</code> that will be used as the title of the Atom feed.</p>
  <p>Here's an example, defined using a <code>metadata.yaml</code> file:</p>
  <div class="highlight highlight-source-yaml position-relative" data-snippet-clipboard-copy-content="databases:
    browse:
      queries:
        feed:
          title: Niche Museums
          sql: |-
            select
              'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id,
              name as atom_title,
              created as atom_updated,
              'https://www.niche-museums.com/browse/museums/' || id as atom_link,
              coalesce(
                '&lt;img src=&quot;' || photo_url || '?w=800&amp;amp;h=400&amp;amp;fit=crop&amp;amp;auto=compress&quot;&gt;',
                ''
              ) || '&lt;p&gt;' || description || '&lt;/p&gt;' as atom_content_html
            from
              museums
            order by
              created desc
            limit
              15
  "><pre><span class="pl-ent">databases</span>:
    <span class="pl-ent">browse</span>:
      <span class="pl-ent">queries</span>:
        <span class="pl-ent">feed</span>:
          <span class="pl-ent">title</span>: <span class="pl-s">Niche Museums</span>
          <span class="pl-ent">sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">          select</span>
  <span class="pl-s">            'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id,</span>
  <span class="pl-s">            name as atom_title,</span>
  <span class="pl-s">            created as atom_updated,</span>
  <span class="pl-s">            'https://www.niche-museums.com/browse/museums/' || id as atom_link,</span>
  <span class="pl-s">            coalesce(</span>
  <span class="pl-s">              '&lt;img src="' || photo_url || '?w=800&amp;amp;h=400&amp;amp;fit=crop&amp;amp;auto=compress"&gt;',</span>
  <span class="pl-s">              ''</span>
  <span class="pl-s">            ) || '&lt;p&gt;' || description || '&lt;/p&gt;' as atom_content_html</span>
  <span class="pl-s">          from</span>
  <span class="pl-s">            museums</span>
  <span class="pl-s">          order by</span>
  <span class="pl-s">            created desc</span>
  <span class="pl-s">          limit</span>
  <span class="pl-s">            15</span></pre></div>
  <h2><a id="user-content-disabling-html-filtering" class="anchor" aria-hidden="true" href="#user-content-disabling-html-filtering"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Disabling HTML filtering</h2>
  <p>The HTML allow-list used by Bleach for the <code>atom_content_html</code> column can be found in the <code>clean(html)</code> function at the bottom of <a href="https://github.com/simonw/datasette-atom/blob/main/datasette_atom/__init__.py">datasette_atom/<strong>init</strong>.py</a>.</p>
  <p>You can disable Bleach entirely for Atom feeds generated using a canned query. You should only do this if you are certain that no user-provided HTML could be included in that value.</p>
  <p>Here's how to do that in <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
    &quot;plugins&quot;: {
      &quot;datasette-atom&quot;: {
        &quot;allow_unsafe_html_in_canned_queries&quot;: true
      }
    }
  }
  "><pre>{
    <span class="pl-s"><span class="pl-pds">"</span>plugins<span class="pl-pds">"</span></span>: {
      <span class="pl-s"><span class="pl-pds">"</span>datasette-atom<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>allow_unsafe_html_in_canned_queries<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>
      }
    }
  }</pre></div>
  <p>Setting this to <code>true</code> will disable Bleach filtering for all canned queries across all databases.</p>
  <p>You can disable Bleach filtering just for a specific list of canned queries like so:</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
    &quot;plugins&quot;: {
      &quot;datasette-atom&quot;: {
        &quot;allow_unsafe_html_in_canned_queries&quot;: {
          &quot;museums&quot;: [&quot;latest&quot;, &quot;moderation&quot;]
        }
      }
    }
  }
  "><pre>{
    <span class="pl-s"><span class="pl-pds">"</span>plugins<span class="pl-pds">"</span></span>: {
      <span class="pl-s"><span class="pl-pds">"</span>datasette-atom<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>allow_unsafe_html_in_canned_queries<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>museums<span class="pl-pds">"</span></span>: [<span class="pl-s"><span class="pl-pds">"</span>latest<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>moderation<span class="pl-pds">"</span></span>]
        }
      }
    }
  }</pre></div>
  <p>This will disable Bleach just for the canned queries called <code>latest</code> and <code>moderation</code> in the <code>museums.db</code> database.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMDk1OTAzNDU=
name: genome-to-sqlite
full_name: dogsheep/genome-to-sqlite
html_url: https://github.com/dogsheep/genome-to-sqlite
description: Import your genome into a SQLite database
created_at: 2019-09-19T15:38:39Z
updated_at: 2021-01-18T19:39:48Z
pushed_at: 2019-09-19T15:41:17Z
language: Python
license: apache-2.0
default_branch: master
permissions: {"admin": false, "push": false, "pull": false}
topics:
  ["genetics", "sqlite", "23andme", "personal-analytics", "datasette", "dogsheep", "datasette-io", "datasette-tool"]

readme:
  # genome-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/genome-to-sqlite.svg)](https://pypi.org/project/genome-to-sqlite/)
  [![CircleCI](https://circleci.com/gh/dogsheep/genome-to-sqlite.svg?style=svg)](https://circleci.com/gh/dogsheep/genome-to-sqlite)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/genome-to-sqlite/blob/master/LICENSE)

  Import your genome into a SQLite database.

  ## How to install

      $ pip install genome-to-sqlite

  ## How to use

  First, export your genome. This tool has only been tested against 23andMe so far. You can request an export of your genome from https://you.23andme.com/tools/data/download/

  Now you can convert the resulting `export.zip` file to SQLite like so:

      $ genome-to-sqlite export.zip genome.db

  A progress bar will be displayed. You can disable this using `--silent`.

  ```
  Importing genome  [#----------------]    5%  00:01:33
  ```

  You can explore the resulting data using [Datasette](https://datasette.readthedocs.io/) like this:

      $ datasette genome.db --config facet_time_limit_ms:1000

  Bumping up the facet time limit is useful in order to enable faceting by chromosome:

  http://127.0.0.1:8001/genome/genome?_facet=chromosome&_sort=position

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-genome-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-genome-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>genome-to-sqlite</h1>
  <p><a href="https://pypi.org/project/genome-to-sqlite/" rel="nofollow"><img src="https://camo.githubusercontent.com/a91d14ecfe9aa2d73da4818a7a898033e301380d4c8477d06a17dc20bd871c84/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f67656e6f6d652d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/genome-to-sqlite.svg" style="max-width:100%;"></a>
  <a href="https://circleci.com/gh/dogsheep/genome-to-sqlite" rel="nofollow"><img src="https://camo.githubusercontent.com/5f7f5be2791ff007c49d3d153ef38baa81770a11c0326e5f6b8ffd7ee939d00f/68747470733a2f2f636972636c6563692e636f6d2f67682f646f6773686565702f67656e6f6d652d746f2d73716c6974652e7376673f7374796c653d737667" alt="CircleCI" data-canonical-src="https://circleci.com/gh/dogsheep/genome-to-sqlite.svg?style=svg" style="max-width:100%;"></a>
  <a href="https://github.com/dogsheep/genome-to-sqlite/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Import your genome into a SQLite database.</p>
  <h2><a id="user-content-how-to-install" class="anchor" aria-hidden="true" href="#user-content-how-to-install"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How to install</h2>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ pip install genome-to-sqlite
  "><pre><code>$ pip install genome-to-sqlite
  </code></pre></div>
  <h2><a id="user-content-how-to-use" class="anchor" aria-hidden="true" href="#user-content-how-to-use"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How to use</h2>
  <p>First, export your genome. This tool has only been tested against 23andMe so far. You can request an export of your genome from <a href="https://you.23andme.com/tools/data/download/" rel="nofollow">https://you.23andme.com/tools/data/download/</a></p>
  <p>Now you can convert the resulting <code>export.zip</code> file to SQLite like so:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ genome-to-sqlite export.zip genome.db
  "><pre><code>$ genome-to-sqlite export.zip genome.db
  </code></pre></div>
  <p>A progress bar will be displayed. You can disable this using <code>--silent</code>.</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="Importing genome  [#----------------]    5%  00:01:33
  "><pre><code>Importing genome  [#----------------]    5%  00:01:33
  </code></pre></div>
  <p>You can explore the resulting data using <a href="https://datasette.readthedocs.io/" rel="nofollow">Datasette</a> like this:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ datasette genome.db --config facet_time_limit_ms:1000
  "><pre><code>$ datasette genome.db --config facet_time_limit_ms:1000
  </code></pre></div>
  <p>Bumping up the facet time limit is useful in order to enable faceting by chromosome:</p>
  <p><a href="http://127.0.0.1:8001/genome/genome?_facet=chromosome&amp;_sort=position" rel="nofollow">http://127.0.0.1:8001/genome/genome?_facet=chromosome&amp;_sort=position</a></p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMTMyODY3NTI=
name: pocket-to-sqlite
full_name: dogsheep/pocket-to-sqlite
html_url: https://github.com/dogsheep/pocket-to-sqlite
description: Create a SQLite database containing data from your Pocket account
created_at: 2019-10-07T03:24:14Z
updated_at: 2022-08-21T21:11:59Z
pushed_at: 2022-08-22T16:21:34Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-tool", "dogsheep", "pocket", "pocket-api", "sqlite"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # pocket-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/pocket-to-sqlite.svg)](https://pypi.org/project/pocket-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/pocket-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/pocket-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/pocket-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/pocket-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/pocket-to-sqlite/blob/main/LICENSE)

  Create a SQLite database containing data from your [Pocket](https://getpocket.com/) account.

  ## How to install

      $ pip install pocket-to-sqlite

  ## Usage

  You will need to first obtain a valid OAuth token for your Pocket account. You can do this by running the `auth` command and following the prompts:

      $ pocket-to-sqlite auth
      Visit this page and sign in with your Pocket account:

      https://getpocket.com/auth/author...

      Once you have signed in there, hit <enter> to continue
      Authentication tokens written to auth.json

  Now you can fetch all of your items from Pocket like this:

      $ pocket-to-sqlite fetch pocket.db

  The first time you run this command it will fetch all of your items, and display a progress bar while it does it.

  On subsequent runs it will only fetch new items.

  You can force it to fetch everything from the beginning again using `--all`. Use `--silent` to disable the progress bar.

  ## Using with Datasette

  The SQLite database produced by this tool is designed to be browsed using [Datasette](https://datasette.readthedocs.io/). Use the [datasette-render-timestamps](https://github.com/simonw/datasette-render-timestamps) plugin to improve the display of the timestamp values.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-pocket-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-pocket-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>pocket-to-sqlite</h1>
  <p dir="auto"><a href="https://pypi.org/project/pocket-to-sqlite/" rel="nofollow"><img src="https://camo.githubusercontent.com/98050bc158f5e93f2c3576b03fa6a25b4a8091bb6d44774e07d95ae2842e4a3e/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f706f636b65742d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/pocket-to-sqlite.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/pocket-to-sqlite/releases"><img src="https://camo.githubusercontent.com/b37c773b526e69ffcc72b9c02cbcd977d1b6e6a9bcf317c3accfc04b975e960c/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f646f6773686565702f706f636b65742d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/dogsheep/pocket-to-sqlite?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/pocket-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/dogsheep/pocket-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/dogsheep/pocket-to-sqlite/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Create a SQLite database containing data from your <a href="https://getpocket.com/" rel="nofollow">Pocket</a> account.</p>
  <h2 dir="auto"><a id="user-content-how-to-install" class="anchor" aria-hidden="true" href="#user-content-how-to-install"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How to install</h2>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install pocket-to-sqlite"><pre class="notranslate"><code>$ pip install pocket-to-sqlite
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">You will need to first obtain a valid OAuth token for your Pocket account. You can do this by running the <code>auth</code> command and following the prompts:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pocket-to-sqlite auth
  Visit this page and sign in with your Pocket account:

  https://getpocket.com/auth/author...

  Once you have signed in there, hit &lt;enter&gt; to continue
  Authentication tokens written to auth.json"><pre class="notranslate"><code>$ pocket-to-sqlite auth
  Visit this page and sign in with your Pocket account:

  https://getpocket.com/auth/author...

  Once you have signed in there, hit &lt;enter&gt; to continue
  Authentication tokens written to auth.json
  </code></pre></div>
  <p dir="auto">Now you can fetch all of your items from Pocket like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pocket-to-sqlite fetch pocket.db"><pre class="notranslate"><code>$ pocket-to-sqlite fetch pocket.db
  </code></pre></div>
  <p dir="auto">The first time you run this command it will fetch all of your items, and display a progress bar while it does it.</p>
  <p dir="auto">On subsequent runs it will only fetch new items.</p>
  <p dir="auto">You can force it to fetch everything from the beginning again using <code>--all</code>. Use <code>--silent</code> to disable the progress bar.</p>
  <h2 dir="auto"><a id="user-content-using-with-datasette" class="anchor" aria-hidden="true" href="#user-content-using-with-datasette"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Using with Datasette</h2>
  <p dir="auto">The SQLite database produced by this tool is designed to be browsed using <a href="https://datasette.readthedocs.io/" rel="nofollow">Datasette</a>. Use the <a href="https://github.com/simonw/datasette-render-timestamps">datasette-render-timestamps</a> plugin to improve the display of the timestamp values.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMTQyOTkyNjc=
name: datasette-render-timestamps
full_name: simonw/datasette-render-timestamps
html_url: https://github.com/simonw/datasette-render-timestamps
description: Datasette plugin for rendering timestamps
created_at: 2019-10-10T22:50:50Z
updated_at: 2020-10-17T11:09:42Z
pushed_at: 2020-03-22T17:57:17Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin", "datasette-io"]
default_branch: master
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # datasette-render-timestamps

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-timestamps.svg)](https://pypi.org/project/datasette-render-timestamps/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-render-timestamps.svg?style=svg)](https://circleci.com/gh/simonw/datasette-render-timestamps)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-timestamps/blob/master/LICENSE)

  Datasette plugin for rendering timestamps.

  ## Installation

  Install this plugin in the same environment as Datasette to enable this new functionality:

      pip install datasette-render-timestamps

  The plugin will then look out for integer numbers that are likely to be timestamps - anything that would be a number of seconds from 5 years ago to 5 years in the future.

  These will then be rendered in a more readable format.

  ## Configuration

  You can disable automatic column detection in favour of explicitly listing the columns that you would like to render using [plugin configuration](https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration) in a `metadata.json` file.

  Add a `"datasette-render-timestamps"` configuration block and use a `"columns"` key to list the columns you would like to treat as timestamp values:

  ```json
  {
      "plugins": {
          "datasette-render-timestamps": {
              "columns": ["created", "updated"]
          }
      }
  }
  ```
  This will cause any `created` or `updated` columns in any table to be treated as timestamps and rendered.

  Save this to `metadata.json` and run datasette with the `--metadata` flag to load this configuration:

      datasette serve mydata.db --metadata metadata.json

  To disable automatic timestamp detection entirely, you can use `"columnns": []`.

  This configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the `entries` table in the `news.db` database:

  ```json
  {
      "databases": {
          "news": {
              "tables": {
                  "entries": {
                      "plugins": {
                          "datasette-render-timestamps": {
                              "columns": ["created", "updated"]
                          }
                      }
                  }
              }
          }
      }
  }
  ```

  And here's how to apply it to every `created` column in every table in the `news.db` database:

  ```json
  {
      "databases": {
          "news": {
              "plugins": {
                  "datasette-render-timestamps": {
                      "columns": ["created", "updated"]
                  }
              }
          }
      }
  }
  ```

  ### Customizing the date format

  The default format is `%B %d, %Y - %H:%M:%S UTC` which renders for example: `October 10, 2019 - 07:18:29 UTC`. If you want another format, the date format can be customized using plugin configuration. Any format string supported by [strftime](http://strftime.org/) may be used. For example:

  ```json
  {
      "plugins": {
          "datasette-render-timestamps": {
              "format": "%Y-%m-%d-%H:%M:%S"
          }
      }
  }
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-render-timestamps" class="anchor" aria-hidden="true" href="#user-content-datasette-render-timestamps"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-render-timestamps</h1>
  <p><a href="https://pypi.org/project/datasette-render-timestamps/" rel="nofollow"><img src="https://camo.githubusercontent.com/6c9b69c5f6aca7ab8a2fbe0dd7b4ce1cca3a75f837750f3e7e7a55f9bd03b96f/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d72656e6465722d74696d657374616d70732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-render-timestamps.svg" style="max-width:100%;"></a>
  <a href="https://circleci.com/gh/simonw/datasette-render-timestamps" rel="nofollow"><img src="https://camo.githubusercontent.com/8bf2320ca702efe91ed80467a992e0bd23b979894eaa4854bbc8affacbfaf8bb/68747470733a2f2f636972636c6563692e636f6d2f67682f73696d6f6e772f6461746173657474652d72656e6465722d74696d657374616d70732e7376673f7374796c653d737667" alt="CircleCI" data-canonical-src="https://circleci.com/gh/simonw/datasette-render-timestamps.svg?style=svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-render-timestamps/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Datasette plugin for rendering timestamps.</p>
  <h2><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p>Install this plugin in the same environment as Datasette to enable this new functionality:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install datasette-render-timestamps
  "><pre><code>pip install datasette-render-timestamps
  </code></pre></div>
  <p>The plugin will then look out for integer numbers that are likely to be timestamps - anything that would be a number of seconds from 5 years ago to 5 years in the future.</p>
  <p>These will then be rendered in a more readable format.</p>
  <h2><a id="user-content-configuration" class="anchor" aria-hidden="true" href="#user-content-configuration"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Configuration</h2>
  <p>You can disable automatic column detection in favour of explicitly listing the columns that you would like to render using <a href="https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration" rel="nofollow">plugin configuration</a> in a <code>metadata.json</code> file.</p>
  <p>Add a <code>"datasette-render-timestamps"</code> configuration block and use a <code>"columns"</code> key to list the columns you would like to treat as timestamp values:</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-render-timestamps&quot;: {
              &quot;columns&quot;: [&quot;created&quot;, &quot;updated&quot;]
          }
      }
  }
  "><pre>{
      <span class="pl-s"><span class="pl-pds">"</span>plugins<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>datasette-render-timestamps<span class="pl-pds">"</span></span>: {
              <span class="pl-s"><span class="pl-pds">"</span>columns<span class="pl-pds">"</span></span>: [<span class="pl-s"><span class="pl-pds">"</span>created<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>updated<span class="pl-pds">"</span></span>]
          }
      }
  }</pre></div>
  <p>This will cause any <code>created</code> or <code>updated</code> columns in any table to be treated as timestamps and rendered.</p>
  <p>Save this to <code>metadata.json</code> and run datasette with the <code>--metadata</code> flag to load this configuration:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="datasette serve mydata.db --metadata metadata.json
  "><pre><code>datasette serve mydata.db --metadata metadata.json
  </code></pre></div>
  <p>To disable automatic timestamp detection entirely, you can use <code>"columnns": []</code>.</p>
  <p>This configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the <code>entries</code> table in the <code>news.db</code> database:</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
      &quot;databases&quot;: {
          &quot;news&quot;: {
              &quot;tables&quot;: {
                  &quot;entries&quot;: {
                      &quot;plugins&quot;: {
                          &quot;datasette-render-timestamps&quot;: {
                              &quot;columns&quot;: [&quot;created&quot;, &quot;updated&quot;]
                          }
                      }
                  }
              }
          }
      }
  }
  "><pre>{
      <span class="pl-s"><span class="pl-pds">"</span>databases<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>news<span class="pl-pds">"</span></span>: {
              <span class="pl-s"><span class="pl-pds">"</span>tables<span class="pl-pds">"</span></span>: {
                  <span class="pl-s"><span class="pl-pds">"</span>entries<span class="pl-pds">"</span></span>: {
                      <span class="pl-s"><span class="pl-pds">"</span>plugins<span class="pl-pds">"</span></span>: {
                          <span class="pl-s"><span class="pl-pds">"</span>datasette-render-timestamps<span class="pl-pds">"</span></span>: {
                              <span class="pl-s"><span class="pl-pds">"</span>columns<span class="pl-pds">"</span></span>: [<span class="pl-s"><span class="pl-pds">"</span>created<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>updated<span class="pl-pds">"</span></span>]
                          }
                      }
                  }
              }
          }
      }
  }</pre></div>
  <p>And here's how to apply it to every <code>created</code> column in every table in the <code>news.db</code> database:</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
      &quot;databases&quot;: {
          &quot;news&quot;: {
              &quot;plugins&quot;: {
                  &quot;datasette-render-timestamps&quot;: {
                      &quot;columns&quot;: [&quot;created&quot;, &quot;updated&quot;]
                  }
              }
          }
      }
  }
  "><pre>{
      <span class="pl-s"><span class="pl-pds">"</span>databases<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>news<span class="pl-pds">"</span></span>: {
              <span class="pl-s"><span class="pl-pds">"</span>plugins<span class="pl-pds">"</span></span>: {
                  <span class="pl-s"><span class="pl-pds">"</span>datasette-render-timestamps<span class="pl-pds">"</span></span>: {
                      <span class="pl-s"><span class="pl-pds">"</span>columns<span class="pl-pds">"</span></span>: [<span class="pl-s"><span class="pl-pds">"</span>created<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>updated<span class="pl-pds">"</span></span>]
                  }
              }
          }
      }
  }</pre></div>
  <h3><a id="user-content-customizing-the-date-format" class="anchor" aria-hidden="true" href="#user-content-customizing-the-date-format"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Customizing the date format</h3>
  <p>The default format is <code>%B %d, %Y - %H:%M:%S UTC</code> which renders for example: <code>October 10, 2019 - 07:18:29 UTC</code>. If you want another format, the date format can be customized using plugin configuration. Any format string supported by <a href="http://strftime.org/" rel="nofollow">strftime</a> may be used. For example:</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-render-timestamps&quot;: {
              &quot;format&quot;: &quot;%Y-%m-%d-%H:%M:%S&quot;
          }
      }
  }
  "><pre>{
      <span class="pl-s"><span class="pl-pds">"</span>plugins<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>datasette-render-timestamps<span class="pl-pds">"</span></span>: {
              <span class="pl-s"><span class="pl-pds">"</span>format<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>%Y-%m-%d-%H:%M:%S<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMTcyMTY3ODc=
name: datasette-haversine
full_name: simonw/datasette-haversine
html_url: https://github.com/simonw/datasette-haversine
description: Datasette plugin that adds a custom SQL function for haversine distances
created_at: 2019-10-24T05:16:14Z
updated_at: 2021-07-28T20:13:38Z
pushed_at: 2021-07-28T20:14:24Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin", "datasette-io"]
default_branch: main
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # datasette-haversine

  [![PyPI](https://img.shields.io/pypi/v/datasette-haversine.svg)](https://pypi.org/project/datasette-haversine/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-haversine?include_prereleases&label=changelog)](https://github.com/simonw/datasette-haversine/releases)
  [![Tests](https://github.com/simonw/datasette-haversine/workflows/Test/badge.svg)](https://github.com/simonw/datasette-haversine/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-haversine/blob/main/LICENSE)

  Datasette plugin that adds a custom SQL function for haversine distances

  Install this plugin in the same environment as Datasette to enable the `haversine()` SQL function.

      $ pip install datasette-haversine

  The plugin is built on top of the [haversine](https://github.com/mapado/haversine) library.

  ## haversine() to calculate distances

  ```sql
  select haversine(lat1, lon1, lat2, lon2);
  ```

  This will return the distance in kilometers between the point defined by `(lat1, lon1)` and the point defined by `(lat2, lon2)`.

  ## Custom units

  By default `haversine()` returns results in km. You can pass an optional third argument to get results in a different unit:

  - `ft` for feet
  - `m` for meters
  - `in` for inches
  - `mi` for miles
  - `nmi` for nautical miles
  - `km` for kilometers (the default)

  ```sql
  select haversine(lat1, lon1, lat2, lon2, 'mi');
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-haversine" class="anchor" aria-hidden="true" href="#user-content-datasette-haversine"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-haversine</h1>
  <p><a href="https://pypi.org/project/datasette-haversine/" rel="nofollow"><img src="https://camo.githubusercontent.com/c15ca12319585eb769d51678b9d275b82c3dd407362c2922244fb4b2e93a9f1e/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d686176657273696e652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-haversine.svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-haversine/releases"><img src="https://camo.githubusercontent.com/2a194d8f840f0491df7941d978f40f1fda33b2edbb9b9896269ff963f72d3bfb/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d686176657273696e653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-haversine?include_prereleases&amp;label=changelog" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-haversine/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-haversine/workflows/Test/badge.svg" alt="Tests" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-haversine/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Datasette plugin that adds a custom SQL function for haversine distances</p>
  <p>Install this plugin in the same environment as Datasette to enable the <code>haversine()</code> SQL function.</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ pip install datasette-haversine
  "><pre><code>$ pip install datasette-haversine
  </code></pre></div>
  <p>The plugin is built on top of the <a href="https://github.com/mapado/haversine">haversine</a> library.</p>
  <h2><a id="user-content-haversine-to-calculate-distances" class="anchor" aria-hidden="true" href="#user-content-haversine-to-calculate-distances"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>haversine() to calculate distances</h2>
  <div class="highlight highlight-source-sql position-relative" data-snippet-clipboard-copy-content="select haversine(lat1, lon1, lat2, lon2);
  "><pre><span class="pl-k">select</span> haversine(lat1, lon1, lat2, lon2);</pre></div>
  <p>This will return the distance in kilometers between the point defined by <code>(lat1, lon1)</code> and the point defined by <code>(lat2, lon2)</code>.</p>
  <h2><a id="user-content-custom-units" class="anchor" aria-hidden="true" href="#user-content-custom-units"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Custom units</h2>
  <p>By default <code>haversine()</code> returns results in km. You can pass an optional third argument to get results in a different unit:</p>
  <ul>
  <li><code>ft</code> for feet</li>
  <li><code>m</code> for meters</li>
  <li><code>in</code> for inches</li>
  <li><code>mi</code> for miles</li>
  <li><code>nmi</code> for nautical miles</li>
  <li><code>km</code> for kilometers (the default)</li>
  </ul>
  <div class="highlight highlight-source-sql position-relative" data-snippet-clipboard-copy-content="select haversine(lat1, lon1, lat2, lon2, 'mi');
  "><pre><span class="pl-k">select</span> haversine(lat1, lon1, lat2, lon2, <span class="pl-s"><span class="pl-pds">'</span>mi<span class="pl-pds">'</span></span>);</pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMTkzNzIxMzM=
name: sqlite-transform
full_name: simonw/sqlite-transform
html_url: https://github.com/simonw/sqlite-transform
description: Tool for running transformations on columns in a SQLite database
created_at: 2019-11-03T22:07:53Z
updated_at: 2021-08-02T22:06:23Z
pushed_at: 2021-08-02T22:07:57Z
language: Python
license: apache-2.0
topics: ["sqlite", "datasette-io", "datasette-tool"]
default_branch: main
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # sqlite-transform

  ![No longer maintained](https://img.shields.io/badge/no%20longer-maintained-red)
  [![PyPI](https://img.shields.io/pypi/v/sqlite-transform.svg)](https://pypi.org/project/sqlite-transform/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-transform?include_prereleases&label=changelog)](https://github.com/simonw/sqlite-transform/releases)
  [![Tests](https://github.com/simonw/sqlite-transform/workflows/Test/badge.svg)](https://github.com/simonw/sqlite-transform/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/sqlite-transform/blob/main/LICENSE)

  Tool for running transformations on columns in a SQLite database.

  > **:warning: This tool is no longer maintained**
  >
  > I added a new tool to [sqlite-utils](https://sqlite-utils.datasette.io/) called [sqlite-utils convert](https://sqlite-utils.datasette.io/en/stable/cli.html#converting-data-in-columns) which provides a super-set of the functionality originally provided here. `sqlite-transform` is no longer maintained, and I recommend switching to using `sqlite-utils convert` instead.

  ## How to install

      pip install sqlite-transform

  ## parsedate and parsedatetime

  These subcommands will run all values in the specified column through `dateutils.parser.parse()` and replace them with the result, formatted as an ISO timestamp or ISO date.

  For example, if a row in the database has an `opened` column which contains `10/10/2019 08:10:00 PM`, running the following command:

      sqlite-transform parsedatetime my.db mytable opened

  Will result in that value being replaced by `2019-10-10T20:10:00`.

  Using the `parsedate` subcommand here would result in `2019-10-10` instead.

  In the case of ambiguous dates such as `03/04/05` these commands both default to assuming American-style `mm/dd/yy` format. You can pass `--dayfirst` to specify that the day should be assumed to be first, or `--yearfirst` for the year.

  ## jsonsplit

  The `jsonsplit` subcommand takes columns that contain a comma-separated list, for example a `tags` column containing records like `"trees,park,dogs"` and converts it into a JSON array `["trees", "park", "dogs"]`.

  This is useful for taking advantage of Datasette's [Facet by JSON array](https://docs.datasette.io/en/stable/facets.html#facet-by-json-array) feature.

      sqlite-transform jsonsplit my.db mytable tags

  It defaults to splitting on commas, but you can specify a different delimiter character using the `--delimiter` option, for example:

      sqlite-transform jsonsplit \
          my.db mytable tags --delimiter ';'

  Values within the array will be treated as strings, so a column containing `123,552,775` will be converted into the JSON array `["123", "552", "775"]`.

  You can specify a different type for these values using `--type int` or `--type float`, for example:

      sqlite-transform jsonsplit \
          my.db mytable tags --type int

  This will result in that column being converted into `[123, 552, 775]`.

  ## lambda for executing your own code

  The `lambda` subcommand lets you specify Python code which will be executed against the column.

  Here's how to convert a column to uppercase:

      sqlite-transform lambda my.db mytable mycolumn --code='str(value).upper()'

  The code you provide will be compiled into a function that takes `value` as a single argument. You can break your function body into multiple lines, provided the last line is a `return` statement:

      sqlite-transform lambda my.db mytable mycolumn --code='value = str(value)
      return value.upper()'

  You can also specify Python modules that should be imported and made available to your code using one or more `--import` options:

      sqlite-transform lambda my.db mytable mycolumn \
          --code='"\n".join(textwrap.wrap(value, 10))' \
          --import=textwrap

  The `--dry-run` option will output a preview of the transformation against the first ten rows, without modifying the database.

  ## Saving the result to a separate column

  Each of these commands accepts optional `--output` and `--output-type` options. These can be used to save the result of the transformation to a separate column, which will be created if the column does not already exist.

  To save the result of `jsonsplit` to a new column called `json_tags`, use the following:

      sqlite-transform jsonsplit my.db mytable tags \
        --output json_tags

  The type of the created column defaults to `text`, but a different column type can be specified using `--output-type`. This example will create a new floating point column called `float_id` with a copy of each item's ID increased by 0.5:

      sqlite-transform lambda my.db mytable id \
        --code 'float(value) + 0.5' \
        --output float_id \
        --output-type float

  You can drop the original column at the end of the operation by adding `--drop`.

  ## Splitting a column into multiple columns

  Sometimes you may wish to convert a single column into multiple derived columns. For example, you may have a `location` column containing `latitude,longitude` values which you wish to split out into separate `latitude` and `longitude` columns.

  You can achieve this using the `--multi` option to `sqlite-transform lambda`. This option expects your `--code` function to return a Python dictionary: new columns well be created and populated for each of the keys in that dictionary.

  For the `latitude,longitude` example you would use the following:

      sqlite-transform lambda demo.db places location \
        --code 'return {
          "latitude": float(value.split(",")[0]),
          "longitude": float(value.split(",")[1]),
        }' --multi

  The type of the returned values will be taken into account when creating the new columns. In this example, the resulting database schema will look like this:

  ```sql
  CREATE TABLE [places] (
      [location] TEXT,
      [latitude] FLOAT,
      [longitude] FLOAT
  );
  ```
  The code function can also return `None`, in which case its output will be ignored.

  You can drop the original column at the end of the operation by adding `--drop`.

  ## Disabling the progress bar

  By default each command will show a progress bar. Pass `-s` or `--silent` to hide that progress bar.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-sqlite-transform" class="anchor" aria-hidden="true" href="#user-content-sqlite-transform"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>sqlite-transform</h1>
  <p><a target="_blank" rel="noopener noreferrer" href="https://camo.githubusercontent.com/818df98789ea0f246ed427c6efefc9450fdab68f50d69b83ecbbda8dda1d82b8/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6e6f2532306c6f6e6765722d6d61696e7461696e65642d726564"><img src="https://camo.githubusercontent.com/818df98789ea0f246ed427c6efefc9450fdab68f50d69b83ecbbda8dda1d82b8/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6e6f2532306c6f6e6765722d6d61696e7461696e65642d726564" alt="No longer maintained" data-canonical-src="https://img.shields.io/badge/no%20longer-maintained-red" style="max-width:100%;"></a>
  <a href="https://pypi.org/project/sqlite-transform/" rel="nofollow"><img src="https://camo.githubusercontent.com/607faf62b18abed6c31fc21c85d2e93f7800da5d89495d6df6355b2bb0b11a38/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d7472616e73666f726d2e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/sqlite-transform.svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/sqlite-transform/releases"><img src="https://camo.githubusercontent.com/351675bc73115e1f71304704ec82e7402dd85f44a866cd2580a9e1ce4194384f/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f73716c6974652d7472616e73666f726d3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/sqlite-transform?include_prereleases&amp;label=changelog" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/sqlite-transform/actions?query=workflow%3ATest"><img src="https://github.com/simonw/sqlite-transform/workflows/Test/badge.svg" alt="Tests" style="max-width:100%;"></a>
  <a href="https://github.com/dogsheep/sqlite-transform/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Tool for running transformations on columns in a SQLite database.</p>
  <blockquote>
  <p><strong><g-emoji class="g-emoji" alias="warning" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/26a0.png">⚠️</g-emoji> This tool is no longer maintained</strong></p>
  <p>I added a new tool to <a href="https://sqlite-utils.datasette.io/" rel="nofollow">sqlite-utils</a> called <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#converting-data-in-columns" rel="nofollow">sqlite-utils convert</a> which provides a super-set of the functionality originally provided here. <code>sqlite-transform</code> is no longer maintained, and I recommend switching to using <code>sqlite-utils convert</code> instead.</p>
  </blockquote>
  <h2><a id="user-content-how-to-install" class="anchor" aria-hidden="true" href="#user-content-how-to-install"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How to install</h2>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install sqlite-transform
  "><pre><code>pip install sqlite-transform
  </code></pre></div>
  <h2><a id="user-content-parsedate-and-parsedatetime" class="anchor" aria-hidden="true" href="#user-content-parsedate-and-parsedatetime"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>parsedate and parsedatetime</h2>
  <p>These subcommands will run all values in the specified column through <code>dateutils.parser.parse()</code> and replace them with the result, formatted as an ISO timestamp or ISO date.</p>
  <p>For example, if a row in the database has an <code>opened</code> column which contains <code>10/10/2019 08:10:00 PM</code>, running the following command:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="sqlite-transform parsedatetime my.db mytable opened
  "><pre><code>sqlite-transform parsedatetime my.db mytable opened
  </code></pre></div>
  <p>Will result in that value being replaced by <code>2019-10-10T20:10:00</code>.</p>
  <p>Using the <code>parsedate</code> subcommand here would result in <code>2019-10-10</code> instead.</p>
  <p>In the case of ambiguous dates such as <code>03/04/05</code> these commands both default to assuming American-style <code>mm/dd/yy</code> format. You can pass <code>--dayfirst</code> to specify that the day should be assumed to be first, or <code>--yearfirst</code> for the year.</p>
  <h2><a id="user-content-jsonsplit" class="anchor" aria-hidden="true" href="#user-content-jsonsplit"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>jsonsplit</h2>
  <p>The <code>jsonsplit</code> subcommand takes columns that contain a comma-separated list, for example a <code>tags</code> column containing records like <code>"trees,park,dogs"</code> and converts it into a JSON array <code>["trees", "park", "dogs"]</code>.</p>
  <p>This is useful for taking advantage of Datasette's <a href="https://docs.datasette.io/en/stable/facets.html#facet-by-json-array" rel="nofollow">Facet by JSON array</a> feature.</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="sqlite-transform jsonsplit my.db mytable tags
  "><pre><code>sqlite-transform jsonsplit my.db mytable tags
  </code></pre></div>
  <p>It defaults to splitting on commas, but you can specify a different delimiter character using the <code>--delimiter</code> option, for example:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="sqlite-transform jsonsplit \
      my.db mytable tags --delimiter ';'
  "><pre><code>sqlite-transform jsonsplit \
      my.db mytable tags --delimiter ';'
  </code></pre></div>
  <p>Values within the array will be treated as strings, so a column containing <code>123,552,775</code> will be converted into the JSON array <code>["123", "552", "775"]</code>.</p>
  <p>You can specify a different type for these values using <code>--type int</code> or <code>--type float</code>, for example:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="sqlite-transform jsonsplit \
      my.db mytable tags --type int
  "><pre><code>sqlite-transform jsonsplit \
      my.db mytable tags --type int
  </code></pre></div>
  <p>This will result in that column being converted into <code>[123, 552, 775]</code>.</p>
  <h2><a id="user-content-lambda-for-executing-your-own-code" class="anchor" aria-hidden="true" href="#user-content-lambda-for-executing-your-own-code"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>lambda for executing your own code</h2>
  <p>The <code>lambda</code> subcommand lets you specify Python code which will be executed against the column.</p>
  <p>Here's how to convert a column to uppercase:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="sqlite-transform lambda my.db mytable mycolumn --code='str(value).upper()'
  "><pre><code>sqlite-transform lambda my.db mytable mycolumn --code='str(value).upper()'
  </code></pre></div>
  <p>The code you provide will be compiled into a function that takes <code>value</code> as a single argument. You can break your function body into multiple lines, provided the last line is a <code>return</code> statement:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="sqlite-transform lambda my.db mytable mycolumn --code='value = str(value)
  return value.upper()'
  "><pre><code>sqlite-transform lambda my.db mytable mycolumn --code='value = str(value)
  return value.upper()'
  </code></pre></div>
  <p>You can also specify Python modules that should be imported and made available to your code using one or more <code>--import</code> options:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="sqlite-transform lambda my.db mytable mycolumn \
      --code='&quot;\n&quot;.join(textwrap.wrap(value, 10))' \
      --import=textwrap
  "><pre><code>sqlite-transform lambda my.db mytable mycolumn \
      --code='"\n".join(textwrap.wrap(value, 10))' \
      --import=textwrap
  </code></pre></div>
  <p>The <code>--dry-run</code> option will output a preview of the transformation against the first ten rows, without modifying the database.</p>
  <h2><a id="user-content-saving-the-result-to-a-separate-column" class="anchor" aria-hidden="true" href="#user-content-saving-the-result-to-a-separate-column"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Saving the result to a separate column</h2>
  <p>Each of these commands accepts optional <code>--output</code> and <code>--output-type</code> options. These can be used to save the result of the transformation to a separate column, which will be created if the column does not already exist.</p>
  <p>To save the result of <code>jsonsplit</code> to a new column called <code>json_tags</code>, use the following:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="sqlite-transform jsonsplit my.db mytable tags \
    --output json_tags
  "><pre><code>sqlite-transform jsonsplit my.db mytable tags \
    --output json_tags
  </code></pre></div>
  <p>The type of the created column defaults to <code>text</code>, but a different column type can be specified using <code>--output-type</code>. This example will create a new floating point column called <code>float_id</code> with a copy of each item's ID increased by 0.5:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="sqlite-transform lambda my.db mytable id \
    --code 'float(value) + 0.5' \
    --output float_id \
    --output-type float
  "><pre><code>sqlite-transform lambda my.db mytable id \
    --code 'float(value) + 0.5' \
    --output float_id \
    --output-type float
  </code></pre></div>
  <p>You can drop the original column at the end of the operation by adding <code>--drop</code>.</p>
  <h2><a id="user-content-splitting-a-column-into-multiple-columns" class="anchor" aria-hidden="true" href="#user-content-splitting-a-column-into-multiple-columns"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Splitting a column into multiple columns</h2>
  <p>Sometimes you may wish to convert a single column into multiple derived columns. For example, you may have a <code>location</code> column containing <code>latitude,longitude</code> values which you wish to split out into separate <code>latitude</code> and <code>longitude</code> columns.</p>
  <p>You can achieve this using the <code>--multi</code> option to <code>sqlite-transform lambda</code>. This option expects your <code>--code</code> function to return a Python dictionary: new columns well be created and populated for each of the keys in that dictionary.</p>
  <p>For the <code>latitude,longitude</code> example you would use the following:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="sqlite-transform lambda demo.db places location \
    --code 'return {
      &quot;latitude&quot;: float(value.split(&quot;,&quot;)[0]),
      &quot;longitude&quot;: float(value.split(&quot;,&quot;)[1]),
    }' --multi
  "><pre><code>sqlite-transform lambda demo.db places location \
    --code 'return {
      "latitude": float(value.split(",")[0]),
      "longitude": float(value.split(",")[1]),
    }' --multi
  </code></pre></div>
  <p>The type of the returned values will be taken into account when creating the new columns. In this example, the resulting database schema will look like this:</p>
  <div class="highlight highlight-source-sql position-relative" data-snippet-clipboard-copy-content="CREATE TABLE [places] (
      [location] TEXT,
      [latitude] FLOAT,
      [longitude] FLOAT
  );
  "><pre>CREATE TABLE [places] (
      [location] <span class="pl-k">TEXT</span>,
      [latitude] FLOAT,
      [longitude] FLOAT
  );</pre></div>
  <p>The code function can also return <code>None</code>, in which case its output will be ignored.</p>
  <p>You can drop the original column at the end of the operation by adding <code>--drop</code>.</p>
  <h2><a id="user-content-disabling-the-progress-bar" class="anchor" aria-hidden="true" href="#user-content-disabling-the-progress-bar"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Disabling the progress bar</h2>
  <p>By default each command will show a progress bar. Pass <code>-s</code> or <code>--silent</code> to hide that progress bar.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMjA3MTY4MjI=
name: datasette-render-markdown
full_name: simonw/datasette-render-markdown
html_url: https://github.com/simonw/datasette-render-markdown
description: Datasette plugin for rendering Markdown
created_at: 2019-11-09T23:28:31Z
updated_at: 2022-05-26T04:58:56Z
pushed_at: 2022-07-18T19:35:10Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "markdown"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-render-markdown

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-markdown.svg)](https://pypi.org/project/datasette-render-markdown/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-render-markdown?include_prereleases&label=changelog)](https://github.com/simonw/datasette-render-markdown/releases)
  [![Tests](https://github.com/simonw/datasette-render-markdown/workflows/Test/badge.svg)](https://github.com/simonw/datasette-render-markdown/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-markdown/blob/main/LICENSE)

  Datasette plugin for rendering Markdown.

  ## Installation

  Install this plugin in the same environment as Datasette to enable this new functionality:

      $ pip install datasette-render-markdown

  ## Usage

  You can explicitly list the columns you would like to treat as Markdown using [plugin configuration](https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration) in a `metadata.json` file.

  Add a `"datasette-render-markdown"` configuration block and use a `"columns"` key to list the columns you would like to treat as Markdown values:

  ```json
  {
      "plugins": {
          "datasette-render-markdown": {
              "columns": ["body"]
          }
      }
  }
  ```

  This will cause any `body` column in any table to be treated as markdown and safely rendered using [Python-Markdown](https://python-markdown.github.io/). The resulting HTML is then run through [Bleach](https://bleach.readthedocs.io/) to avoid the risk of XSS security problems.

  Save this to `metadata.json` and run Datasette with the `--metadata` flag to load this configuration:

      $ datasette serve mydata.db --metadata metadata.json

  The configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the `entries` table in the `news.db` database:

  ```json
  {
      "databases": {
          "news": {
              "tables": {
                  "entries": {
                      "plugins": {
                          "datasette-render-markdown": {
                              "columns": ["body"]
                          }
                      }
                  }
              }
          }
      }
  }
  ```

  And here's how to apply it to every `body` column in every table in the `news.db` database:

  ```json
  {
      "databases": {
          "news": {
              "plugins": {
                  "datasette-render-markdown": {
                      "columns": ["body"]
                  }
              }
          }
      }
  }
  ```

  ## Columns that match a naming convention

  This plugin can also render markdown in any columns that match a specific naming convention.

  By default, columns that have a name ending in `_markdown` will be rendered.

  You can try this out using the following query:

  ```sql
  select '# Hello there

  * This is a list
  * of items

  [And a link](https://github.com/simonw/datasette-render-markdown).'
  as demo_markdown
  ```

  You can configure a different list of wildcard patterns using the `"patterns"` configuration key. Here's how to render columns that end in either `_markdown` or `_md`:

  ```json
  {
      "plugins": {
          "datasette-render-markdown": {
              "patterns": ["*_markdown", "*_md"]
          }
      }
  }
  ```

  To disable wildcard column matching entirely, set `"patterns": []` in your plugin metadata configuration.

  ## Markdown extensions

  The [Python-Markdown library](https://python-markdown.github.io/) that powers this plugin supports extensions, both [bundled](https://python-markdown.github.io/extensions/) and [third-party](https://github.com/Python-Markdown/markdown/wiki/Third-Party-Extensions). These can be used to enable additional Markdown features such as [table support](https://python-markdown.github.io/extensions/tables/).

  You can configure support for extensions using the `"extensions"` key in your plugin metadata configuration.

  Since extensions may introduce new HTML tags, you will also need to add those tags to the list of tags that are allowed by the [Bleach](https://bleach.readthedocs.io/) sanitizer. You can do that using the `"extra_tags"` key, and you can whitelist additional HTML attributes using `"extra_attrs"`. See [the Bleach documentation](https://bleach.readthedocs.io/en/latest/clean.html#allowed-tags-tags) for more information on this.

  Here's how to enable support for [Markdown tables](https://python-markdown.github.io/extensions/tables/):

  ```json
  {
      "plugins": {
          "datasette-render-markdown": {
              "extensions": ["tables"],
              "extra_tags": ["table", "thead", "tr", "th", "td", "tbody"]
          }
      }
  }
  ```

  ### GitHub-Flavored Markdown

  Enabling [GitHub-Flavored Markdown](https://help.github.com/en/github/writing-on-github) (useful for if you are working with data imported from GitHub using [github-to-sqlite](https://github.com/dogsheep/github-to-sqlite)) is a little more complicated.

  First, you will need to install the [py-gfm](https://py-gfm.readthedocs.io) package:

      $ pip install py-gfm

  Note that `py-gfm` has [a bug](https://github.com/Zopieux/py-gfm/issues/13) that causes it to pin to `Markdown<3.0` - so if you are using it you should install it _before_ installing `datasette-render-markdown` to ensure you get a compatibly version of that dependency.

  Now you can configure it like this. Note that the extension name is `mdx_gfm:GithubFlavoredMarkdownExtension` and you need to whitelist several extra HTML tags and attributes:

  ```json
  {
      "plugins": {
          "datasette-render-markdown": {
              "extra_tags": [
                  "hr",
                  "br",
                  "details",
                  "summary",
                  "input"
              ],
              "extra_attrs": {
                  "input": [
                      "type",
                      "disabled",
                      "checked"
                  ],
              },
              "extensions": [
                  "mdx_gfm:GithubFlavoredMarkdownExtension"
              ]
          }
      }
  }
  ```

  The `<input type="" checked disabled>` attributes are needed to support rendering checkboxes in issue descriptions.

  ## Markdown in templates

  The plugin also adds a new template function: `render_markdown(value)`. You can use this in your templates like so:

  ```html+jinja
  {{ render_markdown("""
  # This is markdown

  * One
  * Two
  * Three
  """) }}
  ```

  You can load additional extensions and whitelist tags by passing extra arguments to the function like this:

  ```html+jinja
  {{ render_markdown("""
  ## Markdown table

  First Header  | Second Header
  ------------- | -------------
  Content Cell  | Content Cell
  Content Cell  | Content Cell
  """, extensions=["tables"],
      extra_tags=["table", "thead", "tr", "th", "td", "tbody"])) }}
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-render-markdown" class="anchor" aria-hidden="true" href="#user-content-datasette-render-markdown"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-render-markdown</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-render-markdown/" rel="nofollow"><img src="https://camo.githubusercontent.com/624f87a070fbf68882456039073cd2f330c4b412199efd1bd7308c97c3b30c40/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d72656e6465722d6d61726b646f776e2e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-render-markdown.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-render-markdown/releases"><img src="https://camo.githubusercontent.com/5b3b13cdd601c70ee249e42c3fb43fce5b707eff875d49f19b1460a319cc42d6/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d72656e6465722d6d61726b646f776e3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-render-markdown?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-render-markdown/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-render-markdown/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-render-markdown/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Datasette plugin for rendering Markdown.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette to enable this new functionality:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-render-markdown"><pre class="notranslate"><code>$ pip install datasette-render-markdown
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">You can explicitly list the columns you would like to treat as Markdown using <a href="https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration" rel="nofollow">plugin configuration</a> in a <code>metadata.json</code> file.</p>
  <p dir="auto">Add a <code>"datasette-render-markdown"</code> configuration block and use a <code>"columns"</code> key to list the columns you would like to treat as Markdown values:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-render-markdown&quot;: {
              &quot;columns&quot;: [&quot;body&quot;]
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-render-markdown"</span>: {
              <span class="pl-ent">"columns"</span>: [<span class="pl-s"><span class="pl-pds">"</span>body<span class="pl-pds">"</span></span>]
          }
      }
  }</pre></div>
  <p dir="auto">This will cause any <code>body</code> column in any table to be treated as markdown and safely rendered using <a href="https://python-markdown.github.io/" rel="nofollow">Python-Markdown</a>. The resulting HTML is then run through <a href="https://bleach.readthedocs.io/" rel="nofollow">Bleach</a> to avoid the risk of XSS security problems.</p>
  <p dir="auto">Save this to <code>metadata.json</code> and run Datasette with the <code>--metadata</code> flag to load this configuration:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette serve mydata.db --metadata metadata.json"><pre class="notranslate"><code>$ datasette serve mydata.db --metadata metadata.json
  </code></pre></div>
  <p dir="auto">The configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the <code>entries</code> table in the <code>news.db</code> database:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;databases&quot;: {
          &quot;news&quot;: {
              &quot;tables&quot;: {
                  &quot;entries&quot;: {
                      &quot;plugins&quot;: {
                          &quot;datasette-render-markdown&quot;: {
                              &quot;columns&quot;: [&quot;body&quot;]
                          }
                      }
                  }
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"databases"</span>: {
          <span class="pl-ent">"news"</span>: {
              <span class="pl-ent">"tables"</span>: {
                  <span class="pl-ent">"entries"</span>: {
                      <span class="pl-ent">"plugins"</span>: {
                          <span class="pl-ent">"datasette-render-markdown"</span>: {
                              <span class="pl-ent">"columns"</span>: [<span class="pl-s"><span class="pl-pds">"</span>body<span class="pl-pds">"</span></span>]
                          }
                      }
                  }
              }
          }
      }
  }</pre></div>
  <p dir="auto">And here's how to apply it to every <code>body</code> column in every table in the <code>news.db</code> database:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;databases&quot;: {
          &quot;news&quot;: {
              &quot;plugins&quot;: {
                  &quot;datasette-render-markdown&quot;: {
                      &quot;columns&quot;: [&quot;body&quot;]
                  }
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"databases"</span>: {
          <span class="pl-ent">"news"</span>: {
              <span class="pl-ent">"plugins"</span>: {
                  <span class="pl-ent">"datasette-render-markdown"</span>: {
                      <span class="pl-ent">"columns"</span>: [<span class="pl-s"><span class="pl-pds">"</span>body<span class="pl-pds">"</span></span>]
                  }
              }
          }
      }
  }</pre></div>
  <h2 dir="auto"><a id="user-content-columns-that-match-a-naming-convention" class="anchor" aria-hidden="true" href="#user-content-columns-that-match-a-naming-convention"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Columns that match a naming convention</h2>
  <p dir="auto">This plugin can also render markdown in any columns that match a specific naming convention.</p>
  <p dir="auto">By default, columns that have a name ending in <code>_markdown</code> will be rendered.</p>
  <p dir="auto">You can try this out using the following query:</p>
  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="select '# Hello there

  * This is a list
  * of items

  [And a link](https://github.com/simonw/datasette-render-markdown).'
  as demo_markdown"><pre><span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span># Hello there</span>
  <span class="pl-s"></span>
  <span class="pl-s">* This is a list</span>
  <span class="pl-s">* of items</span>
  <span class="pl-s"></span>
  <span class="pl-s">[And a link](https://github.com/simonw/datasette-render-markdown).<span class="pl-pds">'</span></span>
  <span class="pl-k">as</span> demo_markdown</pre></div>
  <p dir="auto">You can configure a different list of wildcard patterns using the <code>"patterns"</code> configuration key. Here's how to render columns that end in either <code>_markdown</code> or <code>_md</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-render-markdown&quot;: {
              &quot;patterns&quot;: [&quot;*_markdown&quot;, &quot;*_md&quot;]
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-render-markdown"</span>: {
              <span class="pl-ent">"patterns"</span>: [<span class="pl-s"><span class="pl-pds">"</span>*_markdown<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>*_md<span class="pl-pds">"</span></span>]
          }
      }
  }</pre></div>
  <p dir="auto">To disable wildcard column matching entirely, set <code>"patterns": []</code> in your plugin metadata configuration.</p>
  <h2 dir="auto"><a id="user-content-markdown-extensions" class="anchor" aria-hidden="true" href="#user-content-markdown-extensions"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Markdown extensions</h2>
  <p dir="auto">The <a href="https://python-markdown.github.io/" rel="nofollow">Python-Markdown library</a> that powers this plugin supports extensions, both <a href="https://python-markdown.github.io/extensions/" rel="nofollow">bundled</a> and <a href="https://github.com/Python-Markdown/markdown/wiki/Third-Party-Extensions">third-party</a>. These can be used to enable additional Markdown features such as <a href="https://python-markdown.github.io/extensions/tables/" rel="nofollow">table support</a>.</p>
  <p dir="auto">You can configure support for extensions using the <code>"extensions"</code> key in your plugin metadata configuration.</p>
  <p dir="auto">Since extensions may introduce new HTML tags, you will also need to add those tags to the list of tags that are allowed by the <a href="https://bleach.readthedocs.io/" rel="nofollow">Bleach</a> sanitizer. You can do that using the <code>"extra_tags"</code> key, and you can whitelist additional HTML attributes using <code>"extra_attrs"</code>. See <a href="https://bleach.readthedocs.io/en/latest/clean.html#allowed-tags-tags" rel="nofollow">the Bleach documentation</a> for more information on this.</p>
  <p dir="auto">Here's how to enable support for <a href="https://python-markdown.github.io/extensions/tables/" rel="nofollow">Markdown tables</a>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-render-markdown&quot;: {
              &quot;extensions&quot;: [&quot;tables&quot;],
              &quot;extra_tags&quot;: [&quot;table&quot;, &quot;thead&quot;, &quot;tr&quot;, &quot;th&quot;, &quot;td&quot;, &quot;tbody&quot;]
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-render-markdown"</span>: {
              <span class="pl-ent">"extensions"</span>: [<span class="pl-s"><span class="pl-pds">"</span>tables<span class="pl-pds">"</span></span>],
              <span class="pl-ent">"extra_tags"</span>: [<span class="pl-s"><span class="pl-pds">"</span>table<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>thead<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>tr<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>th<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>td<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>tbody<span class="pl-pds">"</span></span>]
          }
      }
  }</pre></div>
  <h3 dir="auto"><a id="user-content-github-flavored-markdown" class="anchor" aria-hidden="true" href="#user-content-github-flavored-markdown"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>GitHub-Flavored Markdown</h3>
  <p dir="auto">Enabling <a href="https://help.github.com/en/github/writing-on-github">GitHub-Flavored Markdown</a> (useful for if you are working with data imported from GitHub using <a href="https://github.com/dogsheep/github-to-sqlite">github-to-sqlite</a>) is a little more complicated.</p>
  <p dir="auto">First, you will need to install the <a href="https://py-gfm.readthedocs.io" rel="nofollow">py-gfm</a> package:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install py-gfm"><pre class="notranslate"><code>$ pip install py-gfm
  </code></pre></div>
  <p dir="auto">Note that <code>py-gfm</code> has <a href="https://github.com/Zopieux/py-gfm/issues/13" data-hovercard-type="issue" data-hovercard-url="/zopieux/py-gfm/issues/13/hovercard">a bug</a> that causes it to pin to <code>Markdown&lt;3.0</code> - so if you are using it you should install it <em>before</em> installing <code>datasette-render-markdown</code> to ensure you get a compatibly version of that dependency.</p>
  <p dir="auto">Now you can configure it like this. Note that the extension name is <code>mdx_gfm:GithubFlavoredMarkdownExtension</code> and you need to whitelist several extra HTML tags and attributes:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-render-markdown&quot;: {
              &quot;extra_tags&quot;: [
                  &quot;hr&quot;,
                  &quot;br&quot;,
                  &quot;details&quot;,
                  &quot;summary&quot;,
                  &quot;input&quot;
              ],
              &quot;extra_attrs&quot;: {
                  &quot;input&quot;: [
                      &quot;type&quot;,
                      &quot;disabled&quot;,
                      &quot;checked&quot;
                  ],
              },
              &quot;extensions&quot;: [
                  &quot;mdx_gfm:GithubFlavoredMarkdownExtension&quot;
              ]
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-render-markdown"</span>: {
              <span class="pl-ent">"extra_tags"</span>: [
                  <span class="pl-s"><span class="pl-pds">"</span>hr<span class="pl-pds">"</span></span>,
                  <span class="pl-s"><span class="pl-pds">"</span>br<span class="pl-pds">"</span></span>,
                  <span class="pl-s"><span class="pl-pds">"</span>details<span class="pl-pds">"</span></span>,
                  <span class="pl-s"><span class="pl-pds">"</span>summary<span class="pl-pds">"</span></span>,
                  <span class="pl-s"><span class="pl-pds">"</span>input<span class="pl-pds">"</span></span>
              ],
              <span class="pl-ent">"extra_attrs"</span>: {
                  <span class="pl-ent">"input"</span>: [
                      <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>,
                      <span class="pl-s"><span class="pl-pds">"</span>disabled<span class="pl-pds">"</span></span>,
                      <span class="pl-s"><span class="pl-pds">"</span>checked<span class="pl-pds">"</span></span>
                  ],
              },
              <span class="pl-ent">"extensions"</span>: [
                  <span class="pl-s"><span class="pl-pds">"</span>mdx_gfm:GithubFlavoredMarkdownExtension<span class="pl-pds">"</span></span>
              ]
          }
      }
  }</pre></div>
  <p dir="auto">The <code>&lt;input type="" checked disabled&gt;</code> attributes are needed to support rendering checkboxes in issue descriptions.</p>
  <h2 dir="auto"><a id="user-content-markdown-in-templates" class="anchor" aria-hidden="true" href="#user-content-markdown-in-templates"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Markdown in templates</h2>
  <p dir="auto">The plugin also adds a new template function: <code>render_markdown(value)</code>. You can use this in your templates like so:</p>
  <div class="highlight highlight-text-html-django notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{{ render_markdown(&quot;&quot;&quot;
  # This is markdown

  * One
  * Two
  * Three
  &quot;&quot;&quot;) }}"><pre>{{ render_markdown("""
  # This is markdown

  * One
  * Two
  * Three
  """) }}</pre></div>
  <p dir="auto">You can load additional extensions and whitelist tags by passing extra arguments to the function like this:</p>
  <div class="highlight highlight-text-html-django notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{{ render_markdown(&quot;&quot;&quot;
  ## Markdown table

  First Header  | Second Header
  ------------- | -------------
  Content Cell  | Content Cell
  Content Cell  | Content Cell
  &quot;&quot;&quot;, extensions=[&quot;tables&quot;],
      extra_tags=[&quot;table&quot;, &quot;thead&quot;, &quot;tr&quot;, &quot;th&quot;, &quot;td&quot;, &quot;tbody&quot;])) }}"><pre>{{ render_markdown("""
  ## Markdown table

  First Header  | Second Header
  ------------- | -------------
  Content Cell  | Content Cell
  Content Cell  | Content Cell
  """, extensions=["tables<span class="pl-smi">"]</span>,
      extra_tags=["table", "thead", "tr", "th", "td", "tbody<span class="pl-smi">"]</span>)) }}</pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMjE4MDIyOTY=
name: datasette-template-sql
full_name: simonw/datasette-template-sql
html_url: https://github.com/simonw/datasette-template-sql
description: Datasette plugin for executing SQL queries from templates
created_at: 2019-11-14T23:05:34Z
updated_at: 2021-05-18T17:58:47Z
pushed_at: 2021-05-18T17:58:44Z
homepage: https://datasette.io/plugins/datasette-template-sql
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin", "datasette-io"]
default_branch: main
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # datasette-template-sql

  [![PyPI](https://img.shields.io/pypi/v/datasette-template-sql.svg)](https://pypi.org/project/datasette-template-sql/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-template-sql?include_prereleases&label=changelog)](https://github.com/simonw/datasette-template-sql/releases)
  [![Tests](https://github.com/simonw/datasette-template-sql/workflows/Test/badge.svg)](https://github.com/simonw/datasette-template-sql/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-template-sql/blob/main/LICENSE)

  Datasette plugin for executing SQL queries from templates.

  ## Examples

  [datasette.io](https://datasette.io/) uses this plugin extensively with [custom page templates](https://docs.datasette.io/en/stable/custom_templates.html#custom-pages), check out [simonw/datasette.io](https://github.com/simonw/datasette.io) to see how it works.

  [www.niche-museums.com](https://www.niche-museums.com/) uses this plugin to run a custom themed website on top of Datasette. The full source code for the site [is here](https://github.com/simonw/museums) - see also [niche-museums.com, powered by Datasette](https://simonwillison.net/2019/Nov/25/niche-museums/).

  [simonw/til](https://github.com/simonw/til) is another simple example, described in [Using a self-rewriting README powered by GitHub Actions to track TILs](https://simonwillison.net/2020/Apr/20/self-rewriting-readme/).

  ## Installation

  Run this command to install the plugin in the same environment as Datasette:

      $ pip install datasette-template-sql

  ## Usage

  This plugin makes a new function, `sql(sql_query)`, available to your Datasette templates.

  You can use it like this:

  ```html+jinja
  {% for row in sql("select 1 + 1 as two, 2 * 4 as eight") %}
      {% for key in row.keys() %}
          {{ key }}: {{ row[key] }}<br>
      {% endfor %}
  {% endfor %}
  ```

  The plugin will execute SQL against the current database for the page in  `database.html`, `table.html` and `row.html` templates. If a template does not have a current database (`index.html` for example) the query will execute against the first attached database.

  ### Queries with arguments

  You can construct a SQL query using `?` or `:name` parameter syntax by passing a list or dictionary as a second argument:

  ```html+jinja
  {% for row in sql("select distinct topic from til order by topic") %}
      <h2>{{ row.topic }}</h2>
      <ul>
          {% for til in sql("select * from til where topic = ?", [row.topic]) %}
              <li><a href="{{ til.url }}">{{ til.title }}</a> - {{ til.created[:10] }}</li>
          {% endfor %}
      </ul>
  {% endfor %}
  ```

  Here's the same example using the `:topic` style of parameters:

  ```html+jinja
  {% for row in sql("select distinct topic from til order by topic") %}
      <h2>{{ row.topic }}</h2>
      <ul>
          {% for til in sql("select * from til where topic = :topic", {"topic": row.topic}) %}
              <li><a href="{{ til.url }}">{{ til.title }}</a> - {{ til.created[:10] }}</li>
          {% endfor %}
      </ul>
  {% endfor %}
  ```

  ### Querying a different database

  You can pass an optional `database=` argument to specify a named database to use for the query. For example, if you have attached a `news.db` database you could use this:

  ```html+jinja
  {% for article in sql(
      "select headline, date, summary from articles order by date desc limit 5",
      database="news"
  ) %}
      <h3>{{ article.headline }}</h2>
      <p class="date">{{ article.date }}</p>
      <p>{{ article.summary }}</p>
  {% endfor %}
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-template-sql" class="anchor" aria-hidden="true" href="#user-content-datasette-template-sql"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-template-sql</h1>
  <p><a href="https://pypi.org/project/datasette-template-sql/" rel="nofollow"><img src="https://camo.githubusercontent.com/4de02f94922198f24958d43afbaa62802bf36bc44849bdf7bfb630d8bd7d5dde/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d74656d706c6174652d73716c2e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-template-sql.svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-template-sql/releases"><img src="https://camo.githubusercontent.com/564e7d6e3e049c80e9c446272b50dd5ce9c6d4b573ad51a7f1e9e51756affbcb/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d74656d706c6174652d73716c3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-template-sql?include_prereleases&amp;label=changelog" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-template-sql/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-template-sql/workflows/Test/badge.svg" alt="Tests" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-template-sql/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Datasette plugin for executing SQL queries from templates.</p>
  <h2><a id="user-content-examples" class="anchor" aria-hidden="true" href="#user-content-examples"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Examples</h2>
  <p><a href="https://datasette.io/" rel="nofollow">datasette.io</a> uses this plugin extensively with <a href="https://docs.datasette.io/en/stable/custom_templates.html#custom-pages" rel="nofollow">custom page templates</a>, check out <a href="https://github.com/simonw/datasette.io">simonw/datasette.io</a> to see how it works.</p>
  <p><a href="https://www.niche-museums.com/" rel="nofollow">www.niche-museums.com</a> uses this plugin to run a custom themed website on top of Datasette. The full source code for the site <a href="https://github.com/simonw/museums">is here</a> - see also <a href="https://simonwillison.net/2019/Nov/25/niche-museums/" rel="nofollow">niche-museums.com, powered by Datasette</a>.</p>
  <p><a href="https://github.com/simonw/til">simonw/til</a> is another simple example, described in <a href="https://simonwillison.net/2020/Apr/20/self-rewriting-readme/" rel="nofollow">Using a self-rewriting README powered by GitHub Actions to track TILs</a>.</p>
  <h2><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p>Run this command to install the plugin in the same environment as Datasette:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ pip install datasette-template-sql
  "><pre><code>$ pip install datasette-template-sql
  </code></pre></div>
  <h2><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p>This plugin makes a new function, <code>sql(sql_query)</code>, available to your Datasette templates.</p>
  <p>You can use it like this:</p>
  <div class="highlight highlight-text-html-django position-relative" data-snippet-clipboard-copy-content="{% for row in sql(&quot;select 1 + 1 as two, 2 * 4 as eight&quot;) %}
      {% for key in row.keys() %}
          {{ key }}: {{ row[key] }}&lt;br&gt;
      {% endfor %}
  {% endfor %}
  "><pre><span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">row</span> <span class="pl-k">in</span> <span class="pl-s">sql</span>(<span class="pl-s">"select 1 + 1 as two, 2 * 4 as eight"</span>) <span class="pl-e">%}</span>
      <span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">key</span> <span class="pl-k">in</span> <span class="pl-s">row</span>.<span class="pl-s">keys</span>() <span class="pl-e">%}</span>
          {{ key }}: {{ row[ke<span class="pl-smi">y]</span> }}&lt;<span class="pl-ent">br</span>&gt;
      <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span>
  <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span></pre></div>
  <p>The plugin will execute SQL against the current database for the page in  <code>database.html</code>, <code>table.html</code> and <code>row.html</code> templates. If a template does not have a current database (<code>index.html</code> for example) the query will execute against the first attached database.</p>
  <h3><a id="user-content-queries-with-arguments" class="anchor" aria-hidden="true" href="#user-content-queries-with-arguments"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Queries with arguments</h3>
  <p>You can construct a SQL query using <code>?</code> or <code>:name</code> parameter syntax by passing a list or dictionary as a second argument:</p>
  <div class="highlight highlight-text-html-django position-relative" data-snippet-clipboard-copy-content="{% for row in sql(&quot;select distinct topic from til order by topic&quot;) %}
      &lt;h2&gt;{{ row.topic }}&lt;/h2&gt;
      &lt;ul&gt;
          {% for til in sql(&quot;select * from til where topic = ?&quot;, [row.topic]) %}
              &lt;li&gt;&lt;a href=&quot;{{ til.url }}&quot;&gt;{{ til.title }}&lt;/a&gt; - {{ til.created[:10] }}&lt;/li&gt;
          {% endfor %}
      &lt;/ul&gt;
  {% endfor %}
  "><pre><span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">row</span> <span class="pl-k">in</span> <span class="pl-s">sql</span>(<span class="pl-s">"select distinct topic from til order by topic"</span>) <span class="pl-e">%}</span>
      &lt;<span class="pl-ent">h2</span>&gt;{{ row.topic }}&lt;/<span class="pl-ent">h2</span>&gt;
      &lt;<span class="pl-ent">ul</span>&gt;
          <span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">til</span> <span class="pl-k">in</span> <span class="pl-s">sql</span>(<span class="pl-s">"select * from til where topic = ?"</span>, [<span class="pl-s">row</span>.<span class="pl-s">topic</span>]) <span class="pl-e">%}</span>
              &lt;<span class="pl-ent">li</span>&gt;&lt;<span class="pl-ent">a</span> <span class="pl-e">href</span>=<span class="pl-s"><span class="pl-pds">"</span>{{ til.url }}<span class="pl-pds">"</span></span>&gt;{{ til.title }}&lt;/<span class="pl-ent">a</span>&gt; - {{ til.created[:1<span class="pl-smi">0]</span> }}&lt;/<span class="pl-ent">li</span>&gt;
          <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span>
      &lt;/<span class="pl-ent">ul</span>&gt;
  <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span></pre></div>
  <p>Here's the same example using the <code>:topic</code> style of parameters:</p>
  <div class="highlight highlight-text-html-django position-relative" data-snippet-clipboard-copy-content="{% for row in sql(&quot;select distinct topic from til order by topic&quot;) %}
      &lt;h2&gt;{{ row.topic }}&lt;/h2&gt;
      &lt;ul&gt;
          {% for til in sql(&quot;select * from til where topic = :topic&quot;, {&quot;topic&quot;: row.topic}) %}
              &lt;li&gt;&lt;a href=&quot;{{ til.url }}&quot;&gt;{{ til.title }}&lt;/a&gt; - {{ til.created[:10] }}&lt;/li&gt;
          {% endfor %}
      &lt;/ul&gt;
  {% endfor %}
  "><pre><span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">row</span> <span class="pl-k">in</span> <span class="pl-s">sql</span>(<span class="pl-s">"select distinct topic from til order by topic"</span>) <span class="pl-e">%}</span>
      &lt;<span class="pl-ent">h2</span>&gt;{{ row.topic }}&lt;/<span class="pl-ent">h2</span>&gt;
      &lt;<span class="pl-ent">ul</span>&gt;
          <span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">til</span> <span class="pl-k">in</span> <span class="pl-s">sql</span>(<span class="pl-s">"select * from til where topic = :topic"</span>, {<span class="pl-s">"topic"</span>: <span class="pl-s">row</span>.<span class="pl-s">topic</span>}) <span class="pl-e">%}</span>
              &lt;<span class="pl-ent">li</span>&gt;&lt;<span class="pl-ent">a</span> <span class="pl-e">href</span>=<span class="pl-s"><span class="pl-pds">"</span>{{ til.url }}<span class="pl-pds">"</span></span>&gt;{{ til.title }}&lt;/<span class="pl-ent">a</span>&gt; - {{ til.created[:1<span class="pl-smi">0]</span> }}&lt;/<span class="pl-ent">li</span>&gt;
          <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span>
      &lt;/<span class="pl-ent">ul</span>&gt;
  <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span></pre></div>
  <h3><a id="user-content-querying-a-different-database" class="anchor" aria-hidden="true" href="#user-content-querying-a-different-database"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Querying a different database</h3>
  <p>You can pass an optional <code>database=</code> argument to specify a named database to use for the query. For example, if you have attached a <code>news.db</code> database you could use this:</p>
  <div class="highlight highlight-text-html-django position-relative" data-snippet-clipboard-copy-content="{% for article in sql(
      &quot;select headline, date, summary from articles order by date desc limit 5&quot;,
      database=&quot;news&quot;
  ) %}
      &lt;h3&gt;{{ article.headline }}&lt;/h2&gt;
      &lt;p class=&quot;date&quot;&gt;{{ article.date }}&lt;/p&gt;
      &lt;p&gt;{{ article.summary }}&lt;/p&gt;
  {% endfor %}
  "><pre><span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">article</span> <span class="pl-k">in</span> <span class="pl-s">sql</span>(
      <span class="pl-s">"select headline, date, summary from articles order by date desc limit 5"</span>,
      <span class="pl-s">database</span>=<span class="pl-s">"news"</span>
  ) <span class="pl-e">%}</span>
      &lt;<span class="pl-ent">h3</span>&gt;{{ article.headline }}&lt;/<span class="pl-ent">h2</span>&gt;
      &lt;<span class="pl-ent">p</span> <span class="pl-e">class</span>=<span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>&gt;{{ article.date }}&lt;/<span class="pl-ent">p</span>&gt;
      &lt;<span class="pl-ent">p</span>&gt;{{ article.summary }}&lt;/<span class="pl-ent">p</span>&gt;
  <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMjg0ODU4MDY=
name: datasette-configure-asgi
full_name: simonw/datasette-configure-asgi
html_url: https://github.com/simonw/datasette-configure-asgi
description: Datasette plugin for configuring arbitrary ASGI middleware
created_at: 2019-12-16T22:17:10Z
updated_at: 2020-08-25T15:54:32Z
pushed_at: 2019-12-16T22:19:49Z
language: Python
license: apache-2.0
topics: ["asgi", "datasette", "datasette-plugin", "datasette-io"]
default_branch: master
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # datasette-configure-asgi

  [![PyPI](https://img.shields.io/pypi/v/datasette-configure-asgi.svg)](https://pypi.org/project/datasette-configure-asgi/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-configure-asgi.svg?style=svg)](https://circleci.com/gh/simonw/datasette-configure-asgi)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-configure-asgi/blob/master/LICENSE)

  Datasette plugin for configuring arbitrary ASGI middleware

  ## Installation

      pip install datasette-configure-asgi

  ## Usage

  This plugin only takes effect if your `metadata.json` file contains relevant top-level plugin configuration in a `"datasette-configure-asgi"` configuration key.

  For example, to wrap your Datasette instance in the `asgi-log-to-sqlite` middleware configured to write logs to `/tmp/log.db` you would use the following:

  ```json
  {
      "plugins": {
          "datasette-configure-asgi": [
              {
                  "class": "asgi_log_to_sqlite.AsgiLogToSqlite",
                  "args": {
                      "file": "/tmp/log.db"
                  }
              }
          ]
      }
  }
  ```

  The `"datasette-configure-asgi"` key should be a list of JSON objects. Each object should have a `"class"` key indicating the class to be used, and an optional `"args"` key providing any necessary arguments to be passed to that class constructor.

  ## Plugin structure

  This plugin can be used to wrap your Datasette instance in any ASGI middleware that conforms to the following structure:

  ```python
  class SomeAsgiMiddleware:
      def __init__(self, app, arg1, arg2):
          self.app = app
          self.arg1 = arg1
          self.arg2 = arg2

      async def __call__(self, scope, receive, send):
          start = time.time()
          await self.app(scope, receive, send)
          end = time.time()
          print("Time taken: {}".format(end - start))
  ```

  So the middleware is a class with a constructor which takes the wrapped application as a first argument, `app`, followed by further named arguments to configure the middleware. It provides an `async def __call__(self, scope, receive, send)` method to implement the middleware's behavior.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-configure-asgi" class="anchor" aria-hidden="true" href="#user-content-datasette-configure-asgi"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-configure-asgi</h1>
  <p><a href="https://pypi.org/project/datasette-configure-asgi/" rel="nofollow"><img src="https://camo.githubusercontent.com/f416823edcf74e9efabbee4be66b7db51784d1eeb1b7fd75286cfcbf16a87815/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d636f6e6669677572652d617367692e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-configure-asgi.svg" style="max-width:100%;"></a>
  <a href="https://circleci.com/gh/simonw/datasette-configure-asgi" rel="nofollow"><img src="https://camo.githubusercontent.com/d9cd8121c1d308429b1b5c4f9a87aee4091dfe95aadeeb9e39336aa91fb737e4/68747470733a2f2f636972636c6563692e636f6d2f67682f73696d6f6e772f6461746173657474652d636f6e6669677572652d617367692e7376673f7374796c653d737667" alt="CircleCI" data-canonical-src="https://circleci.com/gh/simonw/datasette-configure-asgi.svg?style=svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-configure-asgi/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Datasette plugin for configuring arbitrary ASGI middleware</p>
  <h2><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install datasette-configure-asgi
  "><pre><code>pip install datasette-configure-asgi
  </code></pre></div>
  <h2><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p>This plugin only takes effect if your <code>metadata.json</code> file contains relevant top-level plugin configuration in a <code>"datasette-configure-asgi"</code> configuration key.</p>
  <p>For example, to wrap your Datasette instance in the <code>asgi-log-to-sqlite</code> middleware configured to write logs to <code>/tmp/log.db</code> you would use the following:</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-configure-asgi&quot;: [
              {
                  &quot;class&quot;: &quot;asgi_log_to_sqlite.AsgiLogToSqlite&quot;,
                  &quot;args&quot;: {
                      &quot;file&quot;: &quot;/tmp/log.db&quot;
                  }
              }
          ]
      }
  }
  "><pre>{
      <span class="pl-s"><span class="pl-pds">"</span>plugins<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>datasette-configure-asgi<span class="pl-pds">"</span></span>: [
              {
                  <span class="pl-s"><span class="pl-pds">"</span>class<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>asgi_log_to_sqlite.AsgiLogToSqlite<span class="pl-pds">"</span></span>,
                  <span class="pl-s"><span class="pl-pds">"</span>args<span class="pl-pds">"</span></span>: {
                      <span class="pl-s"><span class="pl-pds">"</span>file<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>/tmp/log.db<span class="pl-pds">"</span></span>
                  }
              }
          ]
      }
  }</pre></div>
  <p>The <code>"datasette-configure-asgi"</code> key should be a list of JSON objects. Each object should have a <code>"class"</code> key indicating the class to be used, and an optional <code>"args"</code> key providing any necessary arguments to be passed to that class constructor.</p>
  <h2><a id="user-content-plugin-structure" class="anchor" aria-hidden="true" href="#user-content-plugin-structure"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Plugin structure</h2>
  <p>This plugin can be used to wrap your Datasette instance in any ASGI middleware that conforms to the following structure:</p>
  <div class="highlight highlight-source-python position-relative" data-snippet-clipboard-copy-content="class SomeAsgiMiddleware:
      def __init__(self, app, arg1, arg2):
          self.app = app
          self.arg1 = arg1
          self.arg2 = arg2

      async def __call__(self, scope, receive, send):
          start = time.time()
          await self.app(scope, receive, send)
          end = time.time()
          print(&quot;Time taken: {}&quot;.format(end - start))
  "><pre><span class="pl-k">class</span> <span class="pl-v">SomeAsgiMiddleware</span>:
      <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">app</span>, <span class="pl-s1">arg1</span>, <span class="pl-s1">arg2</span>):
          <span class="pl-s1">self</span>.<span class="pl-s1">app</span> <span class="pl-c1">=</span> <span class="pl-s1">app</span>
          <span class="pl-s1">self</span>.<span class="pl-s1">arg1</span> <span class="pl-c1">=</span> <span class="pl-s1">arg1</span>
          <span class="pl-s1">self</span>.<span class="pl-s1">arg2</span> <span class="pl-c1">=</span> <span class="pl-s1">arg2</span>

      <span class="pl-k">async</span> <span class="pl-k">def</span> <span class="pl-en">__call__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">scope</span>, <span class="pl-s1">receive</span>, <span class="pl-s1">send</span>):
          <span class="pl-s1">start</span> <span class="pl-c1">=</span> <span class="pl-s1">time</span>.<span class="pl-en">time</span>()
          <span class="pl-k">await</span> <span class="pl-s1">self</span>.<span class="pl-en">app</span>(<span class="pl-s1">scope</span>, <span class="pl-s1">receive</span>, <span class="pl-s1">send</span>)
          <span class="pl-s1">end</span> <span class="pl-c1">=</span> <span class="pl-s1">time</span>.<span class="pl-en">time</span>()
          <span class="pl-en">print</span>(<span class="pl-s">"Time taken: {}"</span>.<span class="pl-en">format</span>(<span class="pl-s1">end</span> <span class="pl-c1">-</span> <span class="pl-s1">start</span>))</pre></div>
  <p>So the middleware is a class with a constructor which takes the wrapped application as a first argument, <code>app</code>, followed by further named arguments to configure the middleware. It provides an <code>async def __call__(self, scope, receive, send)</code> method to implement the middleware's behavior.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMzQ4MjU3OTA=
name: datasette-upload-csvs
full_name: simonw/datasette-upload-csvs
html_url: https://github.com/simonw/datasette-upload-csvs
description: Datasette plugin for uploading CSV files and converting them to database tables
created_at: 2020-01-19T02:07:05Z
updated_at: 2022-07-03T20:58:20Z
pushed_at: 2022-09-09T16:23:59Z
homepage: https://datasette.io/plugins/datasette-upload-csvs
language: Python
license: apache-2.0
topics: ["csvs", "datasette", "datasette-io", "datasette-plugin"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-upload-csvs

  [![PyPI](https://img.shields.io/pypi/v/datasette-upload-csvs.svg)](https://pypi.org/project/datasette-upload-csvs/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-upload-csvs?include_prereleases&label=changelog)](https://github.com/simonw/datasette-upload-csvs/releases)
  [![Tests](https://github.com/simonw/datasette-upload-csvs/workflows/Test/badge.svg)](https://github.com/simonw/datasette-upload-csvs/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-upload-csvs/blob/main/LICENSE)

  Datasette plugin for uploading CSV files and converting them to database tables

  ## Installation

      datasette install datasette-upload-csvs

  ## Usage

  The plugin adds an interface at `/-/upload-csvs` for uploading a CSV file and using it to create a new database table.

  By default only [the root actor](https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor) can access the page - so you'll need to run Datasette with the `--root` option and click on the link shown in the terminal to sign in and access the page.

  The `upload-csvs` permission governs access. You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to the write interface.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-upload-csvs" class="anchor" aria-hidden="true" href="#user-content-datasette-upload-csvs"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-upload-csvs</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-upload-csvs/" rel="nofollow"><img src="https://camo.githubusercontent.com/0579577329c092ea76dd2672cfa6077ecd82c918c9d3caa9c8ff26be02104f78/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d75706c6f61642d637376732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-upload-csvs.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-upload-csvs/releases"><img src="https://camo.githubusercontent.com/34f7f9b455f75c80e8b706ff050dc59f70a926b93af51457f14f0a1d290f47bc/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d75706c6f61642d637376733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-upload-csvs?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-upload-csvs/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-upload-csvs/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-upload-csvs/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Datasette plugin for uploading CSV files and converting them to database tables</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-upload-csvs"><pre class="notranslate"><code>datasette install datasette-upload-csvs
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">The plugin adds an interface at <code>/-/upload-csvs</code> for uploading a CSV file and using it to create a new database table.</p>
  <p dir="auto">By default only <a href="https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor" rel="nofollow">the root actor</a> can access the page - so you'll need to run Datasette with the <code>--root</code> option and click on the link shown in the terminal to sign in and access the page.</p>
  <p dir="auto">The <code>upload-csvs</code> permission governs access. You can use permission plugins such as <a href="https://github.com/simonw/datasette-permissions-sql">datasette-permissions-sql</a> to grant additional access to the write interface.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMzYxMTA3NTk=
name: datasette-auth-existing-cookies
full_name: simonw/datasette-auth-existing-cookies
html_url: https://github.com/simonw/datasette-auth-existing-cookies
description: Datasette plugin that authenticates users based on existing domain cookies
created_at: 2020-01-25T01:20:31Z
updated_at: 2022-05-28T01:50:15Z
pushed_at: 2022-05-30T17:10:11Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-auth-existing-cookies

  [![PyPI](https://img.shields.io/pypi/v/datasette-auth-existing-cookies.svg)](https://pypi.org/project/datasette-auth-existing-cookies/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-auth-existing-cookies?include_prereleases&label=changelog)](https://github.com/simonw/datasette-auth-existing-cookies/releases)
  [![Tests](https://github.com/simonw/datasette-auth-existing-cookies/workflows/Test/badge.svg)](https://github.com/simonw/datasette-auth-existing-cookies/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-auth-existing-cookies/blob/master/LICENSE)

  Datasette plugin that authenticates users based on existing domain cookies.

  ## When to use this

  This plugin allows you to build custom authentication for Datasette when you are hosting a Datasette instance on the same domain as another, authenticated website.

  Consider a website on `www.example.com` which supports user authentication.

  You could run Datasette on `data.example.com` in a way that lets it see cookies that were set for the `.example.com` domain.

  Using this plugin, you could build an API endpoint at `www.example.com/user-for-cookies` which returns a JSON object representing the currently signed-in user, based on their cookies.

  The plugin running on `data.example.com` will then make the `actor` available to the rest of Datasette based on the response from that API.

  Read about [Datasette's authentication and permissions system](https://docs.datasette.io/en/stable/authentication.html) for more on how actors and permissions work.

  ## Configuration

  This plugin requires some configuration in the Datasette [metadata.json file](https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration).

  The following configuration options are supported:

  - `api_url`: this is the API endpoint that Datasette should call with the user's cookies in order to identify the logged in user.
  - `cookies`: optional. A list of cookie names that should be passed through to the API endpoint - if left blank, the default is to send all cookies.
  - `ttl`: optional. By default Datasette will make a request to the API endpoint for every HTTP request recieved by Datasette itself. A `ttl` value of 5 will cause Datasette to cache the actor associated with the user's cookies for 5 seconds, reducing that API traffic.
  - `headers`: an optional list of other headers to forward to the API endpoint as query string parameters.

  Here is an example that uses all four of these settings:

  ```json
  {
      "plugins": {
          "datasette-auth-existing-cookies": {
              "api_url": "http://www.example.com/user-from-cookies",
              "cookies": ["sessionid"],
              "headers": ["host"],
              "ttl": 10
          }
      }
  }
  ```
  With this configuration any hit to a Datasette hosted at `data.example.com` will result in the following request being made to the `http://www.example.com/user-from-cookies` API endpoint:
  ```
  GET http://www.example.com/user-from-cookies?host=data.example.com
  Cookie: sessionid=abc123
  ```
  That API is expected to return a JSON object representing the current user:

  ```json
  {
      "id": 1,
      "name": "Barry"
  }
  ```
  Since `ttl` is set to 10 that actor will be cached for ten seconds against that exact combination of cookies and headers. When that cache expires another hit will be made to the API.

  When deciding on a TTL value, take into account that users who lose access to the core site - maybe because their session expires, or their account is disabled - will still be able to access the Datasette instance until that cache expires.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-auth-existing-cookies" class="anchor" aria-hidden="true" href="#user-content-datasette-auth-existing-cookies"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-auth-existing-cookies</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-auth-existing-cookies/" rel="nofollow"><img src="https://camo.githubusercontent.com/590f1fc881f5547ddc2df20c3e78f26fde45e0e616863c3c2111d37bf22f6a61/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d617574682d6578697374696e672d636f6f6b6965732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-auth-existing-cookies.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-auth-existing-cookies/releases"><img src="https://camo.githubusercontent.com/ca9233dbd3b295b882d4e93a38bf3d9d64a6904111ad04a787a4d7201b3cd58f/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d617574682d6578697374696e672d636f6f6b6965733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-auth-existing-cookies?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-auth-existing-cookies/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-auth-existing-cookies/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-auth-existing-cookies/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Datasette plugin that authenticates users based on existing domain cookies.</p>
  <h2 dir="auto"><a id="user-content-when-to-use-this" class="anchor" aria-hidden="true" href="#user-content-when-to-use-this"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>When to use this</h2>
  <p dir="auto">This plugin allows you to build custom authentication for Datasette when you are hosting a Datasette instance on the same domain as another, authenticated website.</p>
  <p dir="auto">Consider a website on <code>www.example.com</code> which supports user authentication.</p>
  <p dir="auto">You could run Datasette on <code>data.example.com</code> in a way that lets it see cookies that were set for the <code>.example.com</code> domain.</p>
  <p dir="auto">Using this plugin, you could build an API endpoint at <code>www.example.com/user-for-cookies</code> which returns a JSON object representing the currently signed-in user, based on their cookies.</p>
  <p dir="auto">The plugin running on <code>data.example.com</code> will then make the <code>actor</code> available to the rest of Datasette based on the response from that API.</p>
  <p dir="auto">Read about <a href="https://docs.datasette.io/en/stable/authentication.html" rel="nofollow">Datasette's authentication and permissions system</a> for more on how actors and permissions work.</p>
  <h2 dir="auto"><a id="user-content-configuration" class="anchor" aria-hidden="true" href="#user-content-configuration"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Configuration</h2>
  <p dir="auto">This plugin requires some configuration in the Datasette <a href="https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration" rel="nofollow">metadata.json file</a>.</p>
  <p dir="auto">The following configuration options are supported:</p>
  <ul dir="auto">
  <li><code>api_url</code>: this is the API endpoint that Datasette should call with the user's cookies in order to identify the logged in user.</li>
  <li><code>cookies</code>: optional. A list of cookie names that should be passed through to the API endpoint - if left blank, the default is to send all cookies.</li>
  <li><code>ttl</code>: optional. By default Datasette will make a request to the API endpoint for every HTTP request recieved by Datasette itself. A <code>ttl</code> value of 5 will cause Datasette to cache the actor associated with the user's cookies for 5 seconds, reducing that API traffic.</li>
  <li><code>headers</code>: an optional list of other headers to forward to the API endpoint as query string parameters.</li>
  </ul>
  <p dir="auto">Here is an example that uses all four of these settings:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-auth-existing-cookies&quot;: {
              &quot;api_url&quot;: &quot;http://www.example.com/user-from-cookies&quot;,
              &quot;cookies&quot;: [&quot;sessionid&quot;],
              &quot;headers&quot;: [&quot;host&quot;],
              &quot;ttl&quot;: 10
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-auth-existing-cookies"</span>: {
              <span class="pl-ent">"api_url"</span>: <span class="pl-s"><span class="pl-pds">"</span>http://www.example.com/user-from-cookies<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"cookies"</span>: [<span class="pl-s"><span class="pl-pds">"</span>sessionid<span class="pl-pds">"</span></span>],
              <span class="pl-ent">"headers"</span>: [<span class="pl-s"><span class="pl-pds">"</span>host<span class="pl-pds">"</span></span>],
              <span class="pl-ent">"ttl"</span>: <span class="pl-c1">10</span>
          }
      }
  }</pre></div>
  <p dir="auto">With this configuration any hit to a Datasette hosted at <code>data.example.com</code> will result in the following request being made to the <code>http://www.example.com/user-from-cookies</code> API endpoint:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="GET http://www.example.com/user-from-cookies?host=data.example.com
  Cookie: sessionid=abc123"><pre class="notranslate"><code>GET http://www.example.com/user-from-cookies?host=data.example.com
  Cookie: sessionid=abc123
  </code></pre></div>
  <p dir="auto">That API is expected to return a JSON object representing the current user:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;id&quot;: 1,
      &quot;name&quot;: &quot;Barry&quot;
  }"><pre>{
      <span class="pl-ent">"id"</span>: <span class="pl-c1">1</span>,
      <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Barry<span class="pl-pds">"</span></span>
  }</pre></div>
  <p dir="auto">Since <code>ttl</code> is set to 10 that actor will be cached for ten seconds against that exact combination of cookies and headers. When that cache expires another hit will be made to the API.</p>
  <p dir="auto">When deciding on a TTL value, take into account that users who lose access to the core site - maybe because their session expires, or their account is disabled - will still be able to access the Datasette instance until that cache expires.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMzY4NjcwMjc=
name: datasette-sentry
full_name: simonw/datasette-sentry
html_url: https://github.com/simonw/datasette-sentry
description: Datasette plugin for configuring Sentry
created_at: 2020-01-28T23:41:27Z
updated_at: 2022-07-18T20:28:25Z
pushed_at: 2022-10-06T22:31:29Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "sentry"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-sentry

  [![PyPI](https://img.shields.io/pypi/v/datasette-sentry.svg)](https://pypi.org/project/datasette-sentry/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-sentry?include_prereleases&label=changelog)](https://github.com/simonw/datasette-sentry/releases)
  [![Tests](https://github.com/simonw/datasette-sentry/workflows/Test/badge.svg)](https://github.com/simonw/datasette-sentry/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-sentry/blob/main/LICENSE)

  Datasette plugin for configuring Sentry for error reporting
   
  ## Installation

      pip install datasette-sentry

  ## Usage

  This plugin only takes effect if your `metadata.json` file contains relevant top-level plugin configuration in a `"datasette-sentry"` configuration key.

  You will need a Sentry DSN - see their [Getting Started instructions](https://docs.sentry.io/error-reporting/quickstart/?platform=python).

  Add it to `metadata.json` like this:

  ```json
  {
      "plugins": {
          "datasette-sentry": {
              "dsn": "https://KEY@sentry.io/PROJECTID"
          }
      }
  }
  ```
  Settings in `metadata.json` are visible to anyone who visits the `/-/metadata` URL so this is a good place to take advantage of Datasette's [secret configuration values](https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values), in which case your configuration will look more like this:
  ```json
  {
      "plugins": {
          "datasette-sentry": {
              "dsn": {
                  "$env": "SENTRY_DSN"
              }
          }
      }
  }
  ```
  Then make a `SENTRY_DSN` environment variable available to Datasette.

  ## Configuration

  In addition to the `dsn` setting, you can also configure the Sentry [sample rate](https://docs.sentry.io/platforms/python/configuration/sampling/) by setting  `sample_rate` to a floating point number between 0 and 1.

  For example, to capture 25% of errors you would do this:

  ```json
  {
      "plugins": {
          "datasette-sentry": {
              "dsn": {
                  "$env": "SENTRY_DSN"
              },
              "sample_rate": 0.25
          }
      }
  }
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-sentry" class="anchor" aria-hidden="true" href="#user-content-datasette-sentry"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-sentry</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-sentry/" rel="nofollow"><img src="https://camo.githubusercontent.com/235a5563b51d4001ae7207d970a8ea72137fd5cdb3443c66b5eb42ef35703683/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73656e7472792e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-sentry.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-sentry/releases"><img src="https://camo.githubusercontent.com/0a9f8c726d4c19b1bde12b99d824cf9269559d1290186a88815f72ee50a2a618/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d73656e7472793f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-sentry?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-sentry/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-sentry/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-sentry/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Datasette plugin for configuring Sentry for error reporting</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install datasette-sentry"><pre class="notranslate"><code>pip install datasette-sentry
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">This plugin only takes effect if your <code>metadata.json</code> file contains relevant top-level plugin configuration in a <code>"datasette-sentry"</code> configuration key.</p>
  <p dir="auto">You will need a Sentry DSN - see their <a href="https://docs.sentry.io/error-reporting/quickstart/?platform=python" rel="nofollow">Getting Started instructions</a>.</p>
  <p dir="auto">Add it to <code>metadata.json</code> like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-sentry&quot;: {
              &quot;dsn&quot;: &quot;https://KEY@sentry.io/PROJECTID&quot;
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-sentry"</span>: {
              <span class="pl-ent">"dsn"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://KEY@sentry.io/PROJECTID<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <p dir="auto">Settings in <code>metadata.json</code> are visible to anyone who visits the <code>/-/metadata</code> URL so this is a good place to take advantage of Datasette's <a href="https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values" rel="nofollow">secret configuration values</a>, in which case your configuration will look more like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-sentry&quot;: {
              &quot;dsn&quot;: {
                  &quot;$env&quot;: &quot;SENTRY_DSN&quot;
              }
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-sentry"</span>: {
              <span class="pl-ent">"dsn"</span>: {
                  <span class="pl-ent">"$env"</span>: <span class="pl-s"><span class="pl-pds">"</span>SENTRY_DSN<span class="pl-pds">"</span></span>
              }
          }
      }
  }</pre></div>
  <p dir="auto">Then make a <code>SENTRY_DSN</code> environment variable available to Datasette.</p>
  <h2 dir="auto"><a id="user-content-configuration" class="anchor" aria-hidden="true" href="#user-content-configuration"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Configuration</h2>
  <p dir="auto">In addition to the <code>dsn</code> setting, you can also configure the Sentry <a href="https://docs.sentry.io/platforms/python/configuration/sampling/" rel="nofollow">sample rate</a> by setting  <code>sample_rate</code> to a floating point number between 0 and 1.</p>
  <p dir="auto">For example, to capture 25% of errors you would do this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-sentry&quot;: {
              &quot;dsn&quot;: {
                  &quot;$env&quot;: &quot;SENTRY_DSN&quot;
              },
              &quot;sample_rate&quot;: 0.25
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-sentry"</span>: {
              <span class="pl-ent">"dsn"</span>: {
                  <span class="pl-ent">"$env"</span>: <span class="pl-s"><span class="pl-pds">"</span>SENTRY_DSN<span class="pl-pds">"</span></span>
              },
              <span class="pl-ent">"sample_rate"</span>: <span class="pl-c1">0.25</span>
          }
      }
  }</pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMzczMjEyNjc=
name: geojson-to-sqlite
full_name: simonw/geojson-to-sqlite
html_url: https://github.com/simonw/geojson-to-sqlite
description: CLI tool for converting GeoJSON files to SQLite (with SpatiaLite)
created_at: 2020-01-30T22:51:05Z
updated_at: 2022-03-05T00:40:56Z
pushed_at: 2022-04-13T23:39:25Z
language: Python
license: apache-2.0
topics: ["datasette-io", "datasette-tool", "geojson", "gis", "sqlite"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # geojson-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/geojson-to-sqlite.svg)](https://pypi.org/project/geojson-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/geojson-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/geojson-to-sqlite/releases)
  [![Tests](https://github.com/simonw/geojson-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/geojson-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/geojson-to-sqlite/blob/main/LICENSE)

  CLI tool for converting GeoJSON to SQLite (optionally with SpatiaLite)

  [RFC 7946: The GeoJSON Format](https://tools.ietf.org/html/rfc7946)

  ## How to install

      $ pip install geojson-to-sqlite

  ## How to use

  You can run this tool against a GeoJSON file like so:

      $ geojson-to-sqlite my.db features features.geojson

  This will load all of the features from the `features.geojson` file into a table called `features`.

  Each row will have a `geometry` column containing the feature geometry, and columns for each of the keys found in any `properties` attached to those features. (To bundle all properties into a single JSON object, use the `--properties` flag.)

  The table will be created the first time you run the command.

  On subsequent runs you can use the `--alter` option to add any new columns that are missing from the table.

  You can pass more than one GeoJSON file, in which case the contents of all of the files will be inserted into the same table.

  If your features have an `"id"` property it will be used as the primary key for the table. You can also use `--pk=PROPERTY` with the name of a different property to use that as the primary key instead. If you don't want to use the `"id"` as the primary key (maybe it contains duplicate values) you can use `--pk ''` to specify no primary key.

  Specifying a primary key also will allow you to upsert data into the rows instead of insert data into new rows.

  If no primary key is specified, a SQLite `rowid` column will be used.

  You can use `-` as the filename to import from standard input. For example:

      $ curl https://eric.clst.org/assets/wiki/uploads/Stuff/gz_2010_us_040_00_20m.json \
          | geojson-to-sqlite my.db states - --pk GEO_ID

  ## Using with SpatiaLite

  By default, the `geometry` column will contain JSON.

  If you have installed the [SpatiaLite](https://www.gaia-gis.it/fossil/libspatialite/index) module for SQLite you can instead import the geometry into a geospatially indexed column.

  You can do this using the `--spatialite` option, like so:

      $ geojson-to-sqlite my.db features features.geojson --spatialite

  The tool will search for the SpatiaLite module in the following locations:

  - `/usr/lib/x86_64-linux-gnu/mod_spatialite.so`
  - `/usr/local/lib/mod_spatialite.dylib`

  If you have installed the module in another location, you can use the `--spatialite_mod=xxx` option to specify where:

      $ geojson-to-sqlite my.db features features.geojson \
          --spatialite_mod=/usr/lib/mod_spatialite.dylib

  You can create a SpatiaLite spatial index on the `geometry` column using the `--spatial-index` option:

      $ geojson-to-sqlite my.db features features.geojson --spatial-index

  Using this option implies `--spatialite` so you do not need to add that.

  ## Streaming large datasets

  For large datasets, consider using newline-delimited JSON to stream features into the database without loading the entire feature collection into memory.

  For example, to load a day of earthquake reports from USGS:

      $ geojson-to-sqlite quakes.db quakes tests/quakes.ndjson \
        --nl --pk=id --spatialite

  When using newline-delimited JSON, tables will also be created from the first feature, instead of guessing types based on the first 100 features.

  If you want to use a larger subset of your data to guess column types (for example, if some fields are inconsistent) you can use [fiona](https://fiona.readthedocs.io/en/latest/cli.html) to collect features into a single collection.

      $ head tests/quakes.ndjson | fio collect | \
        geojson-to-sqlite quakes.db quakes - --spatialite

  This will take the first 10 lines from `tests/quakes.ndjson`, pass them to `fio collect`, which turns them into a single feature collection, and pass that, in turn, to `geojson-to-sqlite`.

  ## Using this with Datasette

  Databases created using this tool can be explored and published using [Datasette](https://datasette.readthedocs.io/).

  The Datasette documentation includes a section on [how to use it to browse SpatiaLite databases](https://datasette.readthedocs.io/en/stable/spatialite.html).

  The [datasette-leaflet-geojson](https://datasette.io/plugins/datasette-leaflet-geojson) plugin can be used to visualize columns containing GeoJSON geometries on a [Leaflet](https://leafletjs.com/) map.

  If you are using SpatiaLite you will need to output the geometry as GeoJSON in order for that plugin to work. You can do that using the SpaitaLite `AsGeoJSON()` function - something like this:

  ```sql
  select rowid, AsGeoJSON(geometry) from mytable limit 10
  ```

  The [datasette-geojson-map](https://datasette.io/plugins/datasette-geojson-map) is an alternative plugin which will automatically render SpatiaLite geometries as a Leaflet map on the corresponding table page, without needing you to call `AsGeoJSON(geometry)`.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-geojson-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-geojson-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>geojson-to-sqlite</h1>
  <p dir="auto"><a href="https://pypi.org/project/geojson-to-sqlite/" rel="nofollow"><img src="https://camo.githubusercontent.com/b0c77834f0d6adf37e62573e121ccadd92711433ad2c5b4dd506610919762dab/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f67656f6a736f6e2d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/geojson-to-sqlite.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/geojson-to-sqlite/releases"><img src="https://camo.githubusercontent.com/e88a12c13acc77ac4d0338f065dfeb664076b3414a7ee2560e8d465e828c1320/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f67656f6a736f6e2d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/geojson-to-sqlite?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/geojson-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/geojson-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/geojson-to-sqlite/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">CLI tool for converting GeoJSON to SQLite (optionally with SpatiaLite)</p>
  <p dir="auto"><a href="https://tools.ietf.org/html/rfc7946" rel="nofollow">RFC 7946: The GeoJSON Format</a></p>
  <h2 dir="auto"><a id="user-content-how-to-install" class="anchor" aria-hidden="true" href="#user-content-how-to-install"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How to install</h2>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install geojson-to-sqlite"><pre><code>$ pip install geojson-to-sqlite
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-how-to-use" class="anchor" aria-hidden="true" href="#user-content-how-to-use"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How to use</h2>
  <p dir="auto">You can run this tool against a GeoJSON file like so:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ geojson-to-sqlite my.db features features.geojson"><pre><code>$ geojson-to-sqlite my.db features features.geojson
  </code></pre></div>
  <p dir="auto">This will load all of the features from the <code>features.geojson</code> file into a table called <code>features</code>.</p>
  <p dir="auto">Each row will have a <code>geometry</code> column containing the feature geometry, and columns for each of the keys found in any <code>properties</code> attached to those features. (To bundle all properties into a single JSON object, use the <code>--properties</code> flag.)</p>
  <p dir="auto">The table will be created the first time you run the command.</p>
  <p dir="auto">On subsequent runs you can use the <code>--alter</code> option to add any new columns that are missing from the table.</p>
  <p dir="auto">You can pass more than one GeoJSON file, in which case the contents of all of the files will be inserted into the same table.</p>
  <p dir="auto">If your features have an <code>"id"</code> property it will be used as the primary key for the table. You can also use <code>--pk=PROPERTY</code> with the name of a different property to use that as the primary key instead. If you don't want to use the <code>"id"</code> as the primary key (maybe it contains duplicate values) you can use <code>--pk ''</code> to specify no primary key.</p>
  <p dir="auto">Specifying a primary key also will allow you to upsert data into the rows instead of insert data into new rows.</p>
  <p dir="auto">If no primary key is specified, a SQLite <code>rowid</code> column will be used.</p>
  <p dir="auto">You can use <code>-</code> as the filename to import from standard input. For example:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ curl https://eric.clst.org/assets/wiki/uploads/Stuff/gz_2010_us_040_00_20m.json \
      | geojson-to-sqlite my.db states - --pk GEO_ID"><pre><code>$ curl https://eric.clst.org/assets/wiki/uploads/Stuff/gz_2010_us_040_00_20m.json \
      | geojson-to-sqlite my.db states - --pk GEO_ID
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-using-with-spatialite" class="anchor" aria-hidden="true" href="#user-content-using-with-spatialite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Using with SpatiaLite</h2>
  <p dir="auto">By default, the <code>geometry</code> column will contain JSON.</p>
  <p dir="auto">If you have installed the <a href="https://www.gaia-gis.it/fossil/libspatialite/index" rel="nofollow">SpatiaLite</a> module for SQLite you can instead import the geometry into a geospatially indexed column.</p>
  <p dir="auto">You can do this using the <code>--spatialite</code> option, like so:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ geojson-to-sqlite my.db features features.geojson --spatialite"><pre><code>$ geojson-to-sqlite my.db features features.geojson --spatialite
  </code></pre></div>
  <p dir="auto">The tool will search for the SpatiaLite module in the following locations:</p>
  <ul dir="auto">
  <li><code>/usr/lib/x86_64-linux-gnu/mod_spatialite.so</code></li>
  <li><code>/usr/local/lib/mod_spatialite.dylib</code></li>
  </ul>
  <p dir="auto">If you have installed the module in another location, you can use the <code>--spatialite_mod=xxx</code> option to specify where:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ geojson-to-sqlite my.db features features.geojson \
      --spatialite_mod=/usr/lib/mod_spatialite.dylib"><pre><code>$ geojson-to-sqlite my.db features features.geojson \
      --spatialite_mod=/usr/lib/mod_spatialite.dylib
  </code></pre></div>
  <p dir="auto">You can create a SpatiaLite spatial index on the <code>geometry</code> column using the <code>--spatial-index</code> option:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ geojson-to-sqlite my.db features features.geojson --spatial-index"><pre><code>$ geojson-to-sqlite my.db features features.geojson --spatial-index
  </code></pre></div>
  <p dir="auto">Using this option implies <code>--spatialite</code> so you do not need to add that.</p>
  <h2 dir="auto"><a id="user-content-streaming-large-datasets" class="anchor" aria-hidden="true" href="#user-content-streaming-large-datasets"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Streaming large datasets</h2>
  <p dir="auto">For large datasets, consider using newline-delimited JSON to stream features into the database without loading the entire feature collection into memory.</p>
  <p dir="auto">For example, to load a day of earthquake reports from USGS:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ geojson-to-sqlite quakes.db quakes tests/quakes.ndjson \
    --nl --pk=id --spatialite"><pre><code>$ geojson-to-sqlite quakes.db quakes tests/quakes.ndjson \
    --nl --pk=id --spatialite
  </code></pre></div>
  <p dir="auto">When using newline-delimited JSON, tables will also be created from the first feature, instead of guessing types based on the first 100 features.</p>
  <p dir="auto">If you want to use a larger subset of your data to guess column types (for example, if some fields are inconsistent) you can use <a href="https://fiona.readthedocs.io/en/latest/cli.html" rel="nofollow">fiona</a> to collect features into a single collection.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ head tests/quakes.ndjson | fio collect | \
    geojson-to-sqlite quakes.db quakes - --spatialite"><pre><code>$ head tests/quakes.ndjson | fio collect | \
    geojson-to-sqlite quakes.db quakes - --spatialite
  </code></pre></div>
  <p dir="auto">This will take the first 10 lines from <code>tests/quakes.ndjson</code>, pass them to <code>fio collect</code>, which turns them into a single feature collection, and pass that, in turn, to <code>geojson-to-sqlite</code>.</p>
  <h2 dir="auto"><a id="user-content-using-this-with-datasette" class="anchor" aria-hidden="true" href="#user-content-using-this-with-datasette"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Using this with Datasette</h2>
  <p dir="auto">Databases created using this tool can be explored and published using <a href="https://datasette.readthedocs.io/" rel="nofollow">Datasette</a>.</p>
  <p dir="auto">The Datasette documentation includes a section on <a href="https://datasette.readthedocs.io/en/stable/spatialite.html" rel="nofollow">how to use it to browse SpatiaLite databases</a>.</p>
  <p dir="auto">The <a href="https://datasette.io/plugins/datasette-leaflet-geojson" rel="nofollow">datasette-leaflet-geojson</a> plugin can be used to visualize columns containing GeoJSON geometries on a <a href="https://leafletjs.com/" rel="nofollow">Leaflet</a> map.</p>
  <p dir="auto">If you are using SpatiaLite you will need to output the geometry as GeoJSON in order for that plugin to work. You can do that using the SpaitaLite <code>AsGeoJSON()</code> function - something like this:</p>
  <div class="highlight highlight-source-sql position-relative overflow-auto" data-snippet-clipboard-copy-content="select rowid, AsGeoJSON(geometry) from mytable limit 10"><pre><span class="pl-k">select</span> rowid, AsGeoJSON(geometry) <span class="pl-k">from</span> mytable <span class="pl-k">limit</span> <span class="pl-c1">10</span></pre></div>
  <p dir="auto">The <a href="https://datasette.io/plugins/datasette-geojson-map" rel="nofollow">datasette-geojson-map</a> is an alternative plugin which will automatically render SpatiaLite geometries as a Leaflet map on the corresponding table page, without needing you to call <code>AsGeoJSON(geometry)</code>.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyMzgzMzk0MTI=
name: datasette-debug-asgi
full_name: simonw/datasette-debug-asgi
html_url: https://github.com/simonw/datasette-debug-asgi
description: Datasette plugin for dumping out the ASGI scope
created_at: 2020-02-05T00:57:09Z
updated_at: 2021-08-17T23:40:02Z
pushed_at: 2021-08-17T23:41:03Z
homepage: https://datasette.io/plugins/datasette-debug-asgi
language: Python
license: apache-2.0
topics: ["asgi", "datasette-io", "datasette-plugin"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
readme:
  # datasette-debug-asgi

  [![PyPI](https://img.shields.io/pypi/v/datasette-debug-asgi.svg)](https://pypi.org/project/datasette-debug-asgi/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-debug-asgi?include_prereleases&label=changelog)](https://github.com/simonw/datasette-debug-asgi/releases)
  [![Tests](https://github.com/simonw/datasette-debug-asgi/workflows/Test/badge.svg)](https://github.com/simonw/datasette-debug-asgi/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-debug-asgi/blob/main/LICENSE)

  Datasette plugin for dumping out the ASGI scope.

  Adds a new URL at `/-/asgi-scope` which shows the current ASGI scope. Demo here: https://datasette.io/-/asgi-scope

  ## Installation

      pip install datasette-debug-asgi

  ## Usage

  Visit `/-/asgi-scope` to see debug output showing the ASGI scope.

  You can add query string parameters such as `/-/asgi-scope?q=hello`.

  You can also add extra path components such as `/-/asgi-scope/more/path/here`.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-debug-asgi" class="anchor" aria-hidden="true" href="#user-content-datasette-debug-asgi"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-debug-asgi</h1>
  <p><a href="https://pypi.org/project/datasette-debug-asgi/" rel="nofollow"><img src="https://camo.githubusercontent.com/59ce0e036bd317f141a4ce0589ec0969f8fd2cfdef49c6b3c7c4dafc4bd9e332/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d64656275672d617367692e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-debug-asgi.svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-debug-asgi/releases"><img src="https://camo.githubusercontent.com/d3b776c6e5f4674675855b8268bbc26631391e18e23a50a4cc92204b89a1b701/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d64656275672d617367693f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-debug-asgi?include_prereleases&amp;label=changelog" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-debug-asgi/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-debug-asgi/workflows/Test/badge.svg" alt="Tests" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-debug-asgi/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Datasette plugin for dumping out the ASGI scope.</p>
  <p>Adds a new URL at <code>/-/asgi-scope</code> which shows the current ASGI scope. Demo here: <a href="https://datasette.io/-/asgi-scope" rel="nofollow">https://datasette.io/-/asgi-scope</a></p>
  <h2><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install datasette-debug-asgi
  "><pre><code>pip install datasette-debug-asgi
  </code></pre></div>
  <h2><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p>Visit <code>/-/asgi-scope</code> to see debug output showing the ASGI scope.</p>
  <p>You can add query string parameters such as <code>/-/asgi-scope?q=hello</code>.</p>
  <p>You can also add extra path components such as <code>/-/asgi-scope/more/path/here</code>.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNDA4MTU5Mzg=
name: shapefile-to-sqlite
full_name: simonw/shapefile-to-sqlite
html_url: https://github.com/simonw/shapefile-to-sqlite
description: Load shapefiles into a SQLite (optionally SpatiaLite) database
created_at: 2020-02-16T01:55:29Z
updated_at: 2021-03-26T08:39:43Z
pushed_at: 2020-08-23T06:00:41Z
language: Python
license: apache-2.0
topics: ["sqlite", "gis", "spatialite", "shapefiles", "datasette", "datasette-io", "datasette-tool"]
default_branch: main
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # shapefile-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/shapefile-to-sqlite.svg)](https://pypi.org/project/shapefile-to-sqlite/)
  [![CircleCI](https://circleci.com/gh/simonw/shapefile-to-sqlite.svg?style=svg)](https://circleci.com/gh/simonw/shapefile-to-sqlite)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/shapefile-to-sqlite/blob/main/LICENSE)

  Load shapefiles into a SQLite (optionally SpatiaLite) database.

  Project background: [Things I learned about shapefiles building shapefile-to-sqlite](https://simonwillison.net/2020/Feb/19/shapefile-to-sqlite/)

  ## How to install

      $ pip install shapefile-to-sqlite

  ## How to use

  You can run this tool against a shapefile file like so:

      $ shapefile-to-sqlite my.db features.shp

  This will load the geometries as GeoJSON in a text column.

  ## Using with SpatiaLite

  If you have [SpatiaLite](https://www.gaia-gis.it/fossil/libspatialite/index) available you can load them as SpatiaLite geometries like this:

      $ shapefile-to-sqlite my.db features.shp --spatialite

  The data will be loaded into a table called `features` - based on the name of the shapefile. You can specify an alternative table name using `--table`:

      $ shapefile-to-sqlite my.db features.shp --table=places --spatialite

  The tool will search for the SpatiaLite module in the following locations:

  - `/usr/lib/x86_64-linux-gnu/mod_spatialite.so`
  - `/usr/local/lib/mod_spatialite.dylib`

  If you have installed the module in another location, you can use the `--spatialite_mod=xxx` option to specify where:

      $ shapefile-to-sqlite my.db features.shp \
          --spatialite_mod=/usr/lib/mod_spatialite.dylib

  You can use the `--spatial-index` option to create a spatial index on the `geometry` column:

      $ shapefile-to-sqlite my.db features.shp --spatial-index

  You can omit `--spatialite` if you use either `--spatialite-mod` or `--spatial-index`.

  ## Projections

  By default, this tool will attempt to convert geometries in the shapefile to the WGS 84 projection, for best conformance with the [GeoJSON specification](https://tools.ietf.org/html/rfc7946).

  If you want it to leave the data in whatever projection was used by the shapefile, use the `--crs=keep` option.

  You can convert the data to another output projection by passing it to the `--crs` option. For example, to convert to [EPSG:2227](https://epsg.io/2227) (California zone 3) use `--crs=espg:2227`.

  The full list of formats accepted by the `--crs` option is [documented here](https://pyproj4.github.io/pyproj/stable/api/crs.html#pyproj.crs.CRS.__init__).

  ## Extracting columns

  If your data contains columns with a small number of heavily duplicated values - the names of specific agencies responsible for parcels of land for example - you can extract those columns into separate lookup tables referenced by foreign keys using the `-c` option:

      $ shapefile-to-sqlite my.db features.shp -c agency

  This will create a `agency` table with `id` and `name` columns, and will create the `agency` column in your main table as an integer foreign key reference to that table.

  The `-c` option can be used multiple times.

  [CPAD_2020a_Units](https://calands.datasettes.com/calands/CPAD_2020a_Units) is an example of a table created using the `-c` option.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-shapefile-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-shapefile-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>shapefile-to-sqlite</h1>
  <p><a href="https://pypi.org/project/shapefile-to-sqlite/" rel="nofollow"><img src="https://camo.githubusercontent.com/491f86f2d61c3cf6ce08c6c019a47686ccbe35187c465aeb784876d3adcc236b/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f736861706566696c652d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/shapefile-to-sqlite.svg" style="max-width:100%;"></a>
  <a href="https://circleci.com/gh/simonw/shapefile-to-sqlite" rel="nofollow"><img src="https://camo.githubusercontent.com/665b44f51d2c08eeb25388a24485a1591ebe793b914807ae371fe96fb242c0aa/68747470733a2f2f636972636c6563692e636f6d2f67682f73696d6f6e772f736861706566696c652d746f2d73716c6974652e7376673f7374796c653d737667" alt="CircleCI" data-canonical-src="https://circleci.com/gh/simonw/shapefile-to-sqlite.svg?style=svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/shapefile-to-sqlite/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Load shapefiles into a SQLite (optionally SpatiaLite) database.</p>
  <p>Project background: <a href="https://simonwillison.net/2020/Feb/19/shapefile-to-sqlite/" rel="nofollow">Things I learned about shapefiles building shapefile-to-sqlite</a></p>
  <h2><a id="user-content-how-to-install" class="anchor" aria-hidden="true" href="#user-content-how-to-install"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How to install</h2>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ pip install shapefile-to-sqlite
  "><pre><code>$ pip install shapefile-to-sqlite
  </code></pre></div>
  <h2><a id="user-content-how-to-use" class="anchor" aria-hidden="true" href="#user-content-how-to-use"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How to use</h2>
  <p>You can run this tool against a shapefile file like so:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ shapefile-to-sqlite my.db features.shp
  "><pre><code>$ shapefile-to-sqlite my.db features.shp
  </code></pre></div>
  <p>This will load the geometries as GeoJSON in a text column.</p>
  <h2><a id="user-content-using-with-spatialite" class="anchor" aria-hidden="true" href="#user-content-using-with-spatialite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Using with SpatiaLite</h2>
  <p>If you have <a href="https://www.gaia-gis.it/fossil/libspatialite/index" rel="nofollow">SpatiaLite</a> available you can load them as SpatiaLite geometries like this:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ shapefile-to-sqlite my.db features.shp --spatialite
  "><pre><code>$ shapefile-to-sqlite my.db features.shp --spatialite
  </code></pre></div>
  <p>The data will be loaded into a table called <code>features</code> - based on the name of the shapefile. You can specify an alternative table name using <code>--table</code>:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ shapefile-to-sqlite my.db features.shp --table=places --spatialite
  "><pre><code>$ shapefile-to-sqlite my.db features.shp --table=places --spatialite
  </code></pre></div>
  <p>The tool will search for the SpatiaLite module in the following locations:</p>
  <ul>
  <li><code>/usr/lib/x86_64-linux-gnu/mod_spatialite.so</code></li>
  <li><code>/usr/local/lib/mod_spatialite.dylib</code></li>
  </ul>
  <p>If you have installed the module in another location, you can use the <code>--spatialite_mod=xxx</code> option to specify where:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ shapefile-to-sqlite my.db features.shp \
      --spatialite_mod=/usr/lib/mod_spatialite.dylib
  "><pre><code>$ shapefile-to-sqlite my.db features.shp \
      --spatialite_mod=/usr/lib/mod_spatialite.dylib
  </code></pre></div>
  <p>You can use the <code>--spatial-index</code> option to create a spatial index on the <code>geometry</code> column:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ shapefile-to-sqlite my.db features.shp --spatial-index
  "><pre><code>$ shapefile-to-sqlite my.db features.shp --spatial-index
  </code></pre></div>
  <p>You can omit <code>--spatialite</code> if you use either <code>--spatialite-mod</code> or <code>--spatial-index</code>.</p>
  <h2><a id="user-content-projections" class="anchor" aria-hidden="true" href="#user-content-projections"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Projections</h2>
  <p>By default, this tool will attempt to convert geometries in the shapefile to the WGS 84 projection, for best conformance with the <a href="https://tools.ietf.org/html/rfc7946" rel="nofollow">GeoJSON specification</a>.</p>
  <p>If you want it to leave the data in whatever projection was used by the shapefile, use the <code>--crs=keep</code> option.</p>
  <p>You can convert the data to another output projection by passing it to the <code>--crs</code> option. For example, to convert to <a href="https://epsg.io/2227" rel="nofollow">EPSG:2227</a> (California zone 3) use <code>--crs=espg:2227</code>.</p>
  <p>The full list of formats accepted by the <code>--crs</code> option is <a href="https://pyproj4.github.io/pyproj/stable/api/crs.html#pyproj.crs.CRS.__init__" rel="nofollow">documented here</a>.</p>
  <h2><a id="user-content-extracting-columns" class="anchor" aria-hidden="true" href="#user-content-extracting-columns"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Extracting columns</h2>
  <p>If your data contains columns with a small number of heavily duplicated values - the names of specific agencies responsible for parcels of land for example - you can extract those columns into separate lookup tables referenced by foreign keys using the <code>-c</code> option:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ shapefile-to-sqlite my.db features.shp -c agency
  "><pre><code>$ shapefile-to-sqlite my.db features.shp -c agency
  </code></pre></div>
  <p>This will create a <code>agency</code> table with <code>id</code> and <code>name</code> columns, and will create the <code>agency</code> column in your main table as an integer foreign key reference to that table.</p>
  <p>The <code>-c</code> option can be used multiple times.</p>
  <p><a href="https://calands.datasettes.com/calands/CPAD_2020a_Units" rel="nofollow">CPAD_2020a_Units</a> is an example of a table created using the <code>-c</code> option.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNDIyNjA1ODM=
name: datasette-mask-columns
full_name: simonw/datasette-mask-columns
html_url: https://github.com/simonw/datasette-mask-columns
description: Datasette plugin that masks specified database columns
created_at: 2020-02-22T01:29:16Z
updated_at: 2021-06-10T19:50:37Z
pushed_at: 2021-06-10T19:51:02Z
homepage: https://datasette.io/plugins/datasette-mask-columns
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin", "datasette-io"]
default_branch: main
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # datasette-mask-columns

  [![PyPI](https://img.shields.io/pypi/v/datasette-mask-columns.svg)](https://pypi.org/project/datasette-mask-columns/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-mask-columns?include_prereleases&label=changelog)](https://github.com/simonw/datasette-mask-columns/releases)
  [![Tests](https://github.com/simonw/datasette-mask-columns/workflows/Test/badge.svg)](https://github.com/simonw/datasette-mask-columns/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-mask-columns/blob/main/LICENSE)

  Datasette plugin that masks specified database columns

  ## Installation

      pip install datasette-mask-columns

  This depends on plugin hook changes in a not-yet released branch of Datasette. See [issue #678](https://github.com/simonw/datasette/issues/678) for details.

  ## Usage

  In your `metadata.json` file add a section like this describing the database and table in which you wish to mask columns:

  ```json
  {
      "databases": {
          "my-database": {
              "plugins": {
                  "datasette-mask-columns": {
                      "users": ["password"]
                  }
              }
          }
      }
  }
  ```
  All SQL queries against the `users` table in `my-database.db` will now return `null` for the `password` column, no matter what value that column actually holds.

  The table page for `users` will display the text `REDACTED` in the masked column. This visual hint will only be available on the table page; it will not display his text for arbitrary queries against the table.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-mask-columns" class="anchor" aria-hidden="true" href="#user-content-datasette-mask-columns"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-mask-columns</h1>
  <p><a href="https://pypi.org/project/datasette-mask-columns/" rel="nofollow"><img src="https://camo.githubusercontent.com/09772a46d884b67ea26bd94868ec94be4bd514fa0a4a8ea3fac1b707801e4d3d/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6d61736b2d636f6c756d6e732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-mask-columns.svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-mask-columns/releases"><img src="https://camo.githubusercontent.com/f3fb962acca8cb0fd0c0caeb303007c6c199287a95f5b2f9bfb2128bec3b1dc3/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6d61736b2d636f6c756d6e733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-mask-columns?include_prereleases&amp;label=changelog" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-mask-columns/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-mask-columns/workflows/Test/badge.svg" alt="Tests" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-mask-columns/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Datasette plugin that masks specified database columns</p>
  <h2><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install datasette-mask-columns
  "><pre><code>pip install datasette-mask-columns
  </code></pre></div>
  <p>This depends on plugin hook changes in a not-yet released branch of Datasette. See <a href="https://github.com/simonw/datasette/issues/678">issue #678</a> for details.</p>
  <h2><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p>In your <code>metadata.json</code> file add a section like this describing the database and table in which you wish to mask columns:</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
      &quot;databases&quot;: {
          &quot;my-database&quot;: {
              &quot;plugins&quot;: {
                  &quot;datasette-mask-columns&quot;: {
                      &quot;users&quot;: [&quot;password&quot;]
                  }
              }
          }
      }
  }
  "><pre>{
      <span class="pl-s"><span class="pl-pds">"</span>databases<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>my-database<span class="pl-pds">"</span></span>: {
              <span class="pl-s"><span class="pl-pds">"</span>plugins<span class="pl-pds">"</span></span>: {
                  <span class="pl-s"><span class="pl-pds">"</span>datasette-mask-columns<span class="pl-pds">"</span></span>: {
                      <span class="pl-s"><span class="pl-pds">"</span>users<span class="pl-pds">"</span></span>: [<span class="pl-s"><span class="pl-pds">"</span>password<span class="pl-pds">"</span></span>]
                  }
              }
          }
      }
  }</pre></div>
  <p>All SQL queries against the <code>users</code> table in <code>my-database.db</code> will now return <code>null</code> for the <code>password</code> column, no matter what value that column actually holds.</p>
  <p>The table page for <code>users</code> will display the text <code>REDACTED</code> in the masked column. This visual hint will only be available on the table page; it will not display his text for arbitrary queries against the table.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkyNDM3MTA3MzM=
name: datasette-ics
full_name: simonw/datasette-ics
html_url: https://github.com/simonw/datasette-ics
description: Datasette plugin for outputting iCalendar files
created_at: 2020-02-28T08:11:01Z
updated_at: 2022-07-07T14:11:49Z
pushed_at: 2022-07-12T02:08:10Z
homepage: https://datasette.io/plugins/datasette-ics
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "icalendar", "ics"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-ics

  [![PyPI](https://img.shields.io/pypi/v/datasette-ics.svg)](https://pypi.org/project/datasette-ics/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-ics?include_prereleases&label=changelog)](https://github.com/simonw/datasette-ics/releases)
  [![Tests](https://github.com/simonw/datasette-ics/workflows/Test/badge.svg)](https://github.com/simonw/datasette-ics/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-ics/blob/main/LICENSE)

  Datasette plugin that adds support for generating [iCalendar .ics files](https://tools.ietf.org/html/rfc5545) with the results of a SQL query.

  ## Installation

  Install this plugin in the same environment as Datasette to enable the `.ics` output extension.

      $ pip install datasette-ics

  ## Usage

  To create an iCalendar file you need to define a custom SQL query that returns a required set of columns:

  * `event_name` - the short name for the event
  * `event_dtstart` - when the event starts

  The following columns are optional:

  * `event_dtend` - when the event ends
  * `event_duration` - the duration of the event (use instead of `dtend`)
  * `event_description` - a longer description of the event
  * `event_uid` - a globally unique identifier for this event
  * `event_tzid` - the timezone for the event, e.g. `America/Chicago`

  A query that returns these columns can then be returned as an ics feed by adding the `.ics` extension.

  ## Demo

  [This SQL query]([https://www.rockybeaches.com/data?sql=with+inner+as+(%0D%0A++select%0D%0A++++datetime%2C%0D%0A++++substr(datetime%2C+0%2C+11)+as+date%2C%0D%0A++++mllw_feet%2C%0D%0A++++lag(mllw_feet)+over+win+as+previous_mllw_feet%2C%0D%0A++++lead(mllw_feet)+over+win+as+next_mllw_feet%0D%0A++from%0D%0A++++tide_predictions%0D%0A++where%0D%0A++++station_id+%3D+%3Astation_id%0D%0A++++and+datetime+%3E%3D+date()%0D%0A++++window+win+as+(%0D%0A++++++order+by%0D%0A++++++++datetime%0D%0A++++)%0D%0A++order+by%0D%0A++++datetime%0D%0A)%2C%0D%0Alowest_tide_per_day+as+(%0D%0A++select%0D%0A++++date%2C%0D%0A++++datetime%2C%0D%0A++++mllw_feet%0D%0A++from%0D%0A++++inner%0D%0A++where%0D%0A++++mllw_feet+%3C%3D+previous_mllw_feet%0D%0A++++and+mllw_feet+%3C%3D+next_mllw_feet%0D%0A)%0D%0Aselect%0D%0A++min(datetime)+as+event_dtstart%2C%0D%0A++%27Low+tide%3A+%27+||+mllw_feet+||+%27+feet%27+as+event_name%2C%0D%0A++%27America%2FLos_Angeles%27+as+event_tzid%0D%0Afrom%0D%0A++lowest_tide_per_day%0D%0Agroup+by%0D%0A++date%0D%0Aorder+by%0D%0A++date&station_id=9414131) calculates the lowest tide per day at Pillar Point in Half Moon Bay, California.

  Since the query returns `event_name`, `event_dtstart` and `event_tzid` columns it produces [this ICS feed](https://www.rockybeaches.com/data.ics?sql=with+inner+as+(%0D%0A++select%0D%0A++++datetime%2C%0D%0A++++substr(datetime%2C+0%2C+11)+as+date%2C%0D%0A++++mllw_feet%2C%0D%0A++++lag(mllw_feet)+over+win+as+previous_mllw_feet%2C%0D%0A++++lead(mllw_feet)+over+win+as+next_mllw_feet%0D%0A++from%0D%0A++++tide_predictions%0D%0A++where%0D%0A++++station_id+%3D+%3Astation_id%0D%0A++++and+datetime+%3E%3D+date()%0D%0A++++window+win+as+(%0D%0A++++++order+by%0D%0A++++++++datetime%0D%0A++++)%0D%0A++order+by%0D%0A++++datetime%0D%0A)%2C%0D%0Alowest_tide_per_day+as+(%0D%0A++select%0D%0A++++date%2C%0D%0A++++datetime%2C%0D%0A++++mllw_feet%0D%0A++from%0D%0A++++inner%0D%0A++where%0D%0A++++mllw_feet+%3C%3D+previous_mllw_feet%0D%0A++++and+mllw_feet+%3C%3D+next_mllw_feet%0D%0A)%0D%0Aselect%0D%0A++min(datetime)+as+event_dtstart%2C%0D%0A++%27Low+tide%3A+%27+||+mllw_feet+||+%27+feet%27+as+event_name%2C%0D%0A++%27America%2FLos_Angeles%27+as+event_tzid%0D%0Afrom%0D%0A++lowest_tide_per_day%0D%0Agroup+by%0D%0A++date%0D%0Aorder+by%0D%0A++date&station_id=9414131). If you subscribe to that in a calendar application such as Apple Calendar you get something that looks like this:

  ![Apple Calendar showing low tides at Pillar Point during a week](https://user-images.githubusercontent.com/9599/173158984-e5ec6bd0-33fc-4fc0-ba9d-17ae674f310a.jpg)

  ## Using a canned query

  Datasette's [canned query mechanism](https://datasette.readthedocs.io/en/stable/sql_queries.html#canned-queries) can be used to configure calendars. If a canned query definition has a `title` that will be used as the title of the calendar.

  Here's an example, defined using a `metadata.yaml` file:

  ```yaml
  databases:
    mydatabase:
      queries:
        calendar:
          title: My Calendar
          sql: |-
            select
              title as event_name,
              start as event_dtstart,
              description as event_description
            from
              events
            order by
              start
            limit
              100
  ```
  This will result in a calendar feed at `http://localhost:8001/mydatabase/calendar.ics`

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-ics" class="anchor" aria-hidden="true" href="#user-content-datasette-ics"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-ics</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-ics/" rel="nofollow"><img src="https://camo.githubusercontent.com/ded2e32b426b79e4545931cf7a056c8c1db91042979ee4e1fa286bd89222af70/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6963732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-ics.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-ics/releases"><img src="https://camo.githubusercontent.com/ac8ba594bbdf3837c5b4a7c54786be6517128f0b6fe17fb569b189024fd0fcfb/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6963733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-ics?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-ics/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-ics/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-ics/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Datasette plugin that adds support for generating <a href="https://tools.ietf.org/html/rfc5545" rel="nofollow">iCalendar .ics files</a> with the results of a SQL query.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette to enable the <code>.ics</code> output extension.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-ics"><pre class="notranslate"><code>$ pip install datasette-ics
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">To create an iCalendar file you need to define a custom SQL query that returns a required set of columns:</p>
  <ul dir="auto">
  <li><code>event_name</code> - the short name for the event</li>
  <li><code>event_dtstart</code> - when the event starts</li>
  </ul>
  <p dir="auto">The following columns are optional:</p>
  <ul dir="auto">
  <li><code>event_dtend</code> - when the event ends</li>
  <li><code>event_duration</code> - the duration of the event (use instead of <code>dtend</code>)</li>
  <li><code>event_description</code> - a longer description of the event</li>
  <li><code>event_uid</code> - a globally unique identifier for this event</li>
  <li><code>event_tzid</code> - the timezone for the event, e.g. <code>America/Chicago</code></li>
  </ul>
  <p dir="auto">A query that returns these columns can then be returned as an ics feed by adding the <code>.ics</code> extension.</p>
  <h2 dir="auto"><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p dir="auto">This SQL query calculates the lowest tide per day at Pillar Point in Half Moon Bay, California.</p>
  <p dir="auto">Since the query returns <code>event_name</code>, <code>event_dtstart</code> and <code>event_tzid</code> columns it produces <a href="https://www.rockybeaches.com/data.ics?sql=with+inner+as+(%0D%0A++select%0D%0A++++datetime%2C%0D%0A++++substr(datetime%2C+0%2C+11)+as+date%2C%0D%0A++++mllw_feet%2C%0D%0A++++lag(mllw_feet)+over+win+as+previous_mllw_feet%2C%0D%0A++++lead(mllw_feet)+over+win+as+next_mllw_feet%0D%0A++from%0D%0A++++tide_predictions%0D%0A++where%0D%0A++++station_id+%3D+%3Astation_id%0D%0A++++and+datetime+%3E%3D+date()%0D%0A++++window+win+as+(%0D%0A++++++order+by%0D%0A++++++++datetime%0D%0A++++)%0D%0A++order+by%0D%0A++++datetime%0D%0A)%2C%0D%0Alowest_tide_per_day+as+(%0D%0A++select%0D%0A++++date%2C%0D%0A++++datetime%2C%0D%0A++++mllw_feet%0D%0A++from%0D%0A++++inner%0D%0A++where%0D%0A++++mllw_feet+%3C%3D+previous_mllw_feet%0D%0A++++and+mllw_feet+%3C%3D+next_mllw_feet%0D%0A)%0D%0Aselect%0D%0A++min(datetime)+as+event_dtstart%2C%0D%0A++%27Low+tide%3A+%27+%7C%7C+mllw_feet+%7C%7C+%27+feet%27+as+event_name%2C%0D%0A++%27America%2FLos_Angeles%27+as+event_tzid%0D%0Afrom%0D%0A++lowest_tide_per_day%0D%0Agroup+by%0D%0A++date%0D%0Aorder+by%0D%0A++date&amp;station_id=9414131" rel="nofollow">this ICS feed</a>. If you subscribe to that in a calendar application such as Apple Calendar you get something that looks like this:</p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer" href="https://user-images.githubusercontent.com/9599/173158984-e5ec6bd0-33fc-4fc0-ba9d-17ae674f310a.jpg"><img src="https://user-images.githubusercontent.com/9599/173158984-e5ec6bd0-33fc-4fc0-ba9d-17ae674f310a.jpg" alt="Apple Calendar showing low tides at Pillar Point during a week" style="max-width: 100%;"></a></p>
  <h2 dir="auto"><a id="user-content-using-a-canned-query" class="anchor" aria-hidden="true" href="#user-content-using-a-canned-query"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Using a canned query</h2>
  <p dir="auto">Datasette's <a href="https://datasette.readthedocs.io/en/stable/sql_queries.html#canned-queries" rel="nofollow">canned query mechanism</a> can be used to configure calendars. If a canned query definition has a <code>title</code> that will be used as the title of the calendar.</p>
  <p dir="auto">Here's an example, defined using a <code>metadata.yaml</code> file:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="databases:
    mydatabase:
      queries:
        calendar:
          title: My Calendar
          sql: |-
            select
              title as event_name,
              start as event_dtstart,
              description as event_description
            from
              events
            order by
              start
            limit
              100"><pre><span class="pl-ent">databases</span>:
    <span class="pl-ent">mydatabase</span>:
      <span class="pl-ent">queries</span>:
        <span class="pl-ent">calendar</span>:
          <span class="pl-ent">title</span>: <span class="pl-s">My Calendar</span>
          <span class="pl-ent">sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">          select</span>
  <span class="pl-s">            title as event_name,</span>
  <span class="pl-s">            start as event_dtstart,</span>
  <span class="pl-s">            description as event_description</span>
  <span class="pl-s">          from</span>
  <span class="pl-s">            events</span>
  <span class="pl-s">          order by</span>
  <span class="pl-s">            start</span>
  <span class="pl-s">          limit</span>
  <span class="pl-s">            100</span></pre></div>
  <p dir="auto">This will result in a calendar feed at <code>http://localhost:8001/mydatabase/calendar.ics</code></p>
  </article></div>


