# repos

node_id: MDEwOlJlcG9zaXRvcnkzOTM5OTk1OTg=
name: datasette-pyinstrument
full_name: simonw/datasette-pyinstrument
html_url: https://github.com/simonw/datasette-pyinstrument
description: Use pyinstrument to analyze Datasette page performance
created_at: 2021-08-08T15:33:29Z
updated_at: 2021-08-08T15:50:54Z
pushed_at: 2021-08-08T15:50:52Z
language: Python
topics: []
default_branch: main
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # datasette-pyinstrument

  [![PyPI](https://img.shields.io/pypi/v/datasette-pyinstrument.svg)](https://pypi.org/project/datasette-pyinstrument/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-pyinstrument?include_prereleases&label=changelog)](https://github.com/simonw/datasette-pyinstrument/releases)
  [![Tests](https://github.com/simonw/datasette-pyinstrument/workflows/Test/badge.svg)](https://github.com/simonw/datasette-pyinstrument/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-pyinstrument/blob/main/LICENSE)

  Use pyinstrument to analyze Datasette page performance

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-pyinstrument

  ## Usage

  Once installed, adding `?_pyinstrument=1` to any URL within Datasette will replace the output of that page with the pyinstrument profiler results for it.

  ## Demo

  You can see the output of this plugin at https://latest-with-plugins.datasette.io/fixtures/sortable?_pyinstrument=1

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-pyinstrument
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-pyinstrument" class="anchor" aria-hidden="true" href="#user-content-datasette-pyinstrument"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-pyinstrument</h1>
  <p><a href="https://pypi.org/project/datasette-pyinstrument/" rel="nofollow"><img src="https://camo.githubusercontent.com/04eb20d5c908480064bd38f51a23efb993e1ab6cd8731310a7fd8762fbf0e589/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d7079696e737472756d656e742e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-pyinstrument.svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-pyinstrument/releases"><img src="https://camo.githubusercontent.com/82a450100507a691eb342ecf1a1e328c5078b2d42918187061fe0f1255431fd7/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d7079696e737472756d656e743f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-pyinstrument?include_prereleases&amp;label=changelog" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-pyinstrument/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-pyinstrument/workflows/Test/badge.svg" alt="Tests" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-pyinstrument/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Use pyinstrument to analyze Datasette page performance</p>
  <h2><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p>Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ datasette install datasette-pyinstrument
  "><pre><code>$ datasette install datasette-pyinstrument
  </code></pre></div>
  <h2><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p>Once installed, adding <code>?_pyinstrument=1</code> to any URL within Datasette will replace the output of that page with the pyinstrument profiler results for it.</p>
  <h2><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p>You can see the output of this plugin at <a href="https://latest-with-plugins.datasette.io/fixtures/sortable?_pyinstrument=1" rel="nofollow">https://latest-with-plugins.datasette.io/fixtures/sortable?_pyinstrument=1</a></p>
  <h2><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p>To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="cd datasette-pyinstrument
  python3 -mvenv venv
  source venv/bin/activate
  "><pre><code>cd datasette-pyinstrument
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p>Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pipenv shell
  "><pre><code>pipenv shell
  </code></pre></div>
  <p>Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install -e '.[test]'
  "><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p>To run the tests:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pytest
  "><pre><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkzOTQxMDc2MTQ=
name: datasette-query-links
full_name: simonw/datasette-query-links
html_url: https://github.com/simonw/datasette-query-links
description: Turn SELECT queries returned by a query into links to execute them
created_at: 2021-08-09T01:16:59Z
updated_at: 2021-08-09T04:31:40Z
pushed_at: 2021-08-09T02:56:40Z
language: Python
topics: []
default_branch: main
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # datasette-query-links

  [![PyPI](https://img.shields.io/pypi/v/datasette-query-links.svg)](https://pypi.org/project/datasette-query-links/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-query-links?include_prereleases&label=changelog)](https://github.com/simonw/datasette-query-links/releases)
  [![Tests](https://github.com/simonw/datasette-query-links/workflows/Test/badge.svg)](https://github.com/simonw/datasette-query-links/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-query-links/blob/main/LICENSE)

  Turn SELECT queries returned by a query into links to execute them

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-query-links

  ## Usage

  This is an experimental plugin, requiring Datasette  0.59a1 or higher.

  Any SQL query that returns a value that itself looks like a valid SQL query will be converted into a link to execute that SQL query when it is displayed in the Datasette interface.

  These links will only show for valid SQL query - if a SQL query would return an error it will not be turned into a link.

  ## Demo

  * [Here's an example query](https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++%27select+*+from+%5Bfacetable%5D%27+as+query%0D%0Aunion%0D%0Aselect%0D%0A++%27select+sqlite_version()%27%0D%0Aunion%0D%0Aselect%0D%0A++%27select+this+is+invalid+SQL+so+will+not+be+linked%27) showing the plugin in action.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-query-links
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-query-links" class="anchor" aria-hidden="true" href="#user-content-datasette-query-links"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-query-links</h1>
  <p><a href="https://pypi.org/project/datasette-query-links/" rel="nofollow"><img src="https://camo.githubusercontent.com/3ab4d04fcda27408dfd62a12ac76681be62dfafccd5c0fffb3837512e08a60fb/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d71756572792d6c696e6b732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-query-links.svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-query-links/releases"><img src="https://camo.githubusercontent.com/2889989730367edad93f0dd54b21beb27e4f8f5b68ae195dec3a4751353b5949/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d71756572792d6c696e6b733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-query-links?include_prereleases&amp;label=changelog" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-query-links/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-query-links/workflows/Test/badge.svg" alt="Tests" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-query-links/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Turn SELECT queries returned by a query into links to execute them</p>
  <h2><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p>Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ datasette install datasette-query-links
  "><pre><code>$ datasette install datasette-query-links
  </code></pre></div>
  <h2><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p>This is an experimental plugin, requiring Datasette  0.59a1 or higher.</p>
  <p>Any SQL query that returns a value that itself looks like a valid SQL query will be converted into a link to execute that SQL query when it is displayed in the Datasette interface.</p>
  <p>These links will only show for valid SQL query - if a SQL query would return an error it will not be turned into a link.</p>
  <h2><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <ul>
  <li><a href="https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++%27select+*+from+%5Bfacetable%5D%27+as+query%0D%0Aunion%0D%0Aselect%0D%0A++%27select+sqlite_version()%27%0D%0Aunion%0D%0Aselect%0D%0A++%27select+this+is+invalid+SQL+so+will+not+be+linked%27" rel="nofollow">Here's an example query</a> showing the plugin in action.</li>
  </ul>
  <h2><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p>To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="cd datasette-query-links
  python3 -mvenv venv
  source venv/bin/activate
  "><pre><code>cd datasette-query-links
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p>Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pipenv shell
  "><pre><code>pipenv shell
  </code></pre></div>
  <p>Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install -e '.[test]'
  "><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p>To run the tests:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pytest
  "><pre><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkzOTUxMzc1MTM=
name: datasette-x-forwarded-host
full_name: simonw/datasette-x-forwarded-host
html_url: https://github.com/simonw/datasette-x-forwarded-host
description: Treat the X-Forwarded-Host header as the Host header
created_at: 2021-08-11T23:10:44Z
updated_at: 2021-11-12T20:48:43Z
pushed_at: 2021-11-12T20:48:41Z
language: Python
topics: ["datasette-io", "datasette-plugin"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-x-forwarded-host

  [![PyPI](https://img.shields.io/pypi/v/datasette-x-forwarded-host.svg)](https://pypi.org/project/datasette-x-forwarded-host/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-x-forwarded-host?include_prereleases&label=changelog)](https://github.com/simonw/datasette-x-forwarded-host/releases)
  [![Tests](https://github.com/simonw/datasette-x-forwarded-host/workflows/Test/badge.svg)](https://github.com/simonw/datasette-x-forwarded-host/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-x-forwarded-host/blob/main/LICENSE)

  Treat the X-Forwarded-Host header as the Host header

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-x-forwarded-host

  ## Usage

  Once installed, Datasette will replace the `host` header with the content of the incoming `x-forwarded-host` header.

  This helps Datasette generate links to new pages that work when hosted behind a proxy that rewrites the `host` header.

  Only use this plugin in deployment environmens where you know the `x-forwarded-host` header can be trusted!

  This has been tested on [GitHub Codespaces](https://github.com/features/codespaces) and [GitPod](https://gitpod.io/).

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-x-forwarded-host
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-x-forwarded-host" class="anchor" aria-hidden="true" href="#user-content-datasette-x-forwarded-host"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-x-forwarded-host</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-x-forwarded-host/" rel="nofollow"><img src="https://camo.githubusercontent.com/b6012c0e79a0401f0d06cf37d220fc5a089c004333d1b82a88864b0e28928921/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d782d666f727761726465642d686f73742e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-x-forwarded-host.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-x-forwarded-host/releases"><img src="https://camo.githubusercontent.com/c6ee077f800a99f5e5e1ca935559745f094792b2db8de155c911d4b8906988fa/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d782d666f727761726465642d686f73743f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-x-forwarded-host?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-x-forwarded-host/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-x-forwarded-host/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-x-forwarded-host/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Treat the X-Forwarded-Host header as the Host header</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-x-forwarded-host
  "><pre><code>$ datasette install datasette-x-forwarded-host
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Once installed, Datasette will replace the <code>host</code> header with the content of the incoming <code>x-forwarded-host</code> header.</p>
  <p dir="auto">This helps Datasette generate links to new pages that work when hosted behind a proxy that rewrites the <code>host</code> header.</p>
  <p dir="auto">Only use this plugin in deployment environmens where you know the <code>x-forwarded-host</code> header can be trusted!</p>
  <p dir="auto">This has been tested on <a href="https://github.com/features/codespaces">GitHub Codespaces</a> and <a href="https://gitpod.io/" rel="nofollow">GitPod</a>.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-x-forwarded-host
  python3 -mvenv venv
  source venv/bin/activate
  "><pre><code>cd datasette-x-forwarded-host
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell
  "><pre><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'
  "><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest
  "><pre><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnk0MDA2NzgzMTc=
name: datasette-verify
full_name: simonw/datasette-verify
html_url: https://github.com/simonw/datasette-verify
description: Verify that files can be opened by Datasette
created_at: 2021-08-28T01:59:12Z
updated_at: 2021-08-28T02:37:03Z
pushed_at: 2021-08-28T02:31:34Z
homepage: https://datasette.io/tools/datasette-verify
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
readme:
  # datasette-verify

  [![PyPI](https://img.shields.io/pypi/v/datasette-verify.svg)](https://pypi.org/project/datasette-verify/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-verify?include_prereleases&label=changelog)](https://github.com/simonw/datasette-verify/releases)
  [![Tests](https://github.com/simonw/datasette-verify/workflows/Test/badge.svg)](https://github.com/simonw/datasette-verify/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-verify/blob/main/LICENSE)

  Verify that SQLite files can be opened using Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-verify

  This plugin depends on [Datasette 0.59a2](https://github.com/simonw/datasette/releases/tag/0.59a2) or higher, as it uses the [register_commands()](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hook-register-commands) plugin hook.

  ## Usage

  To confirm that files can be opened by Datasette, run the following:

      datasette verify file1.db file2.db

  You can pass one or more file paths.

  The command will exit silently with a 0 exit code if the files are all valid SQLite databases that Datasette can open.

  It will exit with a 1 exit code and display an error for the first file it finds that is not valid.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-verify
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-verify" class="anchor" aria-hidden="true" href="#user-content-datasette-verify"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-verify</h1>
  <p><a href="https://pypi.org/project/datasette-verify/" rel="nofollow"><img src="https://camo.githubusercontent.com/054b14e587acf0c25a294d22ff99e43599ca4929bac5e4671b3933f180df398f/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d7665726966792e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-verify.svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-verify/releases"><img src="https://camo.githubusercontent.com/4b18b9f7b4f773b0914da3ef602b9e17772d48cc15af29c19110a11c57195363/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d7665726966793f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-verify?include_prereleases&amp;label=changelog" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-verify/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-verify/workflows/Test/badge.svg" alt="Tests" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-verify/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Verify that SQLite files can be opened using Datasette</p>
  <h2><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p>Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ datasette install datasette-verify
  "><pre><code>$ datasette install datasette-verify
  </code></pre></div>
  <p>This plugin depends on <a href="https://github.com/simonw/datasette/releases/tag/0.59a2">Datasette 0.59a2</a> or higher, as it uses the <a href="https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hook-register-commands" rel="nofollow">register_commands()</a> plugin hook.</p>
  <h2><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p>To confirm that files can be opened by Datasette, run the following:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="datasette verify file1.db file2.db
  "><pre><code>datasette verify file1.db file2.db
  </code></pre></div>
  <p>You can pass one or more file paths.</p>
  <p>The command will exit silently with a 0 exit code if the files are all valid SQLite databases that Datasette can open.</p>
  <p>It will exit with a 1 exit code and display an error for the first file it finds that is not valid.</p>
  <h2><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p>To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="cd datasette-verify
  python3 -mvenv venv
  source venv/bin/activate
  "><pre><code>cd datasette-verify
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p>Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pipenv shell
  "><pre><code>pipenv shell
  </code></pre></div>
  <p>Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install -e '.[test]'
  "><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p>To run the tests:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pytest
  "><pre><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOGGsxew
name: datasette-template-request
full_name: simonw/datasette-template-request
html_url: https://github.com/simonw/datasette-template-request
description: Expose the Datasette request object to custom templates
created_at: 2021-09-23T17:07:00Z
updated_at: 2021-09-23T17:29:08Z
pushed_at: 2021-09-23T17:29:36Z
homepage: https://datasette.io/plugins/datasette-template-request
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
readme:
  # datasette-template-request

  [![PyPI](https://img.shields.io/pypi/v/datasette-template-request.svg)](https://pypi.org/project/datasette-template-request/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-template-request?include_prereleases&label=changelog)](https://github.com/simonw/datasette-template-request/releases)
  [![Tests](https://github.com/simonw/datasette-template-request/workflows/Test/badge.svg)](https://github.com/simonw/datasette-template-request/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-template-request/blob/main/LICENSE)

  Expose the Datasette request object to custom templates

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-template-request

  ## Usage

  Once this plugin is installed, Datasette [custom templates](https://docs.datasette.io/en/stable/custom_templates.html) can use `{{ request }}` to access the current [request object](https://docs.datasette.io/en/stable/internals.html#request-object). For example, to access `?name=Cleo` in the query string a template could use this:

      Name: {{ request.args.name }}

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-template-request
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-template-request" class="anchor" aria-hidden="true" href="#user-content-datasette-template-request"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-template-request</h1>
  <p><a href="https://pypi.org/project/datasette-template-request/" rel="nofollow"><img src="https://camo.githubusercontent.com/a12c5039f3646865c1df5eac80037c393636634d80e2d0830e9a4439f8fbc938/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d74656d706c6174652d726571756573742e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-template-request.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-template-request/releases"><img src="https://camo.githubusercontent.com/2dd3333ece522d57411ec7487f09c3988c7bbb264fd440304f1c699df40531b6/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d74656d706c6174652d726571756573743f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-template-request?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-template-request/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-template-request/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-template-request/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p>Expose the Datasette request object to custom templates</p>
  <h2><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p>Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ datasette install datasette-template-request
  "><pre><code>$ datasette install datasette-template-request
  </code></pre></div>
  <h2><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p>Once this plugin is installed, Datasette <a href="https://docs.datasette.io/en/stable/custom_templates.html" rel="nofollow">custom templates</a> can use <code>{{ request }}</code> to access the current <a href="https://docs.datasette.io/en/stable/internals.html#request-object" rel="nofollow">request object</a>. For example, to access <code>?name=Cleo</code> in the query string a template could use this:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="Name: {{ request.args.name }}
  "><pre><code>Name: {{ request.args.name }}
  </code></pre></div>
  <h2><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p>To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="cd datasette-template-request
  python3 -mvenv venv
  source venv/bin/activate
  "><pre><code>cd datasette-template-request
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p>Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pipenv shell
  "><pre><code>pipenv shell
  </code></pre></div>
  <p>Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install -e '.[test]'
  "><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p>To run the tests:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pytest
  "><pre><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOGT91rg
name: datasette-jupyterlite
full_name: simonw/datasette-jupyterlite
html_url: https://github.com/simonw/datasette-jupyterlite
description: JupyterLite as a Datasette plugin
created_at: 2021-11-01T19:22:51Z
updated_at: 2021-11-05T05:12:17Z
pushed_at: 2021-11-05T05:12:33Z
language: Python
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-jupyterlite

  [![PyPI](https://img.shields.io/pypi/v/datasette-jupyterlite.svg)](https://pypi.org/project/datasette-jupyterlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-jupyterlite?include_prereleases&label=changelog)](https://github.com/simonw/datasette-jupyterlite/releases)
  [![Tests](https://github.com/simonw/datasette-jupyterlite/workflows/Test/badge.svg)](https://github.com/simonw/datasette-jupyterlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-jupyterlite/blob/main/LICENSE)

  [JupyterLite](https://jupyterlite.readthedocs.io/en/latest/) as a Datasette plugin

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-jupyterlite

  ## Demo

  You can try out a demo of the plugin here: https://latest-with-plugins.datasette.io/jupyterlite/

  Run this example code in a Pyolite notebook to pull all of the data from the [github/stars](https://latest-with-plugins.datasette.io/github/stars) table into a Pandas DataFrame:

  ```python
  import pandas, pyodide

  df = pandas.read_csv(pyodide.open_url(
    "https://latest-with-plugins.datasette.io/github/stars.csv?_labels=on&_stream=on&_size=max")
  )
  ```

  ## Usage

  Once installed, visit `/jupyterlite/` to access JupyterLite served from your Datasette instance.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-jupyterlite
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-jupyterlite" class="anchor" aria-hidden="true" href="#user-content-datasette-jupyterlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-jupyterlite</h1>
  <p><a href="https://pypi.org/project/datasette-jupyterlite/" rel="nofollow"><img src="https://camo.githubusercontent.com/583483a5498ff95bfe0084890b0bbad55390e616be11f9592292911c6d6bf74f/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6a7570797465726c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-jupyterlite.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-jupyterlite/releases"><img src="https://camo.githubusercontent.com/8bd8e900ccedb77c748e60e5b1dc3a7cace91fb07a1afae015e9de1ca60db907/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6a7570797465726c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-jupyterlite?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-jupyterlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-jupyterlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-jupyterlite/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p><a href="https://jupyterlite.readthedocs.io/en/latest/" rel="nofollow">JupyterLite</a> as a Datasette plugin</p>
  <h2><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p>Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-jupyterlite
  "><pre><code>$ datasette install datasette-jupyterlite
  </code></pre></div>
  <h2><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p>You can try out a demo of the plugin here: <a href="https://latest-with-plugins.datasette.io/jupyterlite/" rel="nofollow">https://latest-with-plugins.datasette.io/jupyterlite/</a></p>
  <p>Run this example code in a Pyolite notebook to pull all of the data from the <a href="https://latest-with-plugins.datasette.io/github/stars" rel="nofollow">github/stars</a> table into a Pandas DataFrame:</p>
  <div class="highlight highlight-source-python position-relative overflow-auto" data-snippet-clipboard-copy-content="import pandas, pyodide

  df = pandas.read_csv(pyodide.open_url(
    &quot;https://latest-with-plugins.datasette.io/github/stars.csv?_labels=on&amp;_stream=on&amp;_size=max&quot;)
  )
  "><pre><span class="pl-k">import</span> <span class="pl-s1">pandas</span>, <span class="pl-s1">pyodide</span>

  <span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pandas</span>.<span class="pl-en">read_csv</span>(<span class="pl-s1">pyodide</span>.<span class="pl-en">open_url</span>(
    <span class="pl-s">"https://latest-with-plugins.datasette.io/github/stars.csv?_labels=on&amp;_stream=on&amp;_size=max"</span>)
  )</pre></div>
  <h2><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p>Once installed, visit <code>/jupyterlite/</code> to access JupyterLite served from your Datasette instance.</p>
  <h2><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p>To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-jupyterlite
  python3 -mvenv venv
  source venv/bin/activate
  "><pre><code>cd datasette-jupyterlite
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p>Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell
  "><pre><code>pipenv shell
  </code></pre></div>
  <p>Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'
  "><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p>To run the tests:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest
  "><pre><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOGUV9ig
name: s3-credentials
full_name: simonw/s3-credentials
html_url: https://github.com/simonw/s3-credentials
description: A tool for creating credentials for accessing S3 buckets
created_at: 2021-11-02T20:09:50Z
updated_at: 2022-09-05T15:12:46Z
pushed_at: 2022-09-15T23:43:10Z
homepage: https://s3-credentials.readthedocs.io
language: Python
license: apache-2.0
topics: ["aws", "boto3", "s3"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # s3-credentials

  [![PyPI](https://img.shields.io/pypi/v/s3-credentials.svg)](https://pypi.org/project/s3-credentials/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/s3-credentials?include_prereleases&label=changelog)](https://github.com/simonw/s3-credentials/releases)
  [![Tests](https://github.com/simonw/s3-credentials/workflows/Test/badge.svg)](https://github.com/simonw/s3-credentials/actions?query=workflow%3ATest)
  [![Documentation Status](https://readthedocs.org/projects/s3-credentials/badge/?version=latest)](https://s3-credentials.readthedocs.org/)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/s3-credentials/blob/master/LICENSE)

  A tool for creating credentials for accessing S3 buckets

  For project background, see [s3-credentials: a tool for creating credentials for S3 buckets](https://simonwillison.net/2021/Nov/3/s3-credentials/) on my blog.

  ## Installation

      pip install s3-credentials

  ## Basic usage

  To create a new S3 bucket and output credentials that can be used with only that bucket:
  ```
  % s3-credentials create my-new-s3-bucket --create-bucket
  Created bucket:  my-new-s3-bucket
  Created user: s3.read-write.my-new-s3-bucket with permissions boundary: arn:aws:iam::aws:policy/AmazonS3FullAccess
  Attached policy s3.read-write.my-new-s3-bucket to user s3.read-write.my-new-s3-bucket
  Created access key for user: s3.read-write.my-new-s3-bucket
  {
      "UserName": "s3.read-write.my-new-s3-bucket",
      "AccessKeyId": "AKIAWXFXAIOZOYLZAEW5",
      "Status": "Active",
      "SecretAccessKey": "...",
      "CreateDate": "2021-11-03 01:38:24+00:00"
  }
  ```
  The tool can do a lot more than this. See the [documentation](https://s3-credentials.readthedocs.io/) for details.

  ## Documentation

  - [Full documentation](https://s3-credentials.readthedocs.io/)
  - [Command help reference](https://s3-credentials.readthedocs.io/en/stable/help.html)
  - [Release notes](https://github.com/simonw/s3-credentials/releases)

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-s3-credentials" class="anchor" aria-hidden="true" href="#user-content-s3-credentials"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>s3-credentials</h1>
  <p dir="auto"><a href="https://pypi.org/project/s3-credentials/" rel="nofollow"><img src="https://camo.githubusercontent.com/843ad3dae900f3f8042c9847a2d4ec012f47c9e7569fb65cdf66d288c152aecf/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73332d63726564656e7469616c732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/s3-credentials.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/s3-credentials/releases"><img src="https://camo.githubusercontent.com/03854eef387e7e67ee77ab44fc4770f3edef39895610bedd2de9aff4ed873dcf/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f73332d63726564656e7469616c733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/s3-credentials?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/s3-credentials/actions?query=workflow%3ATest"><img src="https://github.com/simonw/s3-credentials/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://s3-credentials.readthedocs.org/" rel="nofollow"><img src="https://camo.githubusercontent.com/2d884a293a1442ff42dc0a43da9554054e6153b0b32766a090d82e2add945d73/68747470733a2f2f72656164746865646f63732e6f72672f70726f6a656374732f73332d63726564656e7469616c732f62616467652f3f76657273696f6e3d6c6174657374" alt="Documentation Status" data-canonical-src="https://readthedocs.org/projects/s3-credentials/badge/?version=latest" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/s3-credentials/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">A tool for creating credentials for accessing S3 buckets</p>
  <p dir="auto">For project background, see <a href="https://simonwillison.net/2021/Nov/3/s3-credentials/" rel="nofollow">s3-credentials: a tool for creating credentials for S3 buckets</a> on my blog.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install s3-credentials"><pre class="notranslate"><code>pip install s3-credentials
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-basic-usage" class="anchor" aria-hidden="true" href="#user-content-basic-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Basic usage</h2>
  <p dir="auto">To create a new S3 bucket and output credentials that can be used with only that bucket:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="% s3-credentials create my-new-s3-bucket --create-bucket
  Created bucket:  my-new-s3-bucket
  Created user: s3.read-write.my-new-s3-bucket with permissions boundary: arn:aws:iam::aws:policy/AmazonS3FullAccess
  Attached policy s3.read-write.my-new-s3-bucket to user s3.read-write.my-new-s3-bucket
  Created access key for user: s3.read-write.my-new-s3-bucket
  {
      &quot;UserName&quot;: &quot;s3.read-write.my-new-s3-bucket&quot;,
      &quot;AccessKeyId&quot;: &quot;AKIAWXFXAIOZOYLZAEW5&quot;,
      &quot;Status&quot;: &quot;Active&quot;,
      &quot;SecretAccessKey&quot;: &quot;...&quot;,
      &quot;CreateDate&quot;: &quot;2021-11-03 01:38:24+00:00&quot;
  }"><pre class="notranslate"><code>% s3-credentials create my-new-s3-bucket --create-bucket
  Created bucket:  my-new-s3-bucket
  Created user: s3.read-write.my-new-s3-bucket with permissions boundary: arn:aws:iam::aws:policy/AmazonS3FullAccess
  Attached policy s3.read-write.my-new-s3-bucket to user s3.read-write.my-new-s3-bucket
  Created access key for user: s3.read-write.my-new-s3-bucket
  {
      "UserName": "s3.read-write.my-new-s3-bucket",
      "AccessKeyId": "AKIAWXFXAIOZOYLZAEW5",
      "Status": "Active",
      "SecretAccessKey": "...",
      "CreateDate": "2021-11-03 01:38:24+00:00"
  }
  </code></pre></div>
  <p dir="auto">The tool can do a lot more than this. See the <a href="https://s3-credentials.readthedocs.io/" rel="nofollow">documentation</a> for details.</p>
  <h2 dir="auto"><a id="user-content-documentation" class="anchor" aria-hidden="true" href="#user-content-documentation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Documentation</h2>
  <ul dir="auto">
  <li><a href="https://s3-credentials.readthedocs.io/" rel="nofollow">Full documentation</a></li>
  <li><a href="https://s3-credentials.readthedocs.io/en/stable/help.html" rel="nofollow">Command help reference</a></li>
  <li><a href="https://github.com/simonw/s3-credentials/releases">Release notes</a></li>
  </ul>
  </article></div>


node_id: R_kgDOGXV4Ig
name: git-history
full_name: simonw/git-history
html_url: https://github.com/simonw/git-history
description: Tools for analyzing Git history using SQLite
created_at: 2021-11-11T20:07:06Z
updated_at: 2022-10-20T20:33:28Z
pushed_at: 2022-10-21T23:06:33Z
language: Python
license: apache-2.0
topics: ["git", "sqlite"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # git-history

  [![PyPI](https://img.shields.io/pypi/v/git-history.svg)](https://pypi.org/project/git-history/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/git-history?include_prereleases&label=changelog)](https://github.com/simonw/git-history/releases)
  [![Tests](https://github.com/simonw/git-history/workflows/Test/badge.svg)](https://github.com/simonw/git-history/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/git-history/blob/master/LICENSE)

  Tools for analyzing Git history using SQLite

  For background on this project see [git-history: a tool for analyzing scraped data collected using Git and SQLite](https://simonwillison.net/2021/Dec/7/git-history/).

  [Measuring traffic during the Half Moon Bay Pumpkin Festival](https://simonwillison.net/2022/Oct/19/measuring-traffic/) describes a project using this tool in detail.

  ## Installation

  Install this tool using `pip`:

      $ pip install git-history

  ## Demos

  [git-history-demos.datasette.io](http://git-history-demos.datasette.io/) hosts three example databases created using this tool:

  - [pge-outages](https://git-history-demos.datasette.io/pge-outages) shows a history of PG&E (the electricity supplier) [outages](https://pgealerts.alerts.pge.com/outagecenter/), using data collected in [simonw/pge-outages](https://github.com/simonw/pge-outages) converted using [pge-outages.sh](https://github.com/simonw/git-history/blob/main/demos/pge-outages.sh)
  - [ca-fires](https://git-history-demos.datasette.io/ca-fires) shows a history of fires in California reported on [fire.ca.gov/incidents](https://www.fire.ca.gov/incidents/), from data in [simonw/ca-fires-history](https://github.com/simonw/ca-fires-history) converted using [ca-fires.sh](https://github.com/simonw/git-history/blob/main/demos/ca-fires.sh)
  - [sf-bay-511](https://git-history-demos.datasette.io/sf-bay-511) has records of San Francisco Bay Area traffic and transit incident data from [511.org](https://511.org/), collected in [dbreunig/511-events-history](https://github.com/dbreunig/511-events-history) converted using [sf-bay-511.sh](https://github.com/simonw/git-history/blob/main/demos/sf-bay-511.sh)

  The demos are deployed using [Datasette](https://datasette.io/) on [Google Cloud Run](https://cloud.google.com/run/) by [this GitHub Actions workflow](https://github.com/simonw/git-history/blob/main/.github/workflows/deploy-demos.yml).

  ## Usage

  This tool can be run against a Git repository that holds a file that contains JSON, CSV/TSV or some other format and which has multiple versions tracked in the Git history. Read [Git scraping: track changes over time by scraping to a Git repository](https://simonwillison.net/2020/Oct/9/git-scraping/) to understand how you might create such a repository.

  The `file` command analyzes the history of an individual file within the repository, and generates a SQLite database table that represents the different versions of that file over time.

  The file is assumed to contain multiple objects - for example, the results of scraping an electricity outage map or a CSV file full of records.

  Assuming you have a file called `incidents.json` that is a JSON array of objects, with multiple versions of that file recorded in a repository. Each version of that file might look something like this:

  ```json
  [
      {
          "IncidentID": "abc123",
          "Location": "Corner of 4th and Vermont",
          "Type": "fire"
      },
      {
          "IncidentID": "cde448",
          "Location": "555 West Example Drive",
          "Type": "medical"
      }
  ]
  ```

  Change directory into the GitHub repository in question and run the following:

      git-history file incidents.db incidents.json

  This will create a new SQLite database in the `incidents.db` file with three tables:

  - `commits` containing a row for every commit, with a `hash` column, the `commit_at` date and a foreign key to a `namespace`.
  - `item` containing a row for every item in every version of the `filename.json` file - with an extra `_commit` column that is a foreign key back to the `commit` table.
  - `namespaces` containing a single row. This allows you to build multiple tables for different files, using the `--namespace` option described below.

  The database schema for this example will look like this:

  <!-- [[[cog
  import cog, json
  from git_history import cli
  from click.testing import CliRunner
  from tests.test_git_history import make_repo
  import sqlite_utils
  import tempfile, pathlib
  tmpdir = pathlib.Path(tempfile.mkdtemp())
  db_path = str(tmpdir / "data.db")
  make_repo(tmpdir)
  runner = CliRunner()
  result = runner.invoke(cli.cli, [
      "file", db_path, str(tmpdir / "repo" / "incidents.json"), "--repo", str(tmpdir / "repo")
  ])
  cog.out("```sql\n")
  cog.out(sqlite_utils.Database(db_path).schema)
  cog.out("\n```")
  ]]] -->
  ```sql
  CREATE TABLE [namespaces] (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT
  );
  CREATE UNIQUE INDEX [idx_namespaces_name]
      ON [namespaces] ([name]);
  CREATE TABLE [commits] (
     [id] INTEGER PRIMARY KEY,
     [namespace] INTEGER REFERENCES [namespaces]([id]),
     [hash] TEXT,
     [commit_at] TEXT
  );
  CREATE UNIQUE INDEX [idx_commits_namespace_hash]
      ON [commits] ([namespace], [hash]);
  CREATE TABLE [item] (
     [IncidentID] TEXT,
     [Location] TEXT,
     [Type] TEXT
  );
  ```
  <!-- [[[end]]] -->

  If you have 10 historic versions of the `incidents.json` file and each one contains 30 incidents, you will end up with 10 * 30 = 300 rows in your `item` table.

  ### Track the history of individual items using IDs

  If your objects have a unique identifier - or multiple columns that together form a unique identifier - you can use the `--id` option to de-duplicate and track changes to each of those items over time.

  This provides a much more interesting way to apply this tool.

  If there is a unique identifier column called `IncidentID` you could run the following:

      git-history file incidents.db incidents.json --id IncidentID

  The database schema used here is very different from the one used without the `--id` option.

  If you have already imported history, the command will skip any commits that it has seen already and just process new ones. This means that even though an initial import could be slow subsequent imports should run a lot faster.

  This command will create six tables - `commits`, `item`, `item_version`, `columns`, `item_changed` and `namespaces`.

  Here's the full schema:

  <!-- [[[cog
  db_path2 = str(tmpdir / "data2.db")
  result = runner.invoke(cli.cli, [
      "file", db_path2, str(tmpdir / "repo" / "incidents.json"),
      "--repo", str(tmpdir / "repo"),
      "--id", "IncidentID"
  ])
  cog.out("```sql\n")
  cog.out(sqlite_utils.Database(db_path2).schema)
  cog.out("\n```")
  ]]] -->
  ```sql
  CREATE TABLE [namespaces] (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT
  );
  CREATE UNIQUE INDEX [idx_namespaces_name]
      ON [namespaces] ([name]);
  CREATE TABLE [commits] (
     [id] INTEGER PRIMARY KEY,
     [namespace] INTEGER REFERENCES [namespaces]([id]),
     [hash] TEXT,
     [commit_at] TEXT
  );
  CREATE UNIQUE INDEX [idx_commits_namespace_hash]
      ON [commits] ([namespace], [hash]);
  CREATE TABLE [item] (
     [_id] INTEGER PRIMARY KEY,
     [_item_id] TEXT
  , [IncidentID] TEXT, [Location] TEXT, [Type] TEXT, [_commit] INTEGER);
  CREATE UNIQUE INDEX [idx_item__item_id]
      ON [item] ([_item_id]);
  CREATE TABLE [item_version] (
     [_id] INTEGER PRIMARY KEY,
     [_item] INTEGER REFERENCES [item]([_id]),
     [_version] INTEGER,
     [_commit] INTEGER REFERENCES [commits]([id]),
     [IncidentID] TEXT,
     [Location] TEXT,
     [Type] TEXT,
     [_item_full_hash] TEXT
  );
  CREATE TABLE [columns] (
     [id] INTEGER PRIMARY KEY,
     [namespace] INTEGER REFERENCES [namespaces]([id]),
     [name] TEXT
  );
  CREATE UNIQUE INDEX [idx_columns_namespace_name]
      ON [columns] ([namespace], [name]);
  CREATE TABLE [item_changed] (
     [item_version] INTEGER REFERENCES [item_version]([_id]),
     [column] INTEGER REFERENCES [columns]([id]),
     PRIMARY KEY ([item_version], [column])
  );
  CREATE VIEW item_version_detail AS select
    commits.commit_at as _commit_at,
    commits.hash as _commit_hash,
    item_version.*,
    (
      select json_group_array(name) from columns
      where id in (
        select column from item_changed
        where item_version = item_version._id
      )
  ) as _changed_columns
  from item_version
    join commits on commits.id = item_version._commit;
  CREATE INDEX [idx_item_version__item]
      ON [item_version] ([_item]);
  ```
  <!-- [[[end]]] -->

  #### item table

  The `item` table will contain the most recent version of each row, de-duplicated by ID, plus the following additional columns:

  - `_id` - a numeric integer primary key, used as a foreign key from the `item_version` table.
  - `_item_id` - a hash of the values of the columns specified using the `--id` option to the command. This is used for de-duplication when processing new versions.
  - `_commit` - a foreign key to the `commit` table, representing the most recent commit to modify this item.

  #### item_version table

  The `item_version` table will contain a row for each captured differing version of that item, plus the following columns:

  - `_id` - a numeric ID for the item version record.
  - `_item` - a foreign key to the `item` table.
  - `_version` - the numeric version number, starting at 1 and incrementing for each captured version.
  - `_commit` - a foreign key to the `commit` table.
  - `_item_full_hash` - a hash of this version of the item. This is used internally by the tool to identify items that have changed between commits.

  The other columns in this table represent columns in the original data that have changed since the previous version. If the value has not changed, it will be represented by a `null`.

  If a value was previously set but has been changed back to `null` it will still be represented as `null` in the `item_version` row. You can identify these using the `item_changed` many-to-many table described below.

  You can use the `--full-versions` option to store full copies of the item at each version, rather than just storing the columns that have changed.

  #### item_version_detail view

  This SQL view joins `item_version` against `commits` to add three further columns: `_commit_at` with the date of the commit, and `_commit_hash` with the Git commit hash.

  #### item_changed

  This many-to-many table indicates exactly which columns were changed in an `item_version`.

  - `item_version` is a foreign key to a row in the `item_version` table.
  - `column` is a foreign key to a row in the `columns` table.

  This table with have the largest number of rows, which is why it stores just two integers in order to save space.

  #### columns

  The `columns` table stores column names. It is referenced by `item_changed`.

  - `id` - an integer ID.
  - `name` - the name of the column.
  - `namespace` - a foreign key to `namespaces`, for if multiple file histories are sharing the same database.

  #### Reserved column names

  <!-- [[[cog
  from git_history.utils import RESERVED
  cog.out("Note that ")
  cog.out(", ".join("`{}`".format(r) for r in RESERVED))
  cog.out(" are considered reserved column names for the purposes of this tool.")
  ]]] -->
  Note that `_id`, `_item_full_hash`, `_item`, `_item_id`, `_version`, `_commit`, `_item_id`, `_commit_at`, `_commit_hash`, `_changed_columns`, `rowid` are considered reserved column names for the purposes of this tool.
  <!-- [[[end]]] -->

  If your data contains any of these they will be renamed to add a trailing underscore, for example `_id_`, `_item_`, `_version_`, to avoid clashing with the reserved columns.

  If you have a column with a name such as `_commit_` it will be renamed too, adding an additional trailing underscore, so `_commit_` becomes `_commit__` and `_commit__` becomes `_commit___`.

  ### Additional options

  - `--repo DIRECTORY` - the path to the Git repository, if it is not the current working directory.
  - `--branch TEXT` - the Git branch to analyze - defaults to `main`.
  - `--id TEXT` - as described above: pass one or more columns that uniquely identify a record, so that changes to that record can be calculated over time.
  - `--full-versions` - instead of recording just the columns that have changed in the `item_version` table record a full copy of each version of theh item.
  - `--ignore TEXT` - one or more columns to ignore - they will not be included in the resulting database.
  - `--csv` - treat the data is CSV or TSV rather than JSON, and attempt to guess the correct dialect
  - `--dialect` - use a spcific CSV dialect. Options are `excel`, `excel-tab` and `unix` - see [the Python CSV documentation](https://docs.python.org/3/library/csv.html#csv.excel) for details.
  - `--skip TEXT` - one or more full Git commit hashes that should be skipped. You can use this if some of the data in your revision history is corrupted in a way that prevents this tool from working.
  - `--start-at TEXT` - skip commits prior to the specified commit hash.
  - `--start-after TEXT` - skip commits up to and including the specified commit hash, then start processing from the following commit.
  - `--convert TEXT` - custom Python code for a conversion, described below.
  - `--import TEXT` - additional Python modules to import for `--convert`.
  - `--ignore-duplicate-ids` - if a single version of a file has the same ID in it more than once, the tool will exit with an error. Use this option to ignore this and instead pick just the first of the two duplicates.
  - `--namespace TEXT` - use this if you wish to include the history of multiple different files in the same database. The default is `item` but you can set it to something else, which will produce tables with names like `yournamespace` and `yournamespace_version`.
  - `--wal` - Enable WAL mode on the created database file. Use this if you plan to run queries against the database while `git-history` is creating it.
  - `--silent` - don't show the progress bar.

  ### CSV and TSV data

  If the data in your repository is a CSV or TSV file you can process it by adding the `--csv` option. This will attempt to detect which delimiter is used by the file, so the same option works for both comma- and tab-separated values.

      git-history file trees.db trees.csv --id TreeID

  You can also specify the CSV dialect using the `--dialect` option.

  ### Custom conversions using --convert

  If your data is not already either CSV/TSV or a flat JSON array, you can reshape it using the `--convert` option.

  The format needed by this tool is an array of dictionaries, as demonstrated by the `incidents.json` example above.

  If your data does not fit this shape, you can provide a snippet of Python code to converts the on-disk content of each stored file into a Python list of dictionaries.

  For example, if your stored files each look like this:

  ```json
  {
      "incidents": [
          {
              "id": "552",
              "name": "Hawthorne Fire",
              "engines": 3
          },
          {
              "id": "556",
              "name": "Merlin Fire",
              "engines": 1
          }
      ]
  }
  ```
  You could use the following Python snippet to convert them to the required format:

  ```python
  json.loads(content)["incidents"]
  ```
  (The `json` module is exposed to your custom function by default.)

  You would then run the tool like this:

      git-history file database.db incidents.json \
        --id id \
        --convert 'json.loads(content)["incidents"]'

  The `content` variable is always a `bytes` object representing the content of the file at a specific moment in the repository's history.

  You can import additional modules using `--import`. This example shows how you could read a CSV file that uses `;` as the delimiter:

      git-history file trees.db ../sf-tree-history/Street_Tree_List.csv \
        --repo ../sf-tree-history \
        --import csv \
        --import io \
        --convert '
          fp = io.StringIO(content.decode("utf-8"))
          return list(csv.DictReader(fp, delimiter=";"))
          ' \
        --id TreeID

  You can import nested modules such as [ElementTree](https://docs.python.org/3/library/xml.etree.elementtree.html)  using `--import xml.etree.ElementTree`, then refer to them in your function body as `xml.etree.ElementTree`. For example, if your tracked data was in an `items.xml` file that looked like this:

  ```xml
  <items>
    <item id="1" name="One" />
    <item id="2" name="Two" />
    <item id="3" name="Three" />
  </item>
  ```
  You could load it using the following `--convert` script:
  ```
  git-history file items.xml --convert '
  tree = xml.etree.ElementTree.fromstring(content)
  return [el.attrib for el in tree.iter("item")]
  ' --import xml.etree.ElementTree --id id
  ```

  If your Python code spans more than one line it needs to include a `return` statement.

  You can also use Python generators in your `--convert` code, for example:

      git-history file stats.db package-stats/stats.json \
          --repo package-stats \
          --convert '
          data = json.loads(content)
          for key, counts in data.items():
              for date, count in counts.items():
                  yield {
                      "package": key,
                      "date": date,
                      "count": count
                  }
          ' --id package --id date

  This conversion function expects data that looks like this:

  ```json
  {
      "airtable-export": {
          "2021-05-18": 66,
          "2021-05-19": 60,
          "2021-05-20": 87
      }
  }
  ```

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd git-history
      python -m venv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

  To update the schema examples in this README file:

      cog -r README.md

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-git-history" class="anchor" aria-hidden="true" href="#user-content-git-history"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>git-history</h1>
  <p dir="auto"><a href="https://pypi.org/project/git-history/" rel="nofollow"><img src="https://camo.githubusercontent.com/eb1f825a264577967f0ed804c6ce43e204d4cb64f8a3cea7c45a023a5ae42ff3/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6769742d686973746f72792e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/git-history.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/git-history/releases"><img src="https://camo.githubusercontent.com/19a0b180a4c69384c234f52e0defeb4086138c1edeccbe42e5623e340503c0ec/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6769742d686973746f72793f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/git-history?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/git-history/actions?query=workflow%3ATest"><img src="https://github.com/simonw/git-history/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/git-history/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Tools for analyzing Git history using SQLite</p>
  <p dir="auto">For background on this project see <a href="https://simonwillison.net/2021/Dec/7/git-history/" rel="nofollow">git-history: a tool for analyzing scraped data collected using Git and SQLite</a>.</p>
  <p dir="auto"><a href="https://simonwillison.net/2022/Oct/19/measuring-traffic/" rel="nofollow">Measuring traffic during the Half Moon Bay Pumpkin Festival</a> describes a project using this tool in detail.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this tool using <code>pip</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install git-history"><pre class="notranslate"><code>$ pip install git-history
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-demos" class="anchor" aria-hidden="true" href="#user-content-demos"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demos</h2>
  <p dir="auto"><a href="http://git-history-demos.datasette.io/" rel="nofollow">git-history-demos.datasette.io</a> hosts three example databases created using this tool:</p>
  <ul dir="auto">
  <li><a href="https://git-history-demos.datasette.io/pge-outages" rel="nofollow">pge-outages</a> shows a history of PG&amp;E (the electricity supplier) <a href="https://pgealerts.alerts.pge.com/outagecenter/" rel="nofollow">outages</a>, using data collected in <a href="https://github.com/simonw/pge-outages">simonw/pge-outages</a> converted using <a href="https://github.com/simonw/git-history/blob/main/demos/pge-outages.sh">pge-outages.sh</a></li>
  <li><a href="https://git-history-demos.datasette.io/ca-fires" rel="nofollow">ca-fires</a> shows a history of fires in California reported on <a href="https://www.fire.ca.gov/incidents/" rel="nofollow">fire.ca.gov/incidents</a>, from data in <a href="https://github.com/simonw/ca-fires-history">simonw/ca-fires-history</a> converted using <a href="https://github.com/simonw/git-history/blob/main/demos/ca-fires.sh">ca-fires.sh</a></li>
  <li><a href="https://git-history-demos.datasette.io/sf-bay-511" rel="nofollow">sf-bay-511</a> has records of San Francisco Bay Area traffic and transit incident data from <a href="https://511.org/" rel="nofollow">511.org</a>, collected in <a href="https://github.com/dbreunig/511-events-history">dbreunig/511-events-history</a> converted using <a href="https://github.com/simonw/git-history/blob/main/demos/sf-bay-511.sh">sf-bay-511.sh</a></li>
  </ul>
  <p dir="auto">The demos are deployed using <a href="https://datasette.io/" rel="nofollow">Datasette</a> on <a href="https://cloud.google.com/run/" rel="nofollow">Google Cloud Run</a> by <a href="https://github.com/simonw/git-history/blob/main/.github/workflows/deploy-demos.yml">this GitHub Actions workflow</a>.</p>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">This tool can be run against a Git repository that holds a file that contains JSON, CSV/TSV or some other format and which has multiple versions tracked in the Git history. Read <a href="https://simonwillison.net/2020/Oct/9/git-scraping/" rel="nofollow">Git scraping: track changes over time by scraping to a Git repository</a> to understand how you might create such a repository.</p>
  <p dir="auto">The <code>file</code> command analyzes the history of an individual file within the repository, and generates a SQLite database table that represents the different versions of that file over time.</p>
  <p dir="auto">The file is assumed to contain multiple objects - for example, the results of scraping an electricity outage map or a CSV file full of records.</p>
  <p dir="auto">Assuming you have a file called <code>incidents.json</code> that is a JSON array of objects, with multiple versions of that file recorded in a repository. Each version of that file might look something like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[
      {
          &quot;IncidentID&quot;: &quot;abc123&quot;,
          &quot;Location&quot;: &quot;Corner of 4th and Vermont&quot;,
          &quot;Type&quot;: &quot;fire&quot;
      },
      {
          &quot;IncidentID&quot;: &quot;cde448&quot;,
          &quot;Location&quot;: &quot;555 West Example Drive&quot;,
          &quot;Type&quot;: &quot;medical&quot;
      }
  ]"><pre>[
      {
          <span class="pl-ent">"IncidentID"</span>: <span class="pl-s"><span class="pl-pds">"</span>abc123<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"Location"</span>: <span class="pl-s"><span class="pl-pds">"</span>Corner of 4th and Vermont<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"Type"</span>: <span class="pl-s"><span class="pl-pds">"</span>fire<span class="pl-pds">"</span></span>
      },
      {
          <span class="pl-ent">"IncidentID"</span>: <span class="pl-s"><span class="pl-pds">"</span>cde448<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"Location"</span>: <span class="pl-s"><span class="pl-pds">"</span>555 West Example Drive<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"Type"</span>: <span class="pl-s"><span class="pl-pds">"</span>medical<span class="pl-pds">"</span></span>
      }
  ]</pre></div>
  <p dir="auto">Change directory into the GitHub repository in question and run the following:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="git-history file incidents.db incidents.json"><pre class="notranslate"><code>git-history file incidents.db incidents.json
  </code></pre></div>
  <p dir="auto">This will create a new SQLite database in the <code>incidents.db</code> file with three tables:</p>
  <ul dir="auto">
  <li><code>commits</code> containing a row for every commit, with a <code>hash</code> column, the <code>commit_at</code> date and a foreign key to a <code>namespace</code>.</li>
  <li><code>item</code> containing a row for every item in every version of the <code>filename.json</code> file - with an extra <code>_commit</code> column that is a foreign key back to the <code>commit</code> table.</li>
  <li><code>namespaces</code> containing a single row. This allows you to build multiple tables for different files, using the <code>--namespace</code> option described below.</li>
  </ul>
  <p dir="auto">The database schema for this example will look like this:</p>

  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="CREATE TABLE [namespaces] (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT
  );
  CREATE UNIQUE INDEX [idx_namespaces_name]
      ON [namespaces] ([name]);
  CREATE TABLE [commits] (
     [id] INTEGER PRIMARY KEY,
     [namespace] INTEGER REFERENCES [namespaces]([id]),
     [hash] TEXT,
     [commit_at] TEXT
  );
  CREATE UNIQUE INDEX [idx_commits_namespace_hash]
      ON [commits] ([namespace], [hash]);
  CREATE TABLE [item] (
     [IncidentID] TEXT,
     [Location] TEXT,
     [Type] TEXT
  );"><pre>CREATE TABLE [namespaces] (
     [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
     [name] <span class="pl-k">TEXT</span>
  );
  CREATE UNIQUE INDEX [idx_namespaces_name]
      <span class="pl-k">ON</span> [namespaces] ([name]);
  CREATE TABLE [commits] (
     [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
     [namespace] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [namespaces]([id]),
     [hash] <span class="pl-k">TEXT</span>,
     [commit_at] <span class="pl-k">TEXT</span>
  );
  CREATE UNIQUE INDEX [idx_commits_namespace_hash]
      <span class="pl-k">ON</span> [commits] ([namespace], [hash]);
  CREATE TABLE [item] (
     [IncidentID] <span class="pl-k">TEXT</span>,
     [Location] <span class="pl-k">TEXT</span>,
     [Type] <span class="pl-k">TEXT</span>
  );</pre></div>

  <p dir="auto">If you have 10 historic versions of the <code>incidents.json</code> file and each one contains 30 incidents, you will end up with 10 * 30 = 300 rows in your <code>item</code> table.</p>
  <h3 dir="auto"><a id="user-content-track-the-history-of-individual-items-using-ids" class="anchor" aria-hidden="true" href="#user-content-track-the-history-of-individual-items-using-ids"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Track the history of individual items using IDs</h3>
  <p dir="auto">If your objects have a unique identifier - or multiple columns that together form a unique identifier - you can use the <code>--id</code> option to de-duplicate and track changes to each of those items over time.</p>
  <p dir="auto">This provides a much more interesting way to apply this tool.</p>
  <p dir="auto">If there is a unique identifier column called <code>IncidentID</code> you could run the following:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="git-history file incidents.db incidents.json --id IncidentID"><pre class="notranslate"><code>git-history file incidents.db incidents.json --id IncidentID
  </code></pre></div>
  <p dir="auto">The database schema used here is very different from the one used without the <code>--id</code> option.</p>
  <p dir="auto">If you have already imported history, the command will skip any commits that it has seen already and just process new ones. This means that even though an initial import could be slow subsequent imports should run a lot faster.</p>
  <p dir="auto">This command will create six tables - <code>commits</code>, <code>item</code>, <code>item_version</code>, <code>columns</code>, <code>item_changed</code> and <code>namespaces</code>.</p>
  <p dir="auto">Here's the full schema:</p>

  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="CREATE TABLE [namespaces] (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT
  );
  CREATE UNIQUE INDEX [idx_namespaces_name]
      ON [namespaces] ([name]);
  CREATE TABLE [commits] (
     [id] INTEGER PRIMARY KEY,
     [namespace] INTEGER REFERENCES [namespaces]([id]),
     [hash] TEXT,
     [commit_at] TEXT
  );
  CREATE UNIQUE INDEX [idx_commits_namespace_hash]
      ON [commits] ([namespace], [hash]);
  CREATE TABLE [item] (
     [_id] INTEGER PRIMARY KEY,
     [_item_id] TEXT
  , [IncidentID] TEXT, [Location] TEXT, [Type] TEXT, [_commit] INTEGER);
  CREATE UNIQUE INDEX [idx_item__item_id]
      ON [item] ([_item_id]);
  CREATE TABLE [item_version] (
     [_id] INTEGER PRIMARY KEY,
     [_item] INTEGER REFERENCES [item]([_id]),
     [_version] INTEGER,
     [_commit] INTEGER REFERENCES [commits]([id]),
     [IncidentID] TEXT,
     [Location] TEXT,
     [Type] TEXT,
     [_item_full_hash] TEXT
  );
  CREATE TABLE [columns] (
     [id] INTEGER PRIMARY KEY,
     [namespace] INTEGER REFERENCES [namespaces]([id]),
     [name] TEXT
  );
  CREATE UNIQUE INDEX [idx_columns_namespace_name]
      ON [columns] ([namespace], [name]);
  CREATE TABLE [item_changed] (
     [item_version] INTEGER REFERENCES [item_version]([_id]),
     [column] INTEGER REFERENCES [columns]([id]),
     PRIMARY KEY ([item_version], [column])
  );
  CREATE VIEW item_version_detail AS select
    commits.commit_at as _commit_at,
    commits.hash as _commit_hash,
    item_version.*,
    (
      select json_group_array(name) from columns
      where id in (
        select column from item_changed
        where item_version = item_version._id
      )
  ) as _changed_columns
  from item_version
    join commits on commits.id = item_version._commit;
  CREATE INDEX [idx_item_version__item]
      ON [item_version] ([_item]);"><pre>CREATE TABLE [namespaces] (
     [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
     [name] <span class="pl-k">TEXT</span>
  );
  CREATE UNIQUE INDEX [idx_namespaces_name]
      <span class="pl-k">ON</span> [namespaces] ([name]);
  CREATE TABLE [commits] (
     [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
     [namespace] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [namespaces]([id]),
     [hash] <span class="pl-k">TEXT</span>,
     [commit_at] <span class="pl-k">TEXT</span>
  );
  CREATE UNIQUE INDEX [idx_commits_namespace_hash]
      <span class="pl-k">ON</span> [commits] ([namespace], [hash]);
  CREATE TABLE [item] (
     [_id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
     [_item_id] <span class="pl-k">TEXT</span>
  , [IncidentID] <span class="pl-k">TEXT</span>, [Location] <span class="pl-k">TEXT</span>, [Type] <span class="pl-k">TEXT</span>, [_commit] <span class="pl-k">INTEGER</span>);
  CREATE UNIQUE INDEX [idx_item__item_id]
      <span class="pl-k">ON</span> [item] ([_item_id]);
  CREATE TABLE [item_version] (
     [_id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
     [_item] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [item]([_id]),
     [_version] <span class="pl-k">INTEGER</span>,
     [_commit] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [commits]([id]),
     [IncidentID] <span class="pl-k">TEXT</span>,
     [Location] <span class="pl-k">TEXT</span>,
     [Type] <span class="pl-k">TEXT</span>,
     [_item_full_hash] <span class="pl-k">TEXT</span>
  );
  CREATE TABLE [columns] (
     [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
     [namespace] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [namespaces]([id]),
     [name] <span class="pl-k">TEXT</span>
  );
  CREATE UNIQUE INDEX [idx_columns_namespace_name]
      <span class="pl-k">ON</span> [columns] ([namespace], [name]);
  CREATE TABLE [item_changed] (
     [item_version] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [item_version]([_id]),
     [column] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [columns]([id]),
     <span class="pl-k">PRIMARY KEY</span> ([item_version], [column])
  );
  <span class="pl-k">CREATE</span> <span class="pl-k">VIEW</span> <span class="pl-en">item_version_detail</span> <span class="pl-k">AS</span> <span class="pl-k">select</span>
    <span class="pl-c1">commits</span>.<span class="pl-c1">commit_at</span> <span class="pl-k">as</span> _commit_at,
    <span class="pl-c1">commits</span>.<span class="pl-c1">hash</span> <span class="pl-k">as</span> _commit_hash,
    item_version.<span class="pl-k">*</span>,
    (
      <span class="pl-k">select</span> json_group_array(name) <span class="pl-k">from</span> columns
      <span class="pl-k">where</span> id <span class="pl-k">in</span> (
        <span class="pl-k">select</span> column <span class="pl-k">from</span> item_changed
        <span class="pl-k">where</span> item_version <span class="pl-k">=</span> <span class="pl-c1">item_version</span>.<span class="pl-c1">_id</span>
      )
  ) <span class="pl-k">as</span> _changed_columns
  <span class="pl-k">from</span> item_version
    <span class="pl-k">join</span> commits <span class="pl-k">on</span> <span class="pl-c1">commits</span>.<span class="pl-c1">id</span> <span class="pl-k">=</span> <span class="pl-c1">item_version</span>.<span class="pl-c1">_commit</span>;
  CREATE INDEX [idx_item_version__item]
      <span class="pl-k">ON</span> [item_version] ([_item]);</pre></div>

  <h4 dir="auto"><a id="user-content-item-table" class="anchor" aria-hidden="true" href="#user-content-item-table"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>item table</h4>
  <p dir="auto">The <code>item</code> table will contain the most recent version of each row, de-duplicated by ID, plus the following additional columns:</p>
  <ul dir="auto">
  <li><code>_id</code> - a numeric integer primary key, used as a foreign key from the <code>item_version</code> table.</li>
  <li><code>_item_id</code> - a hash of the values of the columns specified using the <code>--id</code> option to the command. This is used for de-duplication when processing new versions.</li>
  <li><code>_commit</code> - a foreign key to the <code>commit</code> table, representing the most recent commit to modify this item.</li>
  </ul>
  <h4 dir="auto"><a id="user-content-item_version-table" class="anchor" aria-hidden="true" href="#user-content-item_version-table"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>item_version table</h4>
  <p dir="auto">The <code>item_version</code> table will contain a row for each captured differing version of that item, plus the following columns:</p>
  <ul dir="auto">
  <li><code>_id</code> - a numeric ID for the item version record.</li>
  <li><code>_item</code> - a foreign key to the <code>item</code> table.</li>
  <li><code>_version</code> - the numeric version number, starting at 1 and incrementing for each captured version.</li>
  <li><code>_commit</code> - a foreign key to the <code>commit</code> table.</li>
  <li><code>_item_full_hash</code> - a hash of this version of the item. This is used internally by the tool to identify items that have changed between commits.</li>
  </ul>
  <p dir="auto">The other columns in this table represent columns in the original data that have changed since the previous version. If the value has not changed, it will be represented by a <code>null</code>.</p>
  <p dir="auto">If a value was previously set but has been changed back to <code>null</code> it will still be represented as <code>null</code> in the <code>item_version</code> row. You can identify these using the <code>item_changed</code> many-to-many table described below.</p>
  <p dir="auto">You can use the <code>--full-versions</code> option to store full copies of the item at each version, rather than just storing the columns that have changed.</p>
  <h4 dir="auto"><a id="user-content-item_version_detail-view" class="anchor" aria-hidden="true" href="#user-content-item_version_detail-view"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>item_version_detail view</h4>
  <p dir="auto">This SQL view joins <code>item_version</code> against <code>commits</code> to add three further columns: <code>_commit_at</code> with the date of the commit, and <code>_commit_hash</code> with the Git commit hash.</p>
  <h4 dir="auto"><a id="user-content-item_changed" class="anchor" aria-hidden="true" href="#user-content-item_changed"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>item_changed</h4>
  <p dir="auto">This many-to-many table indicates exactly which columns were changed in an <code>item_version</code>.</p>
  <ul dir="auto">
  <li><code>item_version</code> is a foreign key to a row in the <code>item_version</code> table.</li>
  <li><code>column</code> is a foreign key to a row in the <code>columns</code> table.</li>
  </ul>
  <p dir="auto">This table with have the largest number of rows, which is why it stores just two integers in order to save space.</p>
  <h4 dir="auto"><a id="user-content-columns" class="anchor" aria-hidden="true" href="#user-content-columns"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>columns</h4>
  <p dir="auto">The <code>columns</code> table stores column names. It is referenced by <code>item_changed</code>.</p>
  <ul dir="auto">
  <li><code>id</code> - an integer ID.</li>
  <li><code>name</code> - the name of the column.</li>
  <li><code>namespace</code> - a foreign key to <code>namespaces</code>, for if multiple file histories are sharing the same database.</li>
  </ul>
  <h4 dir="auto"><a id="user-content-reserved-column-names" class="anchor" aria-hidden="true" href="#user-content-reserved-column-names"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Reserved column names</h4>

  <p dir="auto">Note that <code>_id</code>, <code>_item_full_hash</code>, <code>_item</code>, <code>_item_id</code>, <code>_version</code>, <code>_commit</code>, <code>_item_id</code>, <code>_commit_at</code>, <code>_commit_hash</code>, <code>_changed_columns</code>, <code>rowid</code> are considered reserved column names for the purposes of this tool.</p>

  <p dir="auto">If your data contains any of these they will be renamed to add a trailing underscore, for example <code>_id_</code>, <code>_item_</code>, <code>_version_</code>, to avoid clashing with the reserved columns.</p>
  <p dir="auto">If you have a column with a name such as <code>_commit_</code> it will be renamed too, adding an additional trailing underscore, so <code>_commit_</code> becomes <code>_commit__</code> and <code>_commit__</code> becomes <code>_commit___</code>.</p>
  <h3 dir="auto"><a id="user-content-additional-options" class="anchor" aria-hidden="true" href="#user-content-additional-options"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Additional options</h3>
  <ul dir="auto">
  <li><code>--repo DIRECTORY</code> - the path to the Git repository, if it is not the current working directory.</li>
  <li><code>--branch TEXT</code> - the Git branch to analyze - defaults to <code>main</code>.</li>
  <li><code>--id TEXT</code> - as described above: pass one or more columns that uniquely identify a record, so that changes to that record can be calculated over time.</li>
  <li><code>--full-versions</code> - instead of recording just the columns that have changed in the <code>item_version</code> table record a full copy of each version of theh item.</li>
  <li><code>--ignore TEXT</code> - one or more columns to ignore - they will not be included in the resulting database.</li>
  <li><code>--csv</code> - treat the data is CSV or TSV rather than JSON, and attempt to guess the correct dialect</li>
  <li><code>--dialect</code> - use a spcific CSV dialect. Options are <code>excel</code>, <code>excel-tab</code> and <code>unix</code> - see <a href="https://docs.python.org/3/library/csv.html#csv.excel" rel="nofollow">the Python CSV documentation</a> for details.</li>
  <li><code>--skip TEXT</code> - one or more full Git commit hashes that should be skipped. You can use this if some of the data in your revision history is corrupted in a way that prevents this tool from working.</li>
  <li><code>--start-at TEXT</code> - skip commits prior to the specified commit hash.</li>
  <li><code>--start-after TEXT</code> - skip commits up to and including the specified commit hash, then start processing from the following commit.</li>
  <li><code>--convert TEXT</code> - custom Python code for a conversion, described below.</li>
  <li><code>--import TEXT</code> - additional Python modules to import for <code>--convert</code>.</li>
  <li><code>--ignore-duplicate-ids</code> - if a single version of a file has the same ID in it more than once, the tool will exit with an error. Use this option to ignore this and instead pick just the first of the two duplicates.</li>
  <li><code>--namespace TEXT</code> - use this if you wish to include the history of multiple different files in the same database. The default is <code>item</code> but you can set it to something else, which will produce tables with names like <code>yournamespace</code> and <code>yournamespace_version</code>.</li>
  <li><code>--wal</code> - Enable WAL mode on the created database file. Use this if you plan to run queries against the database while <code>git-history</code> is creating it.</li>
  <li><code>--silent</code> - don't show the progress bar.</li>
  </ul>
  <h3 dir="auto"><a id="user-content-csv-and-tsv-data" class="anchor" aria-hidden="true" href="#user-content-csv-and-tsv-data"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>CSV and TSV data</h3>
  <p dir="auto">If the data in your repository is a CSV or TSV file you can process it by adding the <code>--csv</code> option. This will attempt to detect which delimiter is used by the file, so the same option works for both comma- and tab-separated values.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="git-history file trees.db trees.csv --id TreeID"><pre class="notranslate"><code>git-history file trees.db trees.csv --id TreeID
  </code></pre></div>
  <p dir="auto">You can also specify the CSV dialect using the <code>--dialect</code> option.</p>
  <h3 dir="auto"><a id="user-content-custom-conversions-using---convert" class="anchor" aria-hidden="true" href="#user-content-custom-conversions-using---convert"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Custom conversions using --convert</h3>
  <p dir="auto">If your data is not already either CSV/TSV or a flat JSON array, you can reshape it using the <code>--convert</code> option.</p>
  <p dir="auto">The format needed by this tool is an array of dictionaries, as demonstrated by the <code>incidents.json</code> example above.</p>
  <p dir="auto">If your data does not fit this shape, you can provide a snippet of Python code to converts the on-disk content of each stored file into a Python list of dictionaries.</p>
  <p dir="auto">For example, if your stored files each look like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      &quot;incidents&quot;: [
          {
              &quot;id&quot;: &quot;552&quot;,
              &quot;name&quot;: &quot;Hawthorne Fire&quot;,
              &quot;engines&quot;: 3
          },
          {
              &quot;id&quot;: &quot;556&quot;,
              &quot;name&quot;: &quot;Merlin Fire&quot;,
              &quot;engines&quot;: 1
          }
      ]
  }"><pre>{
      <span class="pl-ent">"incidents"</span>: [
          {
              <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>552<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Hawthorne Fire<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"engines"</span>: <span class="pl-c1">3</span>
          },
          {
              <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>556<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Merlin Fire<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"engines"</span>: <span class="pl-c1">1</span>
          }
      ]
  }</pre></div>
  <p dir="auto">You could use the following Python snippet to convert them to the required format:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="json.loads(content)[&quot;incidents&quot;]"><pre><span class="pl-s1">json</span>.<span class="pl-en">loads</span>(<span class="pl-s1">content</span>)[<span class="pl-s">"incidents"</span>]</pre></div>
  <p dir="auto">(The <code>json</code> module is exposed to your custom function by default.)</p>
  <p dir="auto">You would then run the tool like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="git-history file database.db incidents.json \
    --id id \
    --convert 'json.loads(content)[&quot;incidents&quot;]'"><pre class="notranslate"><code>git-history file database.db incidents.json \
    --id id \
    --convert 'json.loads(content)["incidents"]'
  </code></pre></div>
  <p dir="auto">The <code>content</code> variable is always a <code>bytes</code> object representing the content of the file at a specific moment in the repository's history.</p>
  <p dir="auto">You can import additional modules using <code>--import</code>. This example shows how you could read a CSV file that uses <code>;</code> as the delimiter:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="git-history file trees.db ../sf-tree-history/Street_Tree_List.csv \
    --repo ../sf-tree-history \
    --import csv \
    --import io \
    --convert '
      fp = io.StringIO(content.decode(&quot;utf-8&quot;))
      return list(csv.DictReader(fp, delimiter=&quot;;&quot;))
      ' \
    --id TreeID"><pre class="notranslate"><code>git-history file trees.db ../sf-tree-history/Street_Tree_List.csv \
    --repo ../sf-tree-history \
    --import csv \
    --import io \
    --convert '
      fp = io.StringIO(content.decode("utf-8"))
      return list(csv.DictReader(fp, delimiter=";"))
      ' \
    --id TreeID
  </code></pre></div>
  <p dir="auto">You can import nested modules such as <a href="https://docs.python.org/3/library/xml.etree.elementtree.html" rel="nofollow">ElementTree</a>  using <code>--import xml.etree.ElementTree</code>, then refer to them in your function body as <code>xml.etree.ElementTree</code>. For example, if your tracked data was in an <code>items.xml</code> file that looked like this:</p>
  <div class="highlight highlight-text-xml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&lt;items&gt;
    &lt;item id=&quot;1&quot; name=&quot;One&quot; /&gt;
    &lt;item id=&quot;2&quot; name=&quot;Two&quot; /&gt;
    &lt;item id=&quot;3&quot; name=&quot;Three&quot; /&gt;
  &lt;/item&gt;"><pre>&lt;<span class="pl-ent">items</span>&gt;
    &lt;<span class="pl-ent">item</span> <span class="pl-e">id</span>=<span class="pl-s"><span class="pl-pds">"</span>1<span class="pl-pds">"</span></span> <span class="pl-e">name</span>=<span class="pl-s"><span class="pl-pds">"</span>One<span class="pl-pds">"</span></span> /&gt;
    &lt;<span class="pl-ent">item</span> <span class="pl-e">id</span>=<span class="pl-s"><span class="pl-pds">"</span>2<span class="pl-pds">"</span></span> <span class="pl-e">name</span>=<span class="pl-s"><span class="pl-pds">"</span>Two<span class="pl-pds">"</span></span> /&gt;
    &lt;<span class="pl-ent">item</span> <span class="pl-e">id</span>=<span class="pl-s"><span class="pl-pds">"</span>3<span class="pl-pds">"</span></span> <span class="pl-e">name</span>=<span class="pl-s"><span class="pl-pds">"</span>Three<span class="pl-pds">"</span></span> /&gt;
  &lt;/<span class="pl-ent">item</span>&gt;</pre></div>
  <p dir="auto">You could load it using the following <code>--convert</code> script:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="git-history file items.xml --convert '
  tree = xml.etree.ElementTree.fromstring(content)
  return [el.attrib for el in tree.iter(&quot;item&quot;)]
  ' --import xml.etree.ElementTree --id id"><pre class="notranslate"><code>git-history file items.xml --convert '
  tree = xml.etree.ElementTree.fromstring(content)
  return [el.attrib for el in tree.iter("item")]
  ' --import xml.etree.ElementTree --id id
  </code></pre></div>
  <p dir="auto">If your Python code spans more than one line it needs to include a <code>return</code> statement.</p>
  <p dir="auto">You can also use Python generators in your <code>--convert</code> code, for example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="git-history file stats.db package-stats/stats.json \
      --repo package-stats \
      --convert '
      data = json.loads(content)
      for key, counts in data.items():
          for date, count in counts.items():
              yield {
                  &quot;package&quot;: key,
                  &quot;date&quot;: date,
                  &quot;count&quot;: count
              }
      ' --id package --id date"><pre class="notranslate"><code>git-history file stats.db package-stats/stats.json \
      --repo package-stats \
      --convert '
      data = json.loads(content)
      for key, counts in data.items():
          for date, count in counts.items():
              yield {
                  "package": key,
                  "date": date,
                  "count": count
              }
      ' --id package --id date
  </code></pre></div>
  <p dir="auto">This conversion function expects data that looks like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
      &quot;airtable-export&quot;: {
          &quot;2021-05-18&quot;: 66,
          &quot;2021-05-19&quot;: 60,
          &quot;2021-05-20&quot;: 87
      }
  }"><pre>{
      <span class="pl-ent">"airtable-export"</span>: {
          <span class="pl-ent">"2021-05-18"</span>: <span class="pl-c1">66</span>,
          <span class="pl-ent">"2021-05-19"</span>: <span class="pl-c1">60</span>,
          <span class="pl-ent">"2021-05-20"</span>: <span class="pl-c1">87</span>
      }
  }</pre></div>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd git-history
  python -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd git-history
  python -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  <p dir="auto">To update the schema examples in this README file:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cog -r README.md"><pre class="notranslate"><code>cog -r README.md
  </code></pre></div>
  </article></div>

template_repository:
  {"id": 401177473, "node_id": "MDEwOlJlcG9zaXRvcnk0MDExNzc0NzM=", "name": "click-app-template-repository", "full_name": "simonw/click-app-template-repository", "private": false, "owner": {"login": "simonw", "id": 9599, "node_id": "MDQ6VXNlcjk1OTk=", "avatar_url": "https://avatars.githubusercontent.com/u/9599?v=4", "gravatar_id": "", "url": "https://api.github.com/users/simonw", "html_url": "https://github.com/simonw", "followers_url": "https://api.github.com/users/simonw/followers", "following_url": "https://api.github.com/users/simonw/following{/other_user}", "gists_url": "https://api.github.com/users/simonw/gists{/gist_id}", "starred_url": "https://api.github.com/users/simonw/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/simonw/subscriptions", "organizations_url": "https://api.github.com/users/simonw/orgs", "repos_url": "https://api.github.com/users/simonw/repos", "events_url": "https://api.github.com/users/simonw/events{/privacy}", "received_events_url": "https://api.github.com/users/simonw/received_events", "type": "User", "site_admin": false}, "html_url": "https://github.com/simonw/click-app-template-repository", "description": "GitHub template repository for creating new Python Click CLI tools, using the simonw/click-app cookiecutter template", "fork": false, "url": "https://api.github.com/repos/simonw/click-app-template-repository", "forks_url": "https://api.github.com/repos/simonw/click-app-template-repository/forks", "keys_url": "https://api.github.com/repos/simonw/click-app-template-repository/keys{/key_id}", "collaborators_url": "https://api.github.com/repos/simonw/click-app-template-repository/collaborators{/collaborator}", "teams_url": "https://api.github.com/repos/simonw/click-app-template-repository/teams", "hooks_url": "https://api.github.com/repos/simonw/click-app-template-repository/hooks", "issue_events_url": "https://api.github.com/repos/simonw/click-app-template-repository/issues/events{/number}", "events_url": "https://api.github.com/repos/simonw/click-app-template-repository/events", "assignees_url": "https://api.github.com/repos/simonw/click-app-template-repository/assignees{/user}", "branches_url": "https://api.github.com/repos/simonw/click-app-template-repository/branches{/branch}", "tags_url": "https://api.github.com/repos/simonw/click-app-template-repository/tags", "blobs_url": "https://api.github.com/repos/simonw/click-app-template-repository/git/blobs{/sha}", "git_tags_url": "https://api.github.com/repos/simonw/click-app-template-repository/git/tags{/sha}", "git_refs_url": "https://api.github.com/repos/simonw/click-app-template-repository/git/refs{/sha}", "trees_url": "https://api.github.com/repos/simonw/click-app-template-repository/git/trees{/sha}", "statuses_url": "https://api.github.com/repos/simonw/click-app-template-repository/statuses/{sha}", "languages_url": "https://api.github.com/repos/simonw/click-app-template-repository/languages", "stargazers_url": "https://api.github.com/repos/simonw/click-app-template-repository/stargazers", "contributors_url": "https://api.github.com/repos/simonw/click-app-template-repository/contributors", "subscribers_url": "https://api.github.com/repos/simonw/click-app-template-repository/subscribers", "subscription_url": "https://api.github.com/repos/simonw/click-app-template-repository/subscription", "commits_url": "https://api.github.com/repos/simonw/click-app-template-repository/commits{/sha}", "git_commits_url": "https://api.github.com/repos/simonw/click-app-template-repository/git/commits{/sha}", "comments_url": "https://api.github.com/repos/simonw/click-app-template-repository/comments{/number}", "issue_comment_url": "https://api.github.com/repos/simonw/click-app-template-repository/issues/comments{/number}", "contents_url": "https://api.github.com/repos/simonw/click-app-template-repository/contents/{+path}", "compare_url": "https://api.github.com/repos/simonw/click-app-template-repository/compare/{base}...{head}", "merges_url": "https://api.github.com/repos/simonw/click-app-template-repository/merges", "archive_url": "https://api.github.com/repos/simonw/click-app-template-repository/{archive_format}{/ref}", "downloads_url": "https://api.github.com/repos/simonw/click-app-template-repository/downloads", "issues_url": "https://api.github.com/repos/simonw/click-app-template-repository/issues{/number}", "pulls_url": "https://api.github.com/repos/simonw/click-app-template-repository/pulls{/number}", "milestones_url": "https://api.github.com/repos/simonw/click-app-template-repository/milestones{/number}", "notifications_url": "https://api.github.com/repos/simonw/click-app-template-repository/notifications{?since,all,participating}", "labels_url": "https://api.github.com/repos/simonw/click-app-template-repository/labels{/name}", "releases_url": "https://api.github.com/repos/simonw/click-app-template-repository/releases{/id}", "deployments_url": "https://api.github.com/repos/simonw/click-app-template-repository/deployments", "created_at": "2021-08-30T01:03:34Z", "updated_at": "2022-07-17T02:01:39Z", "pushed_at": "2022-03-16T23:35:31Z", "git_url": "git://github.com/simonw/click-app-template-repository.git", "ssh_url": "git@github.com:simonw/click-app-template-repository.git", "clone_url": "https://github.com/simonw/click-app-template-repository.git", "svn_url": "https://github.com/simonw/click-app-template-repository", "homepage": "", "size": 12, "stargazers_count": 8, "watchers_count": 8, "language": null, "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": true, "has_pages": false, "forks_count": 0, "mirror_url": null, "archived": false, "disabled": false, "open_issues_count": 0, "license": null, "allow_forking": true, "is_template": true, "web_commit_signoff_required": false, "topics": [], "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 8, "default_branch": "main", "permissions": {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}, "temp_clone_token": ""}


node_id: R_kgDOGaS1TA
name: datasette-redirect-to-https
full_name: simonw/datasette-redirect-to-https
html_url: https://github.com/simonw/datasette-redirect-to-https
description: Datasette plugin that redirects all non-https requests to https
created_at: 2021-11-20T22:43:33Z
updated_at: 2022-04-24T03:48:01Z
pushed_at: 2022-07-07T17:38:32Z
language: Python
topics: ["asgi", "datasette", "datasette-io", "datasette-plugin"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-redirect-to-https

  [![PyPI](https://img.shields.io/pypi/v/datasette-redirect-to-https.svg)](https://pypi.org/project/datasette-redirect-to-https/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-redirect-to-https?include_prereleases&label=changelog)](https://github.com/simonw/datasette-redirect-to-https/releases)
  [![Tests](https://github.com/simonw/datasette-redirect-to-https/workflows/Test/badge.svg)](https://github.com/simonw/datasette-redirect-to-https/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-redirect-to-https/blob/main/LICENSE)

  Datasette plugin that redirects all non-https requests to https

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-redirect-to-https

  ## Usage

  Once installed, incoming GET requests to the `http://` protocol will be 301 redirected to the `https://` equivalent page.

  HTTP verbs other than GET will get a 405 Method Not Allowed HTTP error.

  ## Configuration

  Some hosting providers handle HTTPS for you, passing requests back to your application server over HTTP.

  For this plugin to work correctly, it needs to detect that the original incoming request came in over HTTP.

  Hosting providers like this often set an additional HTTP header such as `x-forwarded-proto: http` identifying the original protocol.

  You can configure `datasette-redirect-to-https` to respect this header using the following plugin configuration in `metadata.json`:

  ```json
  {
    "plugins": {
      "datasette-redirect-to-https": {
        "if_headers": {
          "x-forwarded-proto": "http"
        }
      }
    }
  }
  ```
  The above example will redirect to `https://` if the incoming request has a `x-forwarded-proto: http` request header.

  If multiple `if_headers` are listed, the redirect will occur if any of them match.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-redirect-to-https
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-redirect-to-https" class="anchor" aria-hidden="true" href="#user-content-datasette-redirect-to-https"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-redirect-to-https</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-redirect-to-https/" rel="nofollow"><img src="https://camo.githubusercontent.com/96eb43748c89385b2656ffbe4a6012ccf73035078c0ca1a323cf5f8f3dea31ad/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d72656469726563742d746f2d68747470732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-redirect-to-https.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-redirect-to-https/releases"><img src="https://camo.githubusercontent.com/b0c8deb890a7e5c1deb8c6bd4da6daf75c7ec8211983b8d5c91f42c9de337d00/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d72656469726563742d746f2d68747470733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-redirect-to-https?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-redirect-to-https/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-redirect-to-https/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-redirect-to-https/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Datasette plugin that redirects all non-https requests to https</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-redirect-to-https"><pre class="notranslate"><code>$ datasette install datasette-redirect-to-https
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Once installed, incoming GET requests to the <code>http://</code> protocol will be 301 redirected to the <code>https://</code> equivalent page.</p>
  <p dir="auto">HTTP verbs other than GET will get a 405 Method Not Allowed HTTP error.</p>
  <h2 dir="auto"><a id="user-content-configuration" class="anchor" aria-hidden="true" href="#user-content-configuration"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Configuration</h2>
  <p dir="auto">Some hosting providers handle HTTPS for you, passing requests back to your application server over HTTP.</p>
  <p dir="auto">For this plugin to work correctly, it needs to detect that the original incoming request came in over HTTP.</p>
  <p dir="auto">Hosting providers like this often set an additional HTTP header such as <code>x-forwarded-proto: http</code> identifying the original protocol.</p>
  <p dir="auto">You can configure <code>datasette-redirect-to-https</code> to respect this header using the following plugin configuration in <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
    &quot;plugins&quot;: {
      &quot;datasette-redirect-to-https&quot;: {
        &quot;if_headers&quot;: {
          &quot;x-forwarded-proto&quot;: &quot;http&quot;
        }
      }
    }
  }"><pre>{
    <span class="pl-ent">"plugins"</span>: {
      <span class="pl-ent">"datasette-redirect-to-https"</span>: {
        <span class="pl-ent">"if_headers"</span>: {
          <span class="pl-ent">"x-forwarded-proto"</span>: <span class="pl-s"><span class="pl-pds">"</span>http<span class="pl-pds">"</span></span>
        }
      }
    }
  }</pre></div>
  <p dir="auto">The above example will redirect to <code>https://</code> if the incoming request has a <code>x-forwarded-proto: http</code> request header.</p>
  <p dir="auto">If multiple <code>if_headers</code> are listed, the redirect will occur if any of them match.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-redirect-to-https
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-redirect-to-https
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>

template_repository:
  {"id": 400878073, "node_id": "MDEwOlJlcG9zaXRvcnk0MDA4NzgwNzM=", "name": "datasette-plugin-template-repository", "full_name": "simonw/datasette-plugin-template-repository", "private": false, "owner": {"login": "simonw", "id": 9599, "node_id": "MDQ6VXNlcjk1OTk=", "avatar_url": "https://avatars.githubusercontent.com/u/9599?v=4", "gravatar_id": "", "url": "https://api.github.com/users/simonw", "html_url": "https://github.com/simonw", "followers_url": "https://api.github.com/users/simonw/followers", "following_url": "https://api.github.com/users/simonw/following{/other_user}", "gists_url": "https://api.github.com/users/simonw/gists{/gist_id}", "starred_url": "https://api.github.com/users/simonw/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/simonw/subscriptions", "organizations_url": "https://api.github.com/users/simonw/orgs", "repos_url": "https://api.github.com/users/simonw/repos", "events_url": "https://api.github.com/users/simonw/events{/privacy}", "received_events_url": "https://api.github.com/users/simonw/received_events", "type": "User", "site_admin": false}, "html_url": "https://github.com/simonw/datasette-plugin-template-repository", "description": "GitHub template repository for creating new Datasette plugins, using the simonw/datasette-plugin cookiecutter template", "fork": false, "url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository", "forks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/forks", "keys_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/keys{/key_id}", "collaborators_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/collaborators{/collaborator}", "teams_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/teams", "hooks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/hooks", "issue_events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/events{/number}", "events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/events", "assignees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/assignees{/user}", "branches_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/branches{/branch}", "tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/tags", "blobs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/blobs{/sha}", "git_tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/tags{/sha}", "git_refs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/refs{/sha}", "trees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/trees{/sha}", "statuses_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/statuses/{sha}", "languages_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/languages", "stargazers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/stargazers", "contributors_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contributors", "subscribers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscribers", "subscription_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscription", "commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/commits{/sha}", "git_commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/commits{/sha}", "comments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/comments{/number}", "issue_comment_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/comments{/number}", "contents_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contents/{+path}", "compare_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/compare/{base}...{head}", "merges_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/merges", "archive_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/{archive_format}{/ref}", "downloads_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/downloads", "issues_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues{/number}", "pulls_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/pulls{/number}", "milestones_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/milestones{/number}", "notifications_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/notifications{?since,all,participating}", "labels_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/labels{/name}", "releases_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/releases{/id}", "deployments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/deployments", "created_at": "2021-08-28T19:50:28Z", "updated_at": "2022-06-10T13:28:46Z", "pushed_at": "2022-03-16T23:42:16Z", "git_url": "git://github.com/simonw/datasette-plugin-template-repository.git", "ssh_url": "git@github.com:simonw/datasette-plugin-template-repository.git", "clone_url": "https://github.com/simonw/datasette-plugin-template-repository.git", "svn_url": "https://github.com/simonw/datasette-plugin-template-repository", "homepage": "", "size": 9, "stargazers_count": 15, "watchers_count": 15, "language": null, "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": true, "has_pages": false, "forks_count": 0, "mirror_url": null, "archived": false, "disabled": false, "open_issues_count": 0, "license": null, "allow_forking": true, "is_template": true, "web_commit_signoff_required": false, "topics": [], "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 15, "default_branch": "main", "permissions": {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}, "temp_clone_token": ""}


node_id: R_kgDOGeMHbg
name: datasette-hovercards
full_name: simonw/datasette-hovercards
html_url: https://github.com/simonw/datasette-hovercards
description: Add preview hovercards to links in Datasette
created_at: 2021-12-02T17:11:59Z
updated_at: 2022-02-08T07:22:21Z
pushed_at: 2021-12-02T19:57:32Z
language: JavaScript
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-hovercards

  [![PyPI](https://img.shields.io/pypi/v/datasette-hovercards.svg)](https://pypi.org/project/datasette-hovercards/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-hovercards?include_prereleases&label=changelog)](https://github.com/simonw/datasette-hovercards/releases)
  [![Tests](https://github.com/simonw/datasette-hovercards/workflows/Test/badge.svg)](https://github.com/simonw/datasette-hovercards/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-hovercards/blob/main/LICENSE)

  Add preview hovercards to links in Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-hovercards

  ## Usage

  Once installed, hovering over a link to a row within the Datasette interface - for example a foreign key reference on the table page - should show a hovercard with a preview of that row.

  For a live demo, hover over values in the `user`, `milestone` or `repo` columns on this table page:

  https://latest-with-plugins.datasette.io/github/issues

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-hovercards
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-hovercards" class="anchor" aria-hidden="true" href="#user-content-datasette-hovercards"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-hovercards</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-hovercards/" rel="nofollow"><img src="https://camo.githubusercontent.com/91c843195be9035808f329272cf35cd055592f8ce986e57052b0abdc7b0bfd6b/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d686f76657263617264732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-hovercards.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-hovercards/releases"><img src="https://camo.githubusercontent.com/7c7341054b262bd168e171bebe57dd73f854cc74504172d88610cd890d7827e9/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d686f76657263617264733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-hovercards?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-hovercards/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-hovercards/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-hovercards/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Add preview hovercards to links in Datasette</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-hovercards"><pre><code>$ datasette install datasette-hovercards
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Once installed, hovering over a link to a row within the Datasette interface - for example a foreign key reference on the table page - should show a hovercard with a preview of that row.</p>
  <p dir="auto">For a live demo, hover over values in the <code>user</code>, <code>milestone</code> or <code>repo</code> columns on this table page:</p>
  <p dir="auto"><a href="https://latest-with-plugins.datasette.io/github/issues" rel="nofollow">https://latest-with-plugins.datasette.io/github/issues</a></p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-hovercards
  python3 -mvenv venv
  source venv/bin/activate"><pre><code>cd datasette-hovercards
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOGhtmrg
name: datasette-pretty-traces
full_name: simonw/datasette-pretty-traces
html_url: https://github.com/simonw/datasette-pretty-traces
description: Prettier formatting for ?_trace=1 traces
created_at: 2021-12-13T19:43:28Z
updated_at: 2021-12-19T20:40:10Z
pushed_at: 2022-01-14T02:08:51Z
language: JavaScript
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-pretty-traces

  [![PyPI](https://img.shields.io/pypi/v/datasette-pretty-traces.svg)](https://pypi.org/project/datasette-pretty-traces/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-pretty-traces?include_prereleases&label=changelog)](https://github.com/simonw/datasette-pretty-traces/releases)
  [![Tests](https://github.com/simonw/datasette-pretty-traces/workflows/Test/badge.svg)](https://github.com/simonw/datasette-pretty-traces/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-pretty-traces/blob/main/LICENSE)

  Prettier formatting for `?_trace=1` traces

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-pretty-traces

  ## Usage

  Once installed, run Datasette using `--setting trace_debug 1`:

      datasette fixtures.db --setting trace_debug 1

  Then navigate to any page and add `?_trace=` to the URL:

      http://localhost:8001/?_trace=1

  The plugin will scroll you down the page to the visualized trace information.

  ## Demo

  You can try out the demo here:

  - [/?_trace=1](https://latest-with-plugins.datasette.io/?_trace=1) tracing the homepage
  - [/github/commits?_trace=1](https://latest-with-plugins.datasette.io/github/commits?_trace=1) tracing a table page

  ## Screenshot

  ![Screenshot showing the visualization produced by the plugin](https://user-images.githubusercontent.com/9599/145883732-a53accdd-5feb-4629-94cd-f73407c7943d.png)

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-pretty-traces
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-pretty-traces" class="anchor" aria-hidden="true" href="#user-content-datasette-pretty-traces"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-pretty-traces</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-pretty-traces/" rel="nofollow"><img src="https://camo.githubusercontent.com/ed86d276f9a69a313d94c59c0a68e29ebd828d11646c2cfd55d776d8738ec333/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d7072657474792d7472616365732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-pretty-traces.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-pretty-traces/releases"><img src="https://camo.githubusercontent.com/26aa360a6a2df25aefecf0566189c1b970e68493853d18d55045684e328eb303/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d7072657474792d7472616365733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-pretty-traces?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-pretty-traces/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-pretty-traces/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-pretty-traces/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Prettier formatting for <code>?_trace=1</code> traces</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-pretty-traces"><pre><code>$ datasette install datasette-pretty-traces
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Once installed, run Datasette using <code>--setting trace_debug 1</code>:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette fixtures.db --setting trace_debug 1"><pre><code>datasette fixtures.db --setting trace_debug 1
  </code></pre></div>
  <p dir="auto">Then navigate to any page and add <code>?_trace=</code> to the URL:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="http://localhost:8001/?_trace=1"><pre><code>http://localhost:8001/?_trace=1
  </code></pre></div>
  <p dir="auto">The plugin will scroll you down the page to the visualized trace information.</p>
  <h2 dir="auto"><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p dir="auto">You can try out the demo here:</p>
  <ul dir="auto">
  <li><a href="https://latest-with-plugins.datasette.io/?_trace=1" rel="nofollow">/?_trace=1</a> tracing the homepage</li>
  <li><a href="https://latest-with-plugins.datasette.io/github/commits?_trace=1" rel="nofollow">/github/commits?_trace=1</a> tracing a table page</li>
  </ul>
  <h2 dir="auto"><a id="user-content-screenshot" class="anchor" aria-hidden="true" href="#user-content-screenshot"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Screenshot</h2>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer" href="https://user-images.githubusercontent.com/9599/145883732-a53accdd-5feb-4629-94cd-f73407c7943d.png"><img src="https://user-images.githubusercontent.com/9599/145883732-a53accdd-5feb-4629-94cd-f73407c7943d.png" alt="Screenshot showing the visualization produced by the plugin" style="max-width: 100%;"></a></p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-pretty-traces
  python3 -mvenv venv
  source venv/bin/activate"><pre><code>cd datasette-pretty-traces
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOGkmBIg
name: datasette-tiddlywiki
full_name: simonw/datasette-tiddlywiki
html_url: https://github.com/simonw/datasette-tiddlywiki
description: Run TiddlyWiki in Datasette and save Tiddlers to a SQLite database
created_at: 2021-12-23T01:05:56Z
updated_at: 2022-02-14T08:57:33Z
pushed_at: 2022-03-08T01:36:10Z
language: HTML
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-tiddlywiki

  [![PyPI](https://img.shields.io/pypi/v/datasette-tiddlywiki.svg)](https://pypi.org/project/datasette-tiddlywiki/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-tiddlywiki?include_prereleases&label=changelog)](https://github.com/simonw/datasette-tiddlywiki/releases)
  [![Tests](https://github.com/simonw/datasette-tiddlywiki/workflows/Test/badge.svg)](https://github.com/simonw/datasette-tiddlywiki/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-tiddlywiki/blob/main/LICENSE)

  Run [TiddlyWiki](https://tiddlywiki.com/) in Datasette and save Tiddlers to a SQLite database

  Read more about this project [on my blog](https://simonwillison.net/2021/Dec/24/datasette-tiddlywiki/).

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-tiddlywiki

  ## Usage

  Start Datasette with a `tiddlywiki.db` database. You can create it if it does not yet exist using `--create`.

  You need to be signed in as the `root` user to write to the wiki, so use the `--root` option and click on the link it provides:

      % datasette tiddlywiki.db --create --root
      http://127.0.0.1:8001/-/auth-token?token=456670f1e8d01a8a33b71e17653130de17387336e29afcdfb4ab3d18261e6630
      # ...

  Navigate to `/-/tiddlywiki` on your instance to interact with TiddlyWiki.

  ## Authentication and permissions

  By default, the wiki can be read by anyone who has permission to read the `tiddlywiki.db` database. Only the signed in `root` user can write to it.

  You can sign in using the `--root` option described above, or you can set a password for that user using the [datasette-auth-passwords](https://datasette.io/plugins/datasette-auth-passwords) plugin and sign in using the `/-/login` page.

  You can use the `edit-tiddlywiki` permission to grant edit permisions to other users, using another plugin such as [datasette-permissions-sql](https://datasette.io/plugins/datasette-permissions-sql).

  You can use the `view-database` permission against the `tiddlywiki` database to control who can view the wiki.

  Datasette's permissions mechanism is described in full in [the Datasette documentation](https://docs.datasette.io/en/stable/authentication.html).

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-tiddlywiki
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-tiddlywiki" class="anchor" aria-hidden="true" href="#user-content-datasette-tiddlywiki"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-tiddlywiki</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-tiddlywiki/" rel="nofollow"><img src="https://camo.githubusercontent.com/94d62f40def55ce1280599082836059caa2a8015c30d984cde1872d75a15d0f4/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d746964646c7977696b692e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-tiddlywiki.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-tiddlywiki/releases"><img src="https://camo.githubusercontent.com/c054acb54b94f60a29e4f277807ce46811b9693a170e3e02c1e04798e3a595e8/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d746964646c7977696b693f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-tiddlywiki?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-tiddlywiki/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-tiddlywiki/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-tiddlywiki/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Run <a href="https://tiddlywiki.com/" rel="nofollow">TiddlyWiki</a> in Datasette and save Tiddlers to a SQLite database</p>
  <p dir="auto">Read more about this project <a href="https://simonwillison.net/2021/Dec/24/datasette-tiddlywiki/" rel="nofollow">on my blog</a>.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-tiddlywiki"><pre><code>$ datasette install datasette-tiddlywiki
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Start Datasette with a <code>tiddlywiki.db</code> database. You can create it if it does not yet exist using <code>--create</code>.</p>
  <p dir="auto">You need to be signed in as the <code>root</code> user to write to the wiki, so use the <code>--root</code> option and click on the link it provides:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="% datasette tiddlywiki.db --create --root
  http://127.0.0.1:8001/-/auth-token?token=456670f1e8d01a8a33b71e17653130de17387336e29afcdfb4ab3d18261e6630
  # ..."><pre><code>% datasette tiddlywiki.db --create --root
  http://127.0.0.1:8001/-/auth-token?token=456670f1e8d01a8a33b71e17653130de17387336e29afcdfb4ab3d18261e6630
  # ...
  </code></pre></div>
  <p dir="auto">Navigate to <code>/-/tiddlywiki</code> on your instance to interact with TiddlyWiki.</p>
  <h2 dir="auto"><a id="user-content-authentication-and-permissions" class="anchor" aria-hidden="true" href="#user-content-authentication-and-permissions"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Authentication and permissions</h2>
  <p dir="auto">By default, the wiki can be read by anyone who has permission to read the <code>tiddlywiki.db</code> database. Only the signed in <code>root</code> user can write to it.</p>
  <p dir="auto">You can sign in using the <code>--root</code> option described above, or you can set a password for that user using the <a href="https://datasette.io/plugins/datasette-auth-passwords" rel="nofollow">datasette-auth-passwords</a> plugin and sign in using the <code>/-/login</code> page.</p>
  <p dir="auto">You can use the <code>edit-tiddlywiki</code> permission to grant edit permisions to other users, using another plugin such as <a href="https://datasette.io/plugins/datasette-permissions-sql" rel="nofollow">datasette-permissions-sql</a>.</p>
  <p dir="auto">You can use the <code>view-database</code> permission against the <code>tiddlywiki</code> database to control who can view the wiki.</p>
  <p dir="auto">Datasette's permissions mechanism is described in full in <a href="https://docs.datasette.io/en/stable/authentication.html" rel="nofollow">the Datasette documentation</a>.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-tiddlywiki
  python3 -mvenv venv
  source venv/bin/activate"><pre><code>cd datasette-tiddlywiki
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOG2hQNg
name: google-drive-to-sqlite
full_name: simonw/google-drive-to-sqlite
html_url: https://github.com/simonw/google-drive-to-sqlite
description: Create a SQLite database containing metadata from Google Drive
created_at: 2022-02-16T02:16:29Z
updated_at: 2022-05-17T00:30:43Z
pushed_at: 2022-05-21T16:56:11Z
homepage: https://datasette.io/tools/google-drive-to-sqlite
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # google-drive-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/google-drive-to-sqlite.svg)](https://pypi.org/project/google-drive-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/google-drive-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/google-drive-to-sqlite/releases)
  [![Tests](https://github.com/simonw/google-drive-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/google-drive-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/google-drive-to-sqlite/blob/master/LICENSE)

  Create a SQLite database containing metadata from [Google Drive](https://www.google.com/drive)

  For background on this project, see [Google Drive to SQLite](https://simonwillison.net/2022/Feb/20/google-drive-to-sqlite/) on my blog.

  If you use Google Drive, and especially if you have shared drives with other people there's a good chance you have hundreds or even thousands of files that you may not be fully aware of.

  This tool can download metadata about those files - their names, sizes, folders, content types, permissions, creation dates and more - and store them in a SQLite database.

  This lets you use SQL to analyze your Google Drive contents, using [Datasette](https://datasette.io/) or the SQLite command-line tool or any other SQLite database browsing software.

  ## Installation

  Install this tool using `pip`:

      pip install google-drive-to-sqlite

  ## Quickstart

  Authenticate with Google Drive by running:

      google-drive-to-sqlite auth

  Now create a SQLite database with metadata about all of the files you have starred using:

      google-drive-to-sqlite files starred.db --starred

  You can explore the resulting database using [Datasette](https://datasette.io/):

      $ pip install datasette
      $ datasette starred.db
      INFO:     Started server process [24661]
      INFO:     Uvicorn running on http://127.0.0.1:8001

  ## Authentication

  > :warning: **This application has not yet been verified by Google** - you may find you are unable to authenticate until that verification is complete. [#10](https://github.com/simonw/google-drive-to-sqlite/issues/10)
  >
  > You can work around this issue by [creating your own OAuth client ID key](https://til.simonwillison.net/googlecloud/google-oauth-cli-application) and passing it to the `auth` command using `--google-client-id` and `--google-client-secret`.

  First, authenticate with Google Drive using the `auth` command:

      $ google-drive-to-sqlite auth
      Visit the following URL to authenticate with Google Drive

      https://accounts.google.com/o/oauth2/v2/auth?...

      Then return here and paste in the resulting code:
      Paste code here: 

  Follow the link, sign in with Google Drive and then copy and paste the resulting code back into the tool.

  This will save an authentication token to the file called `auth.json` in the current directory.

  To specify a different location for that file, use the `--auth` option:

      google-drive-to-sqlite auth --auth ~/google-drive-auth.json

  The `auth` command also provides options for using a different scope, Google client ID and Google client secret. You can use these to create your own custom authentication tokens that can work with other Google APIs, see [issue #5](https://github.com/simonw/google-drive-to-sqlite/issues/5) for details.

  Full `--help`:

  <!-- [[[cog
  import cog
  from google_drive_to_sqlite import cli
  from click.testing import CliRunner
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["auth", "--help"])
  help = result.output.replace("Usage: cli", "Usage: google-drive-to-sqlite")
  cog.out(
      "```\n{}\n```\n".format(help)
  )
  ]]] -->
  ```
  Usage: google-drive-to-sqlite auth [OPTIONS]

    Authenticate user and save credentials

  Options:
    -a, --auth FILE              Path to save token, defaults to auth.json
    --google-client-id TEXT      Custom Google client ID
    --google-client-secret TEXT  Custom Google client secret
    --scope TEXT                 Custom token scope
    --help                       Show this message and exit.

  ```
  <!-- [[[end]]] -->

  To revoke the token that is stored in `auth.json`, such that it cannot be used to access Google Drive in the future, run the `revoke` command:

      google-drive-to-sqlite revoke

  Or if your token is stored in another location:

      google-drive-to-sqlite revoke -a ~/google-drive-auth.json

  You will need to obtain a fresh token using the `auth` command in order to continue using this tool.

  ## google-drive-to-sqlite files

  To retrieve metadata about the files in your Google Drive, or a folder or search within it, use the `google-drive-to-sqlite files` command.

  This will default to writing details about every file in your Google Drive to a SQLite database:

      google-drive-to-sqlite files files.db

  Files and folders will be written to databases tables, which will be created if they do not yet exist. The database schema is [shown below](#database-schema).

  If a file or folder already exists, based on a matching `id`, it will be replaced with fresh data.

  Instead of writing to SQLite you can use `--json` to output as JSON, or `--nl` to output as newline-delimited JSON:

      google-drive-to-sqlite files --nl

  Use `--folder ID` to retrieve everything in a specified folder and its sub-folders:

      google-drive-to-sqlite files files.db --folder 1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i

  Use `--q QUERY` to use a [custom search query](https://developers.google.com/drive/api/v3/reference/query-ref):

      google-drive-to-sqlite files files.db -q "viewedByMeTime > '2022-01-01'"

  The following shortcut options help build queries:

  - `--full-text TEXT` to search for files where the full text matches a search term
  - `--starred` for files and folders you have starred
  - `--trashed` for files and folders in the trash
  - `--shared-with-me` for files and folders that have been shared with you
  - `--apps` for Google Apps documents, spreadsheets, presentations and drawings (equivalent to setting all  of the next four options)
  - `--docs` for Google Apps documents
  - `--sheets` for Google Apps spreadsheets
  - `--presentations` for Google Apps presentations
  - `--drawings` for Google Apps drawings

  You can combine these - for example, this returns all files that you have starred and that were shared with you:

      google-drive-to-sqlite files highlights.db \
        --starred --shared-with-me

  Multiple options are treated as AND, with the exception of the Google Apps options which are treated as OR - so the following would retrieve all spreadsheets and presentations that have also been starred:

      google-drive-to-sqlite files highlights.db \
        --starred --sheets --presentations

  You can use `--stop-after X` to stop after retrieving X files, useful for trying out a new search pattern and seeing results straight away.

  The `--import-json` and `--import-nl` options are mainly useful for testing and developing this tool. They allow you to replay the JSON or newline-delimited JSON that was previously fetched using `--json` or `--nl` and use it to create a fresh SQLite database, without needing to make any outbound API calls:

      # Fetch all starred files from the API, write to starred.json
      google-drive-to-sqlite files -q 'starred = true' --json > starred.json
      # Now import that data into a new SQLite database file
      google-drive-to-sqlite files starred.db --import-json starred.json

  Full `--help`:

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["files", "--help"])
  help = result.output.replace("Usage: cli", "Usage: google-drive-to-sqlite")
  cog.out(
      "```\n{}\n```\n".format(help)
  )
  ]]] -->
  ```
  Usage: google-drive-to-sqlite files [OPTIONS] [DATABASE]

    Retrieve metadata for files in Google Drive, and write to a SQLite database or
    output as JSON.

        google-drive-to-sqlite files files.db

    Use --json to output JSON, --nl for newline-delimited JSON:

        google-drive-to-sqlite files files.db --json

    Use a folder ID to recursively fetch every file in that folder and its sub-
    folders:

        google-drive-to-sqlite files files.db --folder
        1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i

    Fetch files you have starred:

        google-drive-to-sqlite files starred.db --starred

  Options:
    -a, --auth FILE       Path to auth.json token file
    --folder TEXT         Files in this folder ID and its sub-folders
    -q TEXT               Files matching this query
    --full-text TEXT      Search for files with text match
    --starred             Files you have starred
    --trashed             Files in the trash
    --shared-with-me      Files that have been shared with you
    --apps                Google Apps docs, spreadsheets, presentations and
                          drawings
    --docs                Google Apps docs
    --sheets              Google Apps spreadsheets
    --presentations       Google Apps presentations
    --drawings            Google Apps drawings
    --json                Output JSON rather than write to DB
    --nl                  Output newline-delimited JSON rather than write to DB
    --stop-after INTEGER  Stop paginating after X results
    --import-json FILE    Import from this JSON file instead of the API
    --import-nl FILE      Import from this newline-delimited JSON file
    -v, --verbose         Send verbose output to stderr
    --help                Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## google-drive-to-sqlite download FILE_ID

  The `download` command can be used to download files from Google Drive.

  You'll need one or more file IDs, which look something like `0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB`.

  To download the file, run this:

      google-drive-to-sqlite download 0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB

  This will detect the content type of the file and use that as the extension - so if this file is a JPEG the file would be downloaded as:

      0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB.jpeg

  You can pass multiple file IDs to the command at once.

  To hide the progress bar and filename output, use `-s` or `--silent`.

  If you are downloading a single file you can use the `-o` output to specify a filename and location:

      google-drive-to-sqlite download 0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB \
        -o my-image.jpeg

  Use `-o -` to write the file contents to standard output:

      google-drive-to-sqlite download 0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB \
        -o - > my-image.jpeg

  Full `--help`:

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["download", "--help"])
  help = result.output.replace("Usage: cli", "Usage: google-drive-to-sqlite")
  cog.out(
      "```\n{}\n```\n".format(help)
  )
  ]]] -->
  ```
  Usage: google-drive-to-sqlite download [OPTIONS] FILE_IDS...

    Download one or more files to disk, based on their file IDs.

    The file content will be saved to a file with the name:

        FILE_ID.ext

    Where the extension is automatically picked based on the type of file.

    If you are downloading a single file you can specify a filename with -o:

        google-drive-to-sqlite download MY_FILE_ID -o myfile.txt

  Options:
    -a, --auth FILE    Path to auth.json token file
    -o, --output FILE  File to write to, or - for standard output
    -s, --silent       Hide progress bar and filename
    --help             Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## google-drive-to-sqlite export FORMAT FILE_ID

  The `export` command can be used to export Google Docs documents, spreadsheets and presentations in a number of different formats.

  You'll need one or more document IDs, which look something like `10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU`. You can find these by looking at the URL of your document on the Google Docs site.

  To export that document as PDF, run this:

      google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU

  The file will be exported as:

      10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU-export.pdf

  You can pass multiple file IDs to the command at once.

  For the `FORMAT` option you can use any of the mime type options listed [on this page](https://developers.google.com/drive/api/v3/ref-export-formats) - for example, to export as an Open Office document you could use:

      google-drive-to-sqlite export \
       application/vnd.oasis.opendocument.text \
       10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU

  For convenience the following shortcuts for common file formats are provided:

  - Google Docs: `html`, `txt`, `rtf`, `pdf`, `doc`, `zip`, `epub`
  - Google Sheets: `xls`, `pdf`, `csv`, `tsv`, `zip`
  - Presentations: `ppt`, `pdf`, `txt`
  - Drawings: `jpeg`, `png`, `svg`

  The `zip` option returns a zip file of HTML. `txt` returns plain text. The others should be self-evident.

  To hide the filename output, use `-s` or `--silent`.

  If you are exporting a single file you can use the `-o` output to specify a filename and location:

      google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU \
        -o my-document.pdf

  Use `-o -` to write the file contents to standard output:

      google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU \
        -o - > my-document.pdf

  Full `--help`:

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["export", "--help"])
  help = result.output.replace("Usage: cli", "Usage: google-drive-to-sqlite")
  cog.out(
      "```\n{}\n```\n".format(help)
  )
  ]]] -->
  ```
  Usage: google-drive-to-sqlite export [OPTIONS] FORMAT FILE_IDS...

    Export one or more files to the specified format.

    Usage:

        google-drive-to-sqlite export pdf FILE_ID_1 FILE_ID_2

    The file content will be saved to a file with the name:

        FILE_ID-export.ext

    Where the extension is based on the format you specified.

    Available export formats can be seen here:
    https://developers.google.com/drive/api/v3/ref-export-formats

    Or you can use one of the following shortcuts:

    - Google Docs: html, txt, rtf, pdf, doc, zip, epub
    - Google Sheets: xls, pdf, csv, tsv, zip
    - Presentations: ppt, pdf, txt
    - Drawings: jpeg, png, svg

    "zip" returns a zip file of HTML.

    If you are exporting a single file you can specify a filename with -o:

        google-drive-to-sqlite export zip MY_FILE_ID -o myfile.zip

  Options:
    -a, --auth FILE    Path to auth.json token file
    -o, --output FILE  File to write to, or - for standard output
    -s, --silent       Hide progress bar and filename
    --help             Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## google-drive-to-sqlite get URL

  The `get` command makes authenticated requests to the specified URL, using credentials derived from the `auth.json` file.

  For example:

      $ google-drive-to-sqlite get 'https://www.googleapis.com/drive/v3/about?fields=*'
      {
          "kind": "drive#about",
          "user": {
              "kind": "drive#user",
              "displayName": "Simon Willison",
      # ...

  If the resource you are fetching supports pagination you can use `--paginate key` to paginate through all of the rows in a specified key. For example, the following API has a `nextPageToken` key and a `files` list, suggesting it supports pagination:

      $ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files
      {
          "kind": "drive#fileList",
          "nextPageToken": "~!!~AI9...wogHHYlc=",
          "incompleteSearch": false,
          "files": [
              {
                  "kind": "drive#file",
                  "id": "1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg",
                  "name": "Title of a spreadsheet",
                  "mimeType": "application/vnd.google-apps.spreadsheet"
              },

  To paginate through everything in the `files` list you would use `--paginate files` like this:

      $ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files --paginate files
      [
        {
          "kind": "drive#file",
          "id": "1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg",
          "name": "Title of a spreadsheet",
          "mimeType": "application/vnd.google-apps.spreadsheet"
        },
        # ...

  Add `--nl` to stream paginated data as newline-delimited JSON:

      $ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files --paginate files --nl
      {"kind": "drive#file", "id": "1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg", "name": "Title of a spreadsheet", "mimeType": "application/vnd.google-apps.spreadsheet"}
      {"kind": "drive#file", "id": "1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i", "name": "Subfolder", "mimeType": "application/vnd.google-apps.folder"}

  Add `--stop-after 5` to stop after 5 records - useful for testing.

  Full `--help`:

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["get", "--help"])
  help = result.output.replace("Usage: cli", "Usage: google-drive-to-sqlite")
  cog.out(
      "```\n{}\n```\n".format(help)
  )
  ]]] -->
  ```
  Usage: google-drive-to-sqlite get [OPTIONS] URL

    Make an authenticated HTTP GET to the specified URL

  Options:
    -a, --auth FILE       Path to auth.json token file
    --paginate TEXT       Paginate through all results in this key
    --nl                  Output paginated data as newline-delimited JSON
    --stop-after INTEGER  Stop paginating after X results
    -v, --verbose         Send verbose output to stderr
    --help                Show this message and exit.

  ```
  <!-- [[[end]]] -->


  ## Database schema

  The database created by this tool has the following schema:

  <!-- [[[cog
  import tempfile, pathlib, sqlite_utils
  tmpdir = pathlib.Path(tempfile.mkdtemp())
  db_path = str(tmpdir / "docs.db")
  result = runner.invoke(cli.cli, [
      "files", db_path, "--import-json", "tests/folder-and-children.json"
  ])
  cog.out("```sql\n")
  schema = sqlite_utils.Database(db_path).schema
  # Tidy up some formatting
  schema = schema.replace(", [", ",\n   [")
  schema = schema.replace("\n,\n", ",\n")
  schema = schema.replace("TEXT);", "TEXT\n);")
  cog.out(schema)
  cog.out("\n```")
  ]]] -->
  ```sql
  CREATE TABLE [drive_users] (
     [permissionId] TEXT PRIMARY KEY,
     [kind] TEXT,
     [displayName] TEXT,
     [photoLink] TEXT,
     [me] INTEGER,
     [emailAddress] TEXT
  );
  CREATE TABLE [drive_folders] (
     [id] TEXT PRIMARY KEY,
     [_parent] TEXT,
     [_owner] TEXT,
     [lastModifyingUser] TEXT,
     [kind] TEXT,
     [name] TEXT,
     [mimeType] TEXT,
     [starred] INTEGER,
     [trashed] INTEGER,
     [explicitlyTrashed] INTEGER,
     [parents] TEXT,
     [spaces] TEXT,
     [version] TEXT,
     [webViewLink] TEXT,
     [iconLink] TEXT,
     [hasThumbnail] INTEGER,
     [thumbnailVersion] TEXT,
     [viewedByMe] INTEGER,
     [createdTime] TEXT,
     [modifiedTime] TEXT,
     [modifiedByMe] INTEGER,
     [shared] INTEGER,
     [ownedByMe] INTEGER,
     [viewersCanCopyContent] INTEGER,
     [copyRequiresWriterPermission] INTEGER,
     [writersCanShare] INTEGER,
     [folderColorRgb] TEXT,
     [quotaBytesUsed] TEXT,
     [isAppAuthorized] INTEGER,
     [linkShareMetadata] TEXT,
     FOREIGN KEY([_parent]) REFERENCES [drive_folders]([id]),
     FOREIGN KEY([_owner]) REFERENCES [drive_users]([permissionId]),
     FOREIGN KEY([lastModifyingUser]) REFERENCES [drive_users]([permissionId])
  );
  CREATE TABLE [drive_files] (
     [id] TEXT PRIMARY KEY,
     [_parent] TEXT,
     [_owner] TEXT,
     [lastModifyingUser] TEXT,
     [kind] TEXT,
     [name] TEXT,
     [mimeType] TEXT,
     [starred] INTEGER,
     [trashed] INTEGER,
     [explicitlyTrashed] INTEGER,
     [parents] TEXT,
     [spaces] TEXT,
     [version] TEXT,
     [webViewLink] TEXT,
     [iconLink] TEXT,
     [hasThumbnail] INTEGER,
     [thumbnailVersion] TEXT,
     [viewedByMe] INTEGER,
     [createdTime] TEXT,
     [modifiedTime] TEXT,
     [modifiedByMe] INTEGER,
     [shared] INTEGER,
     [ownedByMe] INTEGER,
     [viewersCanCopyContent] INTEGER,
     [copyRequiresWriterPermission] INTEGER,
     [writersCanShare] INTEGER,
     [quotaBytesUsed] TEXT,
     [isAppAuthorized] INTEGER,
     [linkShareMetadata] TEXT,
     FOREIGN KEY([_parent]) REFERENCES [drive_folders]([id]),
     FOREIGN KEY([_owner]) REFERENCES [drive_users]([permissionId]),
     FOREIGN KEY([lastModifyingUser]) REFERENCES [drive_users]([permissionId])
  );
  ```
  <!-- [[[end]]] -->

  ## Thumbnails

  You can construct a thumbnail image for a known file ID using the following URL:

      https://drive.google.com/thumbnail?sz=w800-h800&id=FILE_ID

  Users who are signed into Google Drive and have permission to view a file will be redirected to a thumbnail version of that file. You can tweak the `w800` and `h800` parameters to request different thumbnail sizes.

  ## Privacy policy

  This tool requests access to your Google Drive account in order to retrieve metadata about your files there. It also offers a feature that can download the content of those files.

  The credentials used to access your account are stored in the `auth.json` file on your computer. The metadata and content retrieved from Google Drive is also stored only on your own personal computer.

  At no point do the developers of this tool gain access to any of your data.

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd google-drive-to-sqlite
      python -m venv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-google-drive-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-google-drive-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>google-drive-to-sqlite</h1>
  <p dir="auto"><a href="https://pypi.org/project/google-drive-to-sqlite/" rel="nofollow"><img src="https://camo.githubusercontent.com/18fcd4b2930251d5e234ec0832e9dd82dde2d7b5bfc9dc31387e22107a1da0a3/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f676f6f676c652d64726976652d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/google-drive-to-sqlite.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/google-drive-to-sqlite/releases"><img src="https://camo.githubusercontent.com/7c733ea3efff14b60d66656573c9bdc05d5d6a2d84ec3fdec73cef08685decf1/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f676f6f676c652d64726976652d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/google-drive-to-sqlite?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/google-drive-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/google-drive-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/google-drive-to-sqlite/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Create a SQLite database containing metadata from <a href="https://www.google.com/drive" rel="nofollow">Google Drive</a></p>
  <p dir="auto">For background on this project, see <a href="https://simonwillison.net/2022/Feb/20/google-drive-to-sqlite/" rel="nofollow">Google Drive to SQLite</a> on my blog.</p>
  <p dir="auto">If you use Google Drive, and especially if you have shared drives with other people there's a good chance you have hundreds or even thousands of files that you may not be fully aware of.</p>
  <p dir="auto">This tool can download metadata about those files - their names, sizes, folders, content types, permissions, creation dates and more - and store them in a SQLite database.</p>
  <p dir="auto">This lets you use SQL to analyze your Google Drive contents, using <a href="https://datasette.io/" rel="nofollow">Datasette</a> or the SQLite command-line tool or any other SQLite database browsing software.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this tool using <code>pip</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install google-drive-to-sqlite"><pre class="notranslate"><code>pip install google-drive-to-sqlite
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-quickstart" class="anchor" aria-hidden="true" href="#user-content-quickstart"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Quickstart</h2>
  <p dir="auto">Authenticate with Google Drive by running:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite auth"><pre class="notranslate"><code>google-drive-to-sqlite auth
  </code></pre></div>
  <p dir="auto">Now create a SQLite database with metadata about all of the files you have starred using:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite files starred.db --starred"><pre class="notranslate"><code>google-drive-to-sqlite files starred.db --starred
  </code></pre></div>
  <p dir="auto">You can explore the resulting database using <a href="https://datasette.io/" rel="nofollow">Datasette</a>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette
  $ datasette starred.db
  INFO:     Started server process [24661]
  INFO:     Uvicorn running on http://127.0.0.1:8001"><pre class="notranslate"><code>$ pip install datasette
  $ datasette starred.db
  INFO:     Started server process [24661]
  INFO:     Uvicorn running on http://127.0.0.1:8001
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-authentication" class="anchor" aria-hidden="true" href="#user-content-authentication"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Authentication</h2>
  <blockquote>
  <p dir="auto"><g-emoji class="g-emoji" alias="warning" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/26a0.png">⚠️</g-emoji> <strong>This application has not yet been verified by Google</strong> - you may find you are unable to authenticate until that verification is complete. <a href="https://github.com/simonw/google-drive-to-sqlite/issues/10" data-hovercard-type="issue" data-hovercard-url="/simonw/google-drive-to-sqlite/issues/10/hovercard">#10</a></p>
  <p dir="auto">You can work around this issue by <a href="https://til.simonwillison.net/googlecloud/google-oauth-cli-application" rel="nofollow">creating your own OAuth client ID key</a> and passing it to the <code>auth</code> command using <code>--google-client-id</code> and <code>--google-client-secret</code>.</p>
  </blockquote>
  <p dir="auto">First, authenticate with Google Drive using the <code>auth</code> command:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ google-drive-to-sqlite auth
  Visit the following URL to authenticate with Google Drive

  https://accounts.google.com/o/oauth2/v2/auth?...

  Then return here and paste in the resulting code:
  Paste code here: "><pre class="notranslate"><code>$ google-drive-to-sqlite auth
  Visit the following URL to authenticate with Google Drive

  https://accounts.google.com/o/oauth2/v2/auth?...

  Then return here and paste in the resulting code:
  Paste code here: 
  </code></pre></div>
  <p dir="auto">Follow the link, sign in with Google Drive and then copy and paste the resulting code back into the tool.</p>
  <p dir="auto">This will save an authentication token to the file called <code>auth.json</code> in the current directory.</p>
  <p dir="auto">To specify a different location for that file, use the <code>--auth</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite auth --auth ~/google-drive-auth.json"><pre class="notranslate"><code>google-drive-to-sqlite auth --auth ~/google-drive-auth.json
  </code></pre></div>
  <p dir="auto">The <code>auth</code> command also provides options for using a different scope, Google client ID and Google client secret. You can use these to create your own custom authentication tokens that can work with other Google APIs, see <a href="https://github.com/simonw/google-drive-to-sqlite/issues/5" data-hovercard-type="issue" data-hovercard-url="/simonw/google-drive-to-sqlite/issues/5/hovercard">issue #5</a> for details.</p>
  <p dir="auto">Full <code>--help</code>:</p>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: google-drive-to-sqlite auth [OPTIONS]

    Authenticate user and save credentials

  Options:
    -a, --auth FILE              Path to save token, defaults to auth.json
    --google-client-id TEXT      Custom Google client ID
    --google-client-secret TEXT  Custom Google client secret
    --scope TEXT                 Custom token scope
    --help                       Show this message and exit.
  "><pre class="notranslate"><code>Usage: google-drive-to-sqlite auth [OPTIONS]

    Authenticate user and save credentials

  Options:
    -a, --auth FILE              Path to save token, defaults to auth.json
    --google-client-id TEXT      Custom Google client ID
    --google-client-secret TEXT  Custom Google client secret
    --scope TEXT                 Custom token scope
    --help                       Show this message and exit.

  </code></pre></div>

  <p dir="auto">To revoke the token that is stored in <code>auth.json</code>, such that it cannot be used to access Google Drive in the future, run the <code>revoke</code> command:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite revoke"><pre class="notranslate"><code>google-drive-to-sqlite revoke
  </code></pre></div>
  <p dir="auto">Or if your token is stored in another location:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite revoke -a ~/google-drive-auth.json"><pre class="notranslate"><code>google-drive-to-sqlite revoke -a ~/google-drive-auth.json
  </code></pre></div>
  <p dir="auto">You will need to obtain a fresh token using the <code>auth</code> command in order to continue using this tool.</p>
  <h2 dir="auto"><a id="user-content-google-drive-to-sqlite-files" class="anchor" aria-hidden="true" href="#user-content-google-drive-to-sqlite-files"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>google-drive-to-sqlite files</h2>
  <p dir="auto">To retrieve metadata about the files in your Google Drive, or a folder or search within it, use the <code>google-drive-to-sqlite files</code> command.</p>
  <p dir="auto">This will default to writing details about every file in your Google Drive to a SQLite database:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite files files.db"><pre class="notranslate"><code>google-drive-to-sqlite files files.db
  </code></pre></div>
  <p dir="auto">Files and folders will be written to databases tables, which will be created if they do not yet exist. The database schema is <a href="#user-content-database-schema">shown below</a>.</p>
  <p dir="auto">If a file or folder already exists, based on a matching <code>id</code>, it will be replaced with fresh data.</p>
  <p dir="auto">Instead of writing to SQLite you can use <code>--json</code> to output as JSON, or <code>--nl</code> to output as newline-delimited JSON:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite files --nl"><pre class="notranslate"><code>google-drive-to-sqlite files --nl
  </code></pre></div>
  <p dir="auto">Use <code>--folder ID</code> to retrieve everything in a specified folder and its sub-folders:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite files files.db --folder 1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i"><pre class="notranslate"><code>google-drive-to-sqlite files files.db --folder 1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i
  </code></pre></div>
  <p dir="auto">Use <code>--q QUERY</code> to use a <a href="https://developers.google.com/drive/api/v3/reference/query-ref" rel="nofollow">custom search query</a>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite files files.db -q &quot;viewedByMeTime &gt; '2022-01-01'&quot;"><pre class="notranslate"><code>google-drive-to-sqlite files files.db -q "viewedByMeTime &gt; '2022-01-01'"
  </code></pre></div>
  <p dir="auto">The following shortcut options help build queries:</p>
  <ul dir="auto">
  <li><code>--full-text TEXT</code> to search for files where the full text matches a search term</li>
  <li><code>--starred</code> for files and folders you have starred</li>
  <li><code>--trashed</code> for files and folders in the trash</li>
  <li><code>--shared-with-me</code> for files and folders that have been shared with you</li>
  <li><code>--apps</code> for Google Apps documents, spreadsheets, presentations and drawings (equivalent to setting all  of the next four options)</li>
  <li><code>--docs</code> for Google Apps documents</li>
  <li><code>--sheets</code> for Google Apps spreadsheets</li>
  <li><code>--presentations</code> for Google Apps presentations</li>
  <li><code>--drawings</code> for Google Apps drawings</li>
  </ul>
  <p dir="auto">You can combine these - for example, this returns all files that you have starred and that were shared with you:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite files highlights.db \
    --starred --shared-with-me"><pre class="notranslate"><code>google-drive-to-sqlite files highlights.db \
    --starred --shared-with-me
  </code></pre></div>
  <p dir="auto">Multiple options are treated as AND, with the exception of the Google Apps options which are treated as OR - so the following would retrieve all spreadsheets and presentations that have also been starred:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite files highlights.db \
    --starred --sheets --presentations"><pre class="notranslate"><code>google-drive-to-sqlite files highlights.db \
    --starred --sheets --presentations
  </code></pre></div>
  <p dir="auto">You can use <code>--stop-after X</code> to stop after retrieving X files, useful for trying out a new search pattern and seeing results straight away.</p>
  <p dir="auto">The <code>--import-json</code> and <code>--import-nl</code> options are mainly useful for testing and developing this tool. They allow you to replay the JSON or newline-delimited JSON that was previously fetched using <code>--json</code> or <code>--nl</code> and use it to create a fresh SQLite database, without needing to make any outbound API calls:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# Fetch all starred files from the API, write to starred.json
  google-drive-to-sqlite files -q 'starred = true' --json &gt; starred.json
  # Now import that data into a new SQLite database file
  google-drive-to-sqlite files starred.db --import-json starred.json"><pre class="notranslate"><code># Fetch all starred files from the API, write to starred.json
  google-drive-to-sqlite files -q 'starred = true' --json &gt; starred.json
  # Now import that data into a new SQLite database file
  google-drive-to-sqlite files starred.db --import-json starred.json
  </code></pre></div>
  <p dir="auto">Full <code>--help</code>:</p>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: google-drive-to-sqlite files [OPTIONS] [DATABASE]

    Retrieve metadata for files in Google Drive, and write to a SQLite database or
    output as JSON.

        google-drive-to-sqlite files files.db

    Use --json to output JSON, --nl for newline-delimited JSON:

        google-drive-to-sqlite files files.db --json

    Use a folder ID to recursively fetch every file in that folder and its sub-
    folders:

        google-drive-to-sqlite files files.db --folder
        1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i

    Fetch files you have starred:

        google-drive-to-sqlite files starred.db --starred

  Options:
    -a, --auth FILE       Path to auth.json token file
    --folder TEXT         Files in this folder ID and its sub-folders
    -q TEXT               Files matching this query
    --full-text TEXT      Search for files with text match
    --starred             Files you have starred
    --trashed             Files in the trash
    --shared-with-me      Files that have been shared with you
    --apps                Google Apps docs, spreadsheets, presentations and
                          drawings
    --docs                Google Apps docs
    --sheets              Google Apps spreadsheets
    --presentations       Google Apps presentations
    --drawings            Google Apps drawings
    --json                Output JSON rather than write to DB
    --nl                  Output newline-delimited JSON rather than write to DB
    --stop-after INTEGER  Stop paginating after X results
    --import-json FILE    Import from this JSON file instead of the API
    --import-nl FILE      Import from this newline-delimited JSON file
    -v, --verbose         Send verbose output to stderr
    --help                Show this message and exit.
  "><pre class="notranslate"><code>Usage: google-drive-to-sqlite files [OPTIONS] [DATABASE]

    Retrieve metadata for files in Google Drive, and write to a SQLite database or
    output as JSON.

        google-drive-to-sqlite files files.db

    Use --json to output JSON, --nl for newline-delimited JSON:

        google-drive-to-sqlite files files.db --json

    Use a folder ID to recursively fetch every file in that folder and its sub-
    folders:

        google-drive-to-sqlite files files.db --folder
        1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i

    Fetch files you have starred:

        google-drive-to-sqlite files starred.db --starred

  Options:
    -a, --auth FILE       Path to auth.json token file
    --folder TEXT         Files in this folder ID and its sub-folders
    -q TEXT               Files matching this query
    --full-text TEXT      Search for files with text match
    --starred             Files you have starred
    --trashed             Files in the trash
    --shared-with-me      Files that have been shared with you
    --apps                Google Apps docs, spreadsheets, presentations and
                          drawings
    --docs                Google Apps docs
    --sheets              Google Apps spreadsheets
    --presentations       Google Apps presentations
    --drawings            Google Apps drawings
    --json                Output JSON rather than write to DB
    --nl                  Output newline-delimited JSON rather than write to DB
    --stop-after INTEGER  Stop paginating after X results
    --import-json FILE    Import from this JSON file instead of the API
    --import-nl FILE      Import from this newline-delimited JSON file
    -v, --verbose         Send verbose output to stderr
    --help                Show this message and exit.

  </code></pre></div>

  <h2 dir="auto"><a id="user-content-google-drive-to-sqlite-download-file_id" class="anchor" aria-hidden="true" href="#user-content-google-drive-to-sqlite-download-file_id"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>google-drive-to-sqlite download FILE_ID</h2>
  <p dir="auto">The <code>download</code> command can be used to download files from Google Drive.</p>
  <p dir="auto">You'll need one or more file IDs, which look something like <code>0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB</code>.</p>
  <p dir="auto">To download the file, run this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite download 0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB"><pre class="notranslate"><code>google-drive-to-sqlite download 0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB
  </code></pre></div>
  <p dir="auto">This will detect the content type of the file and use that as the extension - so if this file is a JPEG the file would be downloaded as:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB.jpeg"><pre class="notranslate"><code>0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB.jpeg
  </code></pre></div>
  <p dir="auto">You can pass multiple file IDs to the command at once.</p>
  <p dir="auto">To hide the progress bar and filename output, use <code>-s</code> or <code>--silent</code>.</p>
  <p dir="auto">If you are downloading a single file you can use the <code>-o</code> output to specify a filename and location:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite download 0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB \
    -o my-image.jpeg"><pre class="notranslate"><code>google-drive-to-sqlite download 0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB \
    -o my-image.jpeg
  </code></pre></div>
  <p dir="auto">Use <code>-o -</code> to write the file contents to standard output:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite download 0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB \
    -o - &gt; my-image.jpeg"><pre class="notranslate"><code>google-drive-to-sqlite download 0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB \
    -o - &gt; my-image.jpeg
  </code></pre></div>
  <p dir="auto">Full <code>--help</code>:</p>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: google-drive-to-sqlite download [OPTIONS] FILE_IDS...

    Download one or more files to disk, based on their file IDs.

    The file content will be saved to a file with the name:

        FILE_ID.ext

    Where the extension is automatically picked based on the type of file.

    If you are downloading a single file you can specify a filename with -o:

        google-drive-to-sqlite download MY_FILE_ID -o myfile.txt

  Options:
    -a, --auth FILE    Path to auth.json token file
    -o, --output FILE  File to write to, or - for standard output
    -s, --silent       Hide progress bar and filename
    --help             Show this message and exit.
  "><pre class="notranslate"><code>Usage: google-drive-to-sqlite download [OPTIONS] FILE_IDS...

    Download one or more files to disk, based on their file IDs.

    The file content will be saved to a file with the name:

        FILE_ID.ext

    Where the extension is automatically picked based on the type of file.

    If you are downloading a single file you can specify a filename with -o:

        google-drive-to-sqlite download MY_FILE_ID -o myfile.txt

  Options:
    -a, --auth FILE    Path to auth.json token file
    -o, --output FILE  File to write to, or - for standard output
    -s, --silent       Hide progress bar and filename
    --help             Show this message and exit.

  </code></pre></div>

  <h2 dir="auto"><a id="user-content-google-drive-to-sqlite-export-format-file_id" class="anchor" aria-hidden="true" href="#user-content-google-drive-to-sqlite-export-format-file_id"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>google-drive-to-sqlite export FORMAT FILE_ID</h2>
  <p dir="auto">The <code>export</code> command can be used to export Google Docs documents, spreadsheets and presentations in a number of different formats.</p>
  <p dir="auto">You'll need one or more document IDs, which look something like <code>10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU</code>. You can find these by looking at the URL of your document on the Google Docs site.</p>
  <p dir="auto">To export that document as PDF, run this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU"><pre class="notranslate"><code>google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU
  </code></pre></div>
  <p dir="auto">The file will be exported as:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU-export.pdf"><pre class="notranslate"><code>10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU-export.pdf
  </code></pre></div>
  <p dir="auto">You can pass multiple file IDs to the command at once.</p>
  <p dir="auto">For the <code>FORMAT</code> option you can use any of the mime type options listed <a href="https://developers.google.com/drive/api/v3/ref-export-formats" rel="nofollow">on this page</a> - for example, to export as an Open Office document you could use:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite export \
   application/vnd.oasis.opendocument.text \
   10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU"><pre class="notranslate"><code>google-drive-to-sqlite export \
   application/vnd.oasis.opendocument.text \
   10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU
  </code></pre></div>
  <p dir="auto">For convenience the following shortcuts for common file formats are provided:</p>
  <ul dir="auto">
  <li>Google Docs: <code>html</code>, <code>txt</code>, <code>rtf</code>, <code>pdf</code>, <code>doc</code>, <code>zip</code>, <code>epub</code></li>
  <li>Google Sheets: <code>xls</code>, <code>pdf</code>, <code>csv</code>, <code>tsv</code>, <code>zip</code></li>
  <li>Presentations: <code>ppt</code>, <code>pdf</code>, <code>txt</code></li>
  <li>Drawings: <code>jpeg</code>, <code>png</code>, <code>svg</code></li>
  </ul>
  <p dir="auto">The <code>zip</code> option returns a zip file of HTML. <code>txt</code> returns plain text. The others should be self-evident.</p>
  <p dir="auto">To hide the filename output, use <code>-s</code> or <code>--silent</code>.</p>
  <p dir="auto">If you are exporting a single file you can use the <code>-o</code> output to specify a filename and location:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU \
    -o my-document.pdf"><pre class="notranslate"><code>google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU \
    -o my-document.pdf
  </code></pre></div>
  <p dir="auto">Use <code>-o -</code> to write the file contents to standard output:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU \
    -o - &gt; my-document.pdf"><pre class="notranslate"><code>google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU \
    -o - &gt; my-document.pdf
  </code></pre></div>
  <p dir="auto">Full <code>--help</code>:</p>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: google-drive-to-sqlite export [OPTIONS] FORMAT FILE_IDS...

    Export one or more files to the specified format.

    Usage:

        google-drive-to-sqlite export pdf FILE_ID_1 FILE_ID_2

    The file content will be saved to a file with the name:

        FILE_ID-export.ext

    Where the extension is based on the format you specified.

    Available export formats can be seen here:
    https://developers.google.com/drive/api/v3/ref-export-formats

    Or you can use one of the following shortcuts:

    - Google Docs: html, txt, rtf, pdf, doc, zip, epub
    - Google Sheets: xls, pdf, csv, tsv, zip
    - Presentations: ppt, pdf, txt
    - Drawings: jpeg, png, svg

    &quot;zip&quot; returns a zip file of HTML.

    If you are exporting a single file you can specify a filename with -o:

        google-drive-to-sqlite export zip MY_FILE_ID -o myfile.zip

  Options:
    -a, --auth FILE    Path to auth.json token file
    -o, --output FILE  File to write to, or - for standard output
    -s, --silent       Hide progress bar and filename
    --help             Show this message and exit.
  "><pre class="notranslate"><code>Usage: google-drive-to-sqlite export [OPTIONS] FORMAT FILE_IDS...

    Export one or more files to the specified format.

    Usage:

        google-drive-to-sqlite export pdf FILE_ID_1 FILE_ID_2

    The file content will be saved to a file with the name:

        FILE_ID-export.ext

    Where the extension is based on the format you specified.

    Available export formats can be seen here:
    https://developers.google.com/drive/api/v3/ref-export-formats

    Or you can use one of the following shortcuts:

    - Google Docs: html, txt, rtf, pdf, doc, zip, epub
    - Google Sheets: xls, pdf, csv, tsv, zip
    - Presentations: ppt, pdf, txt
    - Drawings: jpeg, png, svg

    "zip" returns a zip file of HTML.

    If you are exporting a single file you can specify a filename with -o:

        google-drive-to-sqlite export zip MY_FILE_ID -o myfile.zip

  Options:
    -a, --auth FILE    Path to auth.json token file
    -o, --output FILE  File to write to, or - for standard output
    -s, --silent       Hide progress bar and filename
    --help             Show this message and exit.

  </code></pre></div>

  <h2 dir="auto"><a id="user-content-google-drive-to-sqlite-get-url" class="anchor" aria-hidden="true" href="#user-content-google-drive-to-sqlite-get-url"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>google-drive-to-sqlite get URL</h2>
  <p dir="auto">The <code>get</code> command makes authenticated requests to the specified URL, using credentials derived from the <code>auth.json</code> file.</p>
  <p dir="auto">For example:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ google-drive-to-sqlite get 'https://www.googleapis.com/drive/v3/about?fields=*'
  {
      &quot;kind&quot;: &quot;drive#about&quot;,
      &quot;user&quot;: {
          &quot;kind&quot;: &quot;drive#user&quot;,
          &quot;displayName&quot;: &quot;Simon Willison&quot;,
  # ..."><pre class="notranslate"><code>$ google-drive-to-sqlite get 'https://www.googleapis.com/drive/v3/about?fields=*'
  {
      "kind": "drive#about",
      "user": {
          "kind": "drive#user",
          "displayName": "Simon Willison",
  # ...
  </code></pre></div>
  <p dir="auto">If the resource you are fetching supports pagination you can use <code>--paginate key</code> to paginate through all of the rows in a specified key. For example, the following API has a <code>nextPageToken</code> key and a <code>files</code> list, suggesting it supports pagination:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files
  {
      &quot;kind&quot;: &quot;drive#fileList&quot;,
      &quot;nextPageToken&quot;: &quot;~!!~AI9...wogHHYlc=&quot;,
      &quot;incompleteSearch&quot;: false,
      &quot;files&quot;: [
          {
              &quot;kind&quot;: &quot;drive#file&quot;,
              &quot;id&quot;: &quot;1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg&quot;,
              &quot;name&quot;: &quot;Title of a spreadsheet&quot;,
              &quot;mimeType&quot;: &quot;application/vnd.google-apps.spreadsheet&quot;
          },"><pre class="notranslate"><code>$ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files
  {
      "kind": "drive#fileList",
      "nextPageToken": "~!!~AI9...wogHHYlc=",
      "incompleteSearch": false,
      "files": [
          {
              "kind": "drive#file",
              "id": "1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg",
              "name": "Title of a spreadsheet",
              "mimeType": "application/vnd.google-apps.spreadsheet"
          },
  </code></pre></div>
  <p dir="auto">To paginate through everything in the <code>files</code> list you would use <code>--paginate files</code> like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files --paginate files
  [
    {
      &quot;kind&quot;: &quot;drive#file&quot;,
      &quot;id&quot;: &quot;1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg&quot;,
      &quot;name&quot;: &quot;Title of a spreadsheet&quot;,
      &quot;mimeType&quot;: &quot;application/vnd.google-apps.spreadsheet&quot;
    },
    # ..."><pre class="notranslate"><code>$ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files --paginate files
  [
    {
      "kind": "drive#file",
      "id": "1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg",
      "name": "Title of a spreadsheet",
      "mimeType": "application/vnd.google-apps.spreadsheet"
    },
    # ...
  </code></pre></div>
  <p dir="auto">Add <code>--nl</code> to stream paginated data as newline-delimited JSON:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files --paginate files --nl
  {&quot;kind&quot;: &quot;drive#file&quot;, &quot;id&quot;: &quot;1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg&quot;, &quot;name&quot;: &quot;Title of a spreadsheet&quot;, &quot;mimeType&quot;: &quot;application/vnd.google-apps.spreadsheet&quot;}
  {&quot;kind&quot;: &quot;drive#file&quot;, &quot;id&quot;: &quot;1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i&quot;, &quot;name&quot;: &quot;Subfolder&quot;, &quot;mimeType&quot;: &quot;application/vnd.google-apps.folder&quot;}"><pre class="notranslate"><code>$ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files --paginate files --nl
  {"kind": "drive#file", "id": "1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg", "name": "Title of a spreadsheet", "mimeType": "application/vnd.google-apps.spreadsheet"}
  {"kind": "drive#file", "id": "1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i", "name": "Subfolder", "mimeType": "application/vnd.google-apps.folder"}
  </code></pre></div>
  <p dir="auto">Add <code>--stop-after 5</code> to stop after 5 records - useful for testing.</p>
  <p dir="auto">Full <code>--help</code>:</p>

  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: google-drive-to-sqlite get [OPTIONS] URL

    Make an authenticated HTTP GET to the specified URL

  Options:
    -a, --auth FILE       Path to auth.json token file
    --paginate TEXT       Paginate through all results in this key
    --nl                  Output paginated data as newline-delimited JSON
    --stop-after INTEGER  Stop paginating after X results
    -v, --verbose         Send verbose output to stderr
    --help                Show this message and exit.
  "><pre class="notranslate"><code>Usage: google-drive-to-sqlite get [OPTIONS] URL

    Make an authenticated HTTP GET to the specified URL

  Options:
    -a, --auth FILE       Path to auth.json token file
    --paginate TEXT       Paginate through all results in this key
    --nl                  Output paginated data as newline-delimited JSON
    --stop-after INTEGER  Stop paginating after X results
    -v, --verbose         Send verbose output to stderr
    --help                Show this message and exit.

  </code></pre></div>

  <h2 dir="auto"><a id="user-content-database-schema" class="anchor" aria-hidden="true" href="#user-content-database-schema"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Database schema</h2>
  <p dir="auto">The database created by this tool has the following schema:</p>

  <div class="highlight highlight-source-sql notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="CREATE TABLE [drive_users] (
     [permissionId] TEXT PRIMARY KEY,
     [kind] TEXT,
     [displayName] TEXT,
     [photoLink] TEXT,
     [me] INTEGER,
     [emailAddress] TEXT
  );
  CREATE TABLE [drive_folders] (
     [id] TEXT PRIMARY KEY,
     [_parent] TEXT,
     [_owner] TEXT,
     [lastModifyingUser] TEXT,
     [kind] TEXT,
     [name] TEXT,
     [mimeType] TEXT,
     [starred] INTEGER,
     [trashed] INTEGER,
     [explicitlyTrashed] INTEGER,
     [parents] TEXT,
     [spaces] TEXT,
     [version] TEXT,
     [webViewLink] TEXT,
     [iconLink] TEXT,
     [hasThumbnail] INTEGER,
     [thumbnailVersion] TEXT,
     [viewedByMe] INTEGER,
     [createdTime] TEXT,
     [modifiedTime] TEXT,
     [modifiedByMe] INTEGER,
     [shared] INTEGER,
     [ownedByMe] INTEGER,
     [viewersCanCopyContent] INTEGER,
     [copyRequiresWriterPermission] INTEGER,
     [writersCanShare] INTEGER,
     [folderColorRgb] TEXT,
     [quotaBytesUsed] TEXT,
     [isAppAuthorized] INTEGER,
     [linkShareMetadata] TEXT,
     FOREIGN KEY([_parent]) REFERENCES [drive_folders]([id]),
     FOREIGN KEY([_owner]) REFERENCES [drive_users]([permissionId]),
     FOREIGN KEY([lastModifyingUser]) REFERENCES [drive_users]([permissionId])
  );
  CREATE TABLE [drive_files] (
     [id] TEXT PRIMARY KEY,
     [_parent] TEXT,
     [_owner] TEXT,
     [lastModifyingUser] TEXT,
     [kind] TEXT,
     [name] TEXT,
     [mimeType] TEXT,
     [starred] INTEGER,
     [trashed] INTEGER,
     [explicitlyTrashed] INTEGER,
     [parents] TEXT,
     [spaces] TEXT,
     [version] TEXT,
     [webViewLink] TEXT,
     [iconLink] TEXT,
     [hasThumbnail] INTEGER,
     [thumbnailVersion] TEXT,
     [viewedByMe] INTEGER,
     [createdTime] TEXT,
     [modifiedTime] TEXT,
     [modifiedByMe] INTEGER,
     [shared] INTEGER,
     [ownedByMe] INTEGER,
     [viewersCanCopyContent] INTEGER,
     [copyRequiresWriterPermission] INTEGER,
     [writersCanShare] INTEGER,
     [quotaBytesUsed] TEXT,
     [isAppAuthorized] INTEGER,
     [linkShareMetadata] TEXT,
     FOREIGN KEY([_parent]) REFERENCES [drive_folders]([id]),
     FOREIGN KEY([_owner]) REFERENCES [drive_users]([permissionId]),
     FOREIGN KEY([lastModifyingUser]) REFERENCES [drive_users]([permissionId])
  );"><pre>CREATE TABLE [drive_users] (
     [permissionId] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>,
     [kind] <span class="pl-k">TEXT</span>,
     [displayName] <span class="pl-k">TEXT</span>,
     [photoLink] <span class="pl-k">TEXT</span>,
     [me] <span class="pl-k">INTEGER</span>,
     [emailAddress] <span class="pl-k">TEXT</span>
  );
  CREATE TABLE [drive_folders] (
     [id] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>,
     [_parent] <span class="pl-k">TEXT</span>,
     [_owner] <span class="pl-k">TEXT</span>,
     [lastModifyingUser] <span class="pl-k">TEXT</span>,
     [kind] <span class="pl-k">TEXT</span>,
     [name] <span class="pl-k">TEXT</span>,
     [mimeType] <span class="pl-k">TEXT</span>,
     [starred] <span class="pl-k">INTEGER</span>,
     [trashed] <span class="pl-k">INTEGER</span>,
     [explicitlyTrashed] <span class="pl-k">INTEGER</span>,
     [parents] <span class="pl-k">TEXT</span>,
     [spaces] <span class="pl-k">TEXT</span>,
     [version] <span class="pl-k">TEXT</span>,
     [webViewLink] <span class="pl-k">TEXT</span>,
     [iconLink] <span class="pl-k">TEXT</span>,
     [hasThumbnail] <span class="pl-k">INTEGER</span>,
     [thumbnailVersion] <span class="pl-k">TEXT</span>,
     [viewedByMe] <span class="pl-k">INTEGER</span>,
     [createdTime] <span class="pl-k">TEXT</span>,
     [modifiedTime] <span class="pl-k">TEXT</span>,
     [modifiedByMe] <span class="pl-k">INTEGER</span>,
     [shared] <span class="pl-k">INTEGER</span>,
     [ownedByMe] <span class="pl-k">INTEGER</span>,
     [viewersCanCopyContent] <span class="pl-k">INTEGER</span>,
     [copyRequiresWriterPermission] <span class="pl-k">INTEGER</span>,
     [writersCanShare] <span class="pl-k">INTEGER</span>,
     [folderColorRgb] <span class="pl-k">TEXT</span>,
     [quotaBytesUsed] <span class="pl-k">TEXT</span>,
     [isAppAuthorized] <span class="pl-k">INTEGER</span>,
     [linkShareMetadata] <span class="pl-k">TEXT</span>,
     <span class="pl-k">FOREIGN KEY</span>([_parent]) <span class="pl-k">REFERENCES</span> [drive_folders]([id]),
     <span class="pl-k">FOREIGN KEY</span>([_owner]) <span class="pl-k">REFERENCES</span> [drive_users]([permissionId]),
     <span class="pl-k">FOREIGN KEY</span>([lastModifyingUser]) <span class="pl-k">REFERENCES</span> [drive_users]([permissionId])
  );
  CREATE TABLE [drive_files] (
     [id] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>,
     [_parent] <span class="pl-k">TEXT</span>,
     [_owner] <span class="pl-k">TEXT</span>,
     [lastModifyingUser] <span class="pl-k">TEXT</span>,
     [kind] <span class="pl-k">TEXT</span>,
     [name] <span class="pl-k">TEXT</span>,
     [mimeType] <span class="pl-k">TEXT</span>,
     [starred] <span class="pl-k">INTEGER</span>,
     [trashed] <span class="pl-k">INTEGER</span>,
     [explicitlyTrashed] <span class="pl-k">INTEGER</span>,
     [parents] <span class="pl-k">TEXT</span>,
     [spaces] <span class="pl-k">TEXT</span>,
     [version] <span class="pl-k">TEXT</span>,
     [webViewLink] <span class="pl-k">TEXT</span>,
     [iconLink] <span class="pl-k">TEXT</span>,
     [hasThumbnail] <span class="pl-k">INTEGER</span>,
     [thumbnailVersion] <span class="pl-k">TEXT</span>,
     [viewedByMe] <span class="pl-k">INTEGER</span>,
     [createdTime] <span class="pl-k">TEXT</span>,
     [modifiedTime] <span class="pl-k">TEXT</span>,
     [modifiedByMe] <span class="pl-k">INTEGER</span>,
     [shared] <span class="pl-k">INTEGER</span>,
     [ownedByMe] <span class="pl-k">INTEGER</span>,
     [viewersCanCopyContent] <span class="pl-k">INTEGER</span>,
     [copyRequiresWriterPermission] <span class="pl-k">INTEGER</span>,
     [writersCanShare] <span class="pl-k">INTEGER</span>,
     [quotaBytesUsed] <span class="pl-k">TEXT</span>,
     [isAppAuthorized] <span class="pl-k">INTEGER</span>,
     [linkShareMetadata] <span class="pl-k">TEXT</span>,
     <span class="pl-k">FOREIGN KEY</span>([_parent]) <span class="pl-k">REFERENCES</span> [drive_folders]([id]),
     <span class="pl-k">FOREIGN KEY</span>([_owner]) <span class="pl-k">REFERENCES</span> [drive_users]([permissionId]),
     <span class="pl-k">FOREIGN KEY</span>([lastModifyingUser]) <span class="pl-k">REFERENCES</span> [drive_users]([permissionId])
  );</pre></div>

  <h2 dir="auto"><a id="user-content-thumbnails" class="anchor" aria-hidden="true" href="#user-content-thumbnails"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Thumbnails</h2>
  <p dir="auto">You can construct a thumbnail image for a known file ID using the following URL:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="https://drive.google.com/thumbnail?sz=w800-h800&amp;id=FILE_ID"><pre class="notranslate"><code>https://drive.google.com/thumbnail?sz=w800-h800&amp;id=FILE_ID
  </code></pre></div>
  <p dir="auto">Users who are signed into Google Drive and have permission to view a file will be redirected to a thumbnail version of that file. You can tweak the <code>w800</code> and <code>h800</code> parameters to request different thumbnail sizes.</p>
  <h2 dir="auto"><a id="user-content-privacy-policy" class="anchor" aria-hidden="true" href="#user-content-privacy-policy"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Privacy policy</h2>
  <p dir="auto">This tool requests access to your Google Drive account in order to retrieve metadata about your files there. It also offers a feature that can download the content of those files.</p>
  <p dir="auto">The credentials used to access your account are stored in the <code>auth.json</code> file on your computer. The metadata and content retrieved from Google Drive is also stored only on your own personal computer.</p>
  <p dir="auto">At no point do the developers of this tool gain access to any of your data.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd google-drive-to-sqlite
  python -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd google-drive-to-sqlite
  python -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre class="notranslate"><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOG383_g
name: sqlite-colorbrewer
full_name: eyeseast/sqlite-colorbrewer
html_url: https://github.com/eyeseast/sqlite-colorbrewer
description: A custom function to use ColorBrewer scales in SQLite queries
created_at: 2022-02-19T21:53:46Z
updated_at: 2022-03-03T17:16:40Z
pushed_at: 2022-03-02T03:04:56Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # sqlite-colorbrewer

  [![PyPI](https://img.shields.io/pypi/v/sqlite-colorbrewer.svg)](https://pypi.org/project/sqlite-colorbrewer/)
  [![Changelog](https://img.shields.io/github/v/release/eyeseast/sqlite-colorbrewer?include_prereleases&label=changelog)](https://github.com/eyeseast/sqlite-colorbrewer/releases)
  [![Tests](https://github.com/eyeseast/sqlite-colorbrewer/workflows/Test/badge.svg)](https://github.com/eyeseast/sqlite-colorbrewer/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/eyeseast/sqlite-colorbrewer/blob/main/LICENSE)

  A custom function to use [ColorBrewer](https://colorbrewer2.org/) scales in SQLite queries.

  Colors are exported from [here](https://colorbrewer2.org/export/colorbrewer.json).

  ## Installation

  To install as a Python library and use with the [standard SQLite3 module](https://docs.python.org/3/library/sqlite3.html):

      pip install sqlite-colorbrewer

  To install this plugin in the same environment as Datasette.

      datasette install sqlite-colorbrewer

  ## Usage

  If you're using this library with Datasette, it will be automatically registered as a plugin and available for use in SQL queries, like so:

  ```sql
  SELECT colorbrewer('Blues', 9, 0);
  ```

  That will return a single value: `"rgb(247,251,255)"`

  To use with a SQLite connection outside of Datasette, use the `register` function:

  ```python
  >>> import sqlite3
  >>> import sqlite_colorbrewer

  >>> conn = sqlite3.connect(':memory')
  >>> sqlite_colorbrewer.register(conn)

  >>> cursor = conn.execute("SELECT colorbrewer('Blues', 9, 0);")
  >>> result = next(cursor)
  >>> print(result)
  rgb(247,251,255)
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd sqlite-colorbrewer
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

  To build `sqlite_colorbrewer/colorbrewer.py`:

      ./json_to_python.py
      black . # to format the resulting file

  ## ColorBrewer

  Copyright (c) 2002 Cynthia Brewer, Mark Harrower, and The Pennsylvania State University.

  Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

  See the [ColorBrewer updates](http://www.personal.psu.edu/cab38/ColorBrewer/ColorBrewer_updates.html) for updates to copyright information.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-sqlite-colorbrewer" class="anchor" aria-hidden="true" href="#user-content-sqlite-colorbrewer"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>sqlite-colorbrewer</h1>
  <p dir="auto"><a href="https://pypi.org/project/sqlite-colorbrewer/" rel="nofollow"><img src="https://camo.githubusercontent.com/04ce9615649f436d9d448e3c317bf8f12473237aa608429218b746755cbd0ef6/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d636f6c6f726272657765722e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/sqlite-colorbrewer.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/eyeseast/sqlite-colorbrewer/releases"><img src="https://camo.githubusercontent.com/638c41b57fe97aeaec1e6be4a0f1d0c69e4d9259585553a48a5f21ef40047542/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f65796573656173742f73716c6974652d636f6c6f726272657765723f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/eyeseast/sqlite-colorbrewer?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/eyeseast/sqlite-colorbrewer/actions?query=workflow%3ATest"><img src="https://github.com/eyeseast/sqlite-colorbrewer/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/eyeseast/sqlite-colorbrewer/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">A custom function to use <a href="https://colorbrewer2.org/" rel="nofollow">ColorBrewer</a> scales in SQLite queries.</p>
  <p dir="auto">Colors are exported from <a href="https://colorbrewer2.org/export/colorbrewer.json" rel="nofollow">here</a>.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">To install as a Python library and use with the <a href="https://docs.python.org/3/library/sqlite3.html" rel="nofollow">standard SQLite3 module</a>:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install sqlite-colorbrewer"><pre><code>pip install sqlite-colorbrewer
  </code></pre></div>
  <p dir="auto">To install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install sqlite-colorbrewer"><pre><code>datasette install sqlite-colorbrewer
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">If you're using this library with Datasette, it will be automatically registered as a plugin and available for use in SQL queries, like so:</p>
  <div class="highlight highlight-source-sql position-relative overflow-auto" data-snippet-clipboard-copy-content="SELECT colorbrewer('Blues', 9, 0);"><pre><span class="pl-k">SELECT</span> colorbrewer(<span class="pl-s"><span class="pl-pds">'</span>Blues<span class="pl-pds">'</span></span>, <span class="pl-c1">9</span>, <span class="pl-c1">0</span>);</pre></div>
  <p dir="auto">That will return a single value: <code>"rgb(247,251,255)"</code></p>
  <p dir="auto">To use with a SQLite connection outside of Datasette, use the <code>register</code> function:</p>
  <div class="highlight highlight-source-python position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; import sqlite3
  &gt;&gt;&gt; import sqlite_colorbrewer

  &gt;&gt;&gt; conn = sqlite3.connect(':memory')
  &gt;&gt;&gt; sqlite_colorbrewer.register(conn)

  &gt;&gt;&gt; cursor = conn.execute(&quot;SELECT colorbrewer('Blues', 9, 0);&quot;)
  &gt;&gt;&gt; result = next(cursor)
  &gt;&gt;&gt; print(result)
  rgb(247,251,255)"><pre><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-k">import</span> <span class="pl-s1">sqlite3</span>
  <span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-k">import</span> <span class="pl-s1">sqlite_colorbrewer</span>

  <span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">conn</span> <span class="pl-c1">=</span> <span class="pl-s1">sqlite3</span>.<span class="pl-en">connect</span>(<span class="pl-s">':memory'</span>)
  <span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">sqlite_colorbrewer</span>.<span class="pl-en">register</span>(<span class="pl-s1">conn</span>)

  <span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">cursor</span> <span class="pl-c1">=</span> <span class="pl-s1">conn</span>.<span class="pl-en">execute</span>(<span class="pl-s">"SELECT colorbrewer('Blues', 9, 0);"</span>)
  <span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-en">next</span>(<span class="pl-s1">cursor</span>)
  <span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-en">print</span>(<span class="pl-s1">result</span>)
  <span class="pl-en">rgb</span>(<span class="pl-c1">247</span>,<span class="pl-c1">251</span>,<span class="pl-c1">255</span>)</pre></div>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="cd sqlite-colorbrewer
  python3 -mvenv venv
  source venv/bin/activate"><pre><code>cd sqlite-colorbrewer
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre><code>pytest
  </code></pre></div>
  <p dir="auto">To build <code>sqlite_colorbrewer/colorbrewer.py</code>:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="./json_to_python.py
  black . # to format the resulting file"><pre><code>./json_to_python.py
  black . # to format the resulting file
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-colorbrewer" class="anchor" aria-hidden="true" href="#user-content-colorbrewer"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>ColorBrewer</h2>
  <p dir="auto">Copyright (c) 2002 Cynthia Brewer, Mark Harrower, and The Pennsylvania State University.</p>
  <p dir="auto">Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at</p>
  <p dir="auto"><a href="http://www.apache.org/licenses/LICENSE-2.0" rel="nofollow">http://www.apache.org/licenses/LICENSE-2.0</a></p>
  <p dir="auto">Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.</p>
  <p dir="auto">See the <a href="http://www.personal.psu.edu/cab38/ColorBrewer/ColorBrewer_updates.html" rel="nofollow">ColorBrewer updates</a> for updates to copyright information.</p>
  </article></div>


node_id: R_kgDOG5dZxg
name: datasette-redirect-forbidden
full_name: simonw/datasette-redirect-forbidden
html_url: https://github.com/simonw/datasette-redirect-forbidden
description: Redirect forbidden requests to a login page
created_at: 2022-02-23T20:59:26Z
updated_at: 2022-02-23T22:00:12Z
pushed_at: 2022-02-23T22:02:38Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-redirect-forbidden

  [![PyPI](https://img.shields.io/pypi/v/datasette-redirect-forbidden.svg)](https://pypi.org/project/datasette-redirect-forbidden/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-redirect-forbidden?include_prereleases&label=changelog)](https://github.com/simonw/datasette-redirect-forbidden/releases)
  [![Tests](https://github.com/simonw/datasette-redirect-forbidden/workflows/Test/badge.svg)](https://github.com/simonw/datasette-redirect-forbidden/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-redirect-forbidden/blob/main/LICENSE)

  Redirect forbidden requests to a login page

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-redirect-forbidden

  ## Usage

  Add the following to your `metadata.yml` (or `metadata.json`) file to configure the plugin:

  ```yaml
  plugins:
    datasette-redirect-forbidden:
      redirect_to: /-/login
  ```
  Any 403 forbidden pages will redirect to the specified page.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-redirect-forbidden
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-redirect-forbidden" class="anchor" aria-hidden="true" href="#user-content-datasette-redirect-forbidden"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-redirect-forbidden</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-redirect-forbidden/" rel="nofollow"><img src="https://camo.githubusercontent.com/5fd755c03dc30d32c237a39392ed5ba94a54e5f20a4bbd2c0522c69736d9ac0c/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d72656469726563742d666f7262696464656e2e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-redirect-forbidden.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-redirect-forbidden/releases"><img src="https://camo.githubusercontent.com/31dd293e33f6786f2b162731eb130a22c26c39fbbc48a29edcc03ad735e643bf/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d72656469726563742d666f7262696464656e3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-redirect-forbidden?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-redirect-forbidden/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-redirect-forbidden/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-redirect-forbidden/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Redirect forbidden requests to a login page</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-redirect-forbidden"><pre><code>$ datasette install datasette-redirect-forbidden
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Add the following to your <code>metadata.yml</code> (or <code>metadata.json</code>) file to configure the plugin:</p>
  <div class="highlight highlight-source-yaml position-relative overflow-auto" data-snippet-clipboard-copy-content="plugins:
    datasette-redirect-forbidden:
      redirect_to: /-/login"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-redirect-forbidden</span>:
      <span class="pl-ent">redirect_to</span>: <span class="pl-s">/-/login</span></pre></div>
  <p dir="auto">Any 403 forbidden pages will redirect to the specified page.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-redirect-forbidden
  python3 -mvenv venv
  source venv/bin/activate"><pre><code>cd datasette-redirect-forbidden
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre><code>pytest
  </code></pre></div>
  </article></div>

template_repository:
  {"id": 400878073, "node_id": "MDEwOlJlcG9zaXRvcnk0MDA4NzgwNzM=", "name": "datasette-plugin-template-repository", "full_name": "simonw/datasette-plugin-template-repository", "private": false, "owner": {"login": "simonw", "id": 9599, "node_id": "MDQ6VXNlcjk1OTk=", "avatar_url": "https://avatars.githubusercontent.com/u/9599?v=4", "gravatar_id": "", "url": "https://api.github.com/users/simonw", "html_url": "https://github.com/simonw", "followers_url": "https://api.github.com/users/simonw/followers", "following_url": "https://api.github.com/users/simonw/following{/other_user}", "gists_url": "https://api.github.com/users/simonw/gists{/gist_id}", "starred_url": "https://api.github.com/users/simonw/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/simonw/subscriptions", "organizations_url": "https://api.github.com/users/simonw/orgs", "repos_url": "https://api.github.com/users/simonw/repos", "events_url": "https://api.github.com/users/simonw/events{/privacy}", "received_events_url": "https://api.github.com/users/simonw/received_events", "type": "User", "site_admin": false}, "html_url": "https://github.com/simonw/datasette-plugin-template-repository", "description": "GitHub template repository for creating new Datasette plugins, using the simonw/datasette-plugin cookiecutter template", "fork": false, "url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository", "forks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/forks", "keys_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/keys{/key_id}", "collaborators_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/collaborators{/collaborator}", "teams_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/teams", "hooks_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/hooks", "issue_events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/events{/number}", "events_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/events", "assignees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/assignees{/user}", "branches_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/branches{/branch}", "tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/tags", "blobs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/blobs{/sha}", "git_tags_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/tags{/sha}", "git_refs_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/refs{/sha}", "trees_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/trees{/sha}", "statuses_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/statuses/{sha}", "languages_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/languages", "stargazers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/stargazers", "contributors_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contributors", "subscribers_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscribers", "subscription_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscription", "commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/commits{/sha}", "git_commits_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/commits{/sha}", "comments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/comments{/number}", "issue_comment_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/comments{/number}", "contents_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/contents/{+path}", "compare_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/compare/{base}...{head}", "merges_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/merges", "archive_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/{archive_format}{/ref}", "downloads_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/downloads", "issues_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues{/number}", "pulls_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/pulls{/number}", "milestones_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/milestones{/number}", "notifications_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/notifications{?since,all,participating}", "labels_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/labels{/name}", "releases_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/releases{/id}", "deployments_url": "https://api.github.com/repos/simonw/datasette-plugin-template-repository/deployments", "created_at": "2021-08-28T19:50:28Z", "updated_at": "2021-12-21T20:45:06Z", "pushed_at": "2021-12-21T20:45:02Z", "git_url": "git://github.com/simonw/datasette-plugin-template-repository.git", "ssh_url": "git@github.com:simonw/datasette-plugin-template-repository.git", "clone_url": "https://github.com/simonw/datasette-plugin-template-repository.git", "svn_url": "https://github.com/simonw/datasette-plugin-template-repository", "homepage": "", "size": 6, "stargazers_count": 12, "watchers_count": 12, "language": null, "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": true, "has_pages": false, "forks_count": 0, "mirror_url": null, "archived": false, "disabled": false, "open_issues_count": 0, "license": null, "allow_forking": true, "is_template": true, "topics": [], "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 12, "default_branch": "main", "permissions": {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}, "temp_clone_token": ""}


node_id: R_kgDOG-A5Ww
name: shot-scraper
full_name: simonw/shot-scraper
html_url: https://github.com/simonw/shot-scraper
description: A command-line utility for taking automated screenshots of websites
created_at: 2022-03-08T21:21:02Z
updated_at: 2022-11-15T14:38:08Z
pushed_at: 2022-11-16T04:28:52Z
homepage: https://shot-scraper.datasette.io
language: Python
license: apache-2.0
topics: ["playwright", "playwright-python", "scraping", "screenshot-utility", "screenshots"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # shot-scraper

  [![PyPI](https://img.shields.io/pypi/v/shot-scraper.svg)](https://pypi.org/project/shot-scraper/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/shot-scraper?include_prereleases&label=changelog)](https://github.com/simonw/shot-scraper/releases)
  [![Tests](https://github.com/simonw/shot-scraper/workflows/Test/badge.svg)](https://github.com/simonw/shot-scraper/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/shot-scraper/blob/master/LICENSE)

  A command-line utility for taking automated screenshots of websites

  For background on this project see [shot-scraper: automated screenshots for documentation, built on Playwright](https://simonwillison.net/2022/Mar/10/shot-scraper/).

  ## Documentation

  - [Full documentation for shot-scraper](https://shot-scraper.datasette.io/)
  - [Tutorial: Automating screenshots for the Datasette documentation using shot-scraper](https://simonwillison.net/2022/Oct/14/automating-screenshots/)
  - [Release notes](https://github.com/simonw/shot-scraper/releases)

  ## Get started with GitHub Actions

  To get started without installing any software, use the [shot-scraper-template](https://github.com/simonw/shot-scraper-template) template to create your own GitHub repository which takes screenshots of a page using `shot-scraper`. See [Instantly create a GitHub repository to take screenshots of a web page](https://simonwillison.net/2022/Mar/14/shot-scraper-template/) for details.

  ## Quick installation

  You can install the `shot-scraper` CLI tool using [pip](https://pip.pypa.io/):

      pip install shot-scraper
      # Now install the browser it needs:
      shot-scraper install

  ## Taking your first screenshot

  You can take a screenshot of a web page like this:

      shot-scraper https://datasette.io/

  This will create a screenshot in a file called `datasette-io.png`.

  Many more options are available, see [Taking a screenshot](https://shot-scraper.datasette.io/en/stable/screenshots.html) for details.

  ## Examples

  - The [shot-scraper-demo](https://github.com/simonw/shot-scraper-demo) repository uses this tool to capture recently spotted owls in El Granada, CA according to [this page](https://www.owlsnearme.com/?place=127871), and to  generate an annotated screenshot illustrating a Datasette feature as described [in my blog](https://simonwillison.net/2022/Mar/10/shot-scraper/#a-complex-example).
  - The [Datasette Documentation](https://docs.datasette.io/en/latest/) uses screenshots taken by `shot-scraper` running in the [simonw/datasette-screenshots](https://github.com/simonw/datasette-screenshots) GitHub repository, described in detail in [Automating screenshots for the Datasette documentation using shot-scraper](https://simonwillison.net/2022/Oct/14/automating-screenshots/).
  - Ben Welsh built [@newshomepages](https://twitter.com/newshomepages), a Twitter bot that uses `shot-scraper` and GitHub Actions to take screenshots of news website homepages and publish them to Twitter. The code for that lives in [palewire/news-homepages](https://github.com/palewire/news-homepages).
  - [scrape-hacker-news-by-domain](https://github.com/simonw/scrape-hacker-news-by-domain) uses `shot-scraper javascript` to scrape a web page. See [Scraping web pages from the command-line with shot-scraper](https://simonwillison.net/2022/Mar/14/scraping-web-pages-shot-scraper/) for details of how this works.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-shot-scraper" class="anchor" aria-hidden="true" href="#user-content-shot-scraper"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>shot-scraper</h1>
  <p dir="auto"><a href="https://pypi.org/project/shot-scraper/" rel="nofollow"><img src="https://camo.githubusercontent.com/286ea85d2fe4edc6a9b1fdb4649d30bdf8021bb7251f1952f5ac780efe108494/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73686f742d736372617065722e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/shot-scraper.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/shot-scraper/releases"><img src="https://camo.githubusercontent.com/efbd5167cbe1861a03ee215da1ed60e73452db7584ea177e1334c63c7b35897d/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f73686f742d736372617065723f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/shot-scraper?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/shot-scraper/actions?query=workflow%3ATest"><img src="https://github.com/simonw/shot-scraper/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/shot-scraper/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">A command-line utility for taking automated screenshots of websites</p>
  <p dir="auto">For background on this project see <a href="https://simonwillison.net/2022/Mar/10/shot-scraper/" rel="nofollow">shot-scraper: automated screenshots for documentation, built on Playwright</a>.</p>
  <h2 dir="auto"><a id="user-content-documentation" class="anchor" aria-hidden="true" href="#user-content-documentation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Documentation</h2>
  <ul dir="auto">
  <li><a href="https://shot-scraper.datasette.io/" rel="nofollow">Full documentation for shot-scraper</a></li>
  <li><a href="https://simonwillison.net/2022/Oct/14/automating-screenshots/" rel="nofollow">Tutorial: Automating screenshots for the Datasette documentation using shot-scraper</a></li>
  <li><a href="https://github.com/simonw/shot-scraper/releases">Release notes</a></li>
  </ul>
  <h2 dir="auto"><a id="user-content-get-started-with-github-actions" class="anchor" aria-hidden="true" href="#user-content-get-started-with-github-actions"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Get started with GitHub Actions</h2>
  <p dir="auto">To get started without installing any software, use the <a href="https://github.com/simonw/shot-scraper-template">shot-scraper-template</a> template to create your own GitHub repository which takes screenshots of a page using <code>shot-scraper</code>. See <a href="https://simonwillison.net/2022/Mar/14/shot-scraper-template/" rel="nofollow">Instantly create a GitHub repository to take screenshots of a web page</a> for details.</p>
  <h2 dir="auto"><a id="user-content-quick-installation" class="anchor" aria-hidden="true" href="#user-content-quick-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Quick installation</h2>
  <p dir="auto">You can install the <code>shot-scraper</code> CLI tool using <a href="https://pip.pypa.io/" rel="nofollow">pip</a>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install shot-scraper
  # Now install the browser it needs:
  shot-scraper install"><pre class="notranslate"><code>pip install shot-scraper
  # Now install the browser it needs:
  shot-scraper install
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-taking-your-first-screenshot" class="anchor" aria-hidden="true" href="#user-content-taking-your-first-screenshot"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Taking your first screenshot</h2>
  <p dir="auto">You can take a screenshot of a web page like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="shot-scraper https://datasette.io/"><pre class="notranslate"><code>shot-scraper https://datasette.io/
  </code></pre></div>
  <p dir="auto">This will create a screenshot in a file called <code>datasette-io.png</code>.</p>
  <p dir="auto">Many more options are available, see <a href="https://shot-scraper.datasette.io/en/stable/screenshots.html" rel="nofollow">Taking a screenshot</a> for details.</p>
  <h2 dir="auto"><a id="user-content-examples" class="anchor" aria-hidden="true" href="#user-content-examples"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Examples</h2>
  <ul dir="auto">
  <li>The <a href="https://github.com/simonw/shot-scraper-demo">shot-scraper-demo</a> repository uses this tool to capture recently spotted owls in El Granada, CA according to <a href="https://www.owlsnearme.com/?place=127871" rel="nofollow">this page</a>, and to  generate an annotated screenshot illustrating a Datasette feature as described <a href="https://simonwillison.net/2022/Mar/10/shot-scraper/#a-complex-example" rel="nofollow">in my blog</a>.</li>
  <li>The <a href="https://docs.datasette.io/en/latest/" rel="nofollow">Datasette Documentation</a> uses screenshots taken by <code>shot-scraper</code> running in the <a href="https://github.com/simonw/datasette-screenshots">simonw/datasette-screenshots</a> GitHub repository, described in detail in <a href="https://simonwillison.net/2022/Oct/14/automating-screenshots/" rel="nofollow">Automating screenshots for the Datasette documentation using shot-scraper</a>.</li>
  <li>Ben Welsh built <a href="https://twitter.com/newshomepages" rel="nofollow">@newshomepages</a>, a Twitter bot that uses <code>shot-scraper</code> and GitHub Actions to take screenshots of news website homepages and publish them to Twitter. The code for that lives in <a href="https://github.com/palewire/news-homepages">palewire/news-homepages</a>.</li>
  <li><a href="https://github.com/simonw/scrape-hacker-news-by-domain">scrape-hacker-news-by-domain</a> uses <code>shot-scraper javascript</code> to scrape a web page. See <a href="https://simonwillison.net/2022/Mar/14/scraping-web-pages-shot-scraper/" rel="nofollow">Scraping web pages from the command-line with shot-scraper</a> for details of how this works.</li>
  </ul>
  </article></div>


node_id: R_kgDOHAjKFQ
name: datasette-hashed-urls
full_name: simonw/datasette-hashed-urls
html_url: https://github.com/simonw/datasette-hashed-urls
description: Optimize Datasette performance behind a caching proxy
created_at: 2022-03-15T21:31:52Z
updated_at: 2022-03-17T03:00:34Z
pushed_at: 2022-03-24T17:58:05Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-hashed-urls

  [![PyPI](https://img.shields.io/pypi/v/datasette-hashed-urls.svg)](https://pypi.org/project/datasette-hashed-urls/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-hashed-urls?include_prereleases&label=changelog)](https://github.com/simonw/datasette-hashed-urls/releases)
  [![Tests](https://github.com/simonw/datasette-hashed-urls/workflows/Test/badge.svg)](https://github.com/simonw/datasette-hashed-urls/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-hashed-urls/blob/main/LICENSE)

  Optimize Datasette performance behind a caching proxy

  When you open a database file in immutable mode using the `-i` option, Datasette calculates a SHA-256 hash of the contents of that file on startup.

  This content hash can then optionally be used to create URLs that are guaranteed to change if the contents of the file changes in the future.

  The result is pages  that can be cached indefinitely by both browsers and caching proxies - providing a significant performance boost.

  ## Demo

  A demo of this plugin is running at https://datasette-hashed-urls.vercel.app/

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-hashed-urls

  ## Usage

  Once installed, this plugin will act on any immutable database files that are loaded into Datasette:

      datasette -i fixtures.db

  The database will automatically be renamed to incorporate a hash of the contents of the SQLite file - so the above database would be served as:

      http://127.0.0.1:8001/fixtures-aa7318b

  Every page that accesss that database, including JSON endpoints, will be served with the following far-future cache expiry header:

      cache-control: max-age=31536000, public

  Here `max-age=31536000` is the number of seconds in a year.

  A caching proxy such as Cloudflare can then be used to cache and accelerate content served by Datasette.

  When the database file is updated and the server is restarted, the hash will change and content will be served from a new URL. Any hits to the previous hashed URLs will be automatically redirected.

  If you run Datasette using the `--crossdb` option to enable [cross-database queries](https://docs.datasette.io/en/stable/sql_queries.html#cross-database-queries) the `_memory` database will also have a hash added to its URL - in this case, the hash will be a combination of the hashes of the other attached databases.

  ## Configuration

  You can use the `max_age` plugin configuration setting to change the cache duration specified in the `cache-control` HTTP header.

  To set the cache expiry time to one hour you would add this to your Datasette `metadata.json` configuration file:

  ```json
  {
      "plugins": {
          "datasette-hashed-urls": {
              "max_age": 3600
          }
      }
  }
  ```

  ## History

  This functionality used to ship as part of Datasette itself, as a feature called [Hashed URL mode](https://docs.datasette.io/en/0.60.2/performance.html#hashed-url-mode).

  That feature has been deprecated and will be removed in Datasette 1.0. This plugin should be used as an alternative.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-hashed-urls
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-hashed-urls" class="anchor" aria-hidden="true" href="#user-content-datasette-hashed-urls"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-hashed-urls</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-hashed-urls/" rel="nofollow"><img src="https://camo.githubusercontent.com/55153811ecb81f551b5a9e808170c99673e8aa31647e65e9c7d93780f7082f42/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6861736865642d75726c732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-hashed-urls.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-hashed-urls/releases"><img src="https://camo.githubusercontent.com/a583f3c3cf3ad04b522018036f27666125f6dc718c2addd80ed64834d9068677/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6861736865642d75726c733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-hashed-urls?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-hashed-urls/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-hashed-urls/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-hashed-urls/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Optimize Datasette performance behind a caching proxy</p>
  <p dir="auto">When you open a database file in immutable mode using the <code>-i</code> option, Datasette calculates a SHA-256 hash of the contents of that file on startup.</p>
  <p dir="auto">This content hash can then optionally be used to create URLs that are guaranteed to change if the contents of the file changes in the future.</p>
  <p dir="auto">The result is pages  that can be cached indefinitely by both browsers and caching proxies - providing a significant performance boost.</p>
  <h2 dir="auto"><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p dir="auto">A demo of this plugin is running at <a href="https://datasette-hashed-urls.vercel.app/" rel="nofollow">https://datasette-hashed-urls.vercel.app/</a></p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-hashed-urls"><pre><code>$ datasette install datasette-hashed-urls
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Once installed, this plugin will act on any immutable database files that are loaded into Datasette:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette -i fixtures.db"><pre><code>datasette -i fixtures.db
  </code></pre></div>
  <p dir="auto">The database will automatically be renamed to incorporate a hash of the contents of the SQLite file - so the above database would be served as:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="http://127.0.0.1:8001/fixtures-aa7318b"><pre><code>http://127.0.0.1:8001/fixtures-aa7318b
  </code></pre></div>
  <p dir="auto">Every page that accesss that database, including JSON endpoints, will be served with the following far-future cache expiry header:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="cache-control: max-age=31536000, public"><pre><code>cache-control: max-age=31536000, public
  </code></pre></div>
  <p dir="auto">Here <code>max-age=31536000</code> is the number of seconds in a year.</p>
  <p dir="auto">A caching proxy such as Cloudflare can then be used to cache and accelerate content served by Datasette.</p>
  <p dir="auto">When the database file is updated and the server is restarted, the hash will change and content will be served from a new URL. Any hits to the previous hashed URLs will be automatically redirected.</p>
  <p dir="auto">If you run Datasette using the <code>--crossdb</code> option to enable <a href="https://docs.datasette.io/en/stable/sql_queries.html#cross-database-queries" rel="nofollow">cross-database queries</a> the <code>_memory</code> database will also have a hash added to its URL - in this case, the hash will be a combination of the hashes of the other attached databases.</p>
  <h2 dir="auto"><a id="user-content-configuration" class="anchor" aria-hidden="true" href="#user-content-configuration"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Configuration</h2>
  <p dir="auto">You can use the <code>max_age</code> plugin configuration setting to change the cache duration specified in the <code>cache-control</code> HTTP header.</p>
  <p dir="auto">To set the cache expiry time to one hour you would add this to your Datasette <code>metadata.json</code> configuration file:</p>
  <div class="highlight highlight-source-json position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-hashed-urls&quot;: {
              &quot;max_age&quot;: 3600
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-hashed-urls"</span>: {
              <span class="pl-ent">"max_age"</span>: <span class="pl-c1">3600</span>
          }
      }
  }</pre></div>
  <h2 dir="auto"><a id="user-content-history" class="anchor" aria-hidden="true" href="#user-content-history"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>History</h2>
  <p dir="auto">This functionality used to ship as part of Datasette itself, as a feature called <a href="https://docs.datasette.io/en/0.60.2/performance.html#hashed-url-mode" rel="nofollow">Hashed URL mode</a>.</p>
  <p dir="auto">That feature has been deprecated and will be removed in Datasette 1.0. This plugin should be used as an alternative.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-hashed-urls
  python3 -mvenv venv
  source venv/bin/activate"><pre><code>cd datasette-hashed-urls
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell"><pre><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHENadA
name: datasette-packages
full_name: simonw/datasette-packages
html_url: https://github.com/simonw/datasette-packages
description: Show a list of currently installed Python packages
created_at: 2022-03-25T22:04:23Z
updated_at: 2022-03-25T22:04:45Z
pushed_at: 2022-07-03T02:41:55Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-packages

  [![PyPI](https://img.shields.io/pypi/v/datasette-packages.svg)](https://pypi.org/project/datasette-packages/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-packages?include_prereleases&label=changelog)](https://github.com/simonw/datasette-packages/releases)
  [![Tests](https://github.com/simonw/datasette-packages/workflows/Test/badge.svg)](https://github.com/simonw/datasette-packages/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-packages/blob/main/LICENSE)

  Show a list of currently installed Python packages

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-packages

  ## Usage

  Visit `/-/packages` to see a list of installed Python packages.

  Visit `/-/packages.json` to get that back as JSON.

  ## Demo

  The output of this plugin can be seen here:

  - https://latest-with-plugins.datasette.io/-/packages
  - https://latest-with-plugins.datasette.io/-/packages.json

  ## With datasette-graphql

  if you have version 2.1 or higher of the [datasette-graphql](https://datasette.io/plugins/datasette-graphql) plugin installed you can also query the list of packages using this GraphQL query:

  ```graphql
  {
    packages {
      name
      version
    }
  }
  ```
  [Demo of this query](https://latest-with-plugins.datasette.io/graphql?query=%7B%0A%20%20packages%20%7B%0A%20%20%20%20name%0A%20%20%20%20version%0A%20%20%7D%0A%7D).

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-packages
      python3 -mvenv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-packages" class="anchor" aria-hidden="true" href="#user-content-datasette-packages"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-packages</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-packages/" rel="nofollow"><img src="https://camo.githubusercontent.com/e627d38389969d85cfcb630a701966524caf79b08068f56eb1143679614ef233/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d7061636b616765732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-packages.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-packages/releases"><img src="https://camo.githubusercontent.com/091e917c5f8e80e63def650f03bc9ab9990fbc7e68f8bf87e280aa8d6002ed98/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d7061636b616765733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-packages?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-packages/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-packages/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-packages/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Show a list of currently installed Python packages</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-packages"><pre class="notranslate"><code>$ datasette install datasette-packages
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Visit <code>/-/packages</code> to see a list of installed Python packages.</p>
  <p dir="auto">Visit <code>/-/packages.json</code> to get that back as JSON.</p>
  <h2 dir="auto"><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p dir="auto">The output of this plugin can be seen here:</p>
  <ul dir="auto">
  <li><a href="https://latest-with-plugins.datasette.io/-/packages" rel="nofollow">https://latest-with-plugins.datasette.io/-/packages</a></li>
  <li><a href="https://latest-with-plugins.datasette.io/-/packages.json" rel="nofollow">https://latest-with-plugins.datasette.io/-/packages.json</a></li>
  </ul>
  <h2 dir="auto"><a id="user-content-with-datasette-graphql" class="anchor" aria-hidden="true" href="#user-content-with-datasette-graphql"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>With datasette-graphql</h2>
  <p dir="auto">if you have version 2.1 or higher of the <a href="https://datasette.io/plugins/datasette-graphql" rel="nofollow">datasette-graphql</a> plugin installed you can also query the list of packages using this GraphQL query:</p>
  <div class="highlight highlight-source-graphql notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
    packages {
      name
      version
    }
  }"><pre>{
    <span class="pl-v">packages</span> {
      <span class="pl-v">name</span>
      <span class="pl-v">version</span>
    }
  }</pre></div>
  <p dir="auto"><a href="https://latest-with-plugins.datasette.io/graphql?query=%7B%0A%20%20packages%20%7B%0A%20%20%20%20name%0A%20%20%20%20version%0A%20%20%7D%0A%7D" rel="nofollow">Demo of this query</a>.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-packages
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-packages
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHEfRqA
name: datasette-auth0
full_name: simonw/datasette-auth0
html_url: https://github.com/simonw/datasette-auth0
description: Datasette plugin that authenticates users using Auth0
created_at: 2022-03-26T21:19:31Z
updated_at: 2022-03-27T17:59:49Z
pushed_at: 2022-03-28T03:04:52Z
language: Python
license: apache-2.0
topics: ["auth0", "datasette", "datasette-plugin"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-auth0

  [![PyPI](https://img.shields.io/pypi/v/datasette-auth0.svg)](https://pypi.org/project/datasette-auth0/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-auth0?include_prereleases&label=changelog)](https://github.com/simonw/datasette-auth0/releases)
  [![Tests](https://github.com/simonw/datasette-auth0/workflows/Test/badge.svg)](https://github.com/simonw/datasette-auth0/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-auth0/blob/main/LICENSE)

  Datasette plugin that authenticates users using [Auth0](https://auth0.com/)

  See [Simplest possible OAuth authentication with Auth0](https://til.simonwillison.net/auth0/oauth-with-auth0) for more about how this plugin works.

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-auth0

  ## Demo

  You can try this out at [datasette-auth0-demo.datasette.io](https://datasette-auth0-demo.datasette.io/) - click on the top right menu icon and select "Sign in with Auth0".

  ## Initial configuration

  First, create a new application in Auth0. You will need the domain, client ID and client secret for that application.

  The domain should be something like `mysite.us.auth0.com`.

  Add `http://127.0.0.1:8001/-/auth0-callback` to the list of Allowed Callback URLs.

  Then configure these plugin secrets using `metadata.yml`:

  ```yaml
  plugins:
    datasette-auth0:
      domain:
        "$env": AUTH0_DOMAIN
      client_id:
        "$env": AUTH0_CLIENT_ID
      client_secret:
        "$env": AUTH0_CLIENT_SECRET
  ```
  Only the `client_secret` needs to be kept secret, but for consistency I recommend using the `$env` mechanism for all three.

  In development, you can run Datasette and pass in environment variables like this:
  ```
  AUTH0_DOMAIN="your-domain.us.auth0.com" \
  AUTH0_CLIENT_ID="...client-id-goes-here..." \
  AUTH0_CLIENT_SECRET="...secret-goes-here..." \
  datasette -m metadata.yml
  ```

  If you are deploying using `datasette publish` you can pass these using `--plugin-secret`. For example, to deploy using Cloud Run you might run the following:
  ```
  datasette publish cloudrun mydatabase.db \
  --install datasette-auth0 \
  --plugin-secret datasette-auth0 domain "your-domain.us.auth0.com" \
  --plugin-secret datasette-auth0 client_id "your-client-id" \
  --plugin-secret datasette-auth0 client_secret "your-client-secret" \
  --service datasette-auth0-demo
  ```
  Once your Datasette instance is deployed, you will need to add its callback URL to the "Allowed Callback URLs" list in Auth0.

  The callback URL should be something like:

      https://url-to-your-datasette/-/auth0-callback

  ## Usage

  Once installed, a "Sign in with Auth0" menu item will appear in the Datasette main menu.

  You can sign in and then visit the `/-/actor` page to see full details of the `auth0` profile that has been authenticated.

  You can then use [Datasette permissions](https://docs.datasette.io/en/stable/authentication.html#configuring-permissions-in-metadata-json) to grant or deny access to different parts of Datasette based on the authenticated user.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-auth0
      python3 -mvenv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-auth0" class="anchor" aria-hidden="true" href="#user-content-datasette-auth0"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-auth0</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-auth0/" rel="nofollow"><img src="https://camo.githubusercontent.com/f43e5887f316cf40043f9c8a820290d3d12ae6b474804d66627cadf27de5145b/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d61757468302e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-auth0.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-auth0/releases"><img src="https://camo.githubusercontent.com/62decb2ba747ff51dd7bd5bdb614a6973179ff5642425c8e8525764285bc372c/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d61757468303f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-auth0?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-auth0/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-auth0/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-auth0/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Datasette plugin that authenticates users using <a href="https://auth0.com/" rel="nofollow">Auth0</a></p>
  <p dir="auto">See <a href="https://til.simonwillison.net/auth0/oauth-with-auth0" rel="nofollow">Simplest possible OAuth authentication with Auth0</a> for more about how this plugin works.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-auth0"><pre><code>$ datasette install datasette-auth0
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p dir="auto">You can try this out at <a href="https://datasette-auth0-demo.datasette.io/" rel="nofollow">datasette-auth0-demo.datasette.io</a> - click on the top right menu icon and select "Sign in with Auth0".</p>
  <h2 dir="auto"><a id="user-content-initial-configuration" class="anchor" aria-hidden="true" href="#user-content-initial-configuration"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Initial configuration</h2>
  <p dir="auto">First, create a new application in Auth0. You will need the domain, client ID and client secret for that application.</p>
  <p dir="auto">The domain should be something like <code>mysite.us.auth0.com</code>.</p>
  <p dir="auto">Add <code>http://127.0.0.1:8001/-/auth0-callback</code> to the list of Allowed Callback URLs.</p>
  <p dir="auto">Then configure these plugin secrets using <code>metadata.yml</code>:</p>
  <div class="highlight highlight-source-yaml position-relative overflow-auto" data-snippet-clipboard-copy-content="plugins:
    datasette-auth0:
      domain:
        &quot;$env&quot;: AUTH0_DOMAIN
      client_id:
        &quot;$env&quot;: AUTH0_CLIENT_ID
      client_secret:
        &quot;$env&quot;: AUTH0_CLIENT_SECRET"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-auth0</span>:
      <span class="pl-ent">domain</span>:
        <span class="pl-s"><span class="pl-pds">"</span><span class="pl-ent">$env</span><span class="pl-pds">"</span></span>: <span class="pl-s">AUTH0_DOMAIN</span>
      <span class="pl-ent">client_id</span>:
        <span class="pl-s"><span class="pl-pds">"</span><span class="pl-ent">$env</span><span class="pl-pds">"</span></span>: <span class="pl-s">AUTH0_CLIENT_ID</span>
      <span class="pl-ent">client_secret</span>:
        <span class="pl-s"><span class="pl-pds">"</span><span class="pl-ent">$env</span><span class="pl-pds">"</span></span>: <span class="pl-s">AUTH0_CLIENT_SECRET</span></pre></div>
  <p dir="auto">Only the <code>client_secret</code> needs to be kept secret, but for consistency I recommend using the <code>$env</code> mechanism for all three.</p>
  <p dir="auto">In development, you can run Datasette and pass in environment variables like this:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="AUTH0_DOMAIN=&quot;your-domain.us.auth0.com&quot; \
  AUTH0_CLIENT_ID=&quot;...client-id-goes-here...&quot; \
  AUTH0_CLIENT_SECRET=&quot;...secret-goes-here...&quot; \
  datasette -m metadata.yml"><pre><code>AUTH0_DOMAIN="your-domain.us.auth0.com" \
  AUTH0_CLIENT_ID="...client-id-goes-here..." \
  AUTH0_CLIENT_SECRET="...secret-goes-here..." \
  datasette -m metadata.yml
  </code></pre></div>
  <p dir="auto">If you are deploying using <code>datasette publish</code> you can pass these using <code>--plugin-secret</code>. For example, to deploy using Cloud Run you might run the following:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish cloudrun mydatabase.db \
  --install datasette-auth0 \
  --plugin-secret datasette-auth0 domain &quot;your-domain.us.auth0.com&quot; \
  --plugin-secret datasette-auth0 client_id &quot;your-client-id&quot; \
  --plugin-secret datasette-auth0 client_secret &quot;your-client-secret&quot; \
  --service datasette-auth0-demo"><pre><code>datasette publish cloudrun mydatabase.db \
  --install datasette-auth0 \
  --plugin-secret datasette-auth0 domain "your-domain.us.auth0.com" \
  --plugin-secret datasette-auth0 client_id "your-client-id" \
  --plugin-secret datasette-auth0 client_secret "your-client-secret" \
  --service datasette-auth0-demo
  </code></pre></div>
  <p dir="auto">Once your Datasette instance is deployed, you will need to add its callback URL to the "Allowed Callback URLs" list in Auth0.</p>
  <p dir="auto">The callback URL should be something like:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="https://url-to-your-datasette/-/auth0-callback"><pre><code>https://url-to-your-datasette/-/auth0-callback
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Once installed, a "Sign in with Auth0" menu item will appear in the Datasette main menu.</p>
  <p dir="auto">You can sign in and then visit the <code>/-/actor</code> page to see full details of the <code>auth0</code> profile that has been authenticated.</p>
  <p dir="auto">You can then use <a href="https://docs.datasette.io/en/stable/authentication.html#configuring-permissions-in-metadata-json" rel="nofollow">Datasette permissions</a> to grant or deny access to different parts of Datasette based on the authenticated user.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-auth0
  python3 -mvenv venv
  source venv/bin/activate"><pre><code>cd datasette-auth0
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHI-jKw
name: pypi-to-sqlite
full_name: simonw/pypi-to-sqlite
html_url: https://github.com/simonw/pypi-to-sqlite
description: Load data about Python packages from PyPI into SQLite
created_at: 2022-04-07T23:09:02Z
updated_at: 2022-04-08T15:16:03Z
pushed_at: 2022-04-08T16:29:53Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # pypi-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/pypi-to-sqlite.svg)](https://pypi.org/project/pypi-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/pypi-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/pypi-to-sqlite/releases)
  [![Tests](https://github.com/simonw/pypi-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/pypi-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/pypi-to-sqlite/blob/master/LICENSE)

  Load data about Python packages from PyPI into SQLite

  ## Installation

  Install this tool using `pip`:

      pip install pypi-to-sqlite

  ## Usage

  To create a SQLite database with details of one or more packages, run:

      pypi-to-sqlite pypi.db datasette sqlite-utils

  You can also process JSON that you have previously saved to disk like so:

      curl -o datasette.json https://pypi.org/pypi/datasette/json
      pypi-to-sqlite pypi.db -f datasette.json

  The tool will create three tables: `packages`, `versions` and `releases`. The full table schema is shown below.

  To create the tables with a prefix, use `--prefix prefix`. For example:

      pypi-to-sqlite pypi.db datasette --prefix pypi_

  This will create tables called `pypi_packages`, `pypi_versions` and `pypi_releases`.

  ## Demo

  You can see examples of tables created using this tool running in [Datasette](https://datasette.io/) here:

  - [packages](https://datasette.io/content/pypi_packages)
  - [versions](https://datasette.io/content/pypi_versions)
  - [releases](https://datasette.io/content/pypi_releases)

  ## Database schema

  <!-- [[[cog
  import cog, json
  from pypi_to_sqlite import cli
  from click.testing import CliRunner
  import sqlite_utils
  import tempfile, pathlib
  tmpdir = pathlib.Path(tempfile.mkdtemp())
  db_path = str(tmpdir / "pypi.db")
  runner = CliRunner()
  result = runner.invoke(cli.cli, [db_path, "-f", "tests/datasette-block.json"])
  cog.out("```sql\n")
  cog.out(sqlite_utils.Database(db_path).schema)
  cog.out("\n```")
  ]]] -->
  ```sql
  CREATE TABLE [packages] (
     [name] TEXT PRIMARY KEY,
     [summary] TEXT,
     [classifiers] TEXT,
     [description] TEXT,
     [author] TEXT,
     [author_email] TEXT,
     [description_content_type] TEXT,
     [home_page] TEXT,
     [keywords] TEXT,
     [license] TEXT,
     [maintainer] TEXT,
     [maintainer_email] TEXT,
     [package_url] TEXT,
     [platform] TEXT,
     [project_url] TEXT,
     [project_urls] TEXT,
     [release_url] TEXT,
     [requires_dist] TEXT,
     [requires_python] TEXT,
     [version] TEXT,
     [yanked] INTEGER,
     [yanked_reason] TEXT
  );
  CREATE TABLE [versions] (
     [id] TEXT PRIMARY KEY,
     [package] TEXT REFERENCES [packages]([name]),
     [name] TEXT
  );
  CREATE TABLE [releases] (
     [md5_digest] TEXT PRIMARY KEY,
     [package] TEXT REFERENCES [packages]([name]),
     [version] TEXT REFERENCES [versions]([id]),
     [packagetype] TEXT,
     [filename] TEXT,
     [comment_text] TEXT,
     [digests] TEXT,
     [has_sig] INTEGER,
     [python_version] TEXT,
     [requires_python] TEXT,
     [size] INTEGER,
     [upload_time] TEXT,
     [upload_time_iso_8601] TEXT,
     [url] TEXT,
     [yanked] INTEGER,
     [yanked_reason] TEXT
  );
  ```
  <!-- [[[end]]] -->

  ## pypi-to-sqlite --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["--help"])
  cog.out("```\n")
  cog.out(result.output.replace("Usage: cli", "Usage: pypi-to-sqlite"))
  cog.out("\n```")
  ]]] -->
  ```
  Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]...

    Load data about Python packages from PyPI into SQLite

    Usage example:

        pypi-to-sqlite pypy.db datasette sqlite-utils

    Use -f to load data from a JSON file instead:

        pypi-to-sqlite pypy.db -f datasette.json

    Created tables will be packages, versions and releases

    To create tables called pypi_packages, pypi_versions, pypi_releases use
    --prefix pypi_:

        pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_

  Options:
    --version            Show the version and exit.
    -f, --file FILENAME  Import JSON from this file
    -d, --delay FLOAT    Wait this many seconds between requests
    --prefix TEXT        Prefix to use for the created database tables
    --help               Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd pypi-to-sqlite
      python -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-pypi-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-pypi-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>pypi-to-sqlite</h1>
  <p dir="auto"><a href="https://pypi.org/project/pypi-to-sqlite/" rel="nofollow"><img src="https://camo.githubusercontent.com/200eab1d4c07550744081184a86be6c72391ed7061054162919ec2ad2b48fa09/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f707970692d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/pypi-to-sqlite.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/pypi-to-sqlite/releases"><img src="https://camo.githubusercontent.com/5efc8be881faf986aca6a6396ff257b787a9be964ef6f1328e46568544be0847/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f707970692d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/pypi-to-sqlite?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/pypi-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/pypi-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/pypi-to-sqlite/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Load data about Python packages from PyPI into SQLite</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this tool using <code>pip</code>:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install pypi-to-sqlite"><pre><code>pip install pypi-to-sqlite
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">To create a SQLite database with details of one or more packages, run:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pypi-to-sqlite pypi.db datasette sqlite-utils"><pre><code>pypi-to-sqlite pypi.db datasette sqlite-utils
  </code></pre></div>
  <p dir="auto">You can also process JSON that you have previously saved to disk like so:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -o datasette.json https://pypi.org/pypi/datasette/json
  pypi-to-sqlite pypi.db -f datasette.json"><pre><code>curl -o datasette.json https://pypi.org/pypi/datasette/json
  pypi-to-sqlite pypi.db -f datasette.json
  </code></pre></div>
  <p dir="auto">The tool will create three tables: <code>packages</code>, <code>versions</code> and <code>releases</code>. The full table schema is shown below.</p>
  <p dir="auto">To create the tables with a prefix, use <code>--prefix prefix</code>. For example:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pypi-to-sqlite pypi.db datasette --prefix pypi_"><pre><code>pypi-to-sqlite pypi.db datasette --prefix pypi_
  </code></pre></div>
  <p dir="auto">This will create tables called <code>pypi_packages</code>, <code>pypi_versions</code> and <code>pypi_releases</code>.</p>
  <h2 dir="auto"><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p dir="auto">You can see examples of tables created using this tool running in <a href="https://datasette.io/" rel="nofollow">Datasette</a> here:</p>
  <ul dir="auto">
  <li><a href="https://datasette.io/content/pypi_packages" rel="nofollow">packages</a></li>
  <li><a href="https://datasette.io/content/pypi_versions" rel="nofollow">versions</a></li>
  <li><a href="https://datasette.io/content/pypi_releases" rel="nofollow">releases</a></li>
  </ul>
  <h2 dir="auto"><a id="user-content-database-schema" class="anchor" aria-hidden="true" href="#user-content-database-schema"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Database schema</h2>

  <div class="highlight highlight-source-sql position-relative overflow-auto" data-snippet-clipboard-copy-content="CREATE TABLE [packages] (
     [name] TEXT PRIMARY KEY,
     [summary] TEXT,
     [classifiers] TEXT,
     [description] TEXT,
     [author] TEXT,
     [author_email] TEXT,
     [description_content_type] TEXT,
     [home_page] TEXT,
     [keywords] TEXT,
     [license] TEXT,
     [maintainer] TEXT,
     [maintainer_email] TEXT,
     [package_url] TEXT,
     [platform] TEXT,
     [project_url] TEXT,
     [project_urls] TEXT,
     [release_url] TEXT,
     [requires_dist] TEXT,
     [requires_python] TEXT,
     [version] TEXT,
     [yanked] INTEGER,
     [yanked_reason] TEXT
  );
  CREATE TABLE [versions] (
     [id] TEXT PRIMARY KEY,
     [package] TEXT REFERENCES [packages]([name]),
     [name] TEXT
  );
  CREATE TABLE [releases] (
     [md5_digest] TEXT PRIMARY KEY,
     [package] TEXT REFERENCES [packages]([name]),
     [version] TEXT REFERENCES [versions]([id]),
     [packagetype] TEXT,
     [filename] TEXT,
     [comment_text] TEXT,
     [digests] TEXT,
     [has_sig] INTEGER,
     [python_version] TEXT,
     [requires_python] TEXT,
     [size] INTEGER,
     [upload_time] TEXT,
     [upload_time_iso_8601] TEXT,
     [url] TEXT,
     [yanked] INTEGER,
     [yanked_reason] TEXT
  );"><pre>CREATE TABLE [packages] (
     [name] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>,
     [summary] <span class="pl-k">TEXT</span>,
     [classifiers] <span class="pl-k">TEXT</span>,
     [description] <span class="pl-k">TEXT</span>,
     [author] <span class="pl-k">TEXT</span>,
     [author_email] <span class="pl-k">TEXT</span>,
     [description_content_type] <span class="pl-k">TEXT</span>,
     [home_page] <span class="pl-k">TEXT</span>,
     [keywords] <span class="pl-k">TEXT</span>,
     [license] <span class="pl-k">TEXT</span>,
     [maintainer] <span class="pl-k">TEXT</span>,
     [maintainer_email] <span class="pl-k">TEXT</span>,
     [package_url] <span class="pl-k">TEXT</span>,
     [platform] <span class="pl-k">TEXT</span>,
     [project_url] <span class="pl-k">TEXT</span>,
     [project_urls] <span class="pl-k">TEXT</span>,
     [release_url] <span class="pl-k">TEXT</span>,
     [requires_dist] <span class="pl-k">TEXT</span>,
     [requires_python] <span class="pl-k">TEXT</span>,
     [version] <span class="pl-k">TEXT</span>,
     [yanked] <span class="pl-k">INTEGER</span>,
     [yanked_reason] <span class="pl-k">TEXT</span>
  );
  CREATE TABLE [versions] (
     [id] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>,
     [package] <span class="pl-k">TEXT</span> <span class="pl-k">REFERENCES</span> [packages]([name]),
     [name] <span class="pl-k">TEXT</span>
  );
  CREATE TABLE [releases] (
     [md5_digest] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>,
     [package] <span class="pl-k">TEXT</span> <span class="pl-k">REFERENCES</span> [packages]([name]),
     [version] <span class="pl-k">TEXT</span> <span class="pl-k">REFERENCES</span> [versions]([id]),
     [packagetype] <span class="pl-k">TEXT</span>,
     [filename] <span class="pl-k">TEXT</span>,
     [comment_text] <span class="pl-k">TEXT</span>,
     [digests] <span class="pl-k">TEXT</span>,
     [has_sig] <span class="pl-k">INTEGER</span>,
     [python_version] <span class="pl-k">TEXT</span>,
     [requires_python] <span class="pl-k">TEXT</span>,
     [size] <span class="pl-k">INTEGER</span>,
     [upload_time] <span class="pl-k">TEXT</span>,
     [upload_time_iso_8601] <span class="pl-k">TEXT</span>,
     [url] <span class="pl-k">TEXT</span>,
     [yanked] <span class="pl-k">INTEGER</span>,
     [yanked_reason] <span class="pl-k">TEXT</span>
  );</pre></div>

  <h2 dir="auto"><a id="user-content-pypi-to-sqlite---help" class="anchor" aria-hidden="true" href="#user-content-pypi-to-sqlite---help"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>pypi-to-sqlite --help</h2>

  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]...

    Load data about Python packages from PyPI into SQLite

    Usage example:

        pypi-to-sqlite pypy.db datasette sqlite-utils

    Use -f to load data from a JSON file instead:

        pypi-to-sqlite pypy.db -f datasette.json

    Created tables will be packages, versions and releases

    To create tables called pypi_packages, pypi_versions, pypi_releases use
    --prefix pypi_:

        pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_

  Options:
    --version            Show the version and exit.
    -f, --file FILENAME  Import JSON from this file
    -d, --delay FLOAT    Wait this many seconds between requests
    --prefix TEXT        Prefix to use for the created database tables
    --help               Show this message and exit.
  "><pre><code>Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]...

    Load data about Python packages from PyPI into SQLite

    Usage example:

        pypi-to-sqlite pypy.db datasette sqlite-utils

    Use -f to load data from a JSON file instead:

        pypi-to-sqlite pypy.db -f datasette.json

    Created tables will be packages, versions and releases

    To create tables called pypi_packages, pypi_versions, pypi_releases use
    --prefix pypi_:

        pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_

  Options:
    --version            Show the version and exit.
    -f, --file FILENAME  Import JSON from this file
    -d, --delay FLOAT    Wait this many seconds between requests
    --prefix TEXT        Prefix to use for the created database tables
    --help               Show this message and exit.

  </code></pre></div>

  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="cd pypi-to-sqlite
  python -m venv venv
  source venv/bin/activate"><pre><code>cd pypi-to-sqlite
  python -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre><code>pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHPc0Nw
name: datasette-total-page-time
full_name: simonw/datasette-total-page-time
html_url: https://github.com/simonw/datasette-total-page-time
description: Add a note to the Datasette footer measuring the total page load time
created_at: 2022-04-26T22:09:58Z
updated_at: 2022-04-26T22:10:27Z
pushed_at: 2022-04-26T22:11:48Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-total-page-time

  [![PyPI](https://img.shields.io/pypi/v/datasette-total-page-time.svg)](https://pypi.org/project/datasette-total-page-time/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-total-page-time?include_prereleases&label=changelog)](https://github.com/simonw/datasette-total-page-time/releases)
  [![Tests](https://github.com/simonw/datasette-total-page-time/workflows/Test/badge.svg)](https://github.com/simonw/datasette-total-page-time/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-total-page-time/blob/main/LICENSE)

  Add a note to the Datasette footer measuring the total page load time

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-total-page-time

  ## Usage

  Once this plugin is installed, a note will appear in the footer of every page showing how long the page took to generate.

  > Queries took 326.74ms · Page took 386.310ms

  ## How it works

  Measuring how long a page takes to load and then injecting that note into the page is tricky, because you need to finish generating the page before you know how long it took to load it!

  This plugin uses the [asgi_wrapper](https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette) plugin hook to measure the time taken by Datasette and then inject the following JavaScript at the bottom of the response, after the closing `</html>` tag but with the correct measured value:

  ```html
  <script>
  let footer = document.querySelector("footer");
  if (footer) {
      let ms = 37.224;
      let s = ` &middot; Page took ${ms.toFixed(3)}ms`;
      footer.innerHTML += s;
  }
  </script>
  ```
  This script is injected only on pages with the `text/html` content type - so it should not affect JSON or CSV returned by Datasette.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-total-page-time
      python3 -mvenv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-total-page-time" class="anchor" aria-hidden="true" href="#user-content-datasette-total-page-time"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-total-page-time</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-total-page-time/" rel="nofollow"><img src="https://camo.githubusercontent.com/46f2a11c9943f96c31f5d07f9be81db92f572d8d4a0a89fd74eb607647893920/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d746f74616c2d706167652d74696d652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-total-page-time.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-total-page-time/releases"><img src="https://camo.githubusercontent.com/7fb0313d1610a325951cfd81a63bb460a452ac3be321a8a3e275296a9d65dcd2/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d746f74616c2d706167652d74696d653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-total-page-time?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-total-page-time/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-total-page-time/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-total-page-time/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Add a note to the Datasette footer measuring the total page load time</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-total-page-time"><pre class="notranslate"><code class="notranslate">datasette install datasette-total-page-time
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Once this plugin is installed, a note will appear in the footer of every page showing how long the page took to generate.</p>
  <blockquote>
  <p dir="auto">Queries took 326.74ms · Page took 386.310ms</p>
  </blockquote>
  <h2 dir="auto"><a id="user-content-how-it-works" class="anchor" aria-hidden="true" href="#user-content-how-it-works"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How it works</h2>
  <p dir="auto">Measuring how long a page takes to load and then injecting that note into the page is tricky, because you need to finish generating the page before you know how long it took to load it!</p>
  <p dir="auto">This plugin uses the <a href="https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette" rel="nofollow">asgi_wrapper</a> plugin hook to measure the time taken by Datasette and then inject the following JavaScript at the bottom of the response, after the closing <code>&lt;/html&gt;</code> tag but with the correct measured value:</p>
  <div class="highlight highlight-text-html-basic position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;script&gt;
  let footer = document.querySelector(&quot;footer&quot;);
  if (footer) {
      let ms = 37.224;
      let s = ` &amp;middot; Page took ${ms.toFixed(3)}ms`;
      footer.innerHTML += s;
  }
  &lt;/script&gt;"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span>
  <span class="pl-k">let</span> <span class="pl-s1">footer</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">querySelector</span><span class="pl-kos">(</span><span class="pl-s">"footer"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">footer</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
      <span class="pl-k">let</span> <span class="pl-s1">ms</span> <span class="pl-c1">=</span> <span class="pl-c1">37.224</span><span class="pl-kos">;</span>
      <span class="pl-k">let</span> <span class="pl-s1">s</span> <span class="pl-c1">=</span> <span class="pl-s">` &amp;middot; Page took <span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">ms</span><span class="pl-kos">.</span><span class="pl-en">toFixed</span><span class="pl-kos">(</span><span class="pl-c1">3</span><span class="pl-kos">)</span><span class="pl-kos">}</span></span>ms`</span><span class="pl-kos">;</span>
      <span class="pl-s1">footer</span><span class="pl-kos">.</span><span class="pl-c1">innerHTML</span> <span class="pl-c1">+=</span> <span class="pl-s1">s</span><span class="pl-kos">;</span>
  <span class="pl-kos">}</span>
  <span class="pl-kos">&lt;/</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span></pre></div>
  <p dir="auto">This script is injected only on pages with the <code>text/html</code> content type - so it should not affect JSON or CSV returned by Datasette.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-total-page-time
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code class="notranslate">cd datasette-total-page-time
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code class="notranslate">pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code class="notranslate">pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHPj_Zg
name: datasette-gzip
full_name: simonw/datasette-gzip
html_url: https://github.com/simonw/datasette-gzip
description: Add gzip compression to Datasette
created_at: 2022-04-27T06:54:52Z
updated_at: 2022-04-28T03:19:50Z
pushed_at: 2022-04-28T16:06:51Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-gzip

  [![PyPI](https://img.shields.io/pypi/v/datasette-gzip.svg)](https://pypi.org/project/datasette-gzip/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-gzip?include_prereleases&label=changelog)](https://github.com/simonw/datasette-gzip/releases)
  [![Tests](https://github.com/simonw/datasette-gzip/workflows/Test/badge.svg)](https://github.com/simonw/datasette-gzip/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-gzip/blob/main/LICENSE)

  Add gzip compression to Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-gzip

  ## Usage

  Once installed, Datasette will obey the `Accept-Encoding:` header sent by browsers or other user agents and return content compressed in the most appropriate way.

  This plugin is a thin wrapper for the [asgi-gzip library](https://github.com/simonw/asgi-gzip), which extracts the [GzipMiddleware](https://www.starlette.io/middleware/#gzipmiddleware) from Starlette.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-gzip
      python3 -mvenv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-gzip" class="anchor" aria-hidden="true" href="#user-content-datasette-gzip"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-gzip</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-gzip/" rel="nofollow"><img src="https://camo.githubusercontent.com/2710ff5daf8e208e73a7b4cc0c852401cdb1214c7266da2fe9bd9d451a0b6710/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d677a69702e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-gzip.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-gzip/releases"><img src="https://camo.githubusercontent.com/0ac879f01bbec94064e9c423f0d6285e073140ec9583e348c450be58246a8ef5/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d677a69703f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-gzip?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-gzip/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-gzip/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-gzip/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Add gzip compression to Datasette</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-gzip"><pre class="notranslate"><code class="notranslate">datasette install datasette-gzip
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Once installed, Datasette will obey the <code>Accept-Encoding:</code> header sent by browsers or other user agents and return content compressed in the most appropriate way.</p>
  <p dir="auto">This plugin is a thin wrapper for the <a href="https://github.com/simonw/asgi-gzip">asgi-gzip library</a>, which extracts the <a href="https://www.starlette.io/middleware/#gzipmiddleware" rel="nofollow">GzipMiddleware</a> from Starlette.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-gzip
  python3 -mvenv venv
  source venv/bin/activate"><pre class="notranslate"><code class="notranslate">cd datasette-gzip
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code class="notranslate">pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code class="notranslate">pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHQLzfQ
name: datasette-copy-to-memory
full_name: simonw/datasette-copy-to-memory
html_url: https://github.com/simonw/datasette-copy-to-memory
description: Copy database files into an in-memory database on startup
created_at: 2022-04-28T20:02:21Z
updated_at: 2022-04-30T21:32:54Z
pushed_at: 2022-04-30T19:49:29Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-copy-to-memory

  [![PyPI](https://img.shields.io/pypi/v/datasette-copy-to-memory.svg)](https://pypi.org/project/datasette-copy-to-memory/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-copy-to-memory?include_prereleases&label=changelog)](https://github.com/simonw/datasette-copy-to-memory/releases)
  [![Tests](https://github.com/simonw/datasette-copy-to-memory/workflows/Test/badge.svg)](https://github.com/simonw/datasette-copy-to-memory/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-copy-to-memory/blob/main/LICENSE)

  Copy database files into an in-memory database on startup

  This plugin is **highly experimental**. It currently exists to support Datasette performance research, and is not designed for actual production usage.

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-copy-to-memory

  ## Usage

  On startup, Datasette will create an in-memory named database for each attached database. This database will have the same name but with `_memory` at the end.

  So running this:

      datasette fixtures.db

  Will serve two databases: the original at `/fixtures` and the in-memory copy at `/fixtures_memory`.

  ## Demo

  A demo is running on [latest-with-plugins.datasette.io](https://latest-with-plugins.datasette.io/) - the [/fixtures_memory](https://latest-with-plugins.datasette.io/fixtures_memory) table there is provided by this plugin.

  ## Configuration

  By default every attached database file will be loaded into a `_memory` copy.

  You can use plugin configuration to specify just a subset of the database. For example, to create `github_memory` but not `fixtures_memory` you would use the following `metadata.yml` file:

  ```yaml
  plugins:
    datasette-copy-to-memory:
      databases:
      - github
  ```
  Then start Datasette like this:

      datasette github.db fixtures.db -m metadata.yml

  If you don't want to have a `fixtures` and `fixtures_memory` database, you can use `replace: true` to have the plugin replace the file-backed database with the new in-memory one, reusing the same database name:

  ```yaml
  plugins:
    datasette-copy-to-memory:
      replace: true
  ```
  Then:

      datasette github.db fixtures.db -m metadata.yml

  This will result in both `/github` and `/fixtures` but no `/github_memory` or `/fixtures_memory`.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-copy-to-memory
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-copy-to-memory" class="anchor" aria-hidden="true" href="#user-content-datasette-copy-to-memory"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-copy-to-memory</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-copy-to-memory/" rel="nofollow"><img src="https://camo.githubusercontent.com/d3a367abdd18cd053406fa7ada0fe34b03d87d6a8ba6572c6c4722e46ab398b3/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d636f70792d746f2d6d656d6f72792e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-copy-to-memory.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-copy-to-memory/releases"><img src="https://camo.githubusercontent.com/688ecff26c66603539b1877da4753ec7bfdfab4555a6c4846e1ee0b1c428f60b/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d636f70792d746f2d6d656d6f72793f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-copy-to-memory?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-copy-to-memory/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-copy-to-memory/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-copy-to-memory/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Copy database files into an in-memory database on startup</p>
  <p dir="auto">This plugin is <strong>highly experimental</strong>. It currently exists to support Datasette performance research, and is not designed for actual production usage.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-copy-to-memory"><pre class="notranslate"><code class="notranslate">datasette install datasette-copy-to-memory
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">On startup, Datasette will create an in-memory named database for each attached database. This database will have the same name but with <code>_memory</code> at the end.</p>
  <p dir="auto">So running this:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette fixtures.db"><pre class="notranslate"><code class="notranslate">datasette fixtures.db
  </code></pre></div>
  <p dir="auto">Will serve two databases: the original at <code>/fixtures</code> and the in-memory copy at <code>/fixtures_memory</code>.</p>
  <h2 dir="auto"><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p dir="auto">A demo is running on <a href="https://latest-with-plugins.datasette.io/" rel="nofollow">latest-with-plugins.datasette.io</a> - the <a href="https://latest-with-plugins.datasette.io/fixtures_memory" rel="nofollow">/fixtures_memory</a> table there is provided by this plugin.</p>
  <h2 dir="auto"><a id="user-content-configuration" class="anchor" aria-hidden="true" href="#user-content-configuration"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Configuration</h2>
  <p dir="auto">By default every attached database file will be loaded into a <code>_memory</code> copy.</p>
  <p dir="auto">You can use plugin configuration to specify just a subset of the database. For example, to create <code>github_memory</code> but not <code>fixtures_memory</code> you would use the following <code>metadata.yml</code> file:</p>
  <div class="highlight highlight-source-yaml position-relative overflow-auto" data-snippet-clipboard-copy-content="plugins:
    datasette-copy-to-memory:
      databases:
      - github"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-copy-to-memory</span>:
      <span class="pl-ent">databases</span>:
      - <span class="pl-s">github</span></pre></div>
  <p dir="auto">Then start Datasette like this:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette github.db fixtures.db -m metadata.yml"><pre class="notranslate"><code class="notranslate">datasette github.db fixtures.db -m metadata.yml
  </code></pre></div>
  <p dir="auto">If you don't want to have a <code>fixtures</code> and <code>fixtures_memory</code> database, you can use <code>replace: true</code> to have the plugin replace the file-backed database with the new in-memory one, reusing the same database name:</p>
  <div class="highlight highlight-source-yaml position-relative overflow-auto" data-snippet-clipboard-copy-content="plugins:
    datasette-copy-to-memory:
      replace: true"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-copy-to-memory</span>:
      <span class="pl-ent">replace</span>: <span class="pl-c1">true</span></pre></div>
  <p dir="auto">Then:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette github.db fixtures.db -m metadata.yml"><pre class="notranslate"><code class="notranslate">datasette github.db fixtures.db -m metadata.yml
  </code></pre></div>
  <p dir="auto">This will result in both <code>/github</code> and <code>/fixtures</code> but no <code>/github_memory</code> or <code>/fixtures_memory</code>.</p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-copy-to-memory
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code class="notranslate">cd datasette-copy-to-memory
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code class="notranslate">pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code class="notranslate">pytest
  </code></pre></div>
  </article></div>


node_id: R_kgDOHSfuMg
name: datasette-upload-dbs
full_name: simonw/datasette-upload-dbs
html_url: https://github.com/simonw/datasette-upload-dbs
description: Upload SQLite database files to Datasette
created_at: 2022-05-05T23:36:51Z
updated_at: 2022-05-17T16:38:00Z
pushed_at: 2022-09-09T16:23:13Z
language: Python
license: apache-2.0
topics: []
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-upload-dbs

  [![PyPI](https://img.shields.io/pypi/v/datasette-upload-dbs.svg)](https://pypi.org/project/datasette-upload-dbs/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-upload-dbs?include_prereleases&label=changelog)](https://github.com/simonw/datasette-upload-dbs/releases)
  [![Tests](https://github.com/simonw/datasette-upload-dbs/workflows/Test/badge.svg)](https://github.com/simonw/datasette-upload-dbs/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-upload-dbs/blob/main/LICENSE)

  Upload SQLite database files to Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-upload-dbs

  ## Configuration

  This plugin requires you to configure a directory in which uploaded files will be stored.

  On startup, Datasette will automatically load any SQLite files that it finds in that directory. This means it is safe to restart your server in between file uploads.

  To configure the directory as `/home/datasette/uploads`, add this to a `metadata.yml` configuration file:

  ```yaml
  plugins:
    datasette-upload-dbs:
      directory: /home/datasette/uploads
  ```

  Or if you are using `metadata.json`:

  ```json
  {
    "plugins": {
      "datasette-upload-dbs": {
        "directory": "/home/datasette/uploads"
      }
    }
  }
  ```
  You can use `"."` for the current folder when the server starts, or `"uploads"` for a folder relative to that folder. The folder will be created on startup if it does not already exist.

  Then start Datasette like this:

      datasette -m metadata.yml

  ## Usage

  Only users with the `upload-dbs` permission will be able to upload files. The `root` user has this permission by default - other users can be granted access using permission plugins, see the [Permissions](https://docs.datasette.io/en/stable/authentication.html#permissions) documentation for details.

  To start Datasette as the root user, run this:

      datasette -m metadata.yml --root

  And follow the link that is displayd on the console.

  If a user has that permission they will see an "Upload database" link in the navigation menu.

  This will take them to `/-/upload-dbs` where they will be able to upload database files, by selecting them or by dragging them onto the drop area.

  ![Animated demo showing a file being dropped onto a box, then uploading and redirecting to the database page](https://github.com/simonw/datasette-upload-dbs/raw/main/upload-demo.gif)

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-upload-dbs
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-upload-dbs" class="anchor" aria-hidden="true" href="#user-content-datasette-upload-dbs"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-upload-dbs</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-upload-dbs/" rel="nofollow"><img src="https://camo.githubusercontent.com/dc98cd1fbc11a52c3610a10dd294832a4489aa3366a5e88b61ab3b7144bf7425/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d75706c6f61642d6462732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-upload-dbs.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-upload-dbs/releases"><img src="https://camo.githubusercontent.com/c00a952891cccd5b0a084a4a8aebf0f38a89669c5e7769a926b45fcaa5fa33bf/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d75706c6f61642d6462733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-upload-dbs?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-upload-dbs/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-upload-dbs/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-upload-dbs/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Upload SQLite database files to Datasette</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette install datasette-upload-dbs"><pre class="notranslate"><code>datasette install datasette-upload-dbs
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-configuration" class="anchor" aria-hidden="true" href="#user-content-configuration"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Configuration</h2>
  <p dir="auto">This plugin requires you to configure a directory in which uploaded files will be stored.</p>
  <p dir="auto">On startup, Datasette will automatically load any SQLite files that it finds in that directory. This means it is safe to restart your server in between file uploads.</p>
  <p dir="auto">To configure the directory as <code>/home/datasette/uploads</code>, add this to a <code>metadata.yml</code> configuration file:</p>
  <div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="plugins:
    datasette-upload-dbs:
      directory: /home/datasette/uploads"><pre><span class="pl-ent">plugins</span>:
    <span class="pl-ent">datasette-upload-dbs</span>:
      <span class="pl-ent">directory</span>: <span class="pl-s">/home/datasette/uploads</span></pre></div>
  <p dir="auto">Or if you are using <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    &quot;plugins&quot;: {
      &quot;datasette-upload-dbs&quot;: {
        &quot;directory&quot;: &quot;/home/datasette/uploads&quot;
      }
    }
  }"><pre>{
    <span class="pl-ent">"plugins"</span>: {
      <span class="pl-ent">"datasette-upload-dbs"</span>: {
        <span class="pl-ent">"directory"</span>: <span class="pl-s"><span class="pl-pds">"</span>/home/datasette/uploads<span class="pl-pds">"</span></span>
      }
    }
  }</pre></div>
  <p dir="auto">You can use <code>"."</code> for the current folder when the server starts, or <code>"uploads"</code> for a folder relative to that folder. The folder will be created on startup if it does not already exist.</p>
  <p dir="auto">Then start Datasette like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette -m metadata.yml"><pre class="notranslate"><code>datasette -m metadata.yml
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Only users with the <code>upload-dbs</code> permission will be able to upload files. The <code>root</code> user has this permission by default - other users can be granted access using permission plugins, see the <a href="https://docs.datasette.io/en/stable/authentication.html#permissions" rel="nofollow">Permissions</a> documentation for details.</p>
  <p dir="auto">To start Datasette as the root user, run this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette -m metadata.yml --root"><pre class="notranslate"><code>datasette -m metadata.yml --root
  </code></pre></div>
  <p dir="auto">And follow the link that is displayd on the console.</p>
  <p dir="auto">If a user has that permission they will see an "Upload database" link in the navigation menu.</p>
  <p dir="auto">This will take them to <code>/-/upload-dbs</code> where they will be able to upload database files, by selecting them or by dragging them onto the drop area.</p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer" href="https://github.com/simonw/datasette-upload-dbs/raw/main/upload-demo.gif"><img src="https://github.com/simonw/datasette-upload-dbs/raw/main/upload-demo.gif" alt="Animated demo showing a file being dropped onto a box, then uploading and redirecting to the database page" data-animated-image="" style="max-width: 100%;"></a></p>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-upload-dbs
  python3 -m venv venv
  source venv/bin/activate"><pre class="notranslate"><code>cd datasette-upload-dbs
  python3 -m venv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre class="notranslate"><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre class="notranslate"><code>pytest
  </code></pre></div>
  </article></div>


