# uses

_id: 61a01ca9a0e3de0a0f0af9691661af44ea6e149d
_path: for/data-journalism.md
title: Datasette for data journalism
text:
  Datasette was originally designed as a tool for data journalists, to help report on data-driven stories, crunch through large datasets and [publish the results](/for/publishing-data).

  [This demo](https://vimeo.com/436903714) from [SRCCON 2020](https://2020.srccon.org/) shows how Datasette can be used to analyze PPP loan data released by the Small Business Administration. 

  <iframe style="max-width: 100%" src="https://player.vimeo.com/video/436903714" width="640" height="400" frameborder="0" allow="autoplay; fullscreen" allowfullscreen></iframe>

  You can try the demo [here](https://sba-loans-covid-19.datasettes.com/), or read more about it in [this article](https://simonwillison.net/2020/Jul/9/sba-covid-19-ppp-loans/).

html:
  <p>Datasette was originally designed as a tool for data journalists, to help report on data-driven stories, crunch through large datasets and <a href="/for/publishing-data">publish the results</a>.</p>
  <p><a href="https://vimeo.com/436903714">This demo</a> from <a href="https://2020.srccon.org/">SRCCON 2020</a> shows how Datasette can be used to analyze PPP loan data released by the Small Business Administration. </p>
  <iframe style="max-width: 100%" src="https://player.vimeo.com/video/436903714" width="640" height="400" frameborder="0" allow="autoplay; fullscreen" allowfullscreen></iframe>

  <p>You can try the demo <a href="https://sba-loans-covid-19.datasettes.com/">here</a>, or read more about it in <a href="https://simonwillison.net/2020/Jul/9/sba-covid-19-ppp-loans/">this article</a>.</p>

summary:
  Datasette was originally designed as a tool for data journalists, to help report on data-driven stories, crunch through large datasets and publish the results.


_id: ddae37b01edec73668ab75f7e9799f91b580a554
_path: for/exploratory-analysis.md
title: Datasette for exploratory analysis
text:
  Your first step with any new data set should be to start exploring it. Datasette provides multiple tools for doing this.

  ## Start browsing the data

  Datasette's table interface lets you start scrolling through data straight away.

  ![Screenshot of the table interface](/static/datasette-screenshot.png)

  ## Facets

  Datasette automatically identifies columns with less than twenty unique values and gives you the option to facet by them - seeing the most common values, and selecting those to filter the set. This provides a powerful way to start spotting patterns in the data.

  ![Screenshot of the facet interface](/static/screenshots/facets.png)

  ## Visualization

  Plugins such as [datasette-vega](https://github.com/simonw/datasette-vega) and [datasette-cluster-map](https://github.com/simonw/datasette-cluster-map) provide tools for interactively visualizing data directly within the Datasette interface.

  ![Screenshot of datasette-cluster-map](/static/screenshots/datasette-cluster-map.png)

html:
  <p>Your first step with any new data set should be to start exploring it. Datasette provides multiple tools for doing this.</p>
  <h2>Start browsing the data</h2>
  <p>Datasette's table interface lets you start scrolling through data straight away.</p>
  <p><img alt="Screenshot of the table interface" src="/static/datasette-screenshot.png" /></p>
  <h2>Facets</h2>
  <p>Datasette automatically identifies columns with less than twenty unique values and gives you the option to facet by them - seeing the most common values, and selecting those to filter the set. This provides a powerful way to start spotting patterns in the data.</p>
  <p><img alt="Screenshot of the facet interface" src="/static/screenshots/facets.png" /></p>
  <h2>Visualization</h2>
  <p>Plugins such as <a href="https://github.com/simonw/datasette-vega">datasette-vega</a> and <a href="https://github.com/simonw/datasette-cluster-map">datasette-cluster-map</a> provide tools for interactively visualizing data directly within the Datasette interface.</p>
  <p><img alt="Screenshot of datasette-cluster-map" src="/static/screenshots/datasette-cluster-map.png" /></p>

summary:
  Import data from CSVs, JSON, database connections and more. Datasette will automatically show you patterns in your data and help you share your findings with your colleagues.


_id: 1940a8c4df2a2e0e7c95a457ce8dc4e7033c24d9
_path: for/publishing-data.md
title: Datasette for publishing data
text:
  Datasette was [originally inspired](https://simonwillison.net/2018/Aug/19/instantly-publish-datasette/) by work on [the Guardian Data Blog](https://www.theguardian.com/news/datablog/2011/jan/27/data-store-office-for-national-statistics), an initiative to publish the raw data behind the stories in the news.

  Datasette encourages sharing data as tables that can be explored by users and exported out as JSON or CSV. It's like publishing a CSV file but with much deeper flexibility for visitors to remix, visualize and export data.

  Data published by Datasette can be deployed to both traditional and serverless hosting platforms.

  The `datasette publish` command, [documented here](https://docs.datasette.io/en/stable/publish.html), can deploy to [Heroku](https://heroku.com/) and [Google Cloud Run](https://cloud.google.com/run). Plugins can be used to deploy Datasette on [Vercel](https://vercel.com/) or [fly.io](https://fly.io/).

  Datasette also works great on [Glitch](https://glitch.com/).

  When combined with GitHub Actions, Datasette can be used to automatically deploy new databases on a scheduled basis. See [Deploying a data API using GitHub Actions and Cloud Run](https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/) for more details on implementing this pattern.

html:
  <p>Datasette was <a href="https://simonwillison.net/2018/Aug/19/instantly-publish-datasette/">originally inspired</a> by work on <a href="https://www.theguardian.com/news/datablog/2011/jan/27/data-store-office-for-national-statistics">the Guardian Data Blog</a>, an initiative to publish the raw data behind the stories in the news.</p>
  <p>Datasette encourages sharing data as tables that can be explored by users and exported out as JSON or CSV. It's like publishing a CSV file but with much deeper flexibility for visitors to remix, visualize and export data.</p>
  <p>Data published by Datasette can be deployed to both traditional and serverless hosting platforms.</p>
  <p>The <code>datasette publish</code> command, <a href="https://docs.datasette.io/en/stable/publish.html">documented here</a>, can deploy to <a href="https://heroku.com/">Heroku</a> and <a href="https://cloud.google.com/run">Google Cloud Run</a>. Plugins can be used to deploy Datasette on <a href="https://vercel.com/">Vercel</a> or <a href="https://fly.io/">fly.io</a>.</p>
  <p>Datasette also works great on <a href="https://glitch.com/">Glitch</a>.</p>
  <p>When combined with GitHub Actions, Datasette can be used to automatically deploy new databases on a scheduled basis. See <a href="https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/">Deploying a data API using GitHub Actions and Cloud Run</a> for more details on implementing this pattern.</p>

summary:
  The built-in "datasette publish" command lets you instantly publish your data to hosting providers like Google Cloud Run, Heroku or Vercel.


_id: 0574db8535765cbe166676a6b73fd0205331313b
_path: for/rapid-prototyping.md
title: Datasette for rapid prototyping
text:
  Want to build a prototype? Datasette lets you turn any source of data into a powerful JSON or [GraphQL](https://github.com/simonw/datasette-graphql) API, ideal for use as the backend for an interactive prototype.

  Use Datasette with **[Observable notebooks](https://observablehq.com/)** to quickly try out new visualizations and explore the Datasette API using JavaScript.

  Datasette works as a great data for **[Jupyter notebooks](https://jupyter.org/)** too.

  - **[Analyzing ScotRail audio announcements with Datasette—from prototype to production](https://simonwillison.net/2022/Aug/21/scotrail/)** talks through the full process of creating an initial rapid prototype using Datasette Lite and growing that into a full production Datasette instance.
  - **[Fast Autocomplete Search for Your Website](https://simonwillison.net/2018/Dec/19/fast-autocomplete-search/)** shows how to build a full autocomplete search engine for your website, powered by Datasette and a few dozen lines of custom JavaScript.

html:
  <p>Want to build a prototype? Datasette lets you turn any source of data into a powerful JSON or <a href="https://github.com/simonw/datasette-graphql">GraphQL</a> API, ideal for use as the backend for an interactive prototype.</p>
  <p>Use Datasette with <strong><a href="https://observablehq.com/">Observable notebooks</a></strong> to quickly try out new visualizations and explore the Datasette API using JavaScript.</p>
  <p>Datasette works as a great data for <strong><a href="https://jupyter.org/">Jupyter notebooks</a></strong> too.</p>
  <ul>
  <li><strong><a href="https://simonwillison.net/2022/Aug/21/scotrail/">Analyzing ScotRail audio announcements with Datasette—from prototype to production</a></strong> talks through the full process of creating an initial rapid prototype using Datasette Lite and growing that into a full production Datasette instance.</li>
  <li><strong><a href="https://simonwillison.net/2018/Dec/19/fast-autocomplete-search/">Fast Autocomplete Search for Your Website</a></strong> shows how to build a full autocomplete search engine for your website, powered by Datasette and a few dozen lines of custom JavaScript.</li>
  </ul>

summary:
  Spin up a JSON (or GraphQL) API for any data in minutes. Use it to prototype and prove your ideas without building a custom backend.


_id: 863b689b3625af13bbba9bb9077c997fe3a04d13
_path: for/search.md
title: Datasette for search
text:
  Datasette is built on top of [SQLite](https://sqlite.org/), which includes a robust, full-featured [full-text search](https://www.sqlite.org/fts5.html) implementation.

  Datasette automatically detects tables which have been configured for full-text search and adds a search box which can be used by human site users or accessed from the JSON or CSV APIs.

  You can try this out now by [searching FARA records](https://fara.datasettes.com/) or [global power plants](https://global-power-plants.datasettes.com/global-power-plants/global-power-plants).

  ## Enabling search for a table

  The [sqlite-utils command-line tool](https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search) can be used to enable full-text search on a specific set of columns for a table:

      % sqlite-utils enable-fts mydb.db documents title summary

  Alternatively, you can configure search directly within the Datasette interface by installing the [datasette-configure-fts](https://github.com/simonw/datasette-configure-fts) plugin.

  [Fast Autocomplete Search for Your Website](https://simonwillison.net/2018/Dec/19/fast-autocomplete-search/) is a tutorial that shows how to write a scraper that populates a full-text search index in Datasette, then build a JavaScript autocomplete search interface using the Datasette JSON API.

  ## Implementing search with a canned query

  Datasette [canned queries](https://docs.datasette.io/en/stable/sql_queries.html#canned-queries) can be used to configure a SQL query which returns search results for a table, ordered by relevance.

  The search on [www.niche-museums.com](https://www.niche-museums.com/) uses this approach. Here's an example search, for `bones`:

  - [www.niche-museums.com/browse/search?q=bones](https://www.niche-museums.com/browse/search?q=bones)

  This uses the `search` canned query, which is [defined here]([https://github.com/simonw/til/blob/8f961be162868c53b5c484272091bdab703a747a/metadata.yaml#L16-L32](https://github.com/simonw/museums/blob/74e999c0e82781302bf0346a761ee5d88e168863/metadata.yaml#L55-L69)) and looks like this:

      select
        museums_fts.rank,
        museums.*
      from
        museums
        join museums_fts on museums.id = museums_fts.rowid
      where
        museums_fts match case
          :q
          when '' then '*'
          else escape_fts_query(:q)
        end
      order by
        museums_fts.rank

  Try [that query here](https://www.niche-museums.com/browse?sql=select%0D%0A++museums_fts.rank%2C%0D%0A++museums.*%0D%0Afrom%0D%0A++museums%0D%0A++join+museums_fts+on+museums.id+%3D+museums_fts.rowid%0D%0Awhere%0D%0A++museums_fts+match+case%0D%0A++++%3Aq%0D%0A++++when+%27%27+then+%27*%27%0D%0A++++else+escape_fts_query%28%3Aq%29%0D%0A++end%0D%0Aorder+by%0D%0A++museums_fts.rank&q=bones).

  The results are then rendered by [this custom template](https://github.com/simonw/museums/blob/74e999c0e82781302bf0346a761ee5d88e168863/templates/query-browse-search.html).

  ## Plugins for search

  The **[datasette-search-all](https://datasette.io/plugins/datasette-search-all)** plugin adds a search box which runs searches in parallel against all of the FTS-configured tables in all of the databases attached to Datasette.

  You can see that in action here: [fara.datasettes.com/-/search?q=manafort](https://fara.datasettes.com/-/search?q=manafort)

  **[dogsheep-beta](https://datasette.io/plugins/dogsheep-beta)** is a plugin which adds a customizable search engine that can create an index to search multiple tables at once with combined relevance scoring. It is used for the search on this site, for example [datasette.io/-/beta?q=fts](https://datasette.io/-/beta?q=fts) - you can read more about how it works in [Building a search engine for datasette.io](https://simonwillison.net/2020/Dec/19/dogsheep-beta/).

html:
  <p>Datasette is built on top of <a href="https://sqlite.org/">SQLite</a>, which includes a robust, full-featured <a href="https://www.sqlite.org/fts5.html">full-text search</a> implementation.</p>
  <p>Datasette automatically detects tables which have been configured for full-text search and adds a search box which can be used by human site users or accessed from the JSON or CSV APIs.</p>
  <p>You can try this out now by <a href="https://fara.datasettes.com/">searching FARA records</a> or <a href="https://global-power-plants.datasettes.com/global-power-plants/global-power-plants">global power plants</a>.</p>
  <h2>Enabling search for a table</h2>
  <p>The <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search">sqlite-utils command-line tool</a> can be used to enable full-text search on a specific set of columns for a table:</p>
  <pre><code>% sqlite-utils enable-fts mydb.db documents title summary
  </code></pre>
  <p>Alternatively, you can configure search directly within the Datasette interface by installing the <a href="https://github.com/simonw/datasette-configure-fts">datasette-configure-fts</a> plugin.</p>
  <p><a href="https://simonwillison.net/2018/Dec/19/fast-autocomplete-search/">Fast Autocomplete Search for Your Website</a> is a tutorial that shows how to write a scraper that populates a full-text search index in Datasette, then build a JavaScript autocomplete search interface using the Datasette JSON API.</p>
  <h2>Implementing search with a canned query</h2>
  <p>Datasette <a href="https://docs.datasette.io/en/stable/sql_queries.html#canned-queries">canned queries</a> can be used to configure a SQL query which returns search results for a table, ordered by relevance.</p>
  <p>The search on <a href="https://www.niche-museums.com/">www.niche-museums.com</a> uses this approach. Here's an example search, for <code>bones</code>:</p>
  <ul>
  <li><a href="https://www.niche-museums.com/browse/search?q=bones">www.niche-museums.com/browse/search?q=bones</a></li>
  </ul>
  <p>This uses the <code>search</code> canned query, which is <a href="[https://github.com/simonw/til/blob/8f961be162868c53b5c484272091bdab703a747a/metadata.yaml#L16-L32](https://github.com/simonw/museums/blob/74e999c0e82781302bf0346a761ee5d88e168863/metadata.yaml#L55-L69)">defined here</a> and looks like this:</p>
  <pre><code>select
    museums_fts.rank,
    museums.*
  from
    museums
    join museums_fts on museums.id = museums_fts.rowid
  where
    museums_fts match case
      :q
      when '' then '*'
      else escape_fts_query(:q)
    end
  order by
    museums_fts.rank
  </code></pre>
  <p>Try <a href="https://www.niche-museums.com/browse?sql=select%0D%0A++museums_fts.rank%2C%0D%0A++museums.*%0D%0Afrom%0D%0A++museums%0D%0A++join+museums_fts+on+museums.id+%3D+museums_fts.rowid%0D%0Awhere%0D%0A++museums_fts+match+case%0D%0A++++%3Aq%0D%0A++++when+%27%27+then+%27*%27%0D%0A++++else+escape_fts_query%28%3Aq%29%0D%0A++end%0D%0Aorder+by%0D%0A++museums_fts.rank&amp;q=bones">that query here</a>.</p>
  <p>The results are then rendered by <a href="https://github.com/simonw/museums/blob/74e999c0e82781302bf0346a761ee5d88e168863/templates/query-browse-search.html">this custom template</a>.</p>
  <h2>Plugins for search</h2>
  <p>The <strong><a href="https://datasette.io/plugins/datasette-search-all">datasette-search-all</a></strong> plugin adds a search box which runs searches in parallel against all of the FTS-configured tables in all of the databases attached to Datasette.</p>
  <p>You can see that in action here: <a href="https://fara.datasettes.com/-/search?q=manafort">fara.datasettes.com/-/search?q=manafort</a></p>
  <p><strong><a href="https://datasette.io/plugins/dogsheep-beta">dogsheep-beta</a></strong> is a plugin which adds a customizable search engine that can create an index to search multiple tables at once with combined relevance scoring. It is used for the search on this site, for example <a href="https://datasette.io/-/beta?q=fts">datasette.io/-/beta?q=fts</a> - you can read more about how it works in <a href="https://simonwillison.net/2020/Dec/19/dogsheep-beta/">Building a search engine for datasette.io</a>.</p>

summary:
  Datasette uses SQLite's built-in full-text search feature to provide faceted search over any correctly configured collection of records.


_id: 2a478757dc7f379d19b6c77d1710024b8f02a1e1
_path: for/serverless.md
title: Serverless read-only databases with Datasette
text:
  Serverless providers such as [Google Cloud Run](https://cloud.google.com/run) and [Vercel](https://vercel.com/) provide robust, scalable, inexpensive hosting for stateless web applications.

  Databases are usually provided as separate add-ons to serverless hosting. If your data is static or changes infrequently, Datasette can be used to unlock the benefits of serverless hosting without needing a separate hosted database.

  Datasette can be used to implement the [<strong>Baked Data</strong>](https://simonwillison.net/2021/Jul/28/baked-data/) pattern, where any structured data needed by your application is bundled and deployed with the application code itself, as part of the same container.

  This site is an example of Baked Data in action: datasette.io is deployed to Google Cloud Run by build scripts running [on GitHub Actions](https://github.com/simonw/datasette.io/blob/main/.github/workflows/deploy.yml).

  Many of the [Datasette examples](/examples) listed on this site are also deployed in this manner.

html:
  <p>Serverless providers such as <a href="https://cloud.google.com/run">Google Cloud Run</a> and <a href="https://vercel.com/">Vercel</a> provide robust, scalable, inexpensive hosting for stateless web applications.</p>
  <p>Databases are usually provided as separate add-ons to serverless hosting. If your data is static or changes infrequently, Datasette can be used to unlock the benefits of serverless hosting without needing a separate hosted database.</p>
  <p>Datasette can be used to implement the <a href="https://simonwillison.net/2021/Jul/28/baked-data/"><strong>Baked Data</strong></a> pattern, where any structured data needed by your application is bundled and deployed with the application code itself, as part of the same container.</p>
  <p>This site is an example of Baked Data in action: datasette.io is deployed to Google Cloud Run by build scripts running <a href="https://github.com/simonw/datasette.io/blob/main/.github/workflows/deploy.yml">on GitHub Actions</a>.</p>
  <p>Many of the <a href="/examples">Datasette examples</a> listed on this site are also deployed in this manner.</p>

summary:
  Serverless hosting usually depends on external hosted databases, but Datasette lets you implement the Baked Data pattern to bundle your data with your application code and deploy it to serverless providers.


_id: 41624e65228293a1622b9eee3a435699801dead1
_path: for/websites.md
title: Datasette for websites
text:
  Datasette templates can be used to build custom websites with Datasette under the hood. Examples include the site you are browsing right now.

  ## [datasette.io](https://datasette.io/)

  The official Datasette website runs on top of a customized Datasette instance.

  - [datasette.io, an official project website for Datasette](https://simonwillison.net/2020/Dec/13/datasette-io/) talks through the basics of how the site works.
  - [Building a search engine for datasette.io](https://simonwillison.net/2020/Dec/19/dogsheep-beta/) explains how the [site search](https://datasette.io/-/beta) works, using the [dogsheep-beta](https://datasette.io/plugins/dogsheep-beta) search plugin.
  - [The Baked Data architectural pattern](https://simonwillison.net/2021/Jul/28/baked-data/) describes the Baked Data architecture pattern used by the site, and provides a detailed description of how the build scripts for the site work.
  - [github.com/simonw/datasette.io](https://github.com/simonw/datasette.io) has the full source code for the site, including the GitHub Actions workflows that build the databases and deploy them to Google Cloud Run.
  - [datasette.io/content](https://datasette.io/content) is the Datasette interface for the main database that powers the site.

  ## [til.simonwillison.net](https://til.simonwillison.net/)

  A blog sharing TILs - Today I Learned snippets.

  - [Using a self-rewriting README powered by GitHub Actions to track TILs](https://simonwillison.net/2020/Apr/20/self-rewriting-readme/) introduces the site.
  - The site uses the [datasette-atom](https://datasette.io/plugins/datasette-atom) plugin to provide an Atom feed of new entries.
  - [til.simonwillison.net/tils](https://til.simonwillison.net/tils) exposes the underlying database.
  - The site is deployed to [Vercel](https://vercel.com/) using the [datasette-publish-vercel](https://datasette.io/plugins/datasette-publish-vercel) plugin.

  ## [Niche Museums](https://www.niche-museums.com/)

  [www.niche-museums.com](https://www.niche-museums.com/) is a directory of small and niche museums, powered by Datasette. Read [niche-museums.com, powered by Datasette](https://simonwillison.net/2019/Nov/25/niche-museums/) for details, or browse [the code](https://github.com/simonw/museums) on GitHub.

  ## [Rocky Beaches](https://www.rockybeaches.com/us/pillar-point)

  [Rocky Beaches](https://www.rockybeaches.com/us/pillar-point) by Natalie Downe shows the best times to go tidepooling at Pillar Point near Half Moon Bay, California. It runs on Datasette, using SQL queries to suggest the best times to go based on low tide times and sunrise/sunset and display recent observations recorded on [iNaturalist](https://www.inaturalist.org/).

  The source code is available at [github.com/natbat/rockybeaches](https://github.com/natbat/rockybeaches). The site uses the [datasette-graphql](https://datasette.io/plugins/datasette-graphql) in some of the templates, for example [here](https://github.com/natbat/rockybeaches/blob/e072d48479f1404baa206ee660ad228d9f95ac7a/templates/row-data-places.html#L273-L290).

html:
  <p>Datasette templates can be used to build custom websites with Datasette under the hood. Examples include the site you are browsing right now.</p>
  <h2><a href="https://datasette.io/">datasette.io</a></h2>
  <p>The official Datasette website runs on top of a customized Datasette instance.</p>
  <ul>
  <li><a href="https://simonwillison.net/2020/Dec/13/datasette-io/">datasette.io, an official project website for Datasette</a> talks through the basics of how the site works.</li>
  <li><a href="https://simonwillison.net/2020/Dec/19/dogsheep-beta/">Building a search engine for datasette.io</a> explains how the <a href="https://datasette.io/-/beta">site search</a> works, using the <a href="https://datasette.io/plugins/dogsheep-beta">dogsheep-beta</a> search plugin.</li>
  <li><a href="https://simonwillison.net/2021/Jul/28/baked-data/">The Baked Data architectural pattern</a> describes the Baked Data architecture pattern used by the site, and provides a detailed description of how the build scripts for the site work.</li>
  <li><a href="https://github.com/simonw/datasette.io">github.com/simonw/datasette.io</a> has the full source code for the site, including the GitHub Actions workflows that build the databases and deploy them to Google Cloud Run.</li>
  <li><a href="https://datasette.io/content">datasette.io/content</a> is the Datasette interface for the main database that powers the site.</li>
  </ul>
  <h2><a href="https://til.simonwillison.net/">til.simonwillison.net</a></h2>
  <p>A blog sharing TILs - Today I Learned snippets.</p>
  <ul>
  <li><a href="https://simonwillison.net/2020/Apr/20/self-rewriting-readme/">Using a self-rewriting README powered by GitHub Actions to track TILs</a> introduces the site.</li>
  <li>The site uses the <a href="https://datasette.io/plugins/datasette-atom">datasette-atom</a> plugin to provide an Atom feed of new entries.</li>
  <li><a href="https://til.simonwillison.net/tils">til.simonwillison.net/tils</a> exposes the underlying database.</li>
  <li>The site is deployed to <a href="https://vercel.com/">Vercel</a> using the <a href="https://datasette.io/plugins/datasette-publish-vercel">datasette-publish-vercel</a> plugin.</li>
  </ul>
  <h2><a href="https://www.niche-museums.com/">Niche Museums</a></h2>
  <p><a href="https://www.niche-museums.com/">www.niche-museums.com</a> is a directory of small and niche museums, powered by Datasette. Read <a href="https://simonwillison.net/2019/Nov/25/niche-museums/">niche-museums.com, powered by Datasette</a> for details, or browse <a href="https://github.com/simonw/museums">the code</a> on GitHub.</p>
  <h2><a href="https://www.rockybeaches.com/us/pillar-point">Rocky Beaches</a></h2>
  <p><a href="https://www.rockybeaches.com/us/pillar-point">Rocky Beaches</a> by Natalie Downe shows the best times to go tidepooling at Pillar Point near Half Moon Bay, California. It runs on Datasette, using SQL queries to suggest the best times to go based on low tide times and sunrise/sunset and display recent observations recorded on <a href="https://www.inaturalist.org/">iNaturalist</a>.</p>
  <p>The source code is available at <a href="https://github.com/natbat/rockybeaches">github.com/natbat/rockybeaches</a>. The site uses the <a href="https://datasette.io/plugins/datasette-graphql">datasette-graphql</a> in some of the templates, for example <a href="https://github.com/natbat/rockybeaches/blob/e072d48479f1404baa206ee660ad228d9f95ac7a/templates/row-data-places.html#L273-L290">here</a>.</p>

summary:
  Datasette can be used to power dynamic-static websites: sites that run on serverless hosting while providing dynamic data-backed functionality.


_id: ce0b74d7bd16f128e2bd4664b3aa910162b5373f
_path: for/geospatial.md
title: Datasette for geospatial analysis
text:
  The [SpatiaLite](https://www.gaia-gis.it/fossil/libspatialite/index) extension for SQLite can be used with Datasette to enable a full suite of geospatial SQL functions.

  ![A screenshot of Datasette running some geospatial plugins](https://camo.githubusercontent.com/56a2ed7e65e28e726d4c3abfbe56bd7960524e6af6512fc2cee083bdd7fcd863/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032312f6461746173657474652d6c6561666c65742d66726565647261772e706e67)

  Datasette has a number of plugins and tools that can be used to work with geospatial data.

  Geospatial plugins include:

  - [datasette-cluster-map](https://datasette.io/plugins/datasette-cluster-map) renders a map of markers for any table or query with `latitude` and `longitude` columns
  - [datasette-geojson](https://datasette.io/plugins/datasette-geojson) adds a `.geojson` extension which can export SpatiaLite geometries as [GeoJSON](https://geojson.org/)
  - [datasette-geojson-map](https://datasette.io/plugins/datasette-geojson-map) renders that GeoJSON output on a map
  - [datasette-leaflet-freedraw](https://datasette.io/plugins/datasette-leaflet-freedraw) adds an interface to filter geometries by drawing a shape on a map
  - [datasette-leaflet-geojson](https://datasette.io/plugins/datasette-leaflet-geojson) shows maps inline for any column that includes GeoJSON
  - [datasette-tiles](https://datasette.io/plugins/datasette-tiles) allows Datasette to serve map tile images stored using the [MBTiles](https://github.com/mapbox/mbtiles-spec) format

  Tools for working with geospatial data include:

  - [geojson-to-sqlite](https://datasette.io/tools/geojson-to-sqlite), a command line utility for loading GeoJSON data into a SQLite or SpatiaLite database
  - [shapefile-to-sqlite](https://datasette.io/tools/shapefile-to-sqlite), a command line utility for loading shapefiles into SQLite or SpatiaLite
  - [download-tiles](https://datasette.io/tools/download-tiles) can be used to download map tiles and store them in MBTiles, suitable for use with [datasette-tiles](https://datasette.io/plugins/datasette-tiles)

  The following tutorials provide more detail on how Datasette can be used for geospatial processing:

  - [GUnion to combine geometries in SpatiaLite](https://til.simonwillison.net/spatialite/gunion-to-combine-geometries) shows how to load Amtrak data using `geojson-to-sqlite` and render it using `datasette-geojson-map`
  - [KNN queries with SpatiaLite](https://til.simonwillison.net/spatialite/knn) shows how to use the KNN feature to run queries showing the nearest geometries to a point
  - [Natural Earth in SpatiaLite and Datasette](https://til.simonwillison.net/gis/natural-earth-in-spatialite-and-datasette) shows how to use the 791MB [Natural Earth](https://www.naturalearthdata.com/) SQLite vector database with Datasette
  - [Drawing shapes on a map to query a SpatiaLite database](https://simonwillison.net/2021/Jan/24/drawing-shapes-spatialite/) introduces the `datasette-leaflet-freedraw` plugin

html:
  <p>The <a href="https://www.gaia-gis.it/fossil/libspatialite/index">SpatiaLite</a> extension for SQLite can be used with Datasette to enable a full suite of geospatial SQL functions.</p>
  <p><img alt="A screenshot of Datasette running some geospatial plugins" src="https://camo.githubusercontent.com/56a2ed7e65e28e726d4c3abfbe56bd7960524e6af6512fc2cee083bdd7fcd863/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032312f6461746173657474652d6c6561666c65742d66726565647261772e706e67" /></p>
  <p>Datasette has a number of plugins and tools that can be used to work with geospatial data.</p>
  <p>Geospatial plugins include:</p>
  <ul>
  <li><a href="https://datasette.io/plugins/datasette-cluster-map">datasette-cluster-map</a> renders a map of markers for any table or query with <code>latitude</code> and <code>longitude</code> columns</li>
  <li><a href="https://datasette.io/plugins/datasette-geojson">datasette-geojson</a> adds a <code>.geojson</code> extension which can export SpatiaLite geometries as <a href="https://geojson.org/">GeoJSON</a></li>
  <li><a href="https://datasette.io/plugins/datasette-geojson-map">datasette-geojson-map</a> renders that GeoJSON output on a map</li>
  <li><a href="https://datasette.io/plugins/datasette-leaflet-freedraw">datasette-leaflet-freedraw</a> adds an interface to filter geometries by drawing a shape on a map</li>
  <li><a href="https://datasette.io/plugins/datasette-leaflet-geojson">datasette-leaflet-geojson</a> shows maps inline for any column that includes GeoJSON</li>
  <li><a href="https://datasette.io/plugins/datasette-tiles">datasette-tiles</a> allows Datasette to serve map tile images stored using the <a href="https://github.com/mapbox/mbtiles-spec">MBTiles</a> format</li>
  </ul>
  <p>Tools for working with geospatial data include:</p>
  <ul>
  <li><a href="https://datasette.io/tools/geojson-to-sqlite">geojson-to-sqlite</a>, a command line utility for loading GeoJSON data into a SQLite or SpatiaLite database</li>
  <li><a href="https://datasette.io/tools/shapefile-to-sqlite">shapefile-to-sqlite</a>, a command line utility for loading shapefiles into SQLite or SpatiaLite</li>
  <li><a href="https://datasette.io/tools/download-tiles">download-tiles</a> can be used to download map tiles and store them in MBTiles, suitable for use with <a href="https://datasette.io/plugins/datasette-tiles">datasette-tiles</a></li>
  </ul>
  <p>The following tutorials provide more detail on how Datasette can be used for geospatial processing:</p>
  <ul>
  <li><a href="https://til.simonwillison.net/spatialite/gunion-to-combine-geometries">GUnion to combine geometries in SpatiaLite</a> shows how to load Amtrak data using <code>geojson-to-sqlite</code> and render it using <code>datasette-geojson-map</code></li>
  <li><a href="https://til.simonwillison.net/spatialite/knn">KNN queries with SpatiaLite</a> shows how to use the KNN feature to run queries showing the nearest geometries to a point</li>
  <li><a href="https://til.simonwillison.net/gis/natural-earth-in-spatialite-and-datasette">Natural Earth in SpatiaLite and Datasette</a> shows how to use the 791MB <a href="https://www.naturalearthdata.com/">Natural Earth</a> SQLite vector database with Datasette</li>
  <li><a href="https://simonwillison.net/2021/Jan/24/drawing-shapes-spatialite/">Drawing shapes on a map to query a SpatiaLite database</a> introduces the <code>datasette-leaflet-freedraw</code> plugin</li>
  </ul>

summary:
  Combining Datasette with SpatiaLite provides a powerful set of tools for analyzing and visualizing geospatial data.


# datasette_repos

id: MDEwOlJlcG9zaXRvcnkzMzAzMDQ2Mjg=
nameWithOwner: bretwalker/datasette-query-history
createdAt: 2021-01-17T03:13:34Z
topics: []
latest_commit: c4d4cceb2a0b6f69dd96db1d7c1bf97194ad841a
openGraphImageUrl:
  https://opengraph.githubassets.com/3673283fecc017eddca2ec1dde793faf127b92808d4de01e1a9d5f2e6fc05bfd/bretwalker/datasette-query-history


id: MDEwOlJlcG9zaXRvcnkxOTc0MzExMDk=
nameWithOwner: dogsheep/dogsheep-beta
createdAt: 2019-07-17T17:07:26Z
topics: ["dogsheep", "search", "datasette-plugin", "datasette-io", "datasette", "datasette-tool"]
latest_commit: a895bc360f2738c7af43deda35c847f1ee5bff51
openGraphImageUrl:
  https://opengraph.githubassets.com/91c5de08e5c3017b42b057eb99c9b74ba05adf9d7b9c17d473b45ce91e7e55fa/dogsheep/dogsheep-beta


id: MDEwOlJlcG9zaXRvcnkyNTY4MzQ5MDc=
nameWithOwner: dogsheep/dogsheep-photos
createdAt: 2020-04-18T19:22:13Z
topics: ["dogsheep", "datasette-io", "datasette", "sqlite", "datasette-tool"]
latest_commit: 325aa38cb23d0757bb1335ee2ea94a082475a66e
openGraphImageUrl:
  https://opengraph.githubassets.com/2dde16b43209033cdff9ac7eb3147f58d2e278a513f1f5668fc3725bf0e09b2f/dogsheep/dogsheep-photos


id: MDEwOlJlcG9zaXRvcnkzMDMyMTgzNjk=
nameWithOwner: dogsheep/evernote-to-sqlite
createdAt: 2020-10-11T21:45:49Z
topics: ["evernote", "dogsheep", "datasette-io", "sqlite", "datasette-tool"]
latest_commit: fff89772b4404995400e33fe1d269050717ff4cf
openGraphImageUrl:
  https://opengraph.githubassets.com/dec637f260135e228652392d93ac4c0c411f6eabd0b28ce40fa0be15a91ece1f/dogsheep/evernote-to-sqlite


id: MDEwOlJlcG9zaXRvcnkyMDk1OTAzNDU=
nameWithOwner: dogsheep/genome-to-sqlite
createdAt: 2019-09-19T15:38:39Z
latest_commit: 6894c8bc767b4c0375ddb8b2f0d2fed857667ff5
openGraphImageUrl:
  https://opengraph.githubassets.com/8ef7ef26e5c309c10099e3592cab4158365f9821fec4d04ff2a881ae5c48908e/dogsheep/genome-to-sqlite

topics:
  ["sqlite", "datasette", "dogsheep", "personal-analytics", "genetics", "23andme", "datasette-io", "datasette-tool"]


id: MDEwOlJlcG9zaXRvcnkyMDY2NDk3NzA=
nameWithOwner: dogsheep/google-takeout-to-sqlite
createdAt: 2019-09-05T20:15:15Z
topics: ["google", "sqlite", "datasette", "dogsheep", "datasette-io", "datasette-tool"]
latest_commit: e54e544427f1cc3ea8189f0e95f54046301a8645
openGraphImageUrl:
  https://opengraph.githubassets.com/ded4b52cd3db17b9ff26efbe505f803baaa31c401e622473c1f86364dbda9105/dogsheep/google-takeout-to-sqlite


id: MDEwOlJlcG9zaXRvcnkyNDg5MDM1NDQ=
nameWithOwner: dogsheep/hacker-news-to-sqlite
createdAt: 2020-03-21T04:02:05Z
topics: ["datasette-io", "hacker-news", "dogsheep", "datasette", "datasette-tool"]
latest_commit: c5585c103d124b23ba1e163f8857d4ba49fe452a
openGraphImageUrl:
  https://opengraph.githubassets.com/f68795520af80280080e73e96d6d34d801b1e33b9a953d2102f0fd51494fc8bf/dogsheep/hacker-news-to-sqlite


id: MDEwOlJlcG9zaXRvcnkxOTc4ODIzODI=
nameWithOwner: dogsheep/healthkit-to-sqlite
createdAt: 2019-07-20T05:03:12Z
topics: ["healthkit", "sqlite", "datasette", "dogsheep", "datasette-io", "datasette-tool"]
latest_commit: 9fe3cb17e03d6c73222b63e643638cf951567c4c
openGraphImageUrl:
  https://opengraph.githubassets.com/0e1f8f3fa32e94e2bd70ebdfff9f36e902b0814c2a84990bf13c4357b2a74c5a/dogsheep/healthkit-to-sqlite


id: MDEwOlJlcG9zaXRvcnkyMDYyMDI4NjQ=
nameWithOwner: dogsheep/inaturalist-to-sqlite
createdAt: 2019-09-04T01:21:21Z
topics: ["inaturalist", "sqlite", "datasette", "dogsheep", "datasette-io", "datasette-tool"]
latest_commit: d888c7c2f02aa0dfb1559603f02357cd0089da11
openGraphImageUrl:
  https://opengraph.githubassets.com/239d2bccf254938574fa2cb9308166019f061537cc4f1f337ce88a9f8cea065c/dogsheep/inaturalist-to-sqlite


id: MDEwOlJlcG9zaXRvcnkyMTMyODY3NTI=
nameWithOwner: dogsheep/pocket-to-sqlite
createdAt: 2019-10-07T03:24:14Z
topics: ["sqlite", "pocket-api", "pocket", "dogsheep", "datasette", "datasette-io", "datasette-tool"]
latest_commit: 452c4925a48819489c0598a0d1875c0e6e9ee962
openGraphImageUrl:
  https://opengraph.githubassets.com/6e3bf2cce5409d69b86450bbf1175aa29a98104502933da6ca4edc6537183901/dogsheep/pocket-to-sqlite


id: MDEwOlJlcG9zaXRvcnkyMDU0MjkzNzU=
nameWithOwner: dogsheep/swarm-to-sqlite
createdAt: 2019-08-30T17:37:29Z
latest_commit: 719b6e96a016d0ca8b316d3bed9c2a7a0cb499ee
openGraphImageUrl:
  https://opengraph.githubassets.com/69f200952366b321648401a8a0d790cbf3a58b5bc4718148d58ec010b2326d41/dogsheep/swarm-to-sqlite

topics:
  ["foursquare", "sqlite", "foursquare-api", "datasette", "dogsheep", "datasette-io", "swarm", "datasette-tool"]


id: MDEwOlJlcG9zaXRvcnkyMDYxNTY4NjY=
nameWithOwner: dogsheep/twitter-to-sqlite
createdAt: 2019-09-03T19:30:08Z
topics: ["sqlite", "twitter", "twitter-api", "datasette", "dogsheep", "datasette-io", "datasette-tool"]
latest_commit: f09d611782a8372cfb002792dfa727325afb4db6
openGraphImageUrl:
  https://opengraph.githubassets.com/1b746f52eb42cc55c6490f314c8ee712b64ef6217829efaff5796a6f2d03f67b/dogsheep/twitter-to-sqlite


id: MDEwOlJlcG9zaXRvcnkzNzQ4NDYzMTE=
nameWithOwner: eyeseast/datasette-geojson
createdAt: 2021-06-08T01:33:19Z
topics: ["geojson", "gis", "sqlite", "datasette-io", "datasette-plugin"]
latest_commit: 602c4477dc7ddadb1c0a156cbcd2ef6688a5921d
openGraphImageUrl:
  https://opengraph.githubassets.com/a0bd228e917ed65b02cb323c8b614f9ede02a63fb933dd2fcd78c0cc90035ac4/eyeseast/datasette-geojson


id: MDEwOlJlcG9zaXRvcnkzODI5ODY1NjQ=
nameWithOwner: eyeseast/datasette-geojson-map
createdAt: 2021-07-05T01:54:13Z
topics: ["geojson", "gis", "datasette-plugin", "leafletjs", "datasette-io", "mapping"]
latest_commit: 32af5f1fd1a07278bbf8071fbb20a61e0f613246
openGraphImageUrl:
  https://opengraph.githubassets.com/f08faadc15eb10ef3e3beb3c4e0bebd46869dcacc3846e93134c413ab0b7f9a6/eyeseast/datasette-geojson-map


id: R_kgDOHcwLAg
nameWithOwner: eyeseast/datasette-query-files
createdAt: 2022-06-04T18:52:07Z
topics: ["datasette", "datasette-plugin", "python", "sql", "sqlite"]
latest_commit: ea4cf949b260fe5700de63cea0eaf4cd3821d767
openGraphImageUrl:
  https://opengraph.githubassets.com/3564eefd217f0a0069ee316bc868f457a5c1a2606d137795a1b5b45dafe69f4b/eyeseast/datasette-query-files


id: MDEwOlJlcG9zaXRvcnkyOTMzNjE1MTQ=
nameWithOwner: eyeseast/geocode-sqlite
createdAt: 2020-09-06T21:05:39Z
topics: []
latest_commit: 9ff6bdbe21a209d3023afee6cb8d85bd80725301
openGraphImageUrl:
  https://opengraph.githubassets.com/2479340d11f2737bd00da90c8d3a262466af97b6b6ece6b6cf7fffe16d27d2f0/eyeseast/geocode-sqlite


id: R_kgDOG383_g
nameWithOwner: eyeseast/sqlite-colorbrewer
createdAt: 2022-02-19T21:53:46Z
topics: []
latest_commit: ce9f1e0dda489e1a032d3579d45639ea7f0b597d
openGraphImageUrl:
  https://opengraph.githubassets.com/b5aa8f721adccfe33fcf41eafde9eb5db6214f56018916c457730f258ff1c3cd/eyeseast/sqlite-colorbrewer


id: R_kgDOIJsqeg
nameWithOwner: ocdtrekkie/datasette-external-links-new-tabs
createdAt: 2022-10-07T03:34:09Z
topics: []
latest_commit: 57f8662156b6b85328cc3d0b86ad984ca8dacd0d
openGraphImageUrl:
  https://opengraph.githubassets.com/0c4e3a16927bdfc6bb923e16aa5b7f487ae4b6f24c68af05644b1edb3f1a6cc7/ocdtrekkie/datasette-external-links-new-tabs


id: MDEwOlJlcG9zaXRvcnkzNjEwMTQyNzM=
nameWithOwner: rclement/datasette-dashboards
createdAt: 2021-04-23T21:56:48Z
openGraphImageUrl: https://repository-images.githubusercontent.com/361014273/89785380-bbae-11eb-804e-f8363a90af8b
latest_commit: 67157e32ce60209f4ff277ecc16e6cc4081d1866
topics:
  ["datasette", "datasette-io", "datasette-plugin", "dashboards", "data-visualization", "sql", "vega-lite"]


id: MDEwOlJlcG9zaXRvcnkyOTEzMzkwODY=
nameWithOwner: simonw/airtable-export
createdAt: 2020-08-29T19:51:37Z
topics: ["airtable", "airtable-api", "yaml", "datasette-io", "datasette-tool"]
latest_commit: fdb3decfaf844c8ce518aa0206bff9c17d019e13
openGraphImageUrl:
  https://opengraph.githubassets.com/d4d238b0fd7820f13840bc3fdc02be0e5cd638b8848dd067419ca6729a3c8292/simonw/airtable-export


id: MDEwOlJlcG9zaXRvcnkxNzUzMjE0OTc=
nameWithOwner: simonw/csv-diff
createdAt: 2019-03-13T01:11:26Z
topics: ["csv", "click", "diff", "git-scraping", "datasette-io", "datasette-tool", "csv-diff", "tsv-diff"]
latest_commit: 89e27428ca5f0a44ec534b8f9e6e2004e8f4e8da
openGraphImageUrl:
  https://opengraph.githubassets.com/89a75e3835b1cefcc2ba0c546211f8f6c5c9e3c8209af87738cb82499c6cae38/simonw/csv-diff


id: MDEwOlJlcG9zaXRvcnkxMTA1MDk4MTY=
nameWithOwner: simonw/csvs-to-sqlite
createdAt: 2017-11-13T06:38:21Z
topics: ["sqlite", "python", "csv", "pandas", "click", "datasette", "datasette-io", "datasette-tool"]
latest_commit: 1a7a9dfea69b5ea7d5c0e664ef85f68d77b53afe
openGraphImageUrl:
  https://opengraph.githubassets.com/bc4a9dabb0080cb6b720537e7f11c63c4a74b30dc79c50fafdbddc0aea163eeb/simonw/csvs-to-sqlite


id: MDEwOlJlcG9zaXRvcnkyMDkwOTEyNTY=
nameWithOwner: simonw/datasette-atom
createdAt: 2019-09-17T15:31:01Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: 2585ed0310aa04e619bf4aa6dc84cc91f578bf7d
openGraphImageUrl:
  https://opengraph.githubassets.com/a2a1fa34032ddacaefd39e6e1b1186084490d117efa5630efcf097ea49d32f0a/simonw/datasette-atom


id: MDEwOlJlcG9zaXRvcnkyMzYxMTA3NTk=
nameWithOwner: simonw/datasette-auth-existing-cookies
createdAt: 2020-01-25T01:20:31Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: 80c34eb84f05ba61684fb49eb1bf7f4d4f1ade06
openGraphImageUrl:
  https://opengraph.githubassets.com/83c46e7287fe28f94a501eac86c77ca3568408e2e044eff5d5df2e751f46a88d/simonw/datasette-auth-existing-cookies


id: MDEwOlJlcG9zaXRvcnkxOTUwODcxMzc=
nameWithOwner: simonw/datasette-auth-github
createdAt: 2019-07-03T16:02:53Z
topics: ["datasette", "datasette-plugin", "asgi", "datasette-io"]
latest_commit: 9564e25d756e328bf60e5d01f63c688ea2e44ca0
openGraphImageUrl:
  https://opengraph.githubassets.com/1c89de62320ffc62a8eedd19e6ce257d1e95c30f7228670489907a1c4a9f80c6/simonw/datasette-auth-github


id: MDEwOlJlcG9zaXRvcnkyNzkzNTcxMjM=
nameWithOwner: simonw/datasette-auth-passwords
createdAt: 2020-07-13T16:34:39Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 2ebd439bfddac4d6283b802ee000b20abce603bb
openGraphImageUrl:
  https://opengraph.githubassets.com/a3902ea86d8bc770496d00c6d98683ee51528b985945c458f068ad7f01ad3910/simonw/datasette-auth-passwords


id: MDEwOlJlcG9zaXRvcnkyNzE2NjUzMzY=
nameWithOwner: simonw/datasette-auth-tokens
createdAt: 2020-06-11T23:23:30Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 0390c07eb6b5c6ffc6053500b58c334ccab30c44
openGraphImageUrl:
  https://opengraph.githubassets.com/1b9a3431ddc985f85eb6aca57bd2efd7b2971f94ff2bd4c44f12db4b77313cb9/simonw/datasette-auth-tokens


id: R_kgDOHEfRqA
nameWithOwner: simonw/datasette-auth0
createdAt: 2022-03-26T21:19:31Z
topics: ["datasette", "datasette-plugin", "auth0"]
latest_commit: 2ffd71812267b4d39c9654449721e1106e3c6d3e
openGraphImageUrl:
  https://opengraph.githubassets.com/37a14d0153ed36d72cefc21824e6220e07435b7d554df54e99f76ffe51f2459e/simonw/datasette-auth0


id: MDEwOlJlcG9zaXRvcnkyOTMxNjQ0NDc=
nameWithOwner: simonw/datasette-backup
createdAt: 2020-09-05T22:33:29Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: b69eb953c480a0de189cb532aaf699d7cb831f47
openGraphImageUrl:
  https://opengraph.githubassets.com/e29df52d7b3c7afef0db5e67c8f6a0be0bb4dbfe170fa4b8aad95f4a000b01ae/simonw/datasette-backup


id: MDEwOlJlcG9zaXRvcnkzNDIxMjY2MTA=
nameWithOwner: simonw/datasette-block
createdAt: 2021-02-25T04:51:08Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 2c071fe71a6ad2fc4d4d397aa69aceda35eb3758
openGraphImageUrl:
  https://opengraph.githubassets.com/63502cd31770bc897665ad4ba6523b8d96994e306d28ea861fe964ab24771b0a/simonw/datasette-block


id: MDEwOlJlcG9zaXRvcnkyNzQyOTM1OTc=
nameWithOwner: simonw/datasette-block-robots
createdAt: 2020-06-23T02:52:23Z
topics: ["datasette", "datasette-io", "datasette-plugin", "robots-txt"]
latest_commit: 503bb6e5bda413ac664d2e9c32badc277d1405a7
openGraphImageUrl:
  https://opengraph.githubassets.com/1ec95a680529d896a79c4dccd7bb181aac994569a66fe4f9893965a159a2d683/simonw/datasette-block-robots


id: MDEwOlJlcG9zaXRvcnkxOTA5NTA3ODE=
nameWithOwner: simonw/datasette-bplist
createdAt: 2019-06-09T01:15:01Z
topics: ["datasette", "datasette-plugin", "bplist", "datasette-io"]
latest_commit: 218ab42bbac06871cab4748ae73497f1c3e5fc6f
openGraphImageUrl:
  https://opengraph.githubassets.com/545bc37ed309628405a73be1808b3597d67f09b3425e8ce68c299de6b349b1cb/simonw/datasette-bplist


id: MDEwOlJlcG9zaXRvcnkyNTU0NjAzNDc=
nameWithOwner: simonw/datasette-clone
createdAt: 2020-04-13T23:05:41Z
topics: ["datasette", "datasette-io", "datasette-tool"]
latest_commit: 4ded5c22a5ef6ca83bfa718a9187cc8c592b6815
openGraphImageUrl:
  https://opengraph.githubassets.com/b6f6f6a47039b739698b622b3123b86aad31beacb443926b9b7655af394592b0/simonw/datasette-clone


id: MDEwOlJlcG9zaXRvcnkyNDYxMDg1NjE=
nameWithOwner: simonw/datasette-column-inspect
createdAt: 2020-03-09T18:11:00Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 58ad947f78d237673730d56f63dd74af293ca6e9
openGraphImageUrl:
  https://opengraph.githubassets.com/537fd938d9717a416583660317e4a94fb5fbe6a90460af2d71544dbc02f3b2ff/simonw/datasette-column-inspect


id: MDEwOlJlcG9zaXRvcnkyMjg0ODU4MDY=
nameWithOwner: simonw/datasette-configure-asgi
createdAt: 2019-12-16T22:17:10Z
topics: ["datasette", "datasette-plugin", "asgi", "datasette-io"]
latest_commit: 82e8796745cc51dc8881e4089614fb05f0010a64
openGraphImageUrl:
  https://opengraph.githubassets.com/93f06191304b87026e0aadd214c87551f54a023ddff8d68997fd670987f4ae81/simonw/datasette-configure-asgi


id: MDEwOlJlcG9zaXRvcnkyNDM4ODcwMzY=
nameWithOwner: simonw/datasette-configure-fts
createdAt: 2020-02-29T01:50:57Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: eca742e5d4b9190fc22d68bc0a406c575e6d09a0
openGraphImageUrl:
  https://opengraph.githubassets.com/80ba8bfe1c7aeabdc21b9da3cbd761703fc38550720683d9d35a60e069d67d84/simonw/datasette-configure-fts


id: R_kgDOHQLzfQ
nameWithOwner: simonw/datasette-copy-to-memory
createdAt: 2022-04-28T20:02:21Z
topics: ["datasette", "datasette-plugin"]
latest_commit: d541c18a78ae6f707a8f9b1e7fc4c020a9f68f2e
openGraphImageUrl:
  https://opengraph.githubassets.com/32c76d6f5065cefc3e4b15e054988d5558afb06ba99062e90b82945101701266/simonw/datasette-copy-to-memory


id: MDEwOlJlcG9zaXRvcnkxOTU2OTY4MDQ=
nameWithOwner: simonw/datasette-cors
createdAt: 2019-07-07T21:03:11Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: 51f19528c2ae3e89a1519afbc15758346f93ad87
openGraphImageUrl:
  https://opengraph.githubassets.com/e50d3042fbccbe0023f0821fa9d613da16f3a1064b7d1bbd2ad2056ed67dbfa0/simonw/datasette-cors


id: MDEwOlJlcG9zaXRvcnkzMjcwODcyMDc=
nameWithOwner: simonw/datasette-css-properties
createdAt: 2021-01-05T18:38:07Z
topics: ["datasette-io", "datasette-plugin"]
latest_commit: b4d89b84bc40750bba1dad63cb5204cd1e0c4a82
openGraphImageUrl:
  https://opengraph.githubassets.com/4baa3d4f4ab1d1101054e88111a0d2bb957eb7bef4b20d241f096f77c28970a7/simonw/datasette-css-properties


id: MDEwOlJlcG9zaXRvcnkyOTkxNDM4NDk=
nameWithOwner: simonw/datasette-dateutil
createdAt: 2020-09-28T00:14:20Z
topics: ["datasette", "datasette-io", "datasette-plugin", "dateutil"]
latest_commit: c308245266d10e36a8f75562c2c353d37964183e
openGraphImageUrl:
  https://opengraph.githubassets.com/160c7e203f3ce176409b46ea6cf7bab5051dc33849abd66822b65ae99cc06298/simonw/datasette-dateutil


id: MDEwOlJlcG9zaXRvcnkyMzgzMzk0MTI=
nameWithOwner: simonw/datasette-debug-asgi
createdAt: 2020-02-05T00:57:09Z
topics: ["datasette-plugin", "datasette-io", "asgi"]
latest_commit: df72e40d779fac56a96ba1d3d40abdecb545da37
openGraphImageUrl:
  https://opengraph.githubassets.com/099c9bb41a728287f5cfa2f877b116b8b1fb646e61af9780cddc8b4e28994ed8/simonw/datasette-debug-asgi


id: MDEwOlJlcG9zaXRvcnkyNDc1Mjc0Mzg=
nameWithOwner: simonw/datasette-edit-schema
createdAt: 2020-03-15T18:34:06Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: a98fcdd8a7956fdf867f38ed88ecf9910918ee2b
openGraphImageUrl:
  https://opengraph.githubassets.com/0c37b3d48c2670882473d4ea3ff1d3c903f1579437b57369397d77a01438040a/simonw/datasette-edit-schema


id: MDEwOlJlcG9zaXRvcnkzMDg5MzAxMTg=
nameWithOwner: simonw/datasette-edit-templates
createdAt: 2020-10-31T16:58:29Z
topics: []
latest_commit: f772aff4a2a4080c949746668a8ec6302dbeb0d9
openGraphImageUrl:
  https://opengraph.githubassets.com/9fcb7f82a37d2a52de8cb6b553df452f283d626d9ae7308b48dce7962d039034/simonw/datasette-edit-templates


id: R_kgDOHmgSAw
nameWithOwner: simonw/datasette-expose-env
createdAt: 2022-07-03T21:14:29Z
topics: ["datasette", "datasette-plugin"]
latest_commit: 0dc709a6b3106681dab878735b0c26dd2697015f
openGraphImageUrl:
  https://opengraph.githubassets.com/936e53f7b6a95c3dfd17151e66d01ea86f9ca695de9843ace254560d9b8541b7/simonw/datasette-expose-env


id: MDEwOlJlcG9zaXRvcnkyNzU2MTU5NDc=
nameWithOwner: simonw/datasette-glitch
createdAt: 2020-06-28T15:41:25Z
topics: ["glitch", "datasette", "datasette-io", "datasette-plugin"]
latest_commit: ab01c36c2ffc3a8cdbfa714f14bf3143d7f88d79
openGraphImageUrl:
  https://opengraph.githubassets.com/14dd007feb26b03889ee2e98c98ab2863005c6b87f0cb78b3d6c59f40f329907/simonw/datasette-glitch


id: R_kgDOISPElA
nameWithOwner: simonw/datasette-gunicorn
createdAt: 2022-10-22T20:23:57Z
topics: ["datasette", "datasette-plugin"]
latest_commit: 8043e811f7a32ae8844cca50f7272c780b00469c
openGraphImageUrl:
  https://opengraph.githubassets.com/7987ec2b3ad8062667158d4eb20849b61d67841139e44983276f840c928d31fb/simonw/datasette-gunicorn


id: R_kgDOHAjKFQ
nameWithOwner: simonw/datasette-hashed-urls
createdAt: 2022-03-15T21:31:52Z
topics: []
latest_commit: 659614c23cbc544915079c44b09b09b090400ff8
openGraphImageUrl:
  https://opengraph.githubassets.com/00de07a3ef18f046e3aa8a6337abdb97f51348fb1becbc15a9b3cd99a7aed27b/simonw/datasette-hashed-urls


id: MDEwOlJlcG9zaXRvcnkyMTcyMTY3ODc=
nameWithOwner: simonw/datasette-haversine
createdAt: 2019-10-24T05:16:14Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: 2266a51f562906b934f79c65980cbf834e922b60
openGraphImageUrl:
  https://opengraph.githubassets.com/ec84c8105db6a8e0175125b9aeb15737b3cc1969f29e649891af2b32f58719f7/simonw/datasette-haversine


id: R_kgDOGeMHbg
nameWithOwner: simonw/datasette-hovercards
createdAt: 2021-12-02T17:11:59Z
topics: []
latest_commit: 9439ba46b7140fb03223faff0d21aeba5615a287
openGraphImageUrl:
  https://opengraph.githubassets.com/c4543cc6e5b38f4471807f931172ea69663f65cb02caab5d0cd609db489ba663/simonw/datasette-hovercards


id: MDEwOlJlcG9zaXRvcnkyOTkxOTgzNjk=
nameWithOwner: simonw/datasette-import-table
createdAt: 2020-09-28T05:30:07Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 29f1056c841507ef935aee6162450346873fa5e7
openGraphImageUrl:
  https://opengraph.githubassets.com/3d738247e55869ae32c48192d318dbae67633527e41e38a79c0e9718ae3b865d/simonw/datasette-import-table


id: MDEwOlJlcG9zaXRvcnkzMTI5MzQwMDE=
nameWithOwner: simonw/datasette-indieauth
createdAt: 2020-11-15T01:18:21Z
topics: ["datasette", "indieauth", "datasette-plugin", "datasette-io"]
latest_commit: a08ce67ddad6098b1240adbeff37d040e4df53b1
openGraphImageUrl:
  https://opengraph.githubassets.com/65a1b7c88841d17ae797774b3e697b565b1558affabeef547508307083c5987f/simonw/datasette-indieauth


id: MDEwOlJlcG9zaXRvcnkyNzU2MjQzNDY=
nameWithOwner: simonw/datasette-init
createdAt: 2020-06-28T16:26:29Z
topics: []
latest_commit: f978d0308f82641fcc6c71bab45d9a542b7bb66d
openGraphImageUrl:
  https://opengraph.githubassets.com/f74b2c72b85083bb03ed764d954d4ffad4baed09441265ac3b7a12970ef2d74c/simonw/datasette-init


id: MDEwOlJlcG9zaXRvcnkyODA1MDAwMjc=
nameWithOwner: simonw/datasette-insert
createdAt: 2020-07-17T18:40:34Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: dc0902c3064b29e267eada87502a4dd758646bcf
openGraphImageUrl:
  https://opengraph.githubassets.com/6f130b8f1a3a2af9f4c9478cc6e5f883f7f222ce8005bd5bf7f34aca9e67ca65/simonw/datasette-insert


id: MDEwOlJlcG9zaXRvcnkxNzQ3MTUxNTM=
nameWithOwner: simonw/datasette-jellyfish
createdAt: 2019-03-09T16:02:01Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: 74bf7fe96fd087fdffb1f665c18a9148e49ed003
openGraphImageUrl:
  https://opengraph.githubassets.com/3d12a06b331906111000ab826a8ed688930402937fb80c53e35f3d8687f127cc/simonw/datasette-jellyfish


id: MDEwOlJlcG9zaXRvcnkxODkzMjE2NzE=
nameWithOwner: simonw/datasette-jq
createdAt: 2019-05-30T01:06:31Z
topics: ["datasette", "datasette-plugin", "jq", "datasette-io"]
latest_commit: 7c67aa7293cab1441d633d5b67c642bd0aaa42c6
openGraphImageUrl:
  https://opengraph.githubassets.com/2d243e1ddd21124bed9fed39c4a845275fd1ed94307a116e8ccc8a9b61228e67/simonw/datasette-jq


id: MDEwOlJlcG9zaXRvcnkzMzE3MjA4MjQ=
nameWithOwner: simonw/datasette-leaflet
createdAt: 2021-01-21T18:41:19Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 04ea172c684554cedd7e8efa40406b76ad4187d4
openGraphImageUrl:
  https://opengraph.githubassets.com/71dc82e5b8d61da347788618e2516b8044fa3ca4b3e84cdfe22fb67f2d8e9aa0/simonw/datasette-leaflet


id: MDEwOlJlcG9zaXRvcnkxMzUwMDcyODc=
nameWithOwner: simonw/datasette-leaflet-geojson
createdAt: 2018-05-27T01:42:30Z
topics: ["datasette", "leaflet", "gis", "datasette-plugin", "datasette-io"]
latest_commit: 64713aa497750400b9ac2c12e8bb6ffab8eb77f3
openGraphImageUrl:
  https://opengraph.githubassets.com/50e9ae3400e19bed74bc4a75603a13b49de29eb21bdc6f81801bede4c56df8b9/simonw/datasette-leaflet-geojson


id: MDEwOlJlcG9zaXRvcnkyNDIyNjA1ODM=
nameWithOwner: simonw/datasette-mask-columns
createdAt: 2020-02-22T01:29:16Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 50acf9833954f230fd7938bffdbbad0f78c9af27
openGraphImageUrl:
  https://opengraph.githubassets.com/300c366683eff7212120a4f1d9226485ef88dc9fda26eaaee3f2ef1a78b7004f/simonw/datasette-mask-columns


id: MDEwOlJlcG9zaXRvcnkyNjE2MzQ4MDc=
nameWithOwner: simonw/datasette-media
createdAt: 2020-05-06T02:42:57Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: d853fa9a71bcbc966f93316d4533a6832d38e3e8
openGraphImageUrl:
  https://opengraph.githubassets.com/7068007b613463161b40c604b1684b7c2c852abf6e8a4e74131757fa73916f02/simonw/datasette-media


id: R_kgDOH2kMZA
nameWithOwner: simonw/datasette-mp3-audio
createdAt: 2022-08-20T16:20:06Z
topics: []
latest_commit: 4402168792f452a46ab7b488e40ec49cd4b12185
openGraphImageUrl:
  https://opengraph.githubassets.com/11494819d102de3d0eee14ea87d9abde7dc657383ca97ae7e584371e44c5ee25/simonw/datasette-mp3-audio


id: R_kgDOH4H41Q
nameWithOwner: simonw/datasette-multiline-links
createdAt: 2022-08-24T22:16:30Z
topics: []
latest_commit: c14032644c6c56bf5e4ea7bf78715db7e1d525b3
openGraphImageUrl:
  https://opengraph.githubassets.com/faeff45c1f87a30c95910f1fb8fa4eb7f4bc0b4f829d04b906460e120beb4585/simonw/datasette-multiline-links


id: R_kgDOHENadA
nameWithOwner: simonw/datasette-packages
createdAt: 2022-03-25T22:04:23Z
topics: []
latest_commit: 268201fede1956915a464a108a4dfb5cca595a22
openGraphImageUrl:
  https://opengraph.githubassets.com/b8d254ac3895e9714e3183fa185b28e13e6f3137d63ab8667adfec51d27b3faf/simonw/datasette-packages


id: MDEwOlJlcG9zaXRvcnkyNzE0MDg4OTU=
nameWithOwner: simonw/datasette-permissions-sql
createdAt: 2020-06-10T23:48:13Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: e0103ea1c13389391a3e40241485df45739aa638
openGraphImageUrl:
  https://opengraph.githubassets.com/5871124e3f41f99149844d2459cb4e1f29dc9296ac238315c27a79b682639d80/simonw/datasette-permissions-sql


id: MDEwOlJlcG9zaXRvcnkzNzU1NDY2NzU=
nameWithOwner: simonw/datasette-placekey
createdAt: 2021-06-10T02:31:27Z
topics: ["datasette", "datasette-io", "datasette-plugin", "placekey"]
latest_commit: 7bf3497bcec6c4cd7b351cfb87d76b6877ca910a
openGraphImageUrl:
  https://opengraph.githubassets.com/94e136f8c215ad1d213923e121e8ad784737725f80ae0474322a3472453c30ae/simonw/datasette-placekey


id: MDEwOlJlcG9zaXRvcnkxNjc3MzAwNzE=
nameWithOwner: simonw/datasette-pretty-json
createdAt: 2019-01-26T19:30:43Z
topics: ["datasette", "datasette-plugin", "json", "datasette-io"]
latest_commit: 61d82f07b3d5b33b86e77d9959ed424a39f69fc7
openGraphImageUrl:
  https://opengraph.githubassets.com/8f4da13fb377ee05554d0ed3b162f322a2fb34a90d07a463eed2e53526c5a695/simonw/datasette-pretty-json


id: MDEwOlJlcG9zaXRvcnkyNzIwOTg0ODY=
nameWithOwner: simonw/datasette-psutil
createdAt: 2020-06-13T22:57:07Z
topics: ["datasette", "datasette-io", "datasette-plugin", "psutil"]
latest_commit: ef767d6276f87257d5b72c93e3de57ad4aec1f34
openGraphImageUrl:
  https://opengraph.githubassets.com/51d9565b7d50968bfc1d544e9b78c618e88b172c944a5e00653056a563698d3b/simonw/datasette-psutil


id: R_kgDOIIe2Hg
nameWithOwner: simonw/datasette-public
createdAt: 2022-10-05T00:03:28Z
topics: []
latest_commit: 32b6a0ba53bd5714b6b41eddd8705b213c105efc
openGraphImageUrl:
  https://opengraph.githubassets.com/5744a13822ea34db4796e58a1c5096841249e61ccbf5d642842d694dbdaf721f/simonw/datasette-public


id: MDEwOlJlcG9zaXRvcnkyNDgzODUyOTk=
nameWithOwner: simonw/datasette-publish-fly
createdAt: 2020-03-19T01:47:01Z
topics: ["datasette", "datasette-io", "datasette-plugin", "fly"]
latest_commit: 90b064c88d40936d33e63f548976030b4a331ac1
openGraphImageUrl:
  https://opengraph.githubassets.com/7e73ecb40c0d01bae01b9bdb474da7bfba9b2aedba01c03e20e8b1cf6acca16d/simonw/datasette-publish-fly


id: MDEwOlJlcG9zaXRvcnkyNTM2MzI5NDg=
nameWithOwner: simonw/datasette-publish-vercel
createdAt: 2020-04-06T22:47:13Z
topics: ["datasette", "datasette-plugin", "zeit-now", "vercel", "datasette-io"]
latest_commit: 1559d979b4e3b1f2f83c51c3c0c10192ff9a6d0c
openGraphImageUrl:
  https://opengraph.githubassets.com/af7f2600698d400322a7b5c5d64ec9724560cc2c33a48ab884489166b7905572/simonw/datasette-publish-vercel


id: MDEwOlJlcG9zaXRvcnkzOTQxMDc2MTQ=
nameWithOwner: simonw/datasette-query-links
createdAt: 2021-08-09T01:16:59Z
topics: []
latest_commit: 130776d43660716193601ad07cb4059eacf0f393
openGraphImageUrl:
  https://opengraph.githubassets.com/2d9d47761ff290c33feecb04cdb1350d8fcc7f181073d018949367c04e49a253/simonw/datasette-query-links


id: R_kgDOG5dZxg
nameWithOwner: simonw/datasette-redirect-forbidden
createdAt: 2022-02-23T20:59:26Z
topics: []
latest_commit: 058e5362de44a550cb0675757f145d461211852f
openGraphImageUrl:
  https://opengraph.githubassets.com/34ffad9edb7082bf6700668b3f2a02d7b47e5de2647bde09f69300c602c6e1db/simonw/datasette-redirect-forbidden


id: R_kgDOGaS1TA
nameWithOwner: simonw/datasette-redirect-to-https
createdAt: 2021-11-20T22:43:33Z
topics: ["datasette", "datasette-io", "datasette-plugin", "asgi"]
latest_commit: 2a5be0f8b339b0558499a36c4d1be2180c35d58f
openGraphImageUrl:
  https://opengraph.githubassets.com/a7c1ccd34931a515f07cbe8de4ab0446e8a856411d3efffe9158d16863d78165/simonw/datasette-redirect-to-https


id: MDEwOlJlcG9zaXRvcnkzOTA1MzU1MDA=
nameWithOwner: simonw/datasette-remote-metadata
createdAt: 2021-07-28T23:17:19Z
topics: []
latest_commit: b4084ae687bc496c9b767d24a6c73d1025e8b646
openGraphImageUrl:
  https://opengraph.githubassets.com/62c1cffbd297cd078e04b971ed0c4e365fce810faa3d63586154cb25003250a7/simonw/datasette-remote-metadata


id: MDEwOlJlcG9zaXRvcnkxOTEwMjI5Mjg=
nameWithOwner: simonw/datasette-render-binary
createdAt: 2019-06-09T15:25:52Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: 05534ae93079f8dd1eedf2b54e499ff9c6269df5
openGraphImageUrl:
  https://opengraph.githubassets.com/bcc8a167153d9b86bae3eca0440c93c24fda358a1809c6e907a64d323de442e4/simonw/datasette-render-binary


id: MDEwOlJlcG9zaXRvcnkxODQxNjg4NjQ=
nameWithOwner: simonw/datasette-render-html
createdAt: 2019-04-30T01:21:25Z
topics: ["datasette-plugin", "datasette-io", "datasette"]
latest_commit: ad30076fae9c3788b917967a776de42853109536
openGraphImageUrl:
  https://opengraph.githubassets.com/2fa86707c64b28f45906bb0b96a687101e0932d33b9a92996a496d2bbe4e9e8e/simonw/datasette-render-html


id: R_kgDOH7vZQw
nameWithOwner: simonw/datasette-render-image-tags
createdAt: 2022-09-04T00:36:44Z
topics: []
latest_commit: 3acf0563b77b2659b1f8b155b5af8844824ff404
openGraphImageUrl:
  https://opengraph.githubassets.com/6e613cf975734ba893fdcad7d735abcb2dbbf7852b8ab7ab063053fb8af64284/simonw/datasette-render-image-tags


id: MDEwOlJlcG9zaXRvcnkxNDU0ODMwNzc=
nameWithOwner: simonw/datasette-render-images
createdAt: 2018-08-21T00:05:47Z
topics: ["datasette", "plugin", "datasette-plugin", "datasette-io"]
latest_commit: e91ee84398c5f684c6e89bcadbfa13827cf53aad
openGraphImageUrl:
  https://opengraph.githubassets.com/b1cd4978ed92b3682b947f5a45b388056959fee1251a2978a5051a0d58418414/simonw/datasette-render-images


id: MDEwOlJlcG9zaXRvcnkyMjA3MTY4MjI=
nameWithOwner: simonw/datasette-render-markdown
createdAt: 2019-11-09T23:28:31Z
topics: ["datasette", "datasette-plugin", "markdown", "datasette-io"]
latest_commit: 51671757c6eceee4e06ed33b49aafb1e44478bc6
openGraphImageUrl:
  https://opengraph.githubassets.com/e60d0afa373796471ef741f68dea854c4a55030318d6edcb52872b28d3238a27/simonw/datasette-render-markdown


id: MDEwOlJlcG9zaXRvcnkyMTQyOTkyNjc=
nameWithOwner: simonw/datasette-render-timestamps
createdAt: 2019-10-10T22:50:50Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: 53b4a495cf2d8145536e5f9dd42bff27fdd2ef91
openGraphImageUrl:
  https://opengraph.githubassets.com/216874812dcddc8754138b95420d1658851f97622ced18fdd900f6d07497fda3/simonw/datasette-render-timestamps


id: MDEwOlJlcG9zaXRvcnkyMDc2MzAxNzQ=
nameWithOwner: simonw/datasette-rure
createdAt: 2019-09-10T18:09:33Z
topics: ["datasette", "datasette-plugin", "sqlite", "regular-expressions", "datasette-io"]
latest_commit: 043fa612c86e0a4a64970b8f28b98ad7507da9a1
openGraphImageUrl:
  https://opengraph.githubassets.com/9c431bf27608f109642961da549297f1b1d1f2b69fb395a993356e138f4399e3/simonw/datasette-rure


id: R_kgDOH99rFw
nameWithOwner: simonw/datasette-sandstorm-support
createdAt: 2022-09-09T17:11:04Z
topics: []
latest_commit: 9d2eafe1345c4aa1ae6355d511286801be8be8ff
openGraphImageUrl:
  https://opengraph.githubassets.com/f6216b8113799d766e11ba448735c466a8f29df54cd65d916dd94b4dc6504426/simonw/datasette-sandstorm-support


id: MDEwOlJlcG9zaXRvcnkyNzM2MDk4Nzk=
nameWithOwner: simonw/datasette-saved-queries
createdAt: 2020-06-20T00:20:42Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 1c6e48676ad3e7366ed5cc9bae532d3bd85267b5
openGraphImageUrl:
  https://opengraph.githubassets.com/e308ec25a3e83a7feb551f7e15a4f308da931dcfaf04b366df0d4645cddb5f61/simonw/datasette-saved-queries


id: R_kgDOHilbpw
nameWithOwner: simonw/datasette-scale-to-zero
createdAt: 2022-06-21T22:49:04Z
topics: ["datasette", "datasette-plugin"]
latest_commit: 255296205750d5addc32b93fb8ed898d77f14ddc
openGraphImageUrl:
  https://opengraph.githubassets.com/45bdd57355f15ffc503c127732a5989abb33271898e2efcab1c885b838f5d403/simonw/datasette-scale-to-zero


id: MDEwOlJlcG9zaXRvcnkyODg2Mjk3NjY=
nameWithOwner: simonw/datasette-schema-versions
createdAt: 2020-08-19T04:04:39Z
topics: ["datasette", "datasette-io", "datasette-plugin"]
latest_commit: 7cbef5d75d0e5d7d40dc469566bb4bd74eea7d85
openGraphImageUrl:
  https://opengraph.githubassets.com/f1539deb6b5316d53d688f5e0be85e23f642e1b85a86e4dd3cf6101dd75c0827/simonw/datasette-schema-versions


id: MDEwOlJlcG9zaXRvcnkyOTQ3MDYyNjc=
nameWithOwner: simonw/datasette-seaborn
createdAt: 2020-09-11T13:43:08Z
topics: ["datasette", "datasette-plugin", "datasette-io", "seaborn", "visualization"]
latest_commit: 61ad490dad04a964d512aebaf3b7af7d965e51ae
openGraphImageUrl:
  https://opengraph.githubassets.com/9aa3baa116f5acc03c35fa47843169b363970f3e629c53691c13faac549996b0/simonw/datasette-seaborn


id: MDEwOlJlcG9zaXRvcnkyMzY4NjcwMjc=
nameWithOwner: simonw/datasette-sentry
createdAt: 2020-01-28T23:41:27Z
topics: ["datasette", "datasette-plugin", "sentry", "datasette-io"]
latest_commit: 0145805451fed758761c914029c0976a7f55b172
openGraphImageUrl:
  https://opengraph.githubassets.com/29f90f4fb2b7be3ae494ec59126936da926a174eba8e250912d0d6348788fe46/simonw/datasette-sentry


id: MDEwOlJlcG9zaXRvcnkyNDg5OTk5OTQ=
nameWithOwner: simonw/datasette-show-errors
createdAt: 2020-03-21T15:06:04Z
topics: ["datasette", "datasette-io", "datasette-plugin", "starlette", "asgi"]
latest_commit: 07c01e54405531fa780e457543de2af99bf53ab6
openGraphImageUrl:
  https://opengraph.githubassets.com/ded7026b31328d63bdd8757fae16f129a7e9eedb195ecf0001cc27fb81f1ba8e/simonw/datasette-show-errors


id: R_kgDOH50_7g
nameWithOwner: simonw/datasette-sitemap
createdAt: 2022-08-29T21:22:46Z
topics: []
latest_commit: a825654faa4b4802c3bf9ccff4c15bbffe04b224
openGraphImageUrl:
  https://opengraph.githubassets.com/61ec7a2b113e650f58d3a6eb8f0f7915419f89e89d01546820e4ca4c9c426649/simonw/datasette-sitemap


id: R_kgDOHeFHyg
nameWithOwner: simonw/datasette-socrata
createdAt: 2022-06-08T15:13:44Z
topics: []
latest_commit: 32fb256a461bf0e790eca10bdc7dd9d96c20f7c4
openGraphImageUrl:
  https://opengraph.githubassets.com/fe8bcd77062451f02be548f012b454cec45c10d0c83df6e3fd7331575ca68a04/simonw/datasette-socrata


id: MDEwOlJlcG9zaXRvcnkxNjM3OTA4MjI=
nameWithOwner: simonw/datasette-sqlite-fts4
createdAt: 2019-01-02T03:40:41Z
topics: ["datasette", "plugin", "datasette-plugin", "datasette-io"]
latest_commit: 7e46394c63f75536a9dce3796ed03734a7d1e9ee
openGraphImageUrl:
  https://opengraph.githubassets.com/dfdb13b42b2338366bbab20558024e54ce158a8ac2c5f7f0f79b296a9d633b46/simonw/datasette-sqlite-fts4


id: R_kgDOGGsxew
nameWithOwner: simonw/datasette-template-request
createdAt: 2021-09-23T17:07:00Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: b379518cffbd0879010babc7d16fb5cbce550432
openGraphImageUrl:
  https://opengraph.githubassets.com/dd254324f82ad6c913ed5089047eda01548982829d90fea77e2a9b91f69a8735/simonw/datasette-template-request


id: MDEwOlJlcG9zaXRvcnkyMjE4MDIyOTY=
nameWithOwner: simonw/datasette-template-sql
createdAt: 2019-11-14T23:05:34Z
topics: ["datasette", "datasette-plugin", "datasette-io"]
latest_commit: 70483fd94bb0349761830e37c9ab928e8966841a
openGraphImageUrl:
  https://opengraph.githubassets.com/bd9dae382648a01cc7b924ea09525f0221e30a901e5f1c67bec3c0876d3b1c5a/simonw/datasette-template-sql


id: R_kgDOGkmBIg
nameWithOwner: simonw/datasette-tiddlywiki
createdAt: 2021-12-23T01:05:56Z
topics: []
latest_commit: d6e0b6b9a4dcbe92e27eb4cfa989ede9e6a8991d
openGraphImageUrl:
  https://opengraph.githubassets.com/6a003184ee94141dbb4d9e32f369d9e8681007b569bb88ebfceef6bdab8961b3/simonw/datasette-tiddlywiki


id: R_kgDOHPc0Nw
nameWithOwner: simonw/datasette-total-page-time
createdAt: 2022-04-26T22:09:58Z
topics: []
latest_commit: 0ee52a9bb7b0b4c9de5afdd04190c61410a7ada1
openGraphImageUrl:
  https://opengraph.githubassets.com/b3a0de0ce5a378c65e4b052f6a3953cd5a5725e5c6195421f0fc46d720f4c1f8/simonw/datasette-total-page-time


id: MDEwOlJlcG9zaXRvcnkyMzQ4MjU3OTA=
nameWithOwner: simonw/datasette-upload-csvs
createdAt: 2020-01-19T02:07:05Z
topics: ["datasette-plugin", "datasette", "csvs", "datasette-io"]
latest_commit: 0636bd08c76be59cbf22c6ac8fac4fe743ea7299
openGraphImageUrl:
  https://opengraph.githubassets.com/9c9b761a5eb64021f97f8584998bbfd037757a7fbbfc7b71eadbbbc7dc718946/simonw/datasette-upload-csvs


id: R_kgDOHSfuMg
nameWithOwner: simonw/datasette-upload-dbs
createdAt: 2022-05-05T23:36:51Z
topics: []
latest_commit: cbf1d130b67de58a6d932c1f430ade0cf2940d85
openGraphImageUrl:
  https://opengraph.githubassets.com/2b60983184a7fa685d1a6e0f5b760437c453e4952ea61a4a6faa16bc5cfc55c1/simonw/datasette-upload-dbs


id: MDEwOlJlcG9zaXRvcnk0MDA2NzgzMTc=
nameWithOwner: simonw/datasette-verify
createdAt: 2021-08-28T01:59:12Z
topics: ["datasette-io", "datasette-plugin", "datasette"]
latest_commit: 77bbc92bd670d4098bd3738d9c7ebc3ed0a33014
openGraphImageUrl:
  https://opengraph.githubassets.com/9cbc866b39b5e89405a2321289e132336ff3bd0d57c321b66a5c6c20649a497c/simonw/datasette-verify


id: MDEwOlJlcG9zaXRvcnkyNzU3MTEyNTQ=
nameWithOwner: simonw/datasette-write
createdAt: 2020-06-29T02:27:31Z
topics: ["datasette-io", "datasette-plugin"]
latest_commit: aaa1dcb39188389f5610962f9f9b52c3d96dc302
openGraphImageUrl:
  https://opengraph.githubassets.com/a52f074310ab8f2d2d6e780ce4bad0d80e8087f70c60e32d67f04dc8d796767c/simonw/datasette-write


id: MDEwOlJlcG9zaXRvcnkzOTUxMzc1MTM=
nameWithOwner: simonw/datasette-x-forwarded-host
createdAt: 2021-08-11T23:10:44Z
topics: ["datasette-io", "datasette-plugin"]
latest_commit: f1f6650cb6695c5e43a4893c06230b96a150b6ba
openGraphImageUrl:
  https://opengraph.githubassets.com/26377d7ed775716a5a0b40bf87765570f875fbcd9fdd7dc986d184141cfd1833/simonw/datasette-x-forwarded-host


id: MDEwOlJlcG9zaXRvcnkyOTEzNTkzNTg=
nameWithOwner: simonw/datasette-yaml
createdAt: 2020-08-29T22:32:15Z
topics: ["datasette", "datasette-plugin", "datasette-io", "yaml"]
latest_commit: c2f00b5d398bb93ec66fbc52720097e4d1a1eddb
openGraphImageUrl:
  https://opengraph.githubassets.com/984385e6e14b2c88a34143aadd0af8569b014f8c31b6a100c6a7e87deb50270f/simonw/datasette-yaml


id: MDEwOlJlcG9zaXRvcnkxNjYxNTkwNzI=
nameWithOwner: simonw/db-to-sqlite
createdAt: 2019-01-17T04:16:48Z
topics: ["sqlite", "datasette", "sqlalchemy", "datasette-io", "datasette-tool"]
latest_commit: 05b62b6fcd2dae67ff326c3c1925873166496adc
openGraphImageUrl:
  https://opengraph.githubassets.com/6a6ea034c4e528b3b83d9f755e48b091e1c23f5eac41cc84df2c6860770c02af/simonw/db-to-sqlite


id: MDEwOlJlcG9zaXRvcnkxNjg0NzQ5NzA=
nameWithOwner: simonw/dbf-to-sqlite
createdAt: 2019-01-31T06:30:46Z
topics: ["sqlite", "dbf", "dbase", "foxpro", "datasette-io", "datasette-tool"]
latest_commit: 1d1e1e8d64feb74d8a96b99200fb558b06a6957b
openGraphImageUrl:
  https://opengraph.githubassets.com/017b9cb71158679c9348e3c3040aea905f57d127a8f0c6cfad3a6aa445bc0c62/simonw/dbf-to-sqlite


id: MDEwOlJlcG9zaXRvcnkzMzUzNzIwNTA=
nameWithOwner: simonw/download-tiles
createdAt: 2021-02-02T17:37:49Z
topics: ["mbtiles", "openstreetmap", "datasette-io", "datasette-tool"]
latest_commit: cf699be717df0766d0f366108b619d217e30522f
openGraphImageUrl:
  https://opengraph.githubassets.com/bf2b9a8168680bc5af634e513baa50ee261364fd00510704d72f6f3402e24329/simonw/download-tiles


id: MDEwOlJlcG9zaXRvcnkyNDU2NzA2NzA=
nameWithOwner: simonw/fec-to-sqlite
createdAt: 2020-03-07T16:52:49Z
topics: ["sqlite", "datasette", "datasette-io", "fec", "datasette-tool"]
latest_commit: d3ec100f4e9d5acbc5798d95b49e6e373c1ce778
openGraphImageUrl:
  https://opengraph.githubassets.com/409e73734f713d59316301253e44f5f0dad565567d12a7780ea095c5495d45cf/simonw/fec-to-sqlite


id: MDEwOlJlcG9zaXRvcnkyMzczMjEyNjc=
nameWithOwner: simonw/geojson-to-sqlite
createdAt: 2020-01-30T22:51:05Z
topics: ["sqlite", "geojson", "gis", "datasette-io", "datasette-tool"]
latest_commit: 9c6c1c8ce2f27c2e1374c9a5f2f90996268ade1f
openGraphImageUrl:
  https://opengraph.githubassets.com/7f2488e3f4f54a16d22c63fa722c6f09fab5acef1057a760361405726b6bc82e/simonw/geojson-to-sqlite


id: R_kgDOGXV4Ig
nameWithOwner: simonw/git-history
createdAt: 2021-11-11T20:07:06Z
topics: ["git", "sqlite"]
latest_commit: 91abda0b599d51122b13cea9e9785a822f43ef28
openGraphImageUrl:
  https://opengraph.githubassets.com/6527a96ae1f1aaf48aa11e9675018180541f1af6a4cf121e876b08049b18db8c/simonw/git-history


id: R_kgDOG2hQNg
nameWithOwner: simonw/google-drive-to-sqlite
createdAt: 2022-02-16T02:16:29Z
topics: []
latest_commit: ea5f9298a6bb54139e2d34025d4b556858e086db
openGraphImageUrl:
  https://opengraph.githubassets.com/a7866a957c7cae081db2776d6f87db2841e3b9baffca19e1e661184c022582da/simonw/google-drive-to-sqlite


id: MDEwOlJlcG9zaXRvcnkxNjc3NTk4NDY=
nameWithOwner: simonw/markdown-to-sqlite
createdAt: 2019-01-27T02:04:54Z
topics: ["sqlite", "markdown", "yaml", "datasette-io", "datasette-tool"]
latest_commit: e6b852a8c1af51d6d42bfb44d8ee6f5f06debf03
openGraphImageUrl:
  https://opengraph.githubassets.com/4d5f451060f77910f4cf829b46a7ffcfdf4cf7aef994d156294e66eb50a63d25/simonw/markdown-to-sqlite


id: R_kgDOHI-jKw
nameWithOwner: simonw/pypi-to-sqlite
createdAt: 2022-04-07T23:09:02Z
topics: []
latest_commit: 09a7815092c40171d6abeb3859a04886018740d6
openGraphImageUrl:
  https://opengraph.githubassets.com/aa3065d4956ddf774e74100beae3afc65a2d22c745f355ee9909f4e085a39aae/simonw/pypi-to-sqlite


id: MDEwOlJlcG9zaXRvcnkyNDA4MTU5Mzg=
nameWithOwner: simonw/shapefile-to-sqlite
createdAt: 2020-02-16T01:55:29Z
topics: ["sqlite", "datasette", "spatialite", "shapefiles", "gis", "datasette-io", "datasette-tool"]
latest_commit: 2b2233d19e99af78e718ffdfc0e43ee8a078ae0f
openGraphImageUrl:
  https://opengraph.githubassets.com/90f0253a8763fca060a0802fd0f06d6edf56bd981a98e156e2277b5c0427aa23/simonw/shapefile-to-sqlite


id: MDEwOlJlcG9zaXRvcnkzMDUxOTk2NjE=
nameWithOwner: simonw/sphinx-to-sqlite
createdAt: 2020-10-18T21:26:55Z
topics: ["sqlite", "sphinx", "datasette-io", "datasette-tool"]
latest_commit: 950a34d7a0aac47a201f079c53d32f806a3bbfc1
openGraphImageUrl:
  https://opengraph.githubassets.com/54d38dca7f16bc3abe684fd22d882d633edea43bf946c51d7ac3479c5e1681ed/simonw/sphinx-to-sqlite


id: R_kgDOHoFAng
nameWithOwner: simonw/sqlite-comprehend
createdAt: 2022-07-08T06:26:15Z
topics: []
latest_commit: 2a1c9baa9664b3a4f9e2e3e59abd88db5ebb32d7
openGraphImageUrl:
  https://opengraph.githubassets.com/c40b9023f6ffd4c77a5a3eb518bfb8375da95f8725407d615201daeb7f544368/simonw/sqlite-comprehend


id: MDEwOlJlcG9zaXRvcnkxOTUxNDU2Nzg=
nameWithOwner: simonw/sqlite-diffable
createdAt: 2019-07-04T00:58:46Z
topics: ["sqlite", "datasette-tool", "datasette-io"]
latest_commit: 9a6d64dd3846f59c6873e138458f5a5fc540856b
openGraphImageUrl:
  https://opengraph.githubassets.com/79c47094f1c0c235dc9f0ddd77a4133d767656318be48172211d35a06e19e906/simonw/sqlite-diffable


id: MDEwOlJlcG9zaXRvcnkyNzQyNjQ0ODQ=
nameWithOwner: simonw/sqlite-generate
createdAt: 2020-06-22T23:36:44Z
topics: ["sqlite", "datasette-io", "datasette-tool"]
latest_commit: 84267539f3874d7aee7f868827268f479b480423
openGraphImageUrl:
  https://opengraph.githubassets.com/31cba3ff16e9ebdc261f8678d850c20ba21618790deee4603a43750495a7390a/simonw/sqlite-generate


id: MDEwOlJlcG9zaXRvcnkyMTkzNzIxMzM=
nameWithOwner: simonw/sqlite-transform
createdAt: 2019-11-03T22:07:53Z
topics: ["sqlite", "datasette-io", "datasette-tool"]
latest_commit: 049b2fcb6671f86a0916392e0a84dedc160d8fa3
openGraphImageUrl:
  https://opengraph.githubassets.com/fcc50d3db518a44faeedbc23dd4194f22ceff39c4aab2720747a524fa14267cf/simonw/sqlite-transform


id: MDEwOlJlcG9zaXRvcnkxNDA5MTI0MzI=
nameWithOwner: simonw/sqlite-utils
createdAt: 2018-07-14T03:21:46Z
openGraphImageUrl: https://repository-images.githubusercontent.com/140912432/871d0480-8b28-11ea-9d87-0068901541de
latest_commit: 52ddb0b9ffa5284be668da088b7600b6ff64a2f9
topics:
  ["sqlite", "python", "datasette", "sqlite-database", "click", "cli", "datasette-io", "datasette-tool"]


id: MDEwOlJlcG9zaXRvcnkzNDY1OTc1NTc=
nameWithOwner: simonw/tableau-to-sqlite
createdAt: 2021-03-11T06:12:02Z
topics: ["datasette-io", "datasette-tool"]
latest_commit: 256aca60d425cea1ef3a2193655bd90f526c171d
openGraphImageUrl:
  https://opengraph.githubassets.com/b5a46fd485ed37ad9f7156b6b5b63ff8b9b24289bf9868ba2cf2344fd9ff36a7/simonw/tableau-to-sqlite


id: MDEwOlJlcG9zaXRvcnkxNzU1NTAxMjc=
nameWithOwner: simonw/yaml-to-sqlite
createdAt: 2019-03-14T04:49:08Z
topics: ["yaml", "sqlite", "datasette-io", "datasette-tool"]
latest_commit: 1405bc83c8f2d4d8cd772ccacc340754f22b153e
openGraphImageUrl:
  https://opengraph.githubassets.com/9c0efd824550e7d552aa9d87635456faadb4fad118b9d71871ee74823effd9eb/simonw/yaml-to-sqlite


# repos

node_id: MDEwOlJlcG9zaXRvcnkxMDc5MTQ0OTM=
name: datasette
full_name: simonw/datasette
html_url: https://github.com/simonw/datasette
description: An open source multi-tool for exploring and publishing data
created_at: 2017-10-23T00:39:03Z
updated_at: 2022-11-15T23:16:27Z
pushed_at: 2022-11-16T03:47:14Z
homepage: https://datasette.io
language: Python
license: apache-2.0
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
topics:
  ["asgi", "automatic-api", "csv", "datasets", "datasette", "datasette-io", "docker", "json", "python", "sql", "sqlite"]


node_id: MDEwOlJlcG9zaXRvcnkxMTA1MDk4MTY=
name: csvs-to-sqlite
full_name: simonw/csvs-to-sqlite
html_url: https://github.com/simonw/csvs-to-sqlite
description: Convert CSV files into a SQLite database
created_at: 2017-11-13T06:38:21Z
updated_at: 2021-11-18T16:33:39Z
pushed_at: 2021-11-18T16:35:33Z
language: Python
license: apache-2.0
topics: ["click", "csv", "datasette", "datasette-io", "datasette-tool", "pandas", "python", "sqlite"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # csvs-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/csvs-to-sqlite.svg)](https://pypi.org/project/csvs-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/csvs-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/csvs-to-sqlite/releases)
  [![Tests](https://github.com/simonw/csvs-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/csvs-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/csvs-to-sqlite/blob/main/LICENSE)

  Convert CSV files into a SQLite database. Browse and publish that SQLite database with [Datasette](https://github.com/simonw/datasette).

  Basic usage:

      csvs-to-sqlite myfile.csv mydatabase.db

  This will create a new SQLite database called `mydatabase.db` containing a
  single table, `myfile`, containing the CSV content.

  You can provide multiple CSV files:

      csvs-to-sqlite one.csv two.csv bundle.db

  The `bundle.db` database will contain two tables, `one` and `two`.

  This means you can use wildcards:

      csvs-to-sqlite ~/Downloads/*.csv my-downloads.db

  If you pass a path to one or more directories, the script will recursively
  search those directories for CSV files and create tables for each one.

      csvs-to-sqlite ~/path/to/directory all-my-csvs.db

  ## Handling TSV (tab-separated values)

  You can use the `-s` option to specify a different delimiter. If you want
  to use a tab character you'll need to apply shell escaping like so:

      csvs-to-sqlite my-file.tsv my-file.db -s $'\t'

  ## Refactoring columns into separate lookup tables

  Let's say you have a CSV file that looks like this:

      county,precinct,office,district,party,candidate,votes
      Clark,1,President,,REP,John R. Kasich,5
      Clark,2,President,,REP,John R. Kasich,0
      Clark,3,President,,REP,John R. Kasich,7

  ([Real example taken from the Open Elections project](https://github.com/openelections/openelections-data-sd/blob/master/2016/20160607__sd__primary__clark__precinct.csv))

  You can now convert selected columns into separate lookup tables using the new
  `--extract-column` option (shortname: `-c`) - for example:

      csvs-to-sqlite openelections-data-*/*.csv \
          -c county:County:name \
          -c precinct:Precinct:name \
          -c office -c district -c party -c candidate \
          openelections.db

  The format is as follows:

      column_name:optional_table_name:optional_table_value_column_name

  If you just specify the column name e.g. `-c office`, the following table will
  be created:

      CREATE TABLE "office" (
          "id" INTEGER PRIMARY KEY,
          "value" TEXT
      );

  If you specify all three options, e.g. `-c precinct:Precinct:name` the table
  will look like this:

      CREATE TABLE "Precinct" (
          "id" INTEGER PRIMARY KEY,
          "name" TEXT
      );

  The original tables will be created like this:

      CREATE TABLE "ca__primary__san_francisco__precinct" (
          "county" INTEGER,
          "precinct" INTEGER,
          "office" INTEGER,
          "district" INTEGER,
          "party" INTEGER,
          "candidate" INTEGER,
          "votes" INTEGER,
          FOREIGN KEY (county) REFERENCES County(id),
          FOREIGN KEY (party) REFERENCES party(id),
          FOREIGN KEY (precinct) REFERENCES Precinct(id),
          FOREIGN KEY (office) REFERENCES office(id),
          FOREIGN KEY (candidate) REFERENCES candidate(id)
      );

  They will be populated with IDs that reference the new derived tables.

  ## Installation

      $ pip install csvs-to-sqlite

  `csvs-to-sqlite` now requires Python 3. If you are running Python 2 you can install the last version to support Python 2:

      $ pip install csvs-to-sqlite==0.9.2

  ## csvs-to-sqlite --help

  <!-- [[[cog
  import cog
  from csvs_to_sqlite import cli
  from click.testing import CliRunner
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["--help"])
  help = result.output.replace("Usage: cli", "Usage: csvs-to-sqlite")
  cog.out(
      "```\n{}\n```".format(help)
  )
  ]]] -->
  ```
  Usage: csvs-to-sqlite [OPTIONS] PATHS... DBNAME

    PATHS: paths to individual .csv files or to directories containing .csvs

    DBNAME: name of the SQLite database file to create

  Options:
    -s, --separator TEXT            Field separator in input .csv
    -q, --quoting INTEGER           Control field quoting behavior per csv.QUOTE_*
                                    constants. Use one of QUOTE_MINIMAL (0),
                                    QUOTE_ALL (1), QUOTE_NONNUMERIC (2) or
                                    QUOTE_NONE (3).

    --skip-errors                   Skip lines with too many fields instead of
                                    stopping the import

    --replace-tables                Replace tables if they already exist
    -t, --table TEXT                Table to use (instead of using CSV filename)
    -c, --extract-column TEXT       One or more columns to 'extract' into a
                                    separate lookup table. If you pass a simple
                                    column name that column will be replaced with
                                    integer foreign key references to a new table
                                    of that name. You can customize the name of
                                    the table like so:     state:States:state_name
                                    
                                    This will pull unique values from the 'state'
                                    column and use them to populate a new 'States'
                                    table, with an id column primary key and a
                                    state_name column containing the strings from
                                    the original column.

    -d, --date TEXT                 One or more columns to parse into ISO
                                    formatted dates

    -dt, --datetime TEXT            One or more columns to parse into ISO
                                    formatted datetimes

    -df, --datetime-format TEXT     One or more custom date format strings to try
                                    when parsing dates/datetimes

    -pk, --primary-key TEXT         One or more columns to use as the primary key
    -f, --fts TEXT                  One or more columns to use to populate a full-
                                    text index

    -i, --index TEXT                Add index on this column (or a compound index
                                    with -i col1,col2)

    --shape TEXT                    Custom shape for the DB table - format is
                                    csvcol:dbcol(TYPE),...

    --filename-column TEXT          Add a column with this name and populate with
                                    CSV file name

    --fixed-column <TEXT TEXT>...   Populate column with a fixed string
    --fixed-column-int <TEXT INTEGER>...
                                    Populate column with a fixed integer
    --fixed-column-float <TEXT FLOAT>...
                                    Populate column with a fixed float
    --no-index-fks                  Skip adding index to foreign key columns
                                    created using --extract-column (default is to
                                    add them)

    --no-fulltext-fks               Skip adding full-text index on values
                                    extracted using --extract-column (default is
                                    to add them)

    --just-strings                  Import all columns as text strings by default
                                    (and, if specified, still obey --shape,
                                    --date/datetime, and --datetime-format)

    --version                       Show the version and exit.
    --help                          Show this message and exit.

  ```
  <!-- [[[end]]] -->

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-csvs-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-csvs-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>csvs-to-sqlite</h1>
  <p dir="auto"><a href="https://pypi.org/project/csvs-to-sqlite/" rel="nofollow"><img src="https://camo.githubusercontent.com/4869c8a624ce3dbf120decc95900d98af23c54cf5c87d65f7164e640a5fcda6b/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f637376732d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/csvs-to-sqlite.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/csvs-to-sqlite/releases"><img src="https://camo.githubusercontent.com/c0cc260f11a725723219d723be3829d23edfc0962d0e8d191fa2bcd6ceaa749f/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f637376732d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/csvs-to-sqlite?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/csvs-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/csvs-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/csvs-to-sqlite/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Convert CSV files into a SQLite database. Browse and publish that SQLite database with <a href="https://github.com/simonw/datasette">Datasette</a>.</p>
  <p dir="auto">Basic usage:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="csvs-to-sqlite myfile.csv mydatabase.db
  "><pre><code>csvs-to-sqlite myfile.csv mydatabase.db
  </code></pre></div>
  <p dir="auto">This will create a new SQLite database called <code>mydatabase.db</code> containing a
  single table, <code>myfile</code>, containing the CSV content.</p>
  <p dir="auto">You can provide multiple CSV files:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="csvs-to-sqlite one.csv two.csv bundle.db
  "><pre><code>csvs-to-sqlite one.csv two.csv bundle.db
  </code></pre></div>
  <p dir="auto">The <code>bundle.db</code> database will contain two tables, <code>one</code> and <code>two</code>.</p>
  <p dir="auto">This means you can use wildcards:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="csvs-to-sqlite ~/Downloads/*.csv my-downloads.db
  "><pre><code>csvs-to-sqlite ~/Downloads/*.csv my-downloads.db
  </code></pre></div>
  <p dir="auto">If you pass a path to one or more directories, the script will recursively
  search those directories for CSV files and create tables for each one.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="csvs-to-sqlite ~/path/to/directory all-my-csvs.db
  "><pre><code>csvs-to-sqlite ~/path/to/directory all-my-csvs.db
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-handling-tsv-tab-separated-values" class="anchor" aria-hidden="true" href="#user-content-handling-tsv-tab-separated-values"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Handling TSV (tab-separated values)</h2>
  <p dir="auto">You can use the <code>-s</code> option to specify a different delimiter. If you want
  to use a tab character you'll need to apply shell escaping like so:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="csvs-to-sqlite my-file.tsv my-file.db -s $'\t'
  "><pre><code>csvs-to-sqlite my-file.tsv my-file.db -s $'\t'
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-refactoring-columns-into-separate-lookup-tables" class="anchor" aria-hidden="true" href="#user-content-refactoring-columns-into-separate-lookup-tables"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Refactoring columns into separate lookup tables</h2>
  <p dir="auto">Let's say you have a CSV file that looks like this:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="county,precinct,office,district,party,candidate,votes
  Clark,1,President,,REP,John R. Kasich,5
  Clark,2,President,,REP,John R. Kasich,0
  Clark,3,President,,REP,John R. Kasich,7
  "><pre><code>county,precinct,office,district,party,candidate,votes
  Clark,1,President,,REP,John R. Kasich,5
  Clark,2,President,,REP,John R. Kasich,0
  Clark,3,President,,REP,John R. Kasich,7
  </code></pre></div>
  <p dir="auto">(<a href="https://github.com/openelections/openelections-data-sd/blob/master/2016/20160607__sd__primary__clark__precinct.csv">Real example taken from the Open Elections project</a>)</p>
  <p dir="auto">You can now convert selected columns into separate lookup tables using the new
  <code>--extract-column</code> option (shortname: <code>-c</code>) - for example:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="csvs-to-sqlite openelections-data-*/*.csv \
      -c county:County:name \
      -c precinct:Precinct:name \
      -c office -c district -c party -c candidate \
      openelections.db
  "><pre><code>csvs-to-sqlite openelections-data-*/*.csv \
      -c county:County:name \
      -c precinct:Precinct:name \
      -c office -c district -c party -c candidate \
      openelections.db
  </code></pre></div>
  <p dir="auto">The format is as follows:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="column_name:optional_table_name:optional_table_value_column_name
  "><pre><code>column_name:optional_table_name:optional_table_value_column_name
  </code></pre></div>
  <p dir="auto">If you just specify the column name e.g. <code>-c office</code>, the following table will
  be created:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="CREATE TABLE &quot;office&quot; (
      &quot;id&quot; INTEGER PRIMARY KEY,
      &quot;value&quot; TEXT
  );
  "><pre><code>CREATE TABLE "office" (
      "id" INTEGER PRIMARY KEY,
      "value" TEXT
  );
  </code></pre></div>
  <p dir="auto">If you specify all three options, e.g. <code>-c precinct:Precinct:name</code> the table
  will look like this:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="CREATE TABLE &quot;Precinct&quot; (
      &quot;id&quot; INTEGER PRIMARY KEY,
      &quot;name&quot; TEXT
  );
  "><pre><code>CREATE TABLE "Precinct" (
      "id" INTEGER PRIMARY KEY,
      "name" TEXT
  );
  </code></pre></div>
  <p dir="auto">The original tables will be created like this:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="CREATE TABLE &quot;ca__primary__san_francisco__precinct&quot; (
      &quot;county&quot; INTEGER,
      &quot;precinct&quot; INTEGER,
      &quot;office&quot; INTEGER,
      &quot;district&quot; INTEGER,
      &quot;party&quot; INTEGER,
      &quot;candidate&quot; INTEGER,
      &quot;votes&quot; INTEGER,
      FOREIGN KEY (county) REFERENCES County(id),
      FOREIGN KEY (party) REFERENCES party(id),
      FOREIGN KEY (precinct) REFERENCES Precinct(id),
      FOREIGN KEY (office) REFERENCES office(id),
      FOREIGN KEY (candidate) REFERENCES candidate(id)
  );
  "><pre><code>CREATE TABLE "ca__primary__san_francisco__precinct" (
      "county" INTEGER,
      "precinct" INTEGER,
      "office" INTEGER,
      "district" INTEGER,
      "party" INTEGER,
      "candidate" INTEGER,
      "votes" INTEGER,
      FOREIGN KEY (county) REFERENCES County(id),
      FOREIGN KEY (party) REFERENCES party(id),
      FOREIGN KEY (precinct) REFERENCES Precinct(id),
      FOREIGN KEY (office) REFERENCES office(id),
      FOREIGN KEY (candidate) REFERENCES candidate(id)
  );
  </code></pre></div>
  <p dir="auto">They will be populated with IDs that reference the new derived tables.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install csvs-to-sqlite
  "><pre><code>$ pip install csvs-to-sqlite
  </code></pre></div>
  <p dir="auto"><code>csvs-to-sqlite</code> now requires Python 3. If you are running Python 2 you can install the last version to support Python 2:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install csvs-to-sqlite==0.9.2
  "><pre><code>$ pip install csvs-to-sqlite==0.9.2
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-csvs-to-sqlite---help" class="anchor" aria-hidden="true" href="#user-content-csvs-to-sqlite---help"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>csvs-to-sqlite --help</h2>

  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: csvs-to-sqlite [OPTIONS] PATHS... DBNAME

    PATHS: paths to individual .csv files or to directories containing .csvs

    DBNAME: name of the SQLite database file to create

  Options:
    -s, --separator TEXT            Field separator in input .csv
    -q, --quoting INTEGER           Control field quoting behavior per csv.QUOTE_*
                                    constants. Use one of QUOTE_MINIMAL (0),
                                    QUOTE_ALL (1), QUOTE_NONNUMERIC (2) or
                                    QUOTE_NONE (3).

    --skip-errors                   Skip lines with too many fields instead of
                                    stopping the import

    --replace-tables                Replace tables if they already exist
    -t, --table TEXT                Table to use (instead of using CSV filename)
    -c, --extract-column TEXT       One or more columns to 'extract' into a
                                    separate lookup table. If you pass a simple
                                    column name that column will be replaced with
                                    integer foreign key references to a new table
                                    of that name. You can customize the name of
                                    the table like so:     state:States:state_name
                                    
                                    This will pull unique values from the 'state'
                                    column and use them to populate a new 'States'
                                    table, with an id column primary key and a
                                    state_name column containing the strings from
                                    the original column.

    -d, --date TEXT                 One or more columns to parse into ISO
                                    formatted dates

    -dt, --datetime TEXT            One or more columns to parse into ISO
                                    formatted datetimes

    -df, --datetime-format TEXT     One or more custom date format strings to try
                                    when parsing dates/datetimes

    -pk, --primary-key TEXT         One or more columns to use as the primary key
    -f, --fts TEXT                  One or more columns to use to populate a full-
                                    text index

    -i, --index TEXT                Add index on this column (or a compound index
                                    with -i col1,col2)

    --shape TEXT                    Custom shape for the DB table - format is
                                    csvcol:dbcol(TYPE),...

    --filename-column TEXT          Add a column with this name and populate with
                                    CSV file name

    --fixed-column &lt;TEXT TEXT&gt;...   Populate column with a fixed string
    --fixed-column-int &lt;TEXT INTEGER&gt;...
                                    Populate column with a fixed integer
    --fixed-column-float &lt;TEXT FLOAT&gt;...
                                    Populate column with a fixed float
    --no-index-fks                  Skip adding index to foreign key columns
                                    created using --extract-column (default is to
                                    add them)

    --no-fulltext-fks               Skip adding full-text index on values
                                    extracted using --extract-column (default is
                                    to add them)

    --just-strings                  Import all columns as text strings by default
                                    (and, if specified, still obey --shape,
                                    --date/datetime, and --datetime-format)

    --version                       Show the version and exit.
    --help                          Show this message and exit.

  "><pre><code>Usage: csvs-to-sqlite [OPTIONS] PATHS... DBNAME

    PATHS: paths to individual .csv files or to directories containing .csvs

    DBNAME: name of the SQLite database file to create

  Options:
    -s, --separator TEXT            Field separator in input .csv
    -q, --quoting INTEGER           Control field quoting behavior per csv.QUOTE_*
                                    constants. Use one of QUOTE_MINIMAL (0),
                                    QUOTE_ALL (1), QUOTE_NONNUMERIC (2) or
                                    QUOTE_NONE (3).

    --skip-errors                   Skip lines with too many fields instead of
                                    stopping the import

    --replace-tables                Replace tables if they already exist
    -t, --table TEXT                Table to use (instead of using CSV filename)
    -c, --extract-column TEXT       One or more columns to 'extract' into a
                                    separate lookup table. If you pass a simple
                                    column name that column will be replaced with
                                    integer foreign key references to a new table
                                    of that name. You can customize the name of
                                    the table like so:     state:States:state_name
                                    
                                    This will pull unique values from the 'state'
                                    column and use them to populate a new 'States'
                                    table, with an id column primary key and a
                                    state_name column containing the strings from
                                    the original column.

    -d, --date TEXT                 One or more columns to parse into ISO
                                    formatted dates

    -dt, --datetime TEXT            One or more columns to parse into ISO
                                    formatted datetimes

    -df, --datetime-format TEXT     One or more custom date format strings to try
                                    when parsing dates/datetimes

    -pk, --primary-key TEXT         One or more columns to use as the primary key
    -f, --fts TEXT                  One or more columns to use to populate a full-
                                    text index

    -i, --index TEXT                Add index on this column (or a compound index
                                    with -i col1,col2)

    --shape TEXT                    Custom shape for the DB table - format is
                                    csvcol:dbcol(TYPE),...

    --filename-column TEXT          Add a column with this name and populate with
                                    CSV file name

    --fixed-column &lt;TEXT TEXT&gt;...   Populate column with a fixed string
    --fixed-column-int &lt;TEXT INTEGER&gt;...
                                    Populate column with a fixed integer
    --fixed-column-float &lt;TEXT FLOAT&gt;...
                                    Populate column with a fixed float
    --no-index-fks                  Skip adding index to foreign key columns
                                    created using --extract-column (default is to
                                    add them)

    --no-fulltext-fks               Skip adding full-text index on values
                                    extracted using --extract-column (default is
                                    to add them)

    --just-strings                  Import all columns as text strings by default
                                    (and, if specified, still obey --shape,
                                    --date/datetime, and --datetime-format)

    --version                       Show the version and exit.
    --help                          Show this message and exit.

  </code></pre></div>

  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxMzAyMzY3NjI=
name: datasette-cluster-map
full_name: simonw/datasette-cluster-map
html_url: https://github.com/simonw/datasette-cluster-map
description: Datasette plugin that shows a map for any data with latitude/longitude columns
created_at: 2018-04-19T15:31:55Z
updated_at: 2021-12-07T21:55:01Z
pushed_at: 2021-12-07T19:39:02Z
language: JavaScript
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "leafletjs"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-cluster-map

  [![PyPI](https://img.shields.io/pypi/v/datasette-cluster-map.svg)](https://pypi.org/project/datasette-cluster-map/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-cluster-map?include_prereleases&label=changelog)](https://github.com/simonw/datasette-cluster-map/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-cluster-map/blob/main/LICENSE)

  A [Datasette plugin](https://docs.datasette.io/en/stable/plugins.html) that detects tables with `latitude` and `longitude` columns and then plots them on a map using [Leaflet.markercluster](https://github.com/Leaflet/Leaflet.markercluster).

  More about this project: [Datasette plugins, and building a clustered map visualization](https://simonwillison.net/2018/Apr/20/datasette-plugins/).

  ## Demo

  [global-power-plants.datasettes.com](https://global-power-plants.datasettes.com/global-power-plants/global-power-plants) hosts a demo of this plugin running against a database of 33,000 power plants around the world.

  ![Cluster map demo](https://static.simonwillison.net/static/2020/global-power-plants.png)

  ## Installation

  Run `datasette install datasette-cluster-map` to add this plugin to your Datasette virtual environment. Datasette will automatically load the plugin if it is installed in this way.

  If you are deploying using the `datasette publish` command you can use the `--install` option:

      datasette publish cloudrun mydb.db --install=datasette-cluster-map

  If any of your tables have a `latitude` and `longitude` column, a map will be automatically displayed.

  ## Configuration

  If your columns are called something else you can configure the column names using [plugin configuration](https://docs.datasette.io/en/stable/plugins.html#plugin-configuration) in a `metadata.json` file. For example, if all of your columns are called `xlat` and `xlng` you can create a `metadata.json` file like this:

  ```json
  {
      "title": "Regular metadata keys can go here too",
      "plugins": {
          "datasette-cluster-map": {
              "latitude_column": "xlat",
              "longitude_column": "xlng"
          }
      }
  }
  ```

  Then run Datasette like this:

      datasette mydata.db -m metadata.json

  This will configure the required column names for every database loaded by that Datasette instance.

  If you want to customize the column names for just one table in one database, you can do something like this:

  ```json
  {
      "databases": {
          "polar-bears": {
              "tables": {
                  "USGS_WC_eartag_deployments_2009-2011": {
                      "plugins": {
                          "datasette-cluster-map": {
                              "latitude_column": "Capture Latitude",
                              "longitude_column": "Capture Longitude"
                          }
                      }
                  }
              }
          }
      }
  }
  ```

  You can also use a custom SQL query to rename those columns to `latitude` and `longitude`, [for example](https://polar-bears.now.sh/polar-bears?sql=select+*%2C%0D%0A++++%22Capture+Latitude%22+as+latitude%2C%0D%0A++++%22Capture+Longitude%22+as+longitude%0D%0Afrom+%5BUSGS_WC_eartag_deployments_2009-2011%5D):

  ```sql
  select *,
      "Capture Latitude" as latitude,
      "Capture Longitude" as longitude
  from [USGS_WC_eartag_deployments_2009-2011]
  ```

  The map defaults to being displayed above the main results table on the page. You can use the `"container"` plugin setting to provide a CSS selector indicating an element that the map should be appended to instead.

  ## Custom tile layers

  You can customize the tile layer used  by the maps using the `tile_layer` and `tile_layer_options` configuration settings. For example, to use the [Stamen Watercolor tiles](http://maps.stamen.com/watercolor/#12/37.7706/-122.3782) you can use these settings:

  ```json
  {
      "plugins": {
          "datasette-cluster-map": {
              "tile_layer": "https://stamen-tiles-{s}.a.ssl.fastly.net/watercolor/{z}/{x}/{y}.{ext}",
              "tile_layer_options": {
                  "attribution": "Map tiles by <a href=\"http://stamen.com\">Stamen Design</a>, <a href=\"http://creativecommons.org/licenses/by/3.0\">CC BY 3.0</a> &mdash; Map data &copy; <a href=\"https://www.openstreetmap.org/copyright\">OpenStreetMap</a> contributors",
                  "subdomains": "abcd",
                  "minZoom": 1,
                  "maxZoom": 16,
                  "ext": "jpg"
              }
          }
      }
  }
  ```
  The [Leaflet Providers preview list](https://leaflet-extras.github.io/leaflet-providers/preview/index.html) has details of many other tile layers you can use.

  ## Custom marker popups

  The marker popup defaults to displaying the data for the underlying database row.

  You can customize this by including a `popup` column in your results containing JSON that defines a more useful popup.

  The JSON in the popup column should look something like this:

  ```json
  {
      "image": "https://niche-museums.imgix.net/dodgems.heic?w=800&h=400&fit=crop",
      "alt": "Dingles Fairground Heritage Centre",
      "title": "Dingles Fairground Heritage Centre",
      "description": "Home of the National Fairground Collection, Dingles has over 45,000 indoor square feet of vintage fairground rides... and you can go on them! Highlights include the last complete surviving and opera",
      "link": "/browse/museums/26"
  }
  ```

  Each of these columns is optional.

  - `title` is the title to show at the top of the popup
  - `image` is the URL to an image to display in the popup
  - `alt` is the alt attribute to use for that image
  - `description` is a longer string of text to use as a description
  - `link` is a URL that the marker content should link to

  You can use the SQLite `json_object()` function to construct this data dynamically as part of your SQL query. Here's an example:

  ```sql
  select json_object(
    'image', photo_url || '?w=800&h=400&fit=crop',
    'title', name,
    'description', substr(description, 0, 200),
    'link', '/browse/museums/' || id
    ) as popup,
    latitude, longitude from museums
  where id in (26, 27) order by id
  ```

  [Try that example here](https://www.niche-museums.com/browse?sql=select+json_object%28%0D%0A++%27image%27%2C+photo_url+%7C%7C+%27%3Fw%3D800%26h%3D400%26fit%3Dcrop%27%2C%0D%0A++%27title%27%2C+name%2C%0D%0A++%27description%27%2C+substr%28description%2C+0%2C+200%29%2C%0D%0A++%27link%27%2C+%27%2Fbrowse%2Fmuseums%2F%27+%7C%7C+id%0D%0A++%29+as+popup%2C%0D%0A++latitude%2C+longitude+from+museums) or take a look at [this demo built using a SQL view](https://dogsheep-photos.dogsheep.net/public/photos_on_a_map).

  ## How I deployed the demo

      datasette publish cloudrun global-power-plants.db \
          --service global-power-plants \
          --metadata metadata.json \
          --install=datasette-cluster-map \
          --extra-options="--config facet_time_limit_ms:1000"

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-cluster-map
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-cluster-map" class="anchor" aria-hidden="true" href="#user-content-datasette-cluster-map"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-cluster-map</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-cluster-map/" rel="nofollow"><img src="https://camo.githubusercontent.com/6450215f6eeea9441e8212cad322848645a5fe49b0a77613b19c24aaf36eec9b/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d636c75737465722d6d61702e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-cluster-map.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-cluster-map/releases"><img src="https://camo.githubusercontent.com/1e99ad2a8a1efa327c27477d9f56b3288ab98216a2860019b872ecdcff59e7e5/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d636c75737465722d6d61703f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-cluster-map?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-cluster-map/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">A <a href="https://docs.datasette.io/en/stable/plugins.html" rel="nofollow">Datasette plugin</a> that detects tables with <code>latitude</code> and <code>longitude</code> columns and then plots them on a map using <a href="https://github.com/Leaflet/Leaflet.markercluster">Leaflet.markercluster</a>.</p>
  <p dir="auto">More about this project: <a href="https://simonwillison.net/2018/Apr/20/datasette-plugins/" rel="nofollow">Datasette plugins, and building a clustered map visualization</a>.</p>
  <h2 dir="auto"><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p dir="auto"><a href="https://global-power-plants.datasettes.com/global-power-plants/global-power-plants" rel="nofollow">global-power-plants.datasettes.com</a> hosts a demo of this plugin running against a database of 33,000 power plants around the world.</p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer" href="https://camo.githubusercontent.com/8b609bcc80b913862801a773ae46d03393ad3f9e1f7c03a015dfc1123097ed71/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032302f676c6f62616c2d706f7765722d706c616e74732e706e67"><img src="https://camo.githubusercontent.com/8b609bcc80b913862801a773ae46d03393ad3f9e1f7c03a015dfc1123097ed71/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032302f676c6f62616c2d706f7765722d706c616e74732e706e67" alt="Cluster map demo" data-canonical-src="https://static.simonwillison.net/static/2020/global-power-plants.png" style="max-width: 100%;"></a></p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Run <code>datasette install datasette-cluster-map</code> to add this plugin to your Datasette virtual environment. Datasette will automatically load the plugin if it is installed in this way.</p>
  <p dir="auto">If you are deploying using the <code>datasette publish</code> command you can use the <code>--install</code> option:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish cloudrun mydb.db --install=datasette-cluster-map
  "><pre><code>datasette publish cloudrun mydb.db --install=datasette-cluster-map
  </code></pre></div>
  <p dir="auto">If any of your tables have a <code>latitude</code> and <code>longitude</code> column, a map will be automatically displayed.</p>
  <h2 dir="auto"><a id="user-content-configuration" class="anchor" aria-hidden="true" href="#user-content-configuration"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Configuration</h2>
  <p dir="auto">If your columns are called something else you can configure the column names using <a href="https://docs.datasette.io/en/stable/plugins.html#plugin-configuration" rel="nofollow">plugin configuration</a> in a <code>metadata.json</code> file. For example, if all of your columns are called <code>xlat</code> and <code>xlng</code> you can create a <code>metadata.json</code> file like this:</p>
  <div class="highlight highlight-source-json position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;title&quot;: &quot;Regular metadata keys can go here too&quot;,
      &quot;plugins&quot;: {
          &quot;datasette-cluster-map&quot;: {
              &quot;latitude_column&quot;: &quot;xlat&quot;,
              &quot;longitude_column&quot;: &quot;xlng&quot;
          }
      }
  }
  "><pre>{
      <span class="pl-ent">"title"</span>: <span class="pl-s"><span class="pl-pds">"</span>Regular metadata keys can go here too<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-cluster-map"</span>: {
              <span class="pl-ent">"latitude_column"</span>: <span class="pl-s"><span class="pl-pds">"</span>xlat<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"longitude_column"</span>: <span class="pl-s"><span class="pl-pds">"</span>xlng<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <p dir="auto">Then run Datasette like this:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette mydata.db -m metadata.json
  "><pre><code>datasette mydata.db -m metadata.json
  </code></pre></div>
  <p dir="auto">This will configure the required column names for every database loaded by that Datasette instance.</p>
  <p dir="auto">If you want to customize the column names for just one table in one database, you can do something like this:</p>
  <div class="highlight highlight-source-json position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;databases&quot;: {
          &quot;polar-bears&quot;: {
              &quot;tables&quot;: {
                  &quot;USGS_WC_eartag_deployments_2009-2011&quot;: {
                      &quot;plugins&quot;: {
                          &quot;datasette-cluster-map&quot;: {
                              &quot;latitude_column&quot;: &quot;Capture Latitude&quot;,
                              &quot;longitude_column&quot;: &quot;Capture Longitude&quot;
                          }
                      }
                  }
              }
          }
      }
  }
  "><pre>{
      <span class="pl-ent">"databases"</span>: {
          <span class="pl-ent">"polar-bears"</span>: {
              <span class="pl-ent">"tables"</span>: {
                  <span class="pl-ent">"USGS_WC_eartag_deployments_2009-2011"</span>: {
                      <span class="pl-ent">"plugins"</span>: {
                          <span class="pl-ent">"datasette-cluster-map"</span>: {
                              <span class="pl-ent">"latitude_column"</span>: <span class="pl-s"><span class="pl-pds">"</span>Capture Latitude<span class="pl-pds">"</span></span>,
                              <span class="pl-ent">"longitude_column"</span>: <span class="pl-s"><span class="pl-pds">"</span>Capture Longitude<span class="pl-pds">"</span></span>
                          }
                      }
                  }
              }
          }
      }
  }</pre></div>
  <p dir="auto">You can also use a custom SQL query to rename those columns to <code>latitude</code> and <code>longitude</code>, <a href="https://polar-bears.now.sh/polar-bears?sql=select+*%2C%0D%0A++++%22Capture+Latitude%22+as+latitude%2C%0D%0A++++%22Capture+Longitude%22+as+longitude%0D%0Afrom+%5BUSGS_WC_eartag_deployments_2009-2011%5D" rel="nofollow">for example</a>:</p>
  <div class="highlight highlight-source-sql position-relative overflow-auto" data-snippet-clipboard-copy-content="select *,
      &quot;Capture Latitude&quot; as latitude,
      &quot;Capture Longitude&quot; as longitude
  from [USGS_WC_eartag_deployments_2009-2011]
  "><pre><span class="pl-k">select</span> <span class="pl-k">*</span>,
      <span class="pl-s"><span class="pl-pds">"</span>Capture Latitude<span class="pl-pds">"</span></span> <span class="pl-k">as</span> latitude,
      <span class="pl-s"><span class="pl-pds">"</span>Capture Longitude<span class="pl-pds">"</span></span> <span class="pl-k">as</span> longitude
  <span class="pl-k">from</span> [USGS_WC_eartag_deployments_2009<span class="pl-k">-</span><span class="pl-c1">2011</span>]</pre></div>
  <p dir="auto">The map defaults to being displayed above the main results table on the page. You can use the <code>"container"</code> plugin setting to provide a CSS selector indicating an element that the map should be appended to instead.</p>
  <h2 dir="auto"><a id="user-content-custom-tile-layers" class="anchor" aria-hidden="true" href="#user-content-custom-tile-layers"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Custom tile layers</h2>
  <p dir="auto">You can customize the tile layer used  by the maps using the <code>tile_layer</code> and <code>tile_layer_options</code> configuration settings. For example, to use the <a href="http://maps.stamen.com/watercolor/#12/37.7706/-122.3782" rel="nofollow">Stamen Watercolor tiles</a> you can use these settings:</p>
  <div class="highlight highlight-source-json position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-cluster-map&quot;: {
              &quot;tile_layer&quot;: &quot;https://stamen-tiles-{s}.a.ssl.fastly.net/watercolor/{z}/{x}/{y}.{ext}&quot;,
              &quot;tile_layer_options&quot;: {
                  &quot;attribution&quot;: &quot;Map tiles by &lt;a href=\&quot;http://stamen.com\&quot;&gt;Stamen Design&lt;/a&gt;, &lt;a href=\&quot;http://creativecommons.org/licenses/by/3.0\&quot;&gt;CC BY 3.0&lt;/a&gt; &amp;mdash; Map data &amp;copy; &lt;a href=\&quot;https://www.openstreetmap.org/copyright\&quot;&gt;OpenStreetMap&lt;/a&gt; contributors&quot;,
                  &quot;subdomains&quot;: &quot;abcd&quot;,
                  &quot;minZoom&quot;: 1,
                  &quot;maxZoom&quot;: 16,
                  &quot;ext&quot;: &quot;jpg&quot;
              }
          }
      }
  }
  "><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-cluster-map"</span>: {
              <span class="pl-ent">"tile_layer"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://stamen-tiles-{s}.a.ssl.fastly.net/watercolor/{z}/{x}/{y}.{ext}<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"tile_layer_options"</span>: {
                  <span class="pl-ent">"attribution"</span>: <span class="pl-s"><span class="pl-pds">"</span>Map tiles by &lt;a href=<span class="pl-cce">\"</span>http://stamen.com<span class="pl-cce">\"</span>&gt;Stamen Design&lt;/a&gt;, &lt;a href=<span class="pl-cce">\"</span>http://creativecommons.org/licenses/by/3.0<span class="pl-cce">\"</span>&gt;CC BY 3.0&lt;/a&gt; &amp;mdash; Map data &amp;copy; &lt;a href=<span class="pl-cce">\"</span>https://www.openstreetmap.org/copyright<span class="pl-cce">\"</span>&gt;OpenStreetMap&lt;/a&gt; contributors<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"subdomains"</span>: <span class="pl-s"><span class="pl-pds">"</span>abcd<span class="pl-pds">"</span></span>,
                  <span class="pl-ent">"minZoom"</span>: <span class="pl-c1">1</span>,
                  <span class="pl-ent">"maxZoom"</span>: <span class="pl-c1">16</span>,
                  <span class="pl-ent">"ext"</span>: <span class="pl-s"><span class="pl-pds">"</span>jpg<span class="pl-pds">"</span></span>
              }
          }
      }
  }</pre></div>
  <p dir="auto">The <a href="https://leaflet-extras.github.io/leaflet-providers/preview/index.html" rel="nofollow">Leaflet Providers preview list</a> has details of many other tile layers you can use.</p>
  <h2 dir="auto"><a id="user-content-custom-marker-popups" class="anchor" aria-hidden="true" href="#user-content-custom-marker-popups"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Custom marker popups</h2>
  <p dir="auto">The marker popup defaults to displaying the data for the underlying database row.</p>
  <p dir="auto">You can customize this by including a <code>popup</code> column in your results containing JSON that defines a more useful popup.</p>
  <p dir="auto">The JSON in the popup column should look something like this:</p>
  <div class="highlight highlight-source-json position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;image&quot;: &quot;https://niche-museums.imgix.net/dodgems.heic?w=800&amp;h=400&amp;fit=crop&quot;,
      &quot;alt&quot;: &quot;Dingles Fairground Heritage Centre&quot;,
      &quot;title&quot;: &quot;Dingles Fairground Heritage Centre&quot;,
      &quot;description&quot;: &quot;Home of the National Fairground Collection, Dingles has over 45,000 indoor square feet of vintage fairground rides... and you can go on them! Highlights include the last complete surviving and opera&quot;,
      &quot;link&quot;: &quot;/browse/museums/26&quot;
  }
  "><pre>{
      <span class="pl-ent">"image"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://niche-museums.imgix.net/dodgems.heic?w=800&amp;h=400&amp;fit=crop<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"alt"</span>: <span class="pl-s"><span class="pl-pds">"</span>Dingles Fairground Heritage Centre<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"title"</span>: <span class="pl-s"><span class="pl-pds">"</span>Dingles Fairground Heritage Centre<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"description"</span>: <span class="pl-s"><span class="pl-pds">"</span>Home of the National Fairground Collection, Dingles has over 45,000 indoor square feet of vintage fairground rides... and you can go on them! Highlights include the last complete surviving and opera<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"link"</span>: <span class="pl-s"><span class="pl-pds">"</span>/browse/museums/26<span class="pl-pds">"</span></span>
  }</pre></div>
  <p dir="auto">Each of these columns is optional.</p>
  <ul dir="auto">
  <li><code>title</code> is the title to show at the top of the popup</li>
  <li><code>image</code> is the URL to an image to display in the popup</li>
  <li><code>alt</code> is the alt attribute to use for that image</li>
  <li><code>description</code> is a longer string of text to use as a description</li>
  <li><code>link</code> is a URL that the marker content should link to</li>
  </ul>
  <p dir="auto">You can use the SQLite <code>json_object()</code> function to construct this data dynamically as part of your SQL query. Here's an example:</p>
  <div class="highlight highlight-source-sql position-relative overflow-auto" data-snippet-clipboard-copy-content="select json_object(
    'image', photo_url || '?w=800&amp;h=400&amp;fit=crop',
    'title', name,
    'description', substr(description, 0, 200),
    'link', '/browse/museums/' || id
    ) as popup,
    latitude, longitude from museums
  where id in (26, 27) order by id
  "><pre><span class="pl-k">select</span> json_object(
    <span class="pl-s"><span class="pl-pds">'</span>image<span class="pl-pds">'</span></span>, photo_url <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>?w=800&amp;h=400&amp;fit=crop<span class="pl-pds">'</span></span>,
    <span class="pl-s"><span class="pl-pds">'</span>title<span class="pl-pds">'</span></span>, name,
    <span class="pl-s"><span class="pl-pds">'</span>description<span class="pl-pds">'</span></span>, substr(description, <span class="pl-c1">0</span>, <span class="pl-c1">200</span>),
    <span class="pl-s"><span class="pl-pds">'</span>link<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>/browse/museums/<span class="pl-pds">'</span></span> <span class="pl-k">||</span> id
    ) <span class="pl-k">as</span> popup,
    latitude, longitude <span class="pl-k">from</span> museums
  <span class="pl-k">where</span> id <span class="pl-k">in</span> (<span class="pl-c1">26</span>, <span class="pl-c1">27</span>) <span class="pl-k">order by</span> id</pre></div>
  <p dir="auto"><a href="https://www.niche-museums.com/browse?sql=select+json_object%28%0D%0A++%27image%27%2C+photo_url+%7C%7C+%27%3Fw%3D800%26h%3D400%26fit%3Dcrop%27%2C%0D%0A++%27title%27%2C+name%2C%0D%0A++%27description%27%2C+substr%28description%2C+0%2C+200%29%2C%0D%0A++%27link%27%2C+%27%2Fbrowse%2Fmuseums%2F%27+%7C%7C+id%0D%0A++%29+as+popup%2C%0D%0A++latitude%2C+longitude+from+museums" rel="nofollow">Try that example here</a> or take a look at <a href="https://dogsheep-photos.dogsheep.net/public/photos_on_a_map" rel="nofollow">this demo built using a SQL view</a>.</p>
  <h2 dir="auto"><a id="user-content-how-i-deployed-the-demo" class="anchor" aria-hidden="true" href="#user-content-how-i-deployed-the-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>How I deployed the demo</h2>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish cloudrun global-power-plants.db \
      --service global-power-plants \
      --metadata metadata.json \
      --install=datasette-cluster-map \
      --extra-options=&quot;--config facet_time_limit_ms:1000&quot;
  "><pre><code>datasette publish cloudrun global-power-plants.db \
      --service global-power-plants \
      --metadata metadata.json \
      --install=datasette-cluster-map \
      --extra-options="--config facet_time_limit_ms:1000"
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p dir="auto">To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="cd datasette-cluster-map
  python3 -mvenv venv
  source venv/bin/activate
  "><pre><code>cd datasette-cluster-map
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p dir="auto">Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pipenv shell
  "><pre><code>pipenv shell
  </code></pre></div>
  <p dir="auto">Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'
  "><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p dir="auto">To run the tests:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest
  "><pre><code>pytest
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxMzUwMDcyODc=
name: datasette-leaflet-geojson
full_name: simonw/datasette-leaflet-geojson
html_url: https://github.com/simonw/datasette-leaflet-geojson
description: Datasette plugin that replaces any GeoJSON column values with a Leaflet map.
created_at: 2018-05-27T01:42:30Z
updated_at: 2022-08-26T23:27:11Z
pushed_at: 2022-08-26T23:27:08Z
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin", "gis", "leaflet"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-leaflet-geojson

  [![PyPI](https://img.shields.io/pypi/v/datasette-leaflet-geojson.svg)](https://pypi.org/project/datasette-leaflet-geojson/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-leaflet-geojson?include_prereleases&label=changelog)](https://github.com/simonw/datasette-leaflet-geojson/releases)
  [![Tests](https://github.com/simonw/datasette-leaflet-geojson/workflows/Test/badge.svg)](https://github.com/simonw/datasette-leaflet-geojson/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-leaflet-geojson/blob/main/LICENSE)

  Datasette plugin that replaces any GeoJSON column values with a Leaflet map

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-leaflet-geojson

  ## Usage

  Any columns containing valid GeoJSON strings will have their contents replaced with a Leaflet map when they are displayed in the Datasette interface.

  ## Demo

  You can try this plugin out at https://calands.datasettes.com/calands/superunits_with_maps

  ![datasette-leaflet-geojson in action](https://raw.github.com/simonw/datasette-leaflet-geojson/main/datasette-leaflet-geojson.png)

  ## Configuration

  By default this plugin displays maps for the first ten rows, and shows a "Click to load map" prompt for rows past the first ten.

  You can change this limit using the `default_maps_to_load` plugin configuration setting. Add this to your `metadata.json`:

  ```json
  {
      "plugins": {
          "datasette-leaflet-geojson": {
              "default_maps_to_load": 20
          }
      }
  }
  ```
  Then run Datasette with `datasette mydb.db -m metadata.json`.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-leaflet-geojson" class="anchor" aria-hidden="true" href="#user-content-datasette-leaflet-geojson"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-leaflet-geojson</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-leaflet-geojson/" rel="nofollow"><img src="https://camo.githubusercontent.com/c4cff7db933b6c868aae197528f088312b804050a0aad5ba2beaa8c2cbf23d72/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6c6561666c65742d67656f6a736f6e2e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-leaflet-geojson.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-leaflet-geojson/releases"><img src="https://camo.githubusercontent.com/2ac8aa347c051365984375e84c15b4887c4a377dbc495db5320ebfbc409bbb2b/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6c6561666c65742d67656f6a736f6e3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-leaflet-geojson?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-leaflet-geojson/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-leaflet-geojson/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-leaflet-geojson/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Datasette plugin that replaces any GeoJSON column values with a Leaflet map</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ datasette install datasette-leaflet-geojson"><pre class="notranslate"><code>$ datasette install datasette-leaflet-geojson
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Any columns containing valid GeoJSON strings will have their contents replaced with a Leaflet map when they are displayed in the Datasette interface.</p>
  <h2 dir="auto"><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p dir="auto">You can try this plugin out at <a href="https://calands.datasettes.com/calands/superunits_with_maps" rel="nofollow">https://calands.datasettes.com/calands/superunits_with_maps</a></p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer" href="https://camo.githubusercontent.com/6c5c71d68f222ee61e234b56d982f9672a8c2707dc9f95fa8084dfe11843a48e/68747470733a2f2f7261772e6769746875622e636f6d2f73696d6f6e772f6461746173657474652d6c6561666c65742d67656f6a736f6e2f6d61696e2f6461746173657474652d6c6561666c65742d67656f6a736f6e2e706e67"><img src="https://camo.githubusercontent.com/6c5c71d68f222ee61e234b56d982f9672a8c2707dc9f95fa8084dfe11843a48e/68747470733a2f2f7261772e6769746875622e636f6d2f73696d6f6e772f6461746173657474652d6c6561666c65742d67656f6a736f6e2f6d61696e2f6461746173657474652d6c6561666c65742d67656f6a736f6e2e706e67" alt="datasette-leaflet-geojson in action" data-canonical-src="https://raw.github.com/simonw/datasette-leaflet-geojson/main/datasette-leaflet-geojson.png" style="max-width: 100%;"></a></p>
  <h2 dir="auto"><a id="user-content-configuration" class="anchor" aria-hidden="true" href="#user-content-configuration"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Configuration</h2>
  <p dir="auto">By default this plugin displays maps for the first ten rows, and shows a "Click to load map" prompt for rows past the first ten.</p>
  <p dir="auto">You can change this limit using the <code>default_maps_to_load</code> plugin configuration setting. Add this to your <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-leaflet-geojson&quot;: {
              &quot;default_maps_to_load&quot;: 20
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-leaflet-geojson"</span>: {
              <span class="pl-ent">"default_maps_to_load"</span>: <span class="pl-c1">20</span>
          }
      }
  }</pre></div>
  <p dir="auto">Then run Datasette with <code>datasette mydb.db -m metadata.json</code>.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxMzg2Njk2NzM=
name: datasette-vega
full_name: simonw/datasette-vega
html_url: https://github.com/simonw/datasette-vega
description: Datasette plugin for visualizing data using Vega
created_at: 2018-06-26T01:40:54Z
updated_at: 2021-12-10T22:20:46Z
pushed_at: 2021-12-10T22:20:43Z
language: JavaScript
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "plugin", "react", "vega"]
default_branch: master
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-vega

  [![PyPI](https://img.shields.io/pypi/v/datasette-vega.svg)](https://pypi.org/project/datasette-vega/)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-vega/blob/master/LICENSE)

  A [Datasette](https://github.com/simonw/datasette) plugin that provides tools
  for generating charts using [Vega](https://vega.github.io/).

  ![Datasette Vega interface](https://raw.githubusercontent.com/simonw/datasette-vega/master/datasette-vega.png)

  Try out the latest master build as a live demo at https://datasette-vega-latest.datasette.io/ or try the latest release installed as a plugin at https://fivethirtyeight.datasettes.com/

  To add this to your Datasette installation, install the plugin like so:

      pip install datasette-vega

  The plugin will then add itself to every Datasette table view.

  If you are publishing data using the `datasette publish` command, you can
  include this plugin like so:

      datasette publish now mydatabase.db --install=datasette-vega

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-vega" class="anchor" aria-hidden="true" href="#user-content-datasette-vega"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-vega</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-vega/" rel="nofollow"><img src="https://camo.githubusercontent.com/3daf7c04104904d77b14dde7eb24240a712bd4eca97aa15afa673a306a7ef31a/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d766567612e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-vega.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-vega/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">A <a href="https://github.com/simonw/datasette">Datasette</a> plugin that provides tools
  for generating charts using <a href="https://vega.github.io/" rel="nofollow">Vega</a>.</p>
  <p dir="auto"><a target="_blank" rel="noopener noreferrer" href="https://raw.githubusercontent.com/simonw/datasette-vega/master/datasette-vega.png"><img src="https://raw.githubusercontent.com/simonw/datasette-vega/master/datasette-vega.png" alt="Datasette Vega interface" style="max-width: 100%;"></a></p>
  <p dir="auto">Try out the latest master build as a live demo at <a href="https://datasette-vega-latest.datasette.io/" rel="nofollow">https://datasette-vega-latest.datasette.io/</a> or try the latest release installed as a plugin at <a href="https://fivethirtyeight.datasettes.com/" rel="nofollow">https://fivethirtyeight.datasettes.com/</a></p>
  <p dir="auto">To add this to your Datasette installation, install the plugin like so:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install datasette-vega"><pre><code>pip install datasette-vega
  </code></pre></div>
  <p dir="auto">The plugin will then add itself to every Datasette table view.</p>
  <p dir="auto">If you are publishing data using the <code>datasette publish</code> command, you can
  include this plugin like so:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish now mydatabase.db --install=datasette-vega"><pre><code>datasette publish now mydatabase.db --install=datasette-vega
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNDA5MTI0MzI=
name: sqlite-utils
full_name: simonw/sqlite-utils
html_url: https://github.com/simonw/sqlite-utils
description: Python CLI utility and library for manipulating SQLite databases
created_at: 2018-07-14T03:21:46Z
updated_at: 2022-11-15T18:12:16Z
pushed_at: 2022-11-15T15:53:38Z
homepage: https://sqlite-utils.datasette.io
language: Python
license: apache-2.0
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
topics:
  ["cli", "click", "datasette", "datasette-io", "datasette-tool", "python", "sqlite", "sqlite-database"]

readme:
  # sqlite-utils

  [![PyPI](https://img.shields.io/pypi/v/sqlite-utils.svg)](https://pypi.org/project/sqlite-utils/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-utils?include_prereleases&label=changelog)](https://sqlite-utils.datasette.io/en/stable/changelog.html)
  [![Python 3.x](https://img.shields.io/pypi/pyversions/sqlite-utils.svg?logo=python&logoColor=white)](https://pypi.org/project/sqlite-utils/)
  [![Tests](https://github.com/simonw/sqlite-utils/workflows/Test/badge.svg)](https://github.com/simonw/sqlite-utils/actions?query=workflow%3ATest)
  [![Documentation Status](https://readthedocs.org/projects/sqlite-utils/badge/?version=stable)](http://sqlite-utils.datasette.io/en/stable/?badge=stable)
  [![codecov](https://codecov.io/gh/simonw/sqlite-utils/branch/main/graph/badge.svg)](https://codecov.io/gh/simonw/sqlite-utils)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sqlite-utils/blob/main/LICENSE)
  [![discord](https://img.shields.io/discord/823971286308356157?label=discord)](https://discord.gg/Ass7bCAMDw)

  Python CLI utility and library for manipulating SQLite databases.

  ## Some feature highlights

  - [Pipe JSON](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data) (or [CSV or TSV](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-csv-or-tsv-data)) directly into a new SQLite database file, automatically creating a table with the appropriate schema
  - [Run in-memory SQL queries](https://sqlite-utils.datasette.io/en/stable/cli.html#querying-data-directly-using-an-in-memory-database), including joins, directly against data in CSV, TSV or JSON files and view the results
  - [Configure SQLite full-text search](https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search) against your database tables and run search queries against them, ordered by relevance
  - Run [transformations against your tables](https://sqlite-utils.datasette.io/en/stable/cli.html#transforming-tables) to make schema changes that SQLite `ALTER TABLE` does not directly support, such as changing the type of a column
  - [Extract columns](https://sqlite-utils.datasette.io/en/stable/cli.html#extracting-columns-into-a-separate-table) into separate tables to better normalize your existing data

  Read more on my blog, in this series of posts on [New features in sqlite-utils](https://simonwillison.net/series/sqlite-utils-features/) and other [entries tagged sqliteutils](https://simonwillison.net/tags/sqliteutils/).

  ## Installation

      pip install sqlite-utils

  Or if you use [Homebrew](https://brew.sh/) for macOS:

      brew install sqlite-utils

  ## Using as a CLI tool

  Now you can do things with the CLI utility like this:

      $ sqlite-utils memory dogs.csv "select * from t"
      [{"id": 1, "age": 4, "name": "Cleo"},
       {"id": 2, "age": 2, "name": "Pancakes"}]

      $ sqlite-utils insert dogs.db dogs dogs.csv --csv
      [####################################]  100%

      $ sqlite-utils tables dogs.db --counts
      [{"table": "dogs", "count": 2}]

      $ sqlite-utils dogs.db "select id, name from dogs"
      [{"id": 1, "name": "Cleo"},
       {"id": 2, "name": "Pancakes"}]

      $ sqlite-utils dogs.db "select * from dogs" --csv
      id,age,name
      1,4,Cleo
      2,2,Pancakes

      $ sqlite-utils dogs.db "select * from dogs" --table
        id    age  name
      ----  -----  --------
         1      4  Cleo
         2      2  Pancakes

  You can import JSON data into a new database table like this:

      $ curl https://api.github.com/repos/simonw/sqlite-utils/releases \
          | sqlite-utils insert releases.db releases - --pk id

  Or for data in a CSV file:

      $ sqlite-utils insert dogs.db dogs dogs.csv --csv

  `sqlite-utils memory` lets you import CSV or JSON data into an in-memory database and run SQL queries against it in a single command:

      $ cat dogs.csv | sqlite-utils memory - "select name, age from stdin"

  See the [full CLI documentation](https://sqlite-utils.datasette.io/en/stable/cli.html) for comprehensive coverage of many more commands.

  ## Using as a library

  You can also `import sqlite_utils` and use it as a Python library like this:

  ```python
  import sqlite_utils
  db = sqlite_utils.Database("demo_database.db")
  # This line creates a "dogs" table if one does not already exist:
  db["dogs"].insert_all([
      {"id": 1, "age": 4, "name": "Cleo"},
      {"id": 2, "age": 2, "name": "Pancakes"}
  ], pk="id")
  ```

  Check out the [full library documentation](https://sqlite-utils.datasette.io/en/stable/python-api.html) for everything else you can do with the Python library.

  ## Related projects

  * [Datasette](https://datasette.io/): A tool for exploring and publishing data
  * [csvs-to-sqlite](https://github.com/simonw/csvs-to-sqlite): Convert CSV files into a SQLite database
  * [db-to-sqlite](https://github.com/simonw/db-to-sqlite): CLI tool for exporting a MySQL or PostgreSQL database as a SQLite file
  * [dogsheep](https://dogsheep.github.io/): A family of tools for personal analytics, built on top of `sqlite-utils`

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-sqlite-utils" class="anchor" aria-hidden="true" href="#user-content-sqlite-utils"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>sqlite-utils</h1>
  <p dir="auto"><a href="https://pypi.org/project/sqlite-utils/" rel="nofollow"><img src="https://camo.githubusercontent.com/f691f8124616f99c7978ecb2a58841aa6a7ace31234b0c87d392c4ef8db853e4/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d7574696c732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/sqlite-utils.svg" style="max-width: 100%;"></a>
  <a href="https://sqlite-utils.datasette.io/en/stable/changelog.html" rel="nofollow"><img src="https://camo.githubusercontent.com/b114e00dffe1e5b24d21c5cd11dc97f8a6ab83da51acbb971e5daebeeb47d690/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f73716c6974652d7574696c733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/sqlite-utils?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://pypi.org/project/sqlite-utils/" rel="nofollow"><img src="https://camo.githubusercontent.com/2413fbc1359bcdcf500aa31b497c8c25acf7db3f3fc3a33c540e809c984a189e/68747470733a2f2f696d672e736869656c64732e696f2f707970692f707976657273696f6e732f73716c6974652d7574696c732e7376673f6c6f676f3d707974686f6e266c6f676f436f6c6f723d7768697465" alt="Python 3.x" data-canonical-src="https://img.shields.io/pypi/pyversions/sqlite-utils.svg?logo=python&amp;logoColor=white" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/sqlite-utils/actions?query=workflow%3ATest"><img src="https://github.com/simonw/sqlite-utils/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="http://sqlite-utils.datasette.io/en/stable/?badge=stable" rel="nofollow"><img src="https://camo.githubusercontent.com/ebfbbefc97415a2c49a4757ef1f7836a0771eb543ba3c1d1267e466ab0ed8bd1/68747470733a2f2f72656164746865646f63732e6f72672f70726f6a656374732f73716c6974652d7574696c732f62616467652f3f76657273696f6e3d737461626c65" alt="Documentation Status" data-canonical-src="https://readthedocs.org/projects/sqlite-utils/badge/?version=stable" style="max-width: 100%;"></a>
  <a href="https://codecov.io/gh/simonw/sqlite-utils" rel="nofollow"><img src="https://camo.githubusercontent.com/762e8034bbc8654dab7f15e85ae5eb836d07b86ebf68ab955a8c18d8035fbf13/68747470733a2f2f636f6465636f762e696f2f67682f73696d6f6e772f73716c6974652d7574696c732f6272616e63682f6d61696e2f67726170682f62616467652e737667" alt="codecov" data-canonical-src="https://codecov.io/gh/simonw/sqlite-utils/branch/main/graph/badge.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/sqlite-utils/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a>
  <a href="https://discord.gg/Ass7bCAMDw" rel="nofollow"><img src="https://camo.githubusercontent.com/1e67d99839ed78183f50f78dee6bb4897d95ff44a4b7182d4bfd82c505d8d835/68747470733a2f2f696d672e736869656c64732e696f2f646973636f72642f3832333937313238363330383335363135373f6c6162656c3d646973636f7264" alt="discord" data-canonical-src="https://img.shields.io/discord/823971286308356157?label=discord" style="max-width: 100%;"></a></p>
  <p dir="auto">Python CLI utility and library for manipulating SQLite databases.</p>
  <h2 dir="auto"><a id="user-content-some-feature-highlights" class="anchor" aria-hidden="true" href="#user-content-some-feature-highlights"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Some feature highlights</h2>
  <ul dir="auto">
  <li><a href="https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data" rel="nofollow">Pipe JSON</a> (or <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-csv-or-tsv-data" rel="nofollow">CSV or TSV</a>) directly into a new SQLite database file, automatically creating a table with the appropriate schema</li>
  <li><a href="https://sqlite-utils.datasette.io/en/stable/cli.html#querying-data-directly-using-an-in-memory-database" rel="nofollow">Run in-memory SQL queries</a>, including joins, directly against data in CSV, TSV or JSON files and view the results</li>
  <li><a href="https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search" rel="nofollow">Configure SQLite full-text search</a> against your database tables and run search queries against them, ordered by relevance</li>
  <li>Run <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#transforming-tables" rel="nofollow">transformations against your tables</a> to make schema changes that SQLite <code>ALTER TABLE</code> does not directly support, such as changing the type of a column</li>
  <li><a href="https://sqlite-utils.datasette.io/en/stable/cli.html#extracting-columns-into-a-separate-table" rel="nofollow">Extract columns</a> into separate tables to better normalize your existing data</li>
  </ul>
  <p dir="auto">Read more on my blog, in this series of posts on <a href="https://simonwillison.net/series/sqlite-utils-features/" rel="nofollow">New features in sqlite-utils</a> and other <a href="https://simonwillison.net/tags/sqliteutils/" rel="nofollow">entries tagged sqliteutils</a>.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install sqlite-utils"><pre class="notranslate"><code>pip install sqlite-utils
  </code></pre></div>
  <p dir="auto">Or if you use <a href="https://brew.sh/" rel="nofollow">Homebrew</a> for macOS:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="brew install sqlite-utils"><pre class="notranslate"><code>brew install sqlite-utils
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-using-as-a-cli-tool" class="anchor" aria-hidden="true" href="#user-content-using-as-a-cli-tool"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Using as a CLI tool</h2>
  <p dir="auto">Now you can do things with the CLI utility like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ sqlite-utils memory dogs.csv &quot;select * from t&quot;
  [{&quot;id&quot;: 1, &quot;age&quot;: 4, &quot;name&quot;: &quot;Cleo&quot;},
   {&quot;id&quot;: 2, &quot;age&quot;: 2, &quot;name&quot;: &quot;Pancakes&quot;}]

  $ sqlite-utils insert dogs.db dogs dogs.csv --csv
  [####################################]  100%

  $ sqlite-utils tables dogs.db --counts
  [{&quot;table&quot;: &quot;dogs&quot;, &quot;count&quot;: 2}]

  $ sqlite-utils dogs.db &quot;select id, name from dogs&quot;
  [{&quot;id&quot;: 1, &quot;name&quot;: &quot;Cleo&quot;},
   {&quot;id&quot;: 2, &quot;name&quot;: &quot;Pancakes&quot;}]

  $ sqlite-utils dogs.db &quot;select * from dogs&quot; --csv
  id,age,name
  1,4,Cleo
  2,2,Pancakes

  $ sqlite-utils dogs.db &quot;select * from dogs&quot; --table
    id    age  name
  ----  -----  --------
     1      4  Cleo
     2      2  Pancakes"><pre class="notranslate"><code>$ sqlite-utils memory dogs.csv "select * from t"
  [{"id": 1, "age": 4, "name": "Cleo"},
   {"id": 2, "age": 2, "name": "Pancakes"}]

  $ sqlite-utils insert dogs.db dogs dogs.csv --csv
  [####################################]  100%

  $ sqlite-utils tables dogs.db --counts
  [{"table": "dogs", "count": 2}]

  $ sqlite-utils dogs.db "select id, name from dogs"
  [{"id": 1, "name": "Cleo"},
   {"id": 2, "name": "Pancakes"}]

  $ sqlite-utils dogs.db "select * from dogs" --csv
  id,age,name
  1,4,Cleo
  2,2,Pancakes

  $ sqlite-utils dogs.db "select * from dogs" --table
    id    age  name
  ----  -----  --------
     1      4  Cleo
     2      2  Pancakes
  </code></pre></div>
  <p dir="auto">You can import JSON data into a new database table like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ curl https://api.github.com/repos/simonw/sqlite-utils/releases \
      | sqlite-utils insert releases.db releases - --pk id"><pre class="notranslate"><code>$ curl https://api.github.com/repos/simonw/sqlite-utils/releases \
      | sqlite-utils insert releases.db releases - --pk id
  </code></pre></div>
  <p dir="auto">Or for data in a CSV file:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ sqlite-utils insert dogs.db dogs dogs.csv --csv"><pre class="notranslate"><code>$ sqlite-utils insert dogs.db dogs dogs.csv --csv
  </code></pre></div>
  <p dir="auto"><code>sqlite-utils memory</code> lets you import CSV or JSON data into an in-memory database and run SQL queries against it in a single command:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ cat dogs.csv | sqlite-utils memory - &quot;select name, age from stdin&quot;"><pre class="notranslate"><code>$ cat dogs.csv | sqlite-utils memory - "select name, age from stdin"
  </code></pre></div>
  <p dir="auto">See the <a href="https://sqlite-utils.datasette.io/en/stable/cli.html" rel="nofollow">full CLI documentation</a> for comprehensive coverage of many more commands.</p>
  <h2 dir="auto"><a id="user-content-using-as-a-library" class="anchor" aria-hidden="true" href="#user-content-using-as-a-library"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Using as a library</h2>
  <p dir="auto">You can also <code>import sqlite_utils</code> and use it as a Python library like this:</p>
  <div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import sqlite_utils
  db = sqlite_utils.Database(&quot;demo_database.db&quot;)
  # This line creates a &quot;dogs&quot; table if one does not already exist:
  db[&quot;dogs&quot;].insert_all([
      {&quot;id&quot;: 1, &quot;age&quot;: 4, &quot;name&quot;: &quot;Cleo&quot;},
      {&quot;id&quot;: 2, &quot;age&quot;: 2, &quot;name&quot;: &quot;Pancakes&quot;}
  ], pk=&quot;id&quot;)"><pre><span class="pl-k">import</span> <span class="pl-s1">sqlite_utils</span>
  <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-s1">sqlite_utils</span>.<span class="pl-v">Database</span>(<span class="pl-s">"demo_database.db"</span>)
  <span class="pl-c"># This line creates a "dogs" table if one does not already exist:</span>
  <span class="pl-s1">db</span>[<span class="pl-s">"dogs"</span>].<span class="pl-en">insert_all</span>([
      {<span class="pl-s">"id"</span>: <span class="pl-c1">1</span>, <span class="pl-s">"age"</span>: <span class="pl-c1">4</span>, <span class="pl-s">"name"</span>: <span class="pl-s">"Cleo"</span>},
      {<span class="pl-s">"id"</span>: <span class="pl-c1">2</span>, <span class="pl-s">"age"</span>: <span class="pl-c1">2</span>, <span class="pl-s">"name"</span>: <span class="pl-s">"Pancakes"</span>}
  ], <span class="pl-s1">pk</span><span class="pl-c1">=</span><span class="pl-s">"id"</span>)</pre></div>
  <p dir="auto">Check out the <a href="https://sqlite-utils.datasette.io/en/stable/python-api.html" rel="nofollow">full library documentation</a> for everything else you can do with the Python library.</p>
  <h2 dir="auto"><a id="user-content-related-projects" class="anchor" aria-hidden="true" href="#user-content-related-projects"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Related projects</h2>
  <ul dir="auto">
  <li><a href="https://datasette.io/" rel="nofollow">Datasette</a>: A tool for exploring and publishing data</li>
  <li><a href="https://github.com/simonw/csvs-to-sqlite">csvs-to-sqlite</a>: Convert CSV files into a SQLite database</li>
  <li><a href="https://github.com/simonw/db-to-sqlite">db-to-sqlite</a>: CLI tool for exporting a MySQL or PostgreSQL database as a SQLite file</li>
  <li><a href="https://dogsheep.github.io/" rel="nofollow">dogsheep</a>: A family of tools for personal analytics, built on top of <code>sqlite-utils</code></li>
  </ul>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNDI5NjczNDc=
name: datasette-json-html
full_name: simonw/datasette-json-html
html_url: https://github.com/simonw/datasette-json-html
description: Datasette plugin for rendering HTML based on JSON values
created_at: 2018-07-31T05:41:39Z
updated_at: 2022-03-15T04:54:15Z
pushed_at: 2022-03-22T01:43:59Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "plugin"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-json-html

  [![PyPI](https://img.shields.io/pypi/v/datasette-json-html.svg)](https://pypi.org/project/datasette-json-html/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-json-html?include_prereleases&label=changelog)](https://github.com/simonw/datasette-json-html/releases)
  [![Tests](https://github.com/simonw/datasette-json-html/workflows/Test/badge.svg)](https://github.com/simonw/datasette-remote-metadata/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-json-html/blob/main/LICENSE)

  Datasette plugin for rendering HTML based on JSON values, using the [render_cell plugin hook](https://docs.datasette.io/en/stable/plugin_hooks.html#render-cell-value-column-table-database-datasette).

  This plugin looks for cell values that match a very specific JSON format and converts them into HTML when they are rendered by the Datasette interface.

  ## Links

      {
          "href": "https://simonwillison.net/",
          "label": "Simon Willison"
      }

  Will be rendered as an `<a href="">` link:

      <a href="https://simonwillison.net/">Simon Willison</a>

  You can set a tooltip on the link using a `"title"` key:


      {
          "href": "https://simonwillison.net/",
          "label": "Simon Willison",
          "title": "My blog"
      }

  Produces:

      <a href="https://simonwillison.net/" title="My blog">Simon Willison</a>

  You can also include a description, which will be displayed below the link. If descriptions include newlines they will be converted to `<br>` elements:

      select json_object(
          "href", "https://simonwillison.net/",
          "label", "Simon Willison",
          "description", "This can contain" || x'0a' || "newlines"
      )

  Produces:

      <strong><a href="https://simonwillison.net/">Simon Willison</a></strong><br>This can contain<br>newlines

  * [Literal JSON link demo](https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22href%22%3A+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++%22label%22%3A+%22Simon+Willison%22%2C%0D%0A++++%22title%22%3A+%22My+blog%22%0D%0A%7D%27)

  ## List of links

      [
          {
              "href": "https://simonwillison.net/",
              "label": "Simon Willison"
          },
          {
              "href": "https://github.com/simonw/datasette",
              "label": "Datasette"
          }
      ]

  Will be rendered as a comma-separated list of `<a href="">` links:

      <a href="https://simonwillison.net/">Simon Willison</a>,
      <a href="https://github.com/simonw/datasette">Datasette</a>

  The `href` property must begin with `https://` or `http://` or `/`, to avoid potential XSS injection attacks (for example URLs that begin with `javascript:`).

  Lists of links cannot include `"description"` keys.

  * [Literal list of links demo](https://datasette-json-html.datasette.io/demo?sql=select+%27%5B%0D%0A++++%7B%0D%0A++++++++%22href%22%3A+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++++++%22label%22%3A+%22Simon+Willison%22%0D%0A++++%7D%2C%0D%0A++++%7B%0D%0A++++++++%22href%22%3A+%22https%3A%2F%2Fgithub.com%2Fsimonw%2Fdatasette%22%2C%0D%0A++++++++%22label%22%3A+%22Datasette%22%0D%0A++++%7D%0D%0A%5D%27)

  ## Images

  The image tag is more complex. The most basic version looks like this:

      {
          "img_src": "https://placekitten.com/200/300"
      }

  This will render as:

      <img src="https://placekitten.com/200/300">

  But you can also include one or more of `alt`, `caption`, `width` and `href`.

  If you include width or alt, they will be added as attributes:

      {
          "img_src": "https://placekitten.com/200/300",
          "alt": "Kitten",
          "width": 200
      }

  Produces:

      <img src="https://placekitten.com/200/300"
          alt="Kitten" width="200">

  * [Literal image demo](https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22img_src%22%3A+%22https%3A%2F%2Fplacekitten.com%2F200%2F300%22%2C%0D%0A++++%22alt%22%3A+%22Kitten%22%2C%0D%0A++++%22width%22%3A+200%0D%0A%7D%27)

  The `href` key will cause the image to be wrapped in a link:

      {
          "img_src": "https://placekitten.com/200/300",
          "href": "http://www.example.com"
      }

  Produces:

      <a href="http://www.example.com">
          <img src="https://placekitten.com/200/300">
      </a>

  The `caption` key wraps everything in a fancy figure/figcaption block:

      {
          "img_src": "https://placekitten.com/200/300",
          "caption": "Kitten caption"
      }

  Produces:

      <figure>
          <img src="https://placekitten.com/200/300"></a>
          <figcaption>Kitten caption</figcaption>
      </figure>

  ## Preformatted text

  You can use `{"pre": "text"}` to render text in a `<pre>` HTML tag:

      {
          "pre": "This\nhas\nnewlines"
      }

  Produces:

      <pre>This
      has
      newlines</pre>

  If the value attached to the `"pre"` key is itself a JSON object, that JSON will be pretty-printed:

      {
          "pre": {
              "this": {
                  "object": ["is", "nested"]
              }
          }
      }

  Produces:

      <pre>{
        &#34;this&#34;: {
          &#34;object&#34;: [
            &#34;is&#34;,
            &#34;nested&#34;
          ]
        }
      }</pre>

  * [Preformatted text with JSON demo](https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22pre%22%3A+%7B%0D%0A++++++++%22this%22%3A+%7B%0D%0A++++++++++++%22object%22%3A+%5B%22is%22%2C+%22nested%22%5D%0D%0A++++++++%7D%0D%0A++++%7D%0D%0A%7D%27)
  * [Preformatted text demo showing the Mandelbrot Set](https://datasette-json-html.datasette.io/demo?sql=WITH+RECURSIVE%0D%0A++xaxis%28x%29+AS+%28VALUES%28-2.0%29+UNION+ALL+SELECT+x%2B0.05+FROM+xaxis+WHERE+x%3C1.2%29%2C%0D%0A++yaxis%28y%29+AS+%28VALUES%28-1.0%29+UNION+ALL+SELECT+y%2B0.1+FROM+yaxis+WHERE+y%3C1.0%29%2C%0D%0A++m%28iter%2C+cx%2C+cy%2C+x%2C+y%29+AS+%28%0D%0A++++SELECT+0%2C+x%2C+y%2C+0.0%2C+0.0+FROM+xaxis%2C+yaxis%0D%0A++++UNION+ALL%0D%0A++++SELECT+iter%2B1%2C+cx%2C+cy%2C+x*x-y*y+%2B+cx%2C+2.0*x*y+%2B+cy+FROM+m+%0D%0A+++++WHERE+%28x*x+%2B+y*y%29+%3C+4.0+AND+iter%3C28%0D%0A++%29%2C%0D%0A++m2%28iter%2C+cx%2C+cy%29+AS+%28%0D%0A++++SELECT+max%28iter%29%2C+cx%2C+cy+FROM+m+GROUP+BY+cx%2C+cy%0D%0A++%29%2C%0D%0A++a%28t%29+AS+%28%0D%0A++++SELECT+group_concat%28+substr%28%27+.%2B*%23%27%2C+1%2Bmin%28iter%2F7%2C4%29%2C+1%29%2C+%27%27%29+%0D%0A++++FROM+m2+GROUP+BY+cy%0D%0A++%29%0D%0ASELECT+json_object%28%27pre%27%2C+group_concat%28rtrim%28t%29%2Cx%270a%27%29%29+FROM+a%3B) using [this example](https://www.sqlite.org/lang_with.html#outlandish_recursive_query_examples) from the SQLite documentation

  ## Using these with SQLite JSON functions

  The most powerful way to make use of this plugin is in conjunction with SQLite's [JSON functions](https://www.sqlite.org/json1.html). For example:

      select json_object(
          "href", "https://simonwillison.net/",
          "label", "Simon Willison"
      );

  * [json_object() link demo](https://datasette-json-html.datasette.io/demo?sql=select+json_object%28%0D%0A++++%22href%22%2C+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++%22label%22%2C+%22Simon+Willison%22%0D%0A%29%3B)

  You can use these functions to construct JSON objects that work with the plugin from data in a table:

      select id, json_object(
          "href", url, "label", text
      ) from mytable;

  * [Demo that builds links against a table](https://datasette-json-html.datasette.io/demo?sql=select+json_object%28%22href%22%2C+url%2C+%22label%22%2C+package%2C+%22title%22%2C+package+%7C%7C+%22+%22+%7C%7C+url%29+as+package+from+packages)

  The `json_group_array()` function is an aggregate function similar to `group_concat()` - it allows you to construct lists of JSON objects in conjunction with a `GROUP BY` clause.

  This means you can use it to construct dynamic lists of links, for example:

      select
          substr(package, 0, 12) as prefix,
          json_group_array(
              json_object(
                  "href", url,
                  "label", package
              )
          ) as package_links
      from packages
      group by prefix

  * [Demo of json_group_array()](https://datasette-json-html.datasette.io/demo?sql=select%0D%0A++++substr%28package%2C+0%2C+12%29+as+prefix%2C%0D%0A++++json_group_array%28%0D%0A++++++++json_object%28%0D%0A++++++++++++%22href%22%2C+url%2C%0D%0A++++++++++++%22label%22%2C+package%0D%0A++++++++%29%0D%0A++++%29+as+package_links%0D%0Afrom+packages%0D%0Agroup+by+prefix)

  ## The `urllib_quote_plus()` SQL function

  Since this plugin is designed to be used with SQL that constructs the underlying JSON structure, it is likely you will need to construct dynamic URLs from results returned by a SQL query.

  This plugin registers a custom SQLite function called `urllib_quote_plus()` to help you do that. It lets you use Python's [urllib.parse.quote\_plus() function](https://docs.python.org/3/library/urllib.parse.html#urllib.parse.quote_plus) from within a SQL query.

  Here's an example of how you might use it:

      select id, json_object(
          "href",
          "/mydatabase/other_table?_search=" || urllib_quote_plus(text),
          "label", text
      ) from mytable;

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-json-html" class="anchor" aria-hidden="true" href="#user-content-datasette-json-html"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-json-html</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-json-html/" rel="nofollow"><img src="https://camo.githubusercontent.com/6781ecd1fa88174e5228eff74ade944f8f69a051d14df112d9dbc18bd3543551/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6a736f6e2d68746d6c2e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-json-html.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-json-html/releases"><img src="https://camo.githubusercontent.com/56a671c6595b71bd6d225b1c6d2ee0c5dbadd9bbbb44e4074ad4afe4c18905ce/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6a736f6e2d68746d6c3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-json-html?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-remote-metadata/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-json-html/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-json-html/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Datasette plugin for rendering HTML based on JSON values, using the <a href="https://docs.datasette.io/en/stable/plugin_hooks.html#render-cell-value-column-table-database-datasette" rel="nofollow">render_cell plugin hook</a>.</p>
  <p dir="auto">This plugin looks for cell values that match a very specific JSON format and converts them into HTML when they are rendered by the Datasette interface.</p>
  <h2 dir="auto"><a id="user-content-links" class="anchor" aria-hidden="true" href="#user-content-links"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Links</h2>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;href&quot;: &quot;https://simonwillison.net/&quot;,
      &quot;label&quot;: &quot;Simon Willison&quot;
  }"><pre><code>{
      "href": "https://simonwillison.net/",
      "label": "Simon Willison"
  }
  </code></pre></div>
  <p dir="auto">Will be rendered as an <code>&lt;a href=""&gt;</code> link:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;a href=&quot;https://simonwillison.net/&quot;&gt;Simon Willison&lt;/a&gt;"><pre><code>&lt;a href="https://simonwillison.net/"&gt;Simon Willison&lt;/a&gt;
  </code></pre></div>
  <p dir="auto">You can set a tooltip on the link using a <code>"title"</code> key:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;href&quot;: &quot;https://simonwillison.net/&quot;,
      &quot;label&quot;: &quot;Simon Willison&quot;,
      &quot;title&quot;: &quot;My blog&quot;
  }"><pre><code>{
      "href": "https://simonwillison.net/",
      "label": "Simon Willison",
      "title": "My blog"
  }
  </code></pre></div>
  <p dir="auto">Produces:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;a href=&quot;https://simonwillison.net/&quot; title=&quot;My blog&quot;&gt;Simon Willison&lt;/a&gt;"><pre><code>&lt;a href="https://simonwillison.net/" title="My blog"&gt;Simon Willison&lt;/a&gt;
  </code></pre></div>
  <p dir="auto">You can also include a description, which will be displayed below the link. If descriptions include newlines they will be converted to <code>&lt;br&gt;</code> elements:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="select json_object(
      &quot;href&quot;, &quot;https://simonwillison.net/&quot;,
      &quot;label&quot;, &quot;Simon Willison&quot;,
      &quot;description&quot;, &quot;This can contain&quot; || x'0a' || &quot;newlines&quot;
  )"><pre><code>select json_object(
      "href", "https://simonwillison.net/",
      "label", "Simon Willison",
      "description", "This can contain" || x'0a' || "newlines"
  )
  </code></pre></div>
  <p dir="auto">Produces:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;strong&gt;&lt;a href=&quot;https://simonwillison.net/&quot;&gt;Simon Willison&lt;/a&gt;&lt;/strong&gt;&lt;br&gt;This can contain&lt;br&gt;newlines"><pre><code>&lt;strong&gt;&lt;a href="https://simonwillison.net/"&gt;Simon Willison&lt;/a&gt;&lt;/strong&gt;&lt;br&gt;This can contain&lt;br&gt;newlines
  </code></pre></div>
  <ul dir="auto">
  <li><a href="https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22href%22%3A+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++%22label%22%3A+%22Simon+Willison%22%2C%0D%0A++++%22title%22%3A+%22My+blog%22%0D%0A%7D%27" rel="nofollow">Literal JSON link demo</a></li>
  </ul>
  <h2 dir="auto"><a id="user-content-list-of-links" class="anchor" aria-hidden="true" href="#user-content-list-of-links"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>List of links</h2>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="[
      {
          &quot;href&quot;: &quot;https://simonwillison.net/&quot;,
          &quot;label&quot;: &quot;Simon Willison&quot;
      },
      {
          &quot;href&quot;: &quot;https://github.com/simonw/datasette&quot;,
          &quot;label&quot;: &quot;Datasette&quot;
      }
  ]"><pre><code>[
      {
          "href": "https://simonwillison.net/",
          "label": "Simon Willison"
      },
      {
          "href": "https://github.com/simonw/datasette",
          "label": "Datasette"
      }
  ]
  </code></pre></div>
  <p dir="auto">Will be rendered as a comma-separated list of <code>&lt;a href=""&gt;</code> links:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;a href=&quot;https://simonwillison.net/&quot;&gt;Simon Willison&lt;/a&gt;,
  &lt;a href=&quot;https://github.com/simonw/datasette&quot;&gt;Datasette&lt;/a&gt;"><pre><code>&lt;a href="https://simonwillison.net/"&gt;Simon Willison&lt;/a&gt;,
  &lt;a href="https://github.com/simonw/datasette"&gt;Datasette&lt;/a&gt;
  </code></pre></div>
  <p dir="auto">The <code>href</code> property must begin with <code>https://</code> or <code>http://</code> or <code>/</code>, to avoid potential XSS injection attacks (for example URLs that begin with <code>javascript:</code>).</p>
  <p dir="auto">Lists of links cannot include <code>"description"</code> keys.</p>
  <ul dir="auto">
  <li><a href="https://datasette-json-html.datasette.io/demo?sql=select+%27%5B%0D%0A++++%7B%0D%0A++++++++%22href%22%3A+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++++++%22label%22%3A+%22Simon+Willison%22%0D%0A++++%7D%2C%0D%0A++++%7B%0D%0A++++++++%22href%22%3A+%22https%3A%2F%2Fgithub.com%2Fsimonw%2Fdatasette%22%2C%0D%0A++++++++%22label%22%3A+%22Datasette%22%0D%0A++++%7D%0D%0A%5D%27" rel="nofollow">Literal list of links demo</a></li>
  </ul>
  <h2 dir="auto"><a id="user-content-images" class="anchor" aria-hidden="true" href="#user-content-images"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Images</h2>
  <p dir="auto">The image tag is more complex. The most basic version looks like this:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;img_src&quot;: &quot;https://placekitten.com/200/300&quot;
  }"><pre><code>{
      "img_src": "https://placekitten.com/200/300"
  }
  </code></pre></div>
  <p dir="auto">This will render as:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;img src=&quot;https://placekitten.com/200/300&quot;&gt;"><pre><code>&lt;img src="https://placekitten.com/200/300"&gt;
  </code></pre></div>
  <p dir="auto">But you can also include one or more of <code>alt</code>, <code>caption</code>, <code>width</code> and <code>href</code>.</p>
  <p dir="auto">If you include width or alt, they will be added as attributes:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;img_src&quot;: &quot;https://placekitten.com/200/300&quot;,
      &quot;alt&quot;: &quot;Kitten&quot;,
      &quot;width&quot;: 200
  }"><pre><code>{
      "img_src": "https://placekitten.com/200/300",
      "alt": "Kitten",
      "width": 200
  }
  </code></pre></div>
  <p dir="auto">Produces:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;img src=&quot;https://placekitten.com/200/300&quot;
      alt=&quot;Kitten&quot; width=&quot;200&quot;&gt;"><pre><code>&lt;img src="https://placekitten.com/200/300"
      alt="Kitten" width="200"&gt;
  </code></pre></div>
  <ul dir="auto">
  <li><a href="https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22img_src%22%3A+%22https%3A%2F%2Fplacekitten.com%2F200%2F300%22%2C%0D%0A++++%22alt%22%3A+%22Kitten%22%2C%0D%0A++++%22width%22%3A+200%0D%0A%7D%27" rel="nofollow">Literal image demo</a></li>
  </ul>
  <p dir="auto">The <code>href</code> key will cause the image to be wrapped in a link:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;img_src&quot;: &quot;https://placekitten.com/200/300&quot;,
      &quot;href&quot;: &quot;http://www.example.com&quot;
  }"><pre><code>{
      "img_src": "https://placekitten.com/200/300",
      "href": "http://www.example.com"
  }
  </code></pre></div>
  <p dir="auto">Produces:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;a href=&quot;http://www.example.com&quot;&gt;
      &lt;img src=&quot;https://placekitten.com/200/300&quot;&gt;
  &lt;/a&gt;"><pre><code>&lt;a href="http://www.example.com"&gt;
      &lt;img src="https://placekitten.com/200/300"&gt;
  &lt;/a&gt;
  </code></pre></div>
  <p dir="auto">The <code>caption</code> key wraps everything in a fancy figure/figcaption block:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;img_src&quot;: &quot;https://placekitten.com/200/300&quot;,
      &quot;caption&quot;: &quot;Kitten caption&quot;
  }"><pre><code>{
      "img_src": "https://placekitten.com/200/300",
      "caption": "Kitten caption"
  }
  </code></pre></div>
  <p dir="auto">Produces:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;figure&gt;
      &lt;img src=&quot;https://placekitten.com/200/300&quot;&gt;&lt;/a&gt;
      &lt;figcaption&gt;Kitten caption&lt;/figcaption&gt;
  &lt;/figure&gt;"><pre><code>&lt;figure&gt;
      &lt;img src="https://placekitten.com/200/300"&gt;&lt;/a&gt;
      &lt;figcaption&gt;Kitten caption&lt;/figcaption&gt;
  &lt;/figure&gt;
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-preformatted-text" class="anchor" aria-hidden="true" href="#user-content-preformatted-text"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Preformatted text</h2>
  <p dir="auto">You can use <code>{"pre": "text"}</code> to render text in a <code>&lt;pre&gt;</code> HTML tag:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;pre&quot;: &quot;This\nhas\nnewlines&quot;
  }"><pre><code>{
      "pre": "This\nhas\nnewlines"
  }
  </code></pre></div>
  <p dir="auto">Produces:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;pre&gt;This
  has
  newlines&lt;/pre&gt;"><pre><code>&lt;pre&gt;This
  has
  newlines&lt;/pre&gt;
  </code></pre></div>
  <p dir="auto">If the value attached to the <code>"pre"</code> key is itself a JSON object, that JSON will be pretty-printed:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;pre&quot;: {
          &quot;this&quot;: {
              &quot;object&quot;: [&quot;is&quot;, &quot;nested&quot;]
          }
      }
  }"><pre><code>{
      "pre": {
          "this": {
              "object": ["is", "nested"]
          }
      }
  }
  </code></pre></div>
  <p dir="auto">Produces:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="&lt;pre&gt;{
    &amp;#34;this&amp;#34;: {
      &amp;#34;object&amp;#34;: [
        &amp;#34;is&amp;#34;,
        &amp;#34;nested&amp;#34;
      ]
    }
  }&lt;/pre&gt;"><pre><code>&lt;pre&gt;{
    &amp;#34;this&amp;#34;: {
      &amp;#34;object&amp;#34;: [
        &amp;#34;is&amp;#34;,
        &amp;#34;nested&amp;#34;
      ]
    }
  }&lt;/pre&gt;
  </code></pre></div>
  <ul dir="auto">
  <li><a href="https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22pre%22%3A+%7B%0D%0A++++++++%22this%22%3A+%7B%0D%0A++++++++++++%22object%22%3A+%5B%22is%22%2C+%22nested%22%5D%0D%0A++++++++%7D%0D%0A++++%7D%0D%0A%7D%27" rel="nofollow">Preformatted text with JSON demo</a></li>
  <li><a href="https://datasette-json-html.datasette.io/demo?sql=WITH+RECURSIVE%0D%0A++xaxis%28x%29+AS+%28VALUES%28-2.0%29+UNION+ALL+SELECT+x%2B0.05+FROM+xaxis+WHERE+x%3C1.2%29%2C%0D%0A++yaxis%28y%29+AS+%28VALUES%28-1.0%29+UNION+ALL+SELECT+y%2B0.1+FROM+yaxis+WHERE+y%3C1.0%29%2C%0D%0A++m%28iter%2C+cx%2C+cy%2C+x%2C+y%29+AS+%28%0D%0A++++SELECT+0%2C+x%2C+y%2C+0.0%2C+0.0+FROM+xaxis%2C+yaxis%0D%0A++++UNION+ALL%0D%0A++++SELECT+iter%2B1%2C+cx%2C+cy%2C+x*x-y*y+%2B+cx%2C+2.0*x*y+%2B+cy+FROM+m+%0D%0A+++++WHERE+%28x*x+%2B+y*y%29+%3C+4.0+AND+iter%3C28%0D%0A++%29%2C%0D%0A++m2%28iter%2C+cx%2C+cy%29+AS+%28%0D%0A++++SELECT+max%28iter%29%2C+cx%2C+cy+FROM+m+GROUP+BY+cx%2C+cy%0D%0A++%29%2C%0D%0A++a%28t%29+AS+%28%0D%0A++++SELECT+group_concat%28+substr%28%27+.%2B*%23%27%2C+1%2Bmin%28iter%2F7%2C4%29%2C+1%29%2C+%27%27%29+%0D%0A++++FROM+m2+GROUP+BY+cy%0D%0A++%29%0D%0ASELECT+json_object%28%27pre%27%2C+group_concat%28rtrim%28t%29%2Cx%270a%27%29%29+FROM+a%3B" rel="nofollow">Preformatted text demo showing the Mandelbrot Set</a> using <a href="https://www.sqlite.org/lang_with.html#outlandish_recursive_query_examples" rel="nofollow">this example</a> from the SQLite documentation</li>
  </ul>
  <h2 dir="auto"><a id="user-content-using-these-with-sqlite-json-functions" class="anchor" aria-hidden="true" href="#user-content-using-these-with-sqlite-json-functions"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Using these with SQLite JSON functions</h2>
  <p dir="auto">The most powerful way to make use of this plugin is in conjunction with SQLite's <a href="https://www.sqlite.org/json1.html" rel="nofollow">JSON functions</a>. For example:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="select json_object(
      &quot;href&quot;, &quot;https://simonwillison.net/&quot;,
      &quot;label&quot;, &quot;Simon Willison&quot;
  );"><pre><code>select json_object(
      "href", "https://simonwillison.net/",
      "label", "Simon Willison"
  );
  </code></pre></div>
  <ul dir="auto">
  <li><a href="https://datasette-json-html.datasette.io/demo?sql=select+json_object%28%0D%0A++++%22href%22%2C+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++%22label%22%2C+%22Simon+Willison%22%0D%0A%29%3B" rel="nofollow">json_object() link demo</a></li>
  </ul>
  <p dir="auto">You can use these functions to construct JSON objects that work with the plugin from data in a table:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="select id, json_object(
      &quot;href&quot;, url, &quot;label&quot;, text
  ) from mytable;"><pre><code>select id, json_object(
      "href", url, "label", text
  ) from mytable;
  </code></pre></div>
  <ul dir="auto">
  <li><a href="https://datasette-json-html.datasette.io/demo?sql=select+json_object%28%22href%22%2C+url%2C+%22label%22%2C+package%2C+%22title%22%2C+package+%7C%7C+%22+%22+%7C%7C+url%29+as+package+from+packages" rel="nofollow">Demo that builds links against a table</a></li>
  </ul>
  <p dir="auto">The <code>json_group_array()</code> function is an aggregate function similar to <code>group_concat()</code> - it allows you to construct lists of JSON objects in conjunction with a <code>GROUP BY</code> clause.</p>
  <p dir="auto">This means you can use it to construct dynamic lists of links, for example:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="select
      substr(package, 0, 12) as prefix,
      json_group_array(
          json_object(
              &quot;href&quot;, url,
              &quot;label&quot;, package
          )
      ) as package_links
  from packages
  group by prefix"><pre><code>select
      substr(package, 0, 12) as prefix,
      json_group_array(
          json_object(
              "href", url,
              "label", package
          )
      ) as package_links
  from packages
  group by prefix
  </code></pre></div>
  <ul dir="auto">
  <li><a href="https://datasette-json-html.datasette.io/demo?sql=select%0D%0A++++substr%28package%2C+0%2C+12%29+as+prefix%2C%0D%0A++++json_group_array%28%0D%0A++++++++json_object%28%0D%0A++++++++++++%22href%22%2C+url%2C%0D%0A++++++++++++%22label%22%2C+package%0D%0A++++++++%29%0D%0A++++%29+as+package_links%0D%0Afrom+packages%0D%0Agroup+by+prefix" rel="nofollow">Demo of json_group_array()</a></li>
  </ul>
  <h2 dir="auto"><a id="user-content-the-urllib_quote_plus-sql-function" class="anchor" aria-hidden="true" href="#user-content-the-urllib_quote_plus-sql-function"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>The <code>urllib_quote_plus()</code> SQL function</h2>
  <p dir="auto">Since this plugin is designed to be used with SQL that constructs the underlying JSON structure, it is likely you will need to construct dynamic URLs from results returned by a SQL query.</p>
  <p dir="auto">This plugin registers a custom SQLite function called <code>urllib_quote_plus()</code> to help you do that. It lets you use Python's <a href="https://docs.python.org/3/library/urllib.parse.html#urllib.parse.quote_plus" rel="nofollow">urllib.parse.quote_plus() function</a> from within a SQL query.</p>
  <p dir="auto">Here's an example of how you might use it:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="select id, json_object(
      &quot;href&quot;,
      &quot;/mydatabase/other_table?_search=&quot; || urllib_quote_plus(text),
      &quot;label&quot;, text
  ) from mytable;"><pre><code>select id, json_object(
      "href",
      "/mydatabase/other_table?_search=" || urllib_quote_plus(text),
      "label", text
  ) from mytable;
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNDU0ODMwNzc=
name: datasette-render-images
full_name: simonw/datasette-render-images
html_url: https://github.com/simonw/datasette-render-images
description: Datasette plugin that renders binary blob images using data-uris
created_at: 2018-08-21T00:05:47Z
updated_at: 2022-08-11T16:06:11Z
pushed_at: 2022-08-11T16:06:08Z
homepage: https://datasette-render-images-demo.datasette.io/favicons/favicons
language: Python
topics: ["datasette", "datasette-io", "datasette-plugin", "plugin"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-render-images

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-images.svg)](https://pypi.org/project/datasette-render-images/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-render-images?include_prereleases&label=changelog)](https://github.com/simonw/datasette-render-images/releases)
  [![Tests](https://github.com/simonw/datasette-render-images/workflows/Test/badge.svg)](https://github.com/simonw/datasette-render-images/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-images/blob/main/LICENSE)

  A Datasette plugin that renders binary blob images with data-uris, using the [render_cell() plugin hook](https://docs.datasette.io/en/stable/plugins.html#render-cell-value-column-table-database-datasette).

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-render-images

  ## Usage

  If a database row contains binary image data (PNG, GIF or JPEG), this plugin will detect that it is an image (using the [imghdr module](https://docs.python.org/3/library/imghdr.html) and render that cell using an `<img src="data:image/png;base64,...">` element.

  Here's a [demo of the plugin in action](https://datasette-render-images-demo.datasette.io/favicons/favicons).

  ## Creating a compatible database table

  You can use the [sqlite-utils insert-files](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-data-from-files) command to insert image files into a database table:

      $ pip install sqlite-utils
      $ sqlite-utils insert-files gifs.db images *.gif

  See [Fun with binary data and SQLite](https://simonwillison.net/2020/Jul/30/fun-binary-data-and-sqlite/) for more on this tool.

  ## Configuration

  By default the plugin will only render images that are smaller than 100KB. You can adjust this limit using the `size_limit` plugin configuration option - for example, to increase the limit to 1MB (1000000 bytes) use the following in `metadata.json`:

  ```json
  {
      "plugins": {
          "datasette-render-images": {
              "size_limit": 1000000
          }
      }
  }
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-render-images" class="anchor" aria-hidden="true" href="#user-content-datasette-render-images"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-render-images</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-render-images/" rel="nofollow"><img src="https://camo.githubusercontent.com/0ff7b8f998f332ba04ece76be0bea5bd59032b36de01743aaee8735b18b89112/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d72656e6465722d696d616765732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-render-images.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-render-images/releases"><img src="https://camo.githubusercontent.com/6933dd025431c89dbe5604f03f999cb71c66c1a7eb492685a576581ec74871fd/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d72656e6465722d696d616765733f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-render-images?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-render-images/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-render-images/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-render-images/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">A Datasette plugin that renders binary blob images with data-uris, using the <a href="https://docs.datasette.io/en/stable/plugins.html#render-cell-value-column-table-database-datasette" rel="nofollow">render_cell() plugin hook</a>.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p dir="auto">Install this plugin in the same environment as Datasette.</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install datasette-render-images"><pre class="notranslate"><code>$ pip install datasette-render-images
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">If a database row contains binary image data (PNG, GIF or JPEG), this plugin will detect that it is an image (using the <a href="https://docs.python.org/3/library/imghdr.html" rel="nofollow">imghdr module</a> and render that cell using an <code>&lt;img src="data:image/png;base64,..."&gt;</code> element.</p>
  <p dir="auto">Here's a <a href="https://datasette-render-images-demo.datasette.io/favicons/favicons" rel="nofollow">demo of the plugin in action</a>.</p>
  <h2 dir="auto"><a id="user-content-creating-a-compatible-database-table" class="anchor" aria-hidden="true" href="#user-content-creating-a-compatible-database-table"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Creating a compatible database table</h2>
  <p dir="auto">You can use the <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-data-from-files" rel="nofollow">sqlite-utils insert-files</a> command to insert image files into a database table:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ pip install sqlite-utils
  $ sqlite-utils insert-files gifs.db images *.gif"><pre class="notranslate"><code>$ pip install sqlite-utils
  $ sqlite-utils insert-files gifs.db images *.gif
  </code></pre></div>
  <p dir="auto">See <a href="https://simonwillison.net/2020/Jul/30/fun-binary-data-and-sqlite/" rel="nofollow">Fun with binary data and SQLite</a> for more on this tool.</p>
  <h2 dir="auto"><a id="user-content-configuration" class="anchor" aria-hidden="true" href="#user-content-configuration"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Configuration</h2>
  <p dir="auto">By default the plugin will only render images that are smaller than 100KB. You can adjust this limit using the <code>size_limit</code> plugin configuration option - for example, to increase the limit to 1MB (1000000 bytes) use the following in <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-render-images&quot;: {
              &quot;size_limit&quot;: 1000000
          }
      }
  }"><pre>{
      <span class="pl-ent">"plugins"</span>: {
          <span class="pl-ent">"datasette-render-images"</span>: {
              <span class="pl-ent">"size_limit"</span>: <span class="pl-c1">1000000</span>
          }
      }
  }</pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNjM3OTA4MjI=
name: datasette-sqlite-fts4
full_name: simonw/datasette-sqlite-fts4
html_url: https://github.com/simonw/datasette-sqlite-fts4
description: Datasette plugin that adds custom SQL functions for working with SQLite FTS4
created_at: 2019-01-02T03:40:41Z
updated_at: 2022-07-31T16:33:25Z
pushed_at: 2022-07-31T14:46:26Z
homepage: https://datasette.io/plugins/datasette-sqlite-fts4
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "plugin"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-sqlite-fts4

  [![PyPI](https://img.shields.io/pypi/v/datasette-sqlite-fts4.svg)](https://pypi.org/project/datasette-sqlite-fts4/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-sqlite-fts4?include_prereleases&label=changelog)](https://github.com/simonw/datasette-sqlite-fts4/releases)
  [![Tests](https://github.com/simonw/datasette-sqlite-fts4/workflows/Test/badge.svg)](https://github.com/simonw/datasette-sqlite-fts4/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-sqlite-fts4/blob/main/LICENSE)

  Datasette plugin that exposes the custom SQL functions from [sqlite-fts4](https://github.com/simonw/sqlite-fts4).

  [Interactive demo](https://datasette-sqlite-fts4.datasette.io/24ways-fts4?sql=select%0D%0A++++json_object%28%0D%0A++++++++"label"%2C+articles.title%2C+"href"%2C+articles.url%0D%0A++++%29+as+article%2C%0D%0A++++articles.author%2C%0D%0A++++rank_score%28matchinfo%28articles_fts%2C+"pcx"%29%29+as+score%2C%0D%0A++++rank_bm25%28matchinfo%28articles_fts%2C+"pcnalx"%29%29+as+bm25%2C%0D%0A++++json_object%28%0D%0A++++++++"pre"%2C+annotate_matchinfo%28matchinfo%28articles_fts%2C+"pcxnalyb"%29%2C+"pcxnalyb"%29%0D%0A++++%29+as+annotated_matchinfo%2C%0D%0A++++matchinfo%28articles_fts%2C+"pcxnalyb"%29+as+matchinfo%2C%0D%0A++++decode_matchinfo%28matchinfo%28articles_fts%2C+"pcxnalyb"%29%29+as+decoded_matchinfo%0D%0Afrom%0D%0A++++articles_fts+join+articles+on+articles.rowid+%3D+articles_fts.rowid%0D%0Awhere%0D%0A++++articles_fts+match+%3Asearch%0D%0Aorder+by+bm25&search=jquery+maps). Read [Exploring search relevance algorithms with SQLite](https://simonwillison.net/2019/Jan/7/exploring-search-relevance-algorithms-sqlite/) for further details on this project.

  ## Installation

      pip install datasette-sqlite-fts4

  If you are deploying a database using `datasette publish` you can include this plugin using the `--install` option:

      datasette publish now mydb.db --install=datasette-sqlite-fts4

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-sqlite-fts4" class="anchor" aria-hidden="true" href="#user-content-datasette-sqlite-fts4"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-sqlite-fts4</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-sqlite-fts4/" rel="nofollow"><img src="https://camo.githubusercontent.com/97851cd6d6e2680d8da8db28d662851a36db50317ea8909c060aef1d1fcec110/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d73716c6974652d667473342e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-sqlite-fts4.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-sqlite-fts4/releases"><img src="https://camo.githubusercontent.com/e86e60059d267b02f60a9af932dbd99e1ec8d011e106ac2e8be952b364fcb7d1/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d73716c6974652d667473343f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-sqlite-fts4?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-sqlite-fts4/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-sqlite-fts4/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-sqlite-fts4/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Datasette plugin that exposes the custom SQL functions from <a href="https://github.com/simonw/sqlite-fts4">sqlite-fts4</a>.</p>
  <p dir="auto"><a href="https://datasette-sqlite-fts4.datasette.io/24ways-fts4?sql=select%0D%0A++++json_object%28%0D%0A++++++++%22label%22%2C+articles.title%2C+%22href%22%2C+articles.url%0D%0A++++%29+as+article%2C%0D%0A++++articles.author%2C%0D%0A++++rank_score%28matchinfo%28articles_fts%2C+%22pcx%22%29%29+as+score%2C%0D%0A++++rank_bm25%28matchinfo%28articles_fts%2C+%22pcnalx%22%29%29+as+bm25%2C%0D%0A++++json_object%28%0D%0A++++++++%22pre%22%2C+annotate_matchinfo%28matchinfo%28articles_fts%2C+%22pcxnalyb%22%29%2C+%22pcxnalyb%22%29%0D%0A++++%29+as+annotated_matchinfo%2C%0D%0A++++matchinfo%28articles_fts%2C+%22pcxnalyb%22%29+as+matchinfo%2C%0D%0A++++decode_matchinfo%28matchinfo%28articles_fts%2C+%22pcxnalyb%22%29%29+as+decoded_matchinfo%0D%0Afrom%0D%0A++++articles_fts+join+articles+on+articles.rowid+%3D+articles_fts.rowid%0D%0Awhere%0D%0A++++articles_fts+match+%3Asearch%0D%0Aorder+by+bm25&amp;search=jquery+maps" rel="nofollow">Interactive demo</a>. Read <a href="https://simonwillison.net/2019/Jan/7/exploring-search-relevance-algorithms-sqlite/" rel="nofollow">Exploring search relevance algorithms with SQLite</a> for further details on this project.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install datasette-sqlite-fts4"><pre class="notranslate"><code>pip install datasette-sqlite-fts4
  </code></pre></div>
  <p dir="auto">If you are deploying a database using <code>datasette publish</code> you can include this plugin using the <code>--install</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="datasette publish now mydb.db --install=datasette-sqlite-fts4"><pre class="notranslate"><code>datasette publish now mydb.db --install=datasette-sqlite-fts4
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNjYxNTkwNzI=
name: db-to-sqlite
full_name: simonw/db-to-sqlite
html_url: https://github.com/simonw/db-to-sqlite
description: CLI tool for exporting tables or queries from any SQL database to a SQLite file
created_at: 2019-01-17T04:16:48Z
updated_at: 2021-06-11T22:52:12Z
pushed_at: 2021-06-11T22:55:56Z
language: Python
license: apache-2.0
topics: ["sqlalchemy", "sqlite", "datasette", "datasette-io", "datasette-tool"]
default_branch: main
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # db-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/db-to-sqlite.svg)](https://pypi.python.org/pypi/db-to-sqlite)
  [![Changelog](https://img.shields.io/github/v/release/simonw/db-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/db-to-sqlite/releases)
  [![Tests](https://github.com/simonw/db-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/db-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/db-to-sqlite/blob/main/LICENSE)

  CLI tool for exporting tables or queries from any SQL database to a SQLite file.

  ## Installation

  Install from PyPI like so:

      pip install db-to-sqlite

  If you want to use it with MySQL, you can install the extra dependency like this:

      pip install 'db-to-sqlite[mysql]'

  Installing the `mysqlclient` library on OS X can be tricky - I've found [this recipe](https://gist.github.com/simonw/90ac0afd204cd0d6d9c3135c3888d116) to work (run that before installing `db-to-sqlite`).

  For PostgreSQL, use this:

      pip install 'db-to-sqlite[postgresql]'

  ## Usage

      Usage: db-to-sqlite [OPTIONS] CONNECTION PATH

        Load data from any database into SQLite.

        PATH is a path to the SQLite file to create, e.c. /tmp/my_database.db

        CONNECTION is a SQLAlchemy connection string, for example:

            postgresql://localhost/my_database
            postgresql://username:passwd@localhost/my_database

            mysql://root@localhost/my_database
            mysql://username:passwd@localhost/my_database

        More: https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls

      Options:
        --version                     Show the version and exit.
        --all                         Detect and copy all tables
        --table TEXT                  Specific tables to copy
        --skip TEXT                   When using --all skip these tables
        --redact TEXT...              (table, column) pairs to redact with ***
        --sql TEXT                    Optional SQL query to run
        --output TEXT                 Table in which to save --sql query results
        --pk TEXT                     Optional column to use as a primary key
        --index-fks / --no-index-fks  Should foreign keys have indexes? Default on
        -p, --progress                Show progress bar
        --postgres-schema TEXT        PostgreSQL schema to use
        --help                        Show this message and exit.

  For example, to save the content of the `blog_entry` table from a PostgreSQL database to a local file called `blog.db` you could do this:

      db-to-sqlite "postgresql://localhost/myblog" blog.db \
          --table=blog_entry

  You can specify `--table` more than once.

  You can also save the data from all of your tables, effectively creating a SQLite copy of your entire database. Any foreign key relationships will be detected and added to the SQLite database. For example:

      db-to-sqlite "postgresql://localhost/myblog" blog.db \
          --all

  When running `--all` you can specify tables to skip using `--skip`:

      db-to-sqlite "postgresql://localhost/myblog" blog.db \
          --all \
          --skip=django_migrations

  If you want to save the results of a custom SQL query, do this:

      db-to-sqlite "postgresql://localhost/myblog" output.db \
          --output=query_results \
          --sql="select id, title, created from blog_entry" \
          --pk=id

  The `--output` option specifies the table that should contain the results of the query.

  ## Using db-to-sqlite with PostgreSQL schemas

  If the tables you want to copy from your PostgreSQL database aren't in the default schema, you can specify an alternate one with the `--postgres-schema` option:

      db-to-sqlite "postgresql://localhost/myblog" blog.db \
          --all \
          --postgres-schema my_schema

  ## Using db-to-sqlite with Heroku Postgres

  If you run an application on [Heroku](https://www.heroku.com/) using their [Postgres database product](https://www.heroku.com/postgres), you can use the `heroku config` command to access a compatible connection string:

      $ heroku config --app myappname | grep HEROKU_POSTG
      HEROKU_POSTGRESQL_OLIVE_URL: postgres://username:password@ec2-xxx-xxx-xxx-x.compute-1.amazonaws.com:5432/dbname

  You can pass this to `db-to-sqlite` to create a local SQLite database with the data from your Heroku instance.

  You can even do this using a bash one-liner:

      $ db-to-sqlite $(heroku config --app myappname | grep HEROKU_POSTG | cut -d: -f 2-) \
          /tmp/heroku.db --all -p
      1/23: django_migrations
      ...
      17/23: blog_blogmark
      [####################################]  100%
      ...

  ## Related projects

  * [Datasette](https://github.com/simonw/datasette): A tool for exploring and publishing data. Works great with SQLite files generated using `db-to-sqlite`.
  * [sqlite-utils](https://github.com/simonw/sqlite-utils): Python CLI utility and library for manipulating SQLite databases.
  * [csvs-to-sqlite](https://github.com/simonw/csvs-to-sqlite): Convert CSV files into a SQLite database.

  ## Development

  To set up this tool locally, first checkout the code. Then create a new virtual environment:

      cd db-to-sqlite
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

  This will skip tests against MySQL or PostgreSQL if you do not have their additional dependencies installed.

  You can install those extra dependencies like so:

      pip install -e '.[test_mysql,test_postgresql]'

  You can alternative use `pip install psycopg2-binary` if you cannot install the `psycopg2` dependency used by the `test_postgresql` extra.

  See [Running a MySQL server using Homebrew](https://til.simonwillison.net/homebrew/mysql-homebrew) for tips on running the tests against MySQL on macOS, including how to install the `mysqlclient` dependency.

  The PostgreSQL and MySQL tests default to expecting to run against servers on localhost. You can use environment variables to point them at different test database servers:

  - `MYSQL_TEST_DB_CONNECTION` - defaults to `mysql://root@localhost/test_db_to_sqlite`
  - `POSTGRESQL_TEST_DB_CONNECTION` - defaults to `postgresql://localhost/test_db_to_sqlite`

  The database you indicate in the environment variable - `test_db_to_sqlite` by default - will be deleted and recreated on every test run.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-db-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-db-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>db-to-sqlite</h1>
  <p><a href="https://pypi.python.org/pypi/db-to-sqlite" rel="nofollow"><img src="https://camo.githubusercontent.com/b34869a6692a0e2ab6754463aad8578fe9f594788d99d4fd7ae2d815735d1660/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f64622d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/db-to-sqlite.svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/db-to-sqlite/releases"><img src="https://camo.githubusercontent.com/0f168b599361c3b2242a18a2b84f49d1c4e5520c7c425a89f43fd1b6f337f299/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f64622d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/db-to-sqlite?include_prereleases&amp;label=changelog" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/db-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/db-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/db-to-sqlite/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>CLI tool for exporting tables or queries from any SQL database to a SQLite file.</p>
  <h2><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p>Install from PyPI like so:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install db-to-sqlite
  "><pre><code>pip install db-to-sqlite
  </code></pre></div>
  <p>If you want to use it with MySQL, you can install the extra dependency like this:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install 'db-to-sqlite[mysql]'
  "><pre><code>pip install 'db-to-sqlite[mysql]'
  </code></pre></div>
  <p>Installing the <code>mysqlclient</code> library on OS X can be tricky - I've found <a href="https://gist.github.com/simonw/90ac0afd204cd0d6d9c3135c3888d116">this recipe</a> to work (run that before installing <code>db-to-sqlite</code>).</p>
  <p>For PostgreSQL, use this:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install 'db-to-sqlite[postgresql]'
  "><pre><code>pip install 'db-to-sqlite[postgresql]'
  </code></pre></div>
  <h2><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="Usage: db-to-sqlite [OPTIONS] CONNECTION PATH

    Load data from any database into SQLite.

    PATH is a path to the SQLite file to create, e.c. /tmp/my_database.db

    CONNECTION is a SQLAlchemy connection string, for example:

        postgresql://localhost/my_database
        postgresql://username:passwd@localhost/my_database

        mysql://root@localhost/my_database
        mysql://username:passwd@localhost/my_database

    More: https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls

  Options:
    --version                     Show the version and exit.
    --all                         Detect and copy all tables
    --table TEXT                  Specific tables to copy
    --skip TEXT                   When using --all skip these tables
    --redact TEXT...              (table, column) pairs to redact with ***
    --sql TEXT                    Optional SQL query to run
    --output TEXT                 Table in which to save --sql query results
    --pk TEXT                     Optional column to use as a primary key
    --index-fks / --no-index-fks  Should foreign keys have indexes? Default on
    -p, --progress                Show progress bar
    --postgres-schema TEXT        PostgreSQL schema to use
    --help                        Show this message and exit.
  "><pre><code>Usage: db-to-sqlite [OPTIONS] CONNECTION PATH

    Load data from any database into SQLite.

    PATH is a path to the SQLite file to create, e.c. /tmp/my_database.db

    CONNECTION is a SQLAlchemy connection string, for example:

        postgresql://localhost/my_database
        postgresql://username:passwd@localhost/my_database

        mysql://root@localhost/my_database
        mysql://username:passwd@localhost/my_database

    More: https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls

  Options:
    --version                     Show the version and exit.
    --all                         Detect and copy all tables
    --table TEXT                  Specific tables to copy
    --skip TEXT                   When using --all skip these tables
    --redact TEXT...              (table, column) pairs to redact with ***
    --sql TEXT                    Optional SQL query to run
    --output TEXT                 Table in which to save --sql query results
    --pk TEXT                     Optional column to use as a primary key
    --index-fks / --no-index-fks  Should foreign keys have indexes? Default on
    -p, --progress                Show progress bar
    --postgres-schema TEXT        PostgreSQL schema to use
    --help                        Show this message and exit.
  </code></pre></div>
  <p>For example, to save the content of the <code>blog_entry</code> table from a PostgreSQL database to a local file called <code>blog.db</code> you could do this:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="db-to-sqlite &quot;postgresql://localhost/myblog&quot; blog.db \
      --table=blog_entry
  "><pre><code>db-to-sqlite "postgresql://localhost/myblog" blog.db \
      --table=blog_entry
  </code></pre></div>
  <p>You can specify <code>--table</code> more than once.</p>
  <p>You can also save the data from all of your tables, effectively creating a SQLite copy of your entire database. Any foreign key relationships will be detected and added to the SQLite database. For example:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="db-to-sqlite &quot;postgresql://localhost/myblog&quot; blog.db \
      --all
  "><pre><code>db-to-sqlite "postgresql://localhost/myblog" blog.db \
      --all
  </code></pre></div>
  <p>When running <code>--all</code> you can specify tables to skip using <code>--skip</code>:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="db-to-sqlite &quot;postgresql://localhost/myblog&quot; blog.db \
      --all \
      --skip=django_migrations
  "><pre><code>db-to-sqlite "postgresql://localhost/myblog" blog.db \
      --all \
      --skip=django_migrations
  </code></pre></div>
  <p>If you want to save the results of a custom SQL query, do this:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="db-to-sqlite &quot;postgresql://localhost/myblog&quot; output.db \
      --output=query_results \
      --sql=&quot;select id, title, created from blog_entry&quot; \
      --pk=id
  "><pre><code>db-to-sqlite "postgresql://localhost/myblog" output.db \
      --output=query_results \
      --sql="select id, title, created from blog_entry" \
      --pk=id
  </code></pre></div>
  <p>The <code>--output</code> option specifies the table that should contain the results of the query.</p>
  <h2><a id="user-content-using-db-to-sqlite-with-postgresql-schemas" class="anchor" aria-hidden="true" href="#user-content-using-db-to-sqlite-with-postgresql-schemas"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Using db-to-sqlite with PostgreSQL schemas</h2>
  <p>If the tables you want to copy from your PostgreSQL database aren't in the default schema, you can specify an alternate one with the <code>--postgres-schema</code> option:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="db-to-sqlite &quot;postgresql://localhost/myblog&quot; blog.db \
      --all \
      --postgres-schema my_schema
  "><pre><code>db-to-sqlite "postgresql://localhost/myblog" blog.db \
      --all \
      --postgres-schema my_schema
  </code></pre></div>
  <h2><a id="user-content-using-db-to-sqlite-with-heroku-postgres" class="anchor" aria-hidden="true" href="#user-content-using-db-to-sqlite-with-heroku-postgres"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Using db-to-sqlite with Heroku Postgres</h2>
  <p>If you run an application on <a href="https://www.heroku.com/" rel="nofollow">Heroku</a> using their <a href="https://www.heroku.com/postgres" rel="nofollow">Postgres database product</a>, you can use the <code>heroku config</code> command to access a compatible connection string:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ heroku config --app myappname | grep HEROKU_POSTG
  HEROKU_POSTGRESQL_OLIVE_URL: postgres://username:password@ec2-xxx-xxx-xxx-x.compute-1.amazonaws.com:5432/dbname
  "><pre><code>$ heroku config --app myappname | grep HEROKU_POSTG
  HEROKU_POSTGRESQL_OLIVE_URL: postgres://username:password@ec2-xxx-xxx-xxx-x.compute-1.amazonaws.com:5432/dbname
  </code></pre></div>
  <p>You can pass this to <code>db-to-sqlite</code> to create a local SQLite database with the data from your Heroku instance.</p>
  <p>You can even do this using a bash one-liner:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ db-to-sqlite $(heroku config --app myappname | grep HEROKU_POSTG | cut -d: -f 2-) \
      /tmp/heroku.db --all -p
  1/23: django_migrations
  ...
  17/23: blog_blogmark
  [####################################]  100%
  ...
  "><pre><code>$ db-to-sqlite $(heroku config --app myappname | grep HEROKU_POSTG | cut -d: -f 2-) \
      /tmp/heroku.db --all -p
  1/23: django_migrations
  ...
  17/23: blog_blogmark
  [####################################]  100%
  ...
  </code></pre></div>
  <h2><a id="user-content-related-projects" class="anchor" aria-hidden="true" href="#user-content-related-projects"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Related projects</h2>
  <ul>
  <li><a href="https://github.com/simonw/datasette">Datasette</a>: A tool for exploring and publishing data. Works great with SQLite files generated using <code>db-to-sqlite</code>.</li>
  <li><a href="https://github.com/simonw/sqlite-utils">sqlite-utils</a>: Python CLI utility and library for manipulating SQLite databases.</li>
  <li><a href="https://github.com/simonw/csvs-to-sqlite">csvs-to-sqlite</a>: Convert CSV files into a SQLite database.</li>
  </ul>
  <h2><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p>To set up this tool locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="cd db-to-sqlite
  python3 -mvenv venv
  source venv/bin/activate
  "><pre><code>cd db-to-sqlite
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p>Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pipenv shell
  "><pre><code>pipenv shell
  </code></pre></div>
  <p>Now install the dependencies and test dependencies:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install -e '.[test]'
  "><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p>To run the tests:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pytest
  "><pre><code>pytest
  </code></pre></div>
  <p>This will skip tests against MySQL or PostgreSQL if you do not have their additional dependencies installed.</p>
  <p>You can install those extra dependencies like so:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install -e '.[test_mysql,test_postgresql]'
  "><pre><code>pip install -e '.[test_mysql,test_postgresql]'
  </code></pre></div>
  <p>You can alternative use <code>pip install psycopg2-binary</code> if you cannot install the <code>psycopg2</code> dependency used by the <code>test_postgresql</code> extra.</p>
  <p>See <a href="https://til.simonwillison.net/homebrew/mysql-homebrew" rel="nofollow">Running a MySQL server using Homebrew</a> for tips on running the tests against MySQL on macOS, including how to install the <code>mysqlclient</code> dependency.</p>
  <p>The PostgreSQL and MySQL tests default to expecting to run against servers on localhost. You can use environment variables to point them at different test database servers:</p>
  <ul>
  <li><code>MYSQL_TEST_DB_CONNECTION</code> - defaults to <code>mysql://root@localhost/test_db_to_sqlite</code></li>
  <li><code>POSTGRESQL_TEST_DB_CONNECTION</code> - defaults to <code>postgresql://localhost/test_db_to_sqlite</code></li>
  </ul>
  <p>The database you indicate in the environment variable - <code>test_db_to_sqlite</code> by default - will be deleted and recreated on every test run.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNjc3MzAwNzE=
name: datasette-pretty-json
full_name: simonw/datasette-pretty-json
html_url: https://github.com/simonw/datasette-pretty-json
description: Datasette plugin that pretty-prints any column values that are valid JSON objects or arrays
created_at: 2019-01-26T19:30:43Z
updated_at: 2022-09-24T06:13:11Z
pushed_at: 2022-09-28T21:06:31Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-io", "datasette-plugin", "json"]
default_branch: master
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # datasette-pretty-json

  [![PyPI](https://img.shields.io/pypi/v/datasette-pretty-json.svg)](https://pypi.org/project/datasette-pretty-json/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-pretty-json?include_prereleases&label=changelog)](https://github.com/simonw/datasette-pretty-json/releases)
  [![Tests](https://github.com/simonw/datasette-pretty-json/workflows/Test/badge.svg)](https://github.com/simonw/datasette-pretty-json/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-pretty-json/blob/main/LICENSE)


  [Datasette](https://github.com/simonw/datasette) plugin that pretty-prints any column values that are valid JSON objects or arrays.

  You may also be interested in [datasette-json-html](https://github.com/simonw/datasette-json-html).

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-datasette-pretty-json" class="anchor" aria-hidden="true" href="#user-content-datasette-pretty-json"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-pretty-json</h1>
  <p dir="auto"><a href="https://pypi.org/project/datasette-pretty-json/" rel="nofollow"><img src="https://camo.githubusercontent.com/3f7d7f3de1cb9fbdf6ac58e04cea35cc1daee4d53bcbcdf5060a0d311db996dc/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d7072657474792d6a736f6e2e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-pretty-json.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-pretty-json/releases"><img src="https://camo.githubusercontent.com/4fbe6b0ade0a8b75e9200545ca5459c3ba7d546a32ac2f1519417e8c2314d20b/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d7072657474792d6a736f6e3f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-pretty-json?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-pretty-json/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-pretty-json/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/datasette-pretty-json/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto"><a href="https://github.com/simonw/datasette">Datasette</a> plugin that pretty-prints any column values that are valid JSON objects or arrays.</p>
  <p dir="auto">You may also be interested in <a href="https://github.com/simonw/datasette-json-html">datasette-json-html</a>.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNjc3NTk4NDY=
name: markdown-to-sqlite
full_name: simonw/markdown-to-sqlite
html_url: https://github.com/simonw/markdown-to-sqlite
description: CLI tool for loading markdown files into a SQLite database
created_at: 2019-01-27T02:04:54Z
updated_at: 2022-05-13T18:09:26Z
pushed_at: 2022-05-13T18:09:22Z
language: Python
license: apache-2.0
topics: ["datasette-io", "datasette-tool", "markdown", "sqlite", "yaml"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # markdown-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/markdown-to-sqlite.svg)](https://pypi.python.org/pypi/markdown-to-sqlite)
  [![Changelog](https://img.shields.io/github/v/release/simonw/markdown-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/markdown-to-sqlite/releases)
  [![Tests](https://github.com/simonw/markdown-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/markdown-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/markdown-to-sqlite/blob/main/LICENSE)

  CLI tool for loading markdown files into a SQLite database.

  YAML embedded in the markdown files will be used to populate additional columns.

      Usage: markdown-to-sqlite [OPTIONS] DBNAME TABLE PATHS...

  For example:

      $ markdown-to-sqlite docs.db documents file1.md file2.md

  ## Breaking change

  Prior to version 1.0 this argument order was different - markdown files were listed before the database and table.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-markdown-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-markdown-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>markdown-to-sqlite</h1>
  <p dir="auto"><a href="https://pypi.python.org/pypi/markdown-to-sqlite" rel="nofollow"><img src="https://camo.githubusercontent.com/e5aa6980c413976937ea698f111c31b580e6186e9f018673df37b3e9fe1047fb/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6d61726b646f776e2d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/markdown-to-sqlite.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/markdown-to-sqlite/releases"><img src="https://camo.githubusercontent.com/29f48f8deb508081ca40341be55a1cdbe25dcd7a637e5277a24b82f167643ab6/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6d61726b646f776e2d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/markdown-to-sqlite?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/markdown-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/markdown-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/markdown-to-sqlite/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">CLI tool for loading markdown files into a SQLite database.</p>
  <p dir="auto">YAML embedded in the markdown files will be used to populate additional columns.</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: markdown-to-sqlite [OPTIONS] DBNAME TABLE PATHS..."><pre class="notranslate"><code class="notranslate">Usage: markdown-to-sqlite [OPTIONS] DBNAME TABLE PATHS...
  </code></pre></div>
  <p dir="auto">For example:</p>
  <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="$ markdown-to-sqlite docs.db documents file1.md file2.md"><pre class="notranslate"><code class="notranslate">$ markdown-to-sqlite docs.db documents file1.md file2.md
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-breaking-change" class="anchor" aria-hidden="true" href="#user-content-breaking-change"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Breaking change</h2>
  <p dir="auto">Prior to version 1.0 this argument order was different - markdown files were listed before the database and table.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNjg0NzQ5NzA=
name: dbf-to-sqlite
full_name: simonw/dbf-to-sqlite
html_url: https://github.com/simonw/dbf-to-sqlite
description: CLI tool for converting DBF files (dBase, FoxPro etc) to SQLite
created_at: 2019-01-31T06:30:46Z
updated_at: 2021-03-23T01:29:41Z
pushed_at: 2020-02-16T00:41:20Z
language: Python
license: apache-2.0
topics: ["sqlite", "foxpro", "dbf", "dbase", "datasette-io", "datasette-tool"]
default_branch: master
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # dbf-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/dbf-to-sqlite.svg)](https://pypi.python.org/pypi/dbf-to-sqlite)
  [![Travis CI](https://travis-ci.com/simonw/dbf-to-sqlite.svg?branch=master)](https://travis-ci.com/simonw/dbf-to-sqlite)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/dbf-to-sqlite/blob/master/LICENSE)

  CLI tool for converting DBF files (dBase, FoxPro etc) to SQLite.

  ## Installation

      pip install dbf-to-sqlite

  ## Usage

      $ dbf-to-sqlite --help
      Usage: dbf-to-sqlite [OPTIONS] DBF_PATHS... SQLITE_DB

        Convert DBF files (dBase, FoxPro etc) to SQLite

        https://github.com/simonw/dbf-to-sqlite

      Options:
        --version      Show the version and exit.
        --table TEXT   Table name to use (only valid for single files)
        -v, --verbose  Show what's going on
        --help         Show this message and exit.

  Example usage:

      $ dbf-to-sqlite *.DBF database.db

  This will create a new SQLite database called `database.db` containing one table for each of the `DBF` files in the current directory.

  Looking for DBF files to try this out on? Try downloading the [Himalayan Database](http://himalayandatabase.com/) of all expeditions that have climbed in the Nepal Himalaya.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-dbf-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-dbf-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>dbf-to-sqlite</h1>
  <p><a href="https://pypi.python.org/pypi/dbf-to-sqlite" rel="nofollow"><img src="https://camo.githubusercontent.com/59d39789edaeb918bb1febd34597470b4d6cab449725f3db3c0bb59d3c02551a/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6462662d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/dbf-to-sqlite.svg" style="max-width:100%;"></a>
  <a href="https://travis-ci.com/simonw/dbf-to-sqlite" rel="nofollow"><img src="https://camo.githubusercontent.com/d7e086b366bfa3de03a5c75c2112c011ac4eaf71863fed90253e3b5c28e13f89/68747470733a2f2f7472617669732d63692e636f6d2f73696d6f6e772f6462662d746f2d73716c6974652e7376673f6272616e63683d6d6173746572" alt="Travis CI" data-canonical-src="https://travis-ci.com/simonw/dbf-to-sqlite.svg?branch=master" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/dbf-to-sqlite/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>CLI tool for converting DBF files (dBase, FoxPro etc) to SQLite.</p>
  <h2><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install dbf-to-sqlite
  "><pre><code>pip install dbf-to-sqlite
  </code></pre></div>
  <h2><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ dbf-to-sqlite --help
  Usage: dbf-to-sqlite [OPTIONS] DBF_PATHS... SQLITE_DB

    Convert DBF files (dBase, FoxPro etc) to SQLite

    https://github.com/simonw/dbf-to-sqlite

  Options:
    --version      Show the version and exit.
    --table TEXT   Table name to use (only valid for single files)
    -v, --verbose  Show what's going on
    --help         Show this message and exit.
  "><pre><code>$ dbf-to-sqlite --help
  Usage: dbf-to-sqlite [OPTIONS] DBF_PATHS... SQLITE_DB

    Convert DBF files (dBase, FoxPro etc) to SQLite

    https://github.com/simonw/dbf-to-sqlite

  Options:
    --version      Show the version and exit.
    --table TEXT   Table name to use (only valid for single files)
    -v, --verbose  Show what's going on
    --help         Show this message and exit.
  </code></pre></div>
  <p>Example usage:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ dbf-to-sqlite *.DBF database.db
  "><pre><code>$ dbf-to-sqlite *.DBF database.db
  </code></pre></div>
  <p>This will create a new SQLite database called <code>database.db</code> containing one table for each of the <code>DBF</code> files in the current directory.</p>
  <p>Looking for DBF files to try this out on? Try downloading the <a href="http://himalayandatabase.com/" rel="nofollow">Himalayan Database</a> of all expeditions that have climbed in the Nepal Himalaya.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNzQ3MTUxNTM=
name: datasette-jellyfish
full_name: simonw/datasette-jellyfish
html_url: https://github.com/simonw/datasette-jellyfish
description: Datasette plugin adding SQL functions for fuzzy text matching powered by Jellyfish
created_at: 2019-03-09T16:02:01Z
updated_at: 2021-02-06T02:33:49Z
pushed_at: 2021-02-06T02:34:18Z
homepage: https://datasette.io/plugins/datasette-jellyfish
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin", "datasette-io"]
default_branch: main
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # datasette-jellyfish

  [![PyPI](https://img.shields.io/pypi/v/datasette-jellyfish.svg)](https://pypi.org/project/datasette-jellyfish/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-jellyfish?include_prereleases&label=changelog)](https://github.com/simonw/datasette-jellyfish/releases)
  [![Tests](https://github.com/simonw/datasette-jellyfish/workflows/Test/badge.svg)](https://github.com/simonw/datasette-jellyfish/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-jellyfish/blob/main/LICENSE)

  Datasette plugin that adds custom SQL functions for fuzzy string matching, built on top of the [Jellyfish](https://github.com/jamesturk/jellyfish) Python library by James Turk and Michael Stephens.

  Interactive demos:

  * [soundex, metaphone, nysiis, match_rating_codex comparison](https://latest-with-plugins.datasette.io/fixtures?sql=SELECT%0D%0A++++soundex%28%3As%29%2C+%0D%0A++++metaphone%28%3As%29%2C+%0D%0A++++nysiis%28%3As%29%2C+%0D%0A++++match_rating_codex%28%3As%29&s=demo).
  * [distance functions comparison](https://latest-with-plugins.datasette.io/fixtures?sql=SELECT%0D%0A++++levenshtein_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++damerau_levenshtein_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++hamming_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++jaro_similarity%28%3As1%2C+%3As2%29%2C%0D%0A++++jaro_winkler_similarity%28%3As1%2C+%3As2%29%2C%0D%0A++++match_rating_comparison%28%3As1%2C+%3As2%29%3B&s1=barrack+obama&s2=barrack+h+obama)

  Examples:

      SELECT soundex("hello");
          -- Outputs H400
      SELECT metaphone("hello");
          -- Outputs HL
      SELECT nysiis("hello");
          -- Outputs HAL
      SELECT match_rating_codex("hello");
          -- Outputs HLL
      SELECT porter_stem("running");
          -- Outputs run
      SELECT levenshtein_distance("hello", "hello world");
          -- Outputs 6
      SELECT damerau_levenshtein_distance("hello", "hello world");
          -- Outputs 6
      SELECT hamming_distance("hello", "hello world");
          -- Outputs 6
      SELECT jaro_similarity("hello", "hello world");
          -- Outputs 0.8181818181818182
      SELECT jaro_winkler_similarity("hello", "hello world");
          -- Outputs 0.890909090909091
      SELECT match_rating_comparison("hello", "helloo");
          -- Outputs 1

  See [the Jellyfish documentation](https://jellyfish.readthedocs.io/en/latest/) for an explanation of each of these functions.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-jellyfish" class="anchor" aria-hidden="true" href="#user-content-datasette-jellyfish"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-jellyfish</h1>
  <p><a href="https://pypi.org/project/datasette-jellyfish/" rel="nofollow"><img src="https://camo.githubusercontent.com/e05229cfd49e2fc6fbff79424545caee53b94da156d85d3f3bdb39ee9e32aef2/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6a656c6c79666973682e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-jellyfish.svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-jellyfish/releases"><img src="https://camo.githubusercontent.com/13a248cf10fa13fb53cb22841b0065f965423a4e0f5c17d19673603b2baca376/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d6a656c6c79666973683f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-jellyfish?include_prereleases&amp;label=changelog" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-jellyfish/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-jellyfish/workflows/Test/badge.svg" alt="Tests" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-jellyfish/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Datasette plugin that adds custom SQL functions for fuzzy string matching, built on top of the <a href="https://github.com/jamesturk/jellyfish">Jellyfish</a> Python library by James Turk and Michael Stephens.</p>
  <p>Interactive demos:</p>
  <ul>
  <li><a href="https://latest-with-plugins.datasette.io/fixtures?sql=SELECT%0D%0A++++soundex%28%3As%29%2C+%0D%0A++++metaphone%28%3As%29%2C+%0D%0A++++nysiis%28%3As%29%2C+%0D%0A++++match_rating_codex%28%3As%29&amp;s=demo" rel="nofollow">soundex, metaphone, nysiis, match_rating_codex comparison</a>.</li>
  <li><a href="https://latest-with-plugins.datasette.io/fixtures?sql=SELECT%0D%0A++++levenshtein_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++damerau_levenshtein_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++hamming_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++jaro_similarity%28%3As1%2C+%3As2%29%2C%0D%0A++++jaro_winkler_similarity%28%3As1%2C+%3As2%29%2C%0D%0A++++match_rating_comparison%28%3As1%2C+%3As2%29%3B&amp;s1=barrack+obama&amp;s2=barrack+h+obama" rel="nofollow">distance functions comparison</a></li>
  </ul>
  <p>Examples:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="SELECT soundex(&quot;hello&quot;);
      -- Outputs H400
  SELECT metaphone(&quot;hello&quot;);
      -- Outputs HL
  SELECT nysiis(&quot;hello&quot;);
      -- Outputs HAL
  SELECT match_rating_codex(&quot;hello&quot;);
      -- Outputs HLL
  SELECT porter_stem(&quot;running&quot;);
      -- Outputs run
  SELECT levenshtein_distance(&quot;hello&quot;, &quot;hello world&quot;);
      -- Outputs 6
  SELECT damerau_levenshtein_distance(&quot;hello&quot;, &quot;hello world&quot;);
      -- Outputs 6
  SELECT hamming_distance(&quot;hello&quot;, &quot;hello world&quot;);
      -- Outputs 6
  SELECT jaro_similarity(&quot;hello&quot;, &quot;hello world&quot;);
      -- Outputs 0.8181818181818182
  SELECT jaro_winkler_similarity(&quot;hello&quot;, &quot;hello world&quot;);
      -- Outputs 0.890909090909091
  SELECT match_rating_comparison(&quot;hello&quot;, &quot;helloo&quot;);
      -- Outputs 1
  "><pre><code>SELECT soundex("hello");
      -- Outputs H400
  SELECT metaphone("hello");
      -- Outputs HL
  SELECT nysiis("hello");
      -- Outputs HAL
  SELECT match_rating_codex("hello");
      -- Outputs HLL
  SELECT porter_stem("running");
      -- Outputs run
  SELECT levenshtein_distance("hello", "hello world");
      -- Outputs 6
  SELECT damerau_levenshtein_distance("hello", "hello world");
      -- Outputs 6
  SELECT hamming_distance("hello", "hello world");
      -- Outputs 6
  SELECT jaro_similarity("hello", "hello world");
      -- Outputs 0.8181818181818182
  SELECT jaro_winkler_similarity("hello", "hello world");
      -- Outputs 0.890909090909091
  SELECT match_rating_comparison("hello", "helloo");
      -- Outputs 1
  </code></pre></div>
  <p>See <a href="https://jellyfish.readthedocs.io/en/latest/" rel="nofollow">the Jellyfish documentation</a> for an explanation of each of these functions.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNzUzMjE0OTc=
name: csv-diff
full_name: simonw/csv-diff
html_url: https://github.com/simonw/csv-diff
description: Python CLI tool and library for diffing CSV and JSON files
created_at: 2019-03-13T01:11:26Z
updated_at: 2022-07-29T20:01:02Z
pushed_at: 2022-07-29T20:00:59Z
language: Python
license: apache-2.0
topics: ["click", "csv", "datasette-io", "datasette-tool", "diff", "git-scraping"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # csv-diff

  [![PyPI](https://img.shields.io/pypi/v/csv-diff.svg)](https://pypi.org/project/csv-diff/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/csv-diff?include_prereleases&label=changelog)](https://github.com/simonw/csv-diff/releases)
  [![Tests](https://github.com/simonw/csv-diff/workflows/Test/badge.svg)](https://github.com/simonw/csv-diff/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/csv-diff/blob/main/LICENSE)

  Tool for viewing the difference between two CSV, TSV or JSON files. See [Generating a commit log for San Francisco’s official list of trees](https://simonwillison.net/2019/Mar/13/tree-history/) (and the [sf-tree-history repo commit log](https://github.com/simonw/sf-tree-history/commits)) for background information on this project.

  ## Installation

      pip install csv-diff

  ## Usage

  Consider two CSV files:

  `one.csv`

      id,name,age
      1,Cleo,4
      2,Pancakes,2

  `two.csv`

      id,name,age
      1,Cleo,5
      3,Bailey,1

  `csv-diff` can show a human-readable summary of differences between the files:

      $ csv-diff one.csv two.csv --key=id
      1 row changed, 1 row added, 1 row removed

      1 row changed

        Row 1
          age: "4" => "5"

      1 row added

        id: 3
        name: Bailey
        age: 1

      1 row removed

        id: 2
        name: Pancakes
        age: 2

  The `--key=id` option means that the `id` column should be treated as the unique key, to identify which records have changed.

  The tool will automatically detect if your files are comma- or tab-separated. You can over-ride this automatic detection and force the tool to use a specific format using `--format=tsv` or `--format=csv`.

  You can also feed it JSON files, provided they are a JSON array of objects where each object has the same keys. Use `--format=json` if your input files are JSON.

  Use `--show-unchanged` to include full details of the unchanged values for rows with at least one change in the diff output:

      % csv-diff one.csv two.csv --key=id --show-unchanged
      1 row changed

        id: 1
          age: "4" => "5"

          Unchanged:
            name: "Cleo"

  You can use the `--json` option to get a machine-readable difference:

      $ csv-diff one.csv two.csv --key=id --json
      {
          "added": [
              {
                  "id": "3",
                  "name": "Bailey",
                  "age": "1"
              }
          ],
          "removed": [
              {
                  "id": "2",
                  "name": "Pancakes",
                  "age": "2"
              }
          ],
          "changed": [
              {
                  "key": "1",
                  "changes": {
                      "age": [
                          "4",
                          "5"
                      ]
                  }
              }
          ],
          "columns_added": [],
          "columns_removed": []
      }

  ## As a Python library

  You can also import the Python library into your own code like so:

      from csv_diff import load_csv, compare
      diff = compare(
          load_csv(open("one.csv"), key="id"),
          load_csv(open("two.csv"), key="id")
      )

  `diff` will now contain the same data structure as the output in the `--json` example above.

  If the columns in the CSV have changed, those added or removed columns will be ignored when calculating changes made to specific rows.

  ## As a Docker container

  ### Build the image

      $ docker build -t csvdiff .

  ### Run the container

      $ docker run --rm -v $(pwd):/files csvdiff

  Suppose current directory contains two csv files : one.csv two.csv

      $ docker run --rm -v $(pwd):/files csvdiff one.csv two.csv
      
  ## Alternatives

  - [csvdiff](https://github.com/aswinkarthik/csvdiff) is a "fast diff tool for comparing CSV files" - you may get better results from this than from `csv-diff` against larger files.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-csv-diff" class="anchor" aria-hidden="true" href="#user-content-csv-diff"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>csv-diff</h1>
  <p dir="auto"><a href="https://pypi.org/project/csv-diff/" rel="nofollow"><img src="https://camo.githubusercontent.com/75784b8c5ee65df6e894c25c0efe54d360e3b6a33714da9aa0c6bb86ede1f153/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6373762d646966662e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/csv-diff.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/csv-diff/releases"><img src="https://camo.githubusercontent.com/a8ece0f4436cb61b1524e3722c02363faf7aa45fe7e62f6634604f2c30421517/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6373762d646966663f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/csv-diff?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/csv-diff/actions?query=workflow%3ATest"><img src="https://github.com/simonw/csv-diff/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/csv-diff/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Tool for viewing the difference between two CSV, TSV or JSON files. See <a href="https://simonwillison.net/2019/Mar/13/tree-history/" rel="nofollow">Generating a commit log for San Francisco’s official list of trees</a> (and the <a href="https://github.com/simonw/sf-tree-history/commits">sf-tree-history repo commit log</a>) for background information on this project.</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install csv-diff"><pre class="notranslate"><code>pip install csv-diff
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p dir="auto">Consider two CSV files:</p>
  <p dir="auto"><code>one.csv</code></p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="id,name,age
  1,Cleo,4
  2,Pancakes,2"><pre class="notranslate"><code>id,name,age
  1,Cleo,4
  2,Pancakes,2
  </code></pre></div>
  <p dir="auto"><code>two.csv</code></p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="id,name,age
  1,Cleo,5
  3,Bailey,1"><pre class="notranslate"><code>id,name,age
  1,Cleo,5
  3,Bailey,1
  </code></pre></div>
  <p dir="auto"><code>csv-diff</code> can show a human-readable summary of differences between the files:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ csv-diff one.csv two.csv --key=id
  1 row changed, 1 row added, 1 row removed

  1 row changed

    Row 1
      age: &quot;4&quot; =&gt; &quot;5&quot;

  1 row added

    id: 3
    name: Bailey
    age: 1

  1 row removed

    id: 2
    name: Pancakes
    age: 2"><pre class="notranslate"><code>$ csv-diff one.csv two.csv --key=id
  1 row changed, 1 row added, 1 row removed

  1 row changed

    Row 1
      age: "4" =&gt; "5"

  1 row added

    id: 3
    name: Bailey
    age: 1

  1 row removed

    id: 2
    name: Pancakes
    age: 2
  </code></pre></div>
  <p dir="auto">The <code>--key=id</code> option means that the <code>id</code> column should be treated as the unique key, to identify which records have changed.</p>
  <p dir="auto">The tool will automatically detect if your files are comma- or tab-separated. You can over-ride this automatic detection and force the tool to use a specific format using <code>--format=tsv</code> or <code>--format=csv</code>.</p>
  <p dir="auto">You can also feed it JSON files, provided they are a JSON array of objects where each object has the same keys. Use <code>--format=json</code> if your input files are JSON.</p>
  <p dir="auto">Use <code>--show-unchanged</code> to include full details of the unchanged values for rows with at least one change in the diff output:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="% csv-diff one.csv two.csv --key=id --show-unchanged
  1 row changed

    id: 1
      age: &quot;4&quot; =&gt; &quot;5&quot;

      Unchanged:
        name: &quot;Cleo&quot;"><pre class="notranslate"><code>% csv-diff one.csv two.csv --key=id --show-unchanged
  1 row changed

    id: 1
      age: "4" =&gt; "5"

      Unchanged:
        name: "Cleo"
  </code></pre></div>
  <p dir="auto">You can use the <code>--json</code> option to get a machine-readable difference:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ csv-diff one.csv two.csv --key=id --json
  {
      &quot;added&quot;: [
          {
              &quot;id&quot;: &quot;3&quot;,
              &quot;name&quot;: &quot;Bailey&quot;,
              &quot;age&quot;: &quot;1&quot;
          }
      ],
      &quot;removed&quot;: [
          {
              &quot;id&quot;: &quot;2&quot;,
              &quot;name&quot;: &quot;Pancakes&quot;,
              &quot;age&quot;: &quot;2&quot;
          }
      ],
      &quot;changed&quot;: [
          {
              &quot;key&quot;: &quot;1&quot;,
              &quot;changes&quot;: {
                  &quot;age&quot;: [
                      &quot;4&quot;,
                      &quot;5&quot;
                  ]
              }
          }
      ],
      &quot;columns_added&quot;: [],
      &quot;columns_removed&quot;: []
  }"><pre class="notranslate"><code>$ csv-diff one.csv two.csv --key=id --json
  {
      "added": [
          {
              "id": "3",
              "name": "Bailey",
              "age": "1"
          }
      ],
      "removed": [
          {
              "id": "2",
              "name": "Pancakes",
              "age": "2"
          }
      ],
      "changed": [
          {
              "key": "1",
              "changes": {
                  "age": [
                      "4",
                      "5"
                  ]
              }
          }
      ],
      "columns_added": [],
      "columns_removed": []
  }
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-as-a-python-library" class="anchor" aria-hidden="true" href="#user-content-as-a-python-library"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>As a Python library</h2>
  <p dir="auto">You can also import the Python library into your own code like so:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="from csv_diff import load_csv, compare
  diff = compare(
      load_csv(open(&quot;one.csv&quot;), key=&quot;id&quot;),
      load_csv(open(&quot;two.csv&quot;), key=&quot;id&quot;)
  )"><pre class="notranslate"><code>from csv_diff import load_csv, compare
  diff = compare(
      load_csv(open("one.csv"), key="id"),
      load_csv(open("two.csv"), key="id")
  )
  </code></pre></div>
  <p dir="auto"><code>diff</code> will now contain the same data structure as the output in the <code>--json</code> example above.</p>
  <p dir="auto">If the columns in the CSV have changed, those added or removed columns will be ignored when calculating changes made to specific rows.</p>
  <h2 dir="auto"><a id="user-content-as-a-docker-container" class="anchor" aria-hidden="true" href="#user-content-as-a-docker-container"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>As a Docker container</h2>
  <h3 dir="auto"><a id="user-content-build-the-image" class="anchor" aria-hidden="true" href="#user-content-build-the-image"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Build the image</h3>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ docker build -t csvdiff ."><pre class="notranslate"><code>$ docker build -t csvdiff .
  </code></pre></div>
  <h3 dir="auto"><a id="user-content-run-the-container" class="anchor" aria-hidden="true" href="#user-content-run-the-container"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Run the container</h3>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ docker run --rm -v $(pwd):/files csvdiff"><pre class="notranslate"><code>$ docker run --rm -v $(pwd):/files csvdiff
  </code></pre></div>
  <p dir="auto">Suppose current directory contains two csv files : one.csv two.csv</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ docker run --rm -v $(pwd):/files csvdiff one.csv two.csv"><pre class="notranslate"><code>$ docker run --rm -v $(pwd):/files csvdiff one.csv two.csv
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-alternatives" class="anchor" aria-hidden="true" href="#user-content-alternatives"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Alternatives</h2>
  <ul dir="auto">
  <li><a href="https://github.com/aswinkarthik/csvdiff">csvdiff</a> is a "fast diff tool for comparing CSV files" - you may get better results from this than from <code>csv-diff</code> against larger files.</li>
  </ul>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxNzU1NTAxMjc=
name: yaml-to-sqlite
full_name: simonw/yaml-to-sqlite
html_url: https://github.com/simonw/yaml-to-sqlite
description: Utility for converting YAML files to SQLite
created_at: 2019-03-14T04:49:08Z
updated_at: 2021-06-13T09:04:40Z
pushed_at: 2021-06-13T04:45:52Z
language: Python
license: apache-2.0
topics: ["yaml", "sqlite", "datasette-io", "datasette-tool"]
default_branch: main
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # yaml-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/yaml-to-sqlite.svg)](https://pypi.org/project/yaml-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/yaml-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/yaml-to-sqlite/releases)
  [![Tests](https://github.com/simonw/yaml-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/yaml-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/yaml-to-sqlite/blob/main/LICENSE)

  Load the contents of a YAML file into a SQLite database table.

  ```
  $ yaml-to-sqlite --help
  Usage: yaml-to-sqlite [OPTIONS] DB_PATH TABLE YAML_FILE

    Convert YAML files to SQLite

  Options:
    --version             Show the version and exit.
    --pk TEXT             Column to use as a primary key
    --single-column TEXT  If YAML file is a list of values, populate this column
    --help                Show this message and exit.
  ```
  ## Usage

  Given a `news.yml` file containing the following:
  ```yaml
  - date: 2021-06-05
    body: |-
      [Datasette 0.57](https://docs.datasette.io/en/stable/changelog.html#v0-57) is out with an important security patch.
  - date: 2021-05-10
    body: |-
      [Django SQL Dashboard](https://simonwillison.net/2021/May/10/django-sql-dashboard/) is a new tool that brings a useful authenticated subset of Datasette to Django projects that are built on top of PostgreSQL.
  ```
  Running this command:
  ```bash
  $ yaml-to-sqlite news.db stories news.yml
  ```
  Will create a database file with this schema:
  ```bash
  $ sqlite-utils schema news.db
  CREATE TABLE [stories] (
     [date] TEXT,
     [body] TEXT
  );
  ```
  The `--pk` option can be used to set a column as the primary key for the table:

  ```bash
  $ yaml-to-sqlite news.db stories news.yml --pk date
  $ sqlite-utils schema news.db
  CREATE TABLE [stories] (
     [date] TEXT PRIMARY KEY,
     [body] TEXT
  );
  ```
  ## Single column YAML lists

  The `--single-column` option can be used when the YAML file is a list of values, for example a file called `dogs.yml` containing the following:

  ```yaml
  - Cleo
  - Pancakes
  - Nixie
  ```
  Running this command:
  ```bash
  $ yaml-to-sqlite dogs.db dogs.yaml --single-column=name
  ```
  Will create a single `dogs` table with a single `name` column that is the primary key:

  ```bash
  $ sqlite-utils schema dogs.db
  CREATE TABLE [dogs] (
     [name] TEXT PRIMARY KEY
  );
  $ sqlite-utils dogs.db 'select * from dogs' -t
  name
  --------
  Cleo
  Pancakes
  Nixie
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-yaml-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-yaml-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>yaml-to-sqlite</h1>
  <p><a href="https://pypi.org/project/yaml-to-sqlite/" rel="nofollow"><img src="https://camo.githubusercontent.com/32afda5e7bc913df42ad343b589f5d20c4fb51d9755037f4af1df86149cd0d94/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f79616d6c2d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/yaml-to-sqlite.svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/yaml-to-sqlite/releases"><img src="https://camo.githubusercontent.com/83cd14b53497376686dfabbadad49ff0e485a8a84f8266713a3c89fe7181ccbc/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f79616d6c2d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/yaml-to-sqlite?include_prereleases&amp;label=changelog" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/yaml-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/yaml-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/yaml-to-sqlite/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Load the contents of a YAML file into a SQLite database table.</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ yaml-to-sqlite --help
  Usage: yaml-to-sqlite [OPTIONS] DB_PATH TABLE YAML_FILE

    Convert YAML files to SQLite

  Options:
    --version             Show the version and exit.
    --pk TEXT             Column to use as a primary key
    --single-column TEXT  If YAML file is a list of values, populate this column
    --help                Show this message and exit.
  "><pre><code>$ yaml-to-sqlite --help
  Usage: yaml-to-sqlite [OPTIONS] DB_PATH TABLE YAML_FILE

    Convert YAML files to SQLite

  Options:
    --version             Show the version and exit.
    --pk TEXT             Column to use as a primary key
    --single-column TEXT  If YAML file is a list of values, populate this column
    --help                Show this message and exit.
  </code></pre></div>
  <h2><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p>Given a <code>news.yml</code> file containing the following:</p>
  <div class="highlight highlight-source-yaml position-relative" data-snippet-clipboard-copy-content="- date: 2021-06-05
    body: |-
      [Datasette 0.57](https://docs.datasette.io/en/stable/changelog.html#v0-57) is out with an important security patch.
  - date: 2021-05-10
    body: |-
      [Django SQL Dashboard](https://simonwillison.net/2021/May/10/django-sql-dashboard/) is a new tool that brings a useful authenticated subset of Datasette to Django projects that are built on top of PostgreSQL.
  "><pre>- <span class="pl-ent">date</span>: <span class="pl-c1">2021-06-05</span>
    <span class="pl-ent">body</span>: <span class="pl-s">|-</span>
  <span class="pl-s">    [Datasette 0.57](https://docs.datasette.io/en/stable/changelog.html#v0-57) is out with an important security patch.</span>
  <span class="pl-s"></span>- <span class="pl-ent">date</span>: <span class="pl-c1">2021-05-10</span>
    <span class="pl-ent">body</span>: <span class="pl-s">|-</span>
  <span class="pl-s">    [Django SQL Dashboard](https://simonwillison.net/2021/May/10/django-sql-dashboard/) is a new tool that brings a useful authenticated subset of Datasette to Django projects that are built on top of PostgreSQL.</span></pre></div>
  <p>Running this command:</p>
  <div class="highlight highlight-source-shell position-relative" data-snippet-clipboard-copy-content="$ yaml-to-sqlite news.db stories news.yml
  "><pre>$ yaml-to-sqlite news.db stories news.yml</pre></div>
  <p>Will create a database file with this schema:</p>
  <div class="highlight highlight-source-shell position-relative" data-snippet-clipboard-copy-content="$ sqlite-utils schema news.db
  CREATE TABLE [stories] (
     [date] TEXT,
     [body] TEXT
  );
  "><pre>$ sqlite-utils schema news.db
  CREATE TABLE [stories] (
     [date] TEXT,
     [body] TEXT
  )<span class="pl-k">;</span></pre></div>
  <p>The <code>--pk</code> option can be used to set a column as the primary key for the table:</p>
  <div class="highlight highlight-source-shell position-relative" data-snippet-clipboard-copy-content="$ yaml-to-sqlite news.db stories news.yml --pk date
  $ sqlite-utils schema news.db
  CREATE TABLE [stories] (
     [date] TEXT PRIMARY KEY,
     [body] TEXT
  );
  "><pre>$ yaml-to-sqlite news.db stories news.yml --pk date
  $ sqlite-utils schema news.db
  CREATE TABLE [stories] (
     [date] TEXT PRIMARY KEY,
     [body] TEXT
  )<span class="pl-k">;</span></pre></div>
  <h2><a id="user-content-single-column-yaml-lists" class="anchor" aria-hidden="true" href="#user-content-single-column-yaml-lists"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Single column YAML lists</h2>
  <p>The <code>--single-column</code> option can be used when the YAML file is a list of values, for example a file called <code>dogs.yml</code> containing the following:</p>
  <div class="highlight highlight-source-yaml position-relative" data-snippet-clipboard-copy-content="- Cleo
  - Pancakes
  - Nixie
  "><pre>- <span class="pl-s">Cleo</span>
  - <span class="pl-s">Pancakes</span>
  - <span class="pl-s">Nixie</span></pre></div>
  <p>Running this command:</p>
  <div class="highlight highlight-source-shell position-relative" data-snippet-clipboard-copy-content="$ yaml-to-sqlite dogs.db dogs.yaml --single-column=name
  "><pre>$ yaml-to-sqlite dogs.db dogs.yaml --single-column=name</pre></div>
  <p>Will create a single <code>dogs</code> table with a single <code>name</code> column that is the primary key:</p>
  <div class="highlight highlight-source-shell position-relative" data-snippet-clipboard-copy-content="$ sqlite-utils schema dogs.db
  CREATE TABLE [dogs] (
     [name] TEXT PRIMARY KEY
  );
  $ sqlite-utils dogs.db 'select * from dogs' -t
  name
  --------
  Cleo
  Pancakes
  Nixie
  "><pre>$ sqlite-utils schema dogs.db
  CREATE TABLE [dogs] (
     [name] TEXT PRIMARY KEY
  )<span class="pl-k">;</span>
  $ sqlite-utils dogs.db <span class="pl-s"><span class="pl-pds">'</span>select * from dogs<span class="pl-pds">'</span></span> -t
  name
  --------
  Cleo
  Pancakes
  Nixie</pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxODQxNjg4NjQ=
name: datasette-render-html
full_name: simonw/datasette-render-html
html_url: https://github.com/simonw/datasette-render-html
description: Plugin for selectively rendering the HTML is specific columns
created_at: 2019-04-30T01:21:25Z
updated_at: 2020-09-24T04:44:47Z
pushed_at: 2021-03-17T03:57:13Z
language: Python
topics: ["datasette", "datasette-plugin", "datasette-io"]
default_branch: master
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # datasette-render-html

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-html.svg)](https://pypi.org/project/datasette-render-html/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-render-html.svg?style=svg)](https://circleci.com/gh/simonw/datasette-render-html)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-html/blob/master/LICENSE)

  This Datasette plugin lets you configure Datasette to render specific columns as HTML in the table and row interfaces.

  This means you can store HTML in those columns and have it rendered as such on those pages.

  If you have a database called `docs.db` containing a `glossary` table and you want the `definition` column in that table to be rendered as HTML, you would use a `metadata.json` file that looks like this:

      {
          "databases": {
              "docs": {
                  "tables": {
                      "glossary": {
                          "plugins": {
                              "datasette-render-html": {
                                  "columns": ["definition"]
                              }
                          }
                      }
                  }
              }
          }
      }

  ## Security

  This plugin allows HTML to be rendered exactly as it is stored in the database. As such, you should be sure only to use this against columns with content that you trust - otherwise you could open yourself up to an [XSS attack](https://owasp.org/www-community/attacks/xss/).

  It's possible to configure this plugin to apply to columns with specific names across whole databases or the full Datasette instance, but doing so is not safe. It could open you up to XSS vulnerabilities where an attacker composes a SQL query that results in a column containing unsafe HTML.

  As such, you should only use this plugin against specific columns in specific tables, as shown in the example above.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-render-html" class="anchor" aria-hidden="true" href="#user-content-datasette-render-html"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-render-html</h1>
  <p><a href="https://pypi.org/project/datasette-render-html/" rel="nofollow"><img src="https://camo.githubusercontent.com/d6cf26d5dc49484049bd4fa79e43a5d4571ca85ca03001436695b0c9c6046bf5/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d72656e6465722d68746d6c2e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-render-html.svg" style="max-width:100%;"></a>
  <a href="https://circleci.com/gh/simonw/datasette-render-html" rel="nofollow"><img src="https://camo.githubusercontent.com/27f16d9427d77d3a778994ee51b122ccd4e070ddb3e698e778506b4ac7d88bd2/68747470733a2f2f636972636c6563692e636f6d2f67682f73696d6f6e772f6461746173657474652d72656e6465722d68746d6c2e7376673f7374796c653d737667" alt="CircleCI" data-canonical-src="https://circleci.com/gh/simonw/datasette-render-html.svg?style=svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-render-html/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>This Datasette plugin lets you configure Datasette to render specific columns as HTML in the table and row interfaces.</p>
  <p>This means you can store HTML in those columns and have it rendered as such on those pages.</p>
  <p>If you have a database called <code>docs.db</code> containing a <code>glossary</code> table and you want the <code>definition</code> column in that table to be rendered as HTML, you would use a <code>metadata.json</code> file that looks like this:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="{
      &quot;databases&quot;: {
          &quot;docs&quot;: {
              &quot;tables&quot;: {
                  &quot;glossary&quot;: {
                      &quot;plugins&quot;: {
                          &quot;datasette-render-html&quot;: {
                              &quot;columns&quot;: [&quot;definition&quot;]
                          }
                      }
                  }
              }
          }
      }
  }
  "><pre><code>{
      "databases": {
          "docs": {
              "tables": {
                  "glossary": {
                      "plugins": {
                          "datasette-render-html": {
                              "columns": ["definition"]
                          }
                      }
                  }
              }
          }
      }
  }
  </code></pre></div>
  <h2><a id="user-content-security" class="anchor" aria-hidden="true" href="#user-content-security"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Security</h2>
  <p>This plugin allows HTML to be rendered exactly as it is stored in the database. As such, you should be sure only to use this against columns with content that you trust - otherwise you could open yourself up to an <a href="https://owasp.org/www-community/attacks/xss/" rel="nofollow">XSS attack</a>.</p>
  <p>It's possible to configure this plugin to apply to columns with specific names across whole databases or the full Datasette instance, but doing so is not safe. It could open you up to XSS vulnerabilities where an attacker composes a SQL query that results in a column containing unsafe HTML.</p>
  <p>As such, you should only use this plugin against specific columns in specific tables, as shown in the example above.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxODkzMjE2NzE=
name: datasette-jq
full_name: simonw/datasette-jq
html_url: https://github.com/simonw/datasette-jq
description: Datasette plugin that adds a custom SQL function for executing jq expressions against JSON values
created_at: 2019-05-30T01:06:31Z
updated_at: 2020-12-24T17:35:27Z
pushed_at: 2020-04-09T05:43:43Z
language: Python
license: apache-2.0
topics: ["jq", "datasette", "datasette-plugin", "datasette-io"]
default_branch: master
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # datasette-jq

  [![PyPI](https://img.shields.io/pypi/v/datasette-jq.svg)](https://pypi.org/project/datasette-jq/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-jq.svg?style=svg)](https://circleci.com/gh/simonw/datasette-jq)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-jq/blob/master/LICENSE)

  Datasette plugin that adds custom SQL functions for executing [jq](https://stedolan.github.io/jq/) expressions against JSON values.

  Install this plugin in the same environment as Datasette to enable the `jq()` SQL function.

  Usage:

      select jq(
          column_with_json,
          "{top_3: .classifiers[:3], v: .version}"
      )

  See [the jq manual](https://stedolan.github.io/jq/manual/#Basicfilters) for full details of supported expression syntax.

  ## Interactive demo

  You can try this plugin out at [datasette-jq-demo.datasette.io](https://datasette-jq-demo.datasette.io/)

  Sample query:

      select package, "https://pypi.org/project/" || package || "/" as url,
      jq(info, "{summary: .info.summary, author: .info.author, versions: .releases|keys|reverse}")
      from packages

  [Try this query out](https://datasette-jq-demo.datasette.io/demo?sql=select+package%2C+%22https%3A%2F%2Fpypi.org%2Fproject%2F%22+%7C%7C+package+%7C%7C+%22%2F%22+as+url%2C%0D%0Ajq%28info%2C+%22%7Bsummary%3A+.info.summary%2C+author%3A+.info.author%2C+versions%3A+.releases%7Ckeys%7Creverse%7D%22%29%0D%0Afrom+packages) in the interactive demo.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-jq" class="anchor" aria-hidden="true" href="#user-content-datasette-jq"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-jq</h1>
  <p><a href="https://pypi.org/project/datasette-jq/" rel="nofollow"><img src="https://camo.githubusercontent.com/c2171714e633d8829fa1aa20a14db54e2fd581574883e34eb1c684abd5c40963/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d6a712e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-jq.svg" style="max-width:100%;"></a>
  <a href="https://circleci.com/gh/simonw/datasette-jq" rel="nofollow"><img src="https://camo.githubusercontent.com/95b100e2cf4b8cda790f4e9b37a3eee67f56566f45acf66097aba367f649209e/68747470733a2f2f636972636c6563692e636f6d2f67682f73696d6f6e772f6461746173657474652d6a712e7376673f7374796c653d737667" alt="CircleCI" data-canonical-src="https://circleci.com/gh/simonw/datasette-jq.svg?style=svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-jq/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Datasette plugin that adds custom SQL functions for executing <a href="https://stedolan.github.io/jq/" rel="nofollow">jq</a> expressions against JSON values.</p>
  <p>Install this plugin in the same environment as Datasette to enable the <code>jq()</code> SQL function.</p>
  <p>Usage:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="select jq(
      column_with_json,
      &quot;{top_3: .classifiers[:3], v: .version}&quot;
  )
  "><pre><code>select jq(
      column_with_json,
      "{top_3: .classifiers[:3], v: .version}"
  )
  </code></pre></div>
  <p>See <a href="https://stedolan.github.io/jq/manual/#Basicfilters" rel="nofollow">the jq manual</a> for full details of supported expression syntax.</p>
  <h2><a id="user-content-interactive-demo" class="anchor" aria-hidden="true" href="#user-content-interactive-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Interactive demo</h2>
  <p>You can try this plugin out at <a href="https://datasette-jq-demo.datasette.io/" rel="nofollow">datasette-jq-demo.datasette.io</a></p>
  <p>Sample query:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="select package, &quot;https://pypi.org/project/&quot; || package || &quot;/&quot; as url,
  jq(info, &quot;{summary: .info.summary, author: .info.author, versions: .releases|keys|reverse}&quot;)
  from packages
  "><pre><code>select package, "https://pypi.org/project/" || package || "/" as url,
  jq(info, "{summary: .info.summary, author: .info.author, versions: .releases|keys|reverse}")
  from packages
  </code></pre></div>
  <p><a href="https://datasette-jq-demo.datasette.io/demo?sql=select+package%2C+%22https%3A%2F%2Fpypi.org%2Fproject%2F%22+%7C%7C+package+%7C%7C+%22%2F%22+as+url%2C%0D%0Ajq%28info%2C+%22%7Bsummary%3A+.info.summary%2C+author%3A+.info.author%2C+versions%3A+.releases%7Ckeys%7Creverse%7D%22%29%0D%0Afrom+packages" rel="nofollow">Try this query out</a> in the interactive demo.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxOTA5NTA3ODE=
name: datasette-bplist
full_name: simonw/datasette-bplist
html_url: https://github.com/simonw/datasette-bplist
description: Datasette plugin for working with Apple's binary plist format
created_at: 2019-06-09T01:15:01Z
updated_at: 2021-06-07T18:05:00Z
pushed_at: 2019-06-09T01:17:19Z
language: Python
license: apache-2.0
topics: ["bplist", "datasette", "datasette-plugin", "datasette-io"]
default_branch: master
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # datasette-bplist

  [![PyPI](https://img.shields.io/pypi/v/datasette-bplist.svg)](https://pypi.org/project/datasette-bplist/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-bplist.svg?style=svg)](https://circleci.com/gh/simonw/datasette-bplist)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-bplist/blob/master/LICENSE)

  Datasette plugin for working with Apple's [binary plist](https://en.wikipedia.org/wiki/Property_list) format.

  This plugin adds two features: a display hook and a SQL function.

  The display hook will detect any database values that are encoded using the binary plist format. It will decode them, convert them into JSON and display them pretty-printed in the Datasette UI.

  The SQL function `bplist_to_json(value)` can be used inside a SQL query to convert a binary plist value into a JSON string. This can then be used with SQLite's `json_extract()` function or with the [datasette-jq](https://github.com/simonw/datasette-jq) plugin to further analyze that data as part of a SQL query.

  Install this plugin in the same environment as Datasette to enable this new functionality:

      pip install datasette-bplist

  ## Trying it out

  If you use a Mac you already have plenty of SQLite databases that contain binary plist data.

  One example is the database that powers the Apple Photos app.

  This database tends to be locked, so you will need to create a copy of the database in order to run queries against it:

      cp ~/Pictures/Photos\ Library.photoslibrary/database/photos.db /tmp/photos.db

  The database also makes use of custom SQLite extensions which prevent it from opening in Datasette.

  You can work around this by exporting the data that you want to experiment with into a new SQLite file.

  I recommend trying this plugin against the `RKMaster_dataNote` table, which contains plist-encoded EXIF metadata about the photos you have taken.

  You can export that table into a fresh database like so:

      sqlite3 /tmp/photos.db ".dump RKMaster_dataNote" | sqlite3 /tmp/exif.db

  Now run `datasette /tmp/exif.db` and you can start trying out the plugin.

  ## Using the bplist_to_json() SQL function

  Once you have the `exif.db` demo working, you can try the `bplist_to_json()` SQL function.

  Here's a query that shows the camera lenses you have used the most often to take photos:

      select
          json_extract(
              bplist_to_json(value),
              "$.{Exif}.LensModel"
          ) as lens,
          count(*) as n
      from RKMaster_dataNote
      group by lens
      order by n desc;

  If you have a large number of photos this query can take a long time to execute, so you may need to increase the SQL time limit enforced by Datasette like so:

      $ datasette /tmp/exif.db \
          --config sql_time_limit_ms:10000

  Here's another query, showing the time at which you took every photo in your library which is classified as as screenshot:

      select
          attachedToId,
          json_extract(
              bplist_to_json(value),
              "$.{Exif}.DateTimeOriginal"
          )
      from RKMaster_dataNote
      where
          json_extract(
              bplist_to_json(value),
              "$.{Exif}.UserComment"
          ) = "Screenshot"

  And if you install the [datasette-cluster-map](https://github.com/simonw/datasette-cluster-map) plugin, this query will show you a map of your most recent 1000 photos:

      select
          *, 
          json_extract(
              bplist_to_json(value),
              "$.{GPS}.Latitude"
          ) as latitude,
          -json_extract(
              bplist_to_json(value),
              "$.{GPS}.Longitude"
          ) as longitude,
          json_extract(
              bplist_to_json(value),
              "$.{Exif}.DateTimeOriginal"
          ) as datetime
      from
          RKMaster_dataNote
      where
          latitude is not null
      order by
          attachedToId desc

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-bplist" class="anchor" aria-hidden="true" href="#user-content-datasette-bplist"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-bplist</h1>
  <p><a href="https://pypi.org/project/datasette-bplist/" rel="nofollow"><img src="https://camo.githubusercontent.com/a6a0093ab5212f3444dc00643b1dddd80bba4eea970443b0cbab79157ebab0ea/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d62706c6973742e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-bplist.svg" style="max-width:100%;"></a>
  <a href="https://circleci.com/gh/simonw/datasette-bplist" rel="nofollow"><img src="https://camo.githubusercontent.com/b4666b829d10d25169928fd8de374c652be7345117ec655c69ba8e6d8b9d2421/68747470733a2f2f636972636c6563692e636f6d2f67682f73696d6f6e772f6461746173657474652d62706c6973742e7376673f7374796c653d737667" alt="CircleCI" data-canonical-src="https://circleci.com/gh/simonw/datasette-bplist.svg?style=svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-bplist/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Datasette plugin for working with Apple's <a href="https://en.wikipedia.org/wiki/Property_list" rel="nofollow">binary plist</a> format.</p>
  <p>This plugin adds two features: a display hook and a SQL function.</p>
  <p>The display hook will detect any database values that are encoded using the binary plist format. It will decode them, convert them into JSON and display them pretty-printed in the Datasette UI.</p>
  <p>The SQL function <code>bplist_to_json(value)</code> can be used inside a SQL query to convert a binary plist value into a JSON string. This can then be used with SQLite's <code>json_extract()</code> function or with the <a href="https://github.com/simonw/datasette-jq">datasette-jq</a> plugin to further analyze that data as part of a SQL query.</p>
  <p>Install this plugin in the same environment as Datasette to enable this new functionality:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install datasette-bplist
  "><pre><code>pip install datasette-bplist
  </code></pre></div>
  <h2><a id="user-content-trying-it-out" class="anchor" aria-hidden="true" href="#user-content-trying-it-out"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Trying it out</h2>
  <p>If you use a Mac you already have plenty of SQLite databases that contain binary plist data.</p>
  <p>One example is the database that powers the Apple Photos app.</p>
  <p>This database tends to be locked, so you will need to create a copy of the database in order to run queries against it:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="cp ~/Pictures/Photos\ Library.photoslibrary/database/photos.db /tmp/photos.db
  "><pre><code>cp ~/Pictures/Photos\ Library.photoslibrary/database/photos.db /tmp/photos.db
  </code></pre></div>
  <p>The database also makes use of custom SQLite extensions which prevent it from opening in Datasette.</p>
  <p>You can work around this by exporting the data that you want to experiment with into a new SQLite file.</p>
  <p>I recommend trying this plugin against the <code>RKMaster_dataNote</code> table, which contains plist-encoded EXIF metadata about the photos you have taken.</p>
  <p>You can export that table into a fresh database like so:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="sqlite3 /tmp/photos.db &quot;.dump RKMaster_dataNote&quot; | sqlite3 /tmp/exif.db
  "><pre><code>sqlite3 /tmp/photos.db ".dump RKMaster_dataNote" | sqlite3 /tmp/exif.db
  </code></pre></div>
  <p>Now run <code>datasette /tmp/exif.db</code> and you can start trying out the plugin.</p>
  <h2><a id="user-content-using-the-bplist_to_json-sql-function" class="anchor" aria-hidden="true" href="#user-content-using-the-bplist_to_json-sql-function"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Using the bplist_to_json() SQL function</h2>
  <p>Once you have the <code>exif.db</code> demo working, you can try the <code>bplist_to_json()</code> SQL function.</p>
  <p>Here's a query that shows the camera lenses you have used the most often to take photos:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="select
      json_extract(
          bplist_to_json(value),
          &quot;$.{Exif}.LensModel&quot;
      ) as lens,
      count(*) as n
  from RKMaster_dataNote
  group by lens
  order by n desc;
  "><pre><code>select
      json_extract(
          bplist_to_json(value),
          "$.{Exif}.LensModel"
      ) as lens,
      count(*) as n
  from RKMaster_dataNote
  group by lens
  order by n desc;
  </code></pre></div>
  <p>If you have a large number of photos this query can take a long time to execute, so you may need to increase the SQL time limit enforced by Datasette like so:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ datasette /tmp/exif.db \
      --config sql_time_limit_ms:10000
  "><pre><code>$ datasette /tmp/exif.db \
      --config sql_time_limit_ms:10000
  </code></pre></div>
  <p>Here's another query, showing the time at which you took every photo in your library which is classified as as screenshot:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="select
      attachedToId,
      json_extract(
          bplist_to_json(value),
          &quot;$.{Exif}.DateTimeOriginal&quot;
      )
  from RKMaster_dataNote
  where
      json_extract(
          bplist_to_json(value),
          &quot;$.{Exif}.UserComment&quot;
      ) = &quot;Screenshot&quot;
  "><pre><code>select
      attachedToId,
      json_extract(
          bplist_to_json(value),
          "$.{Exif}.DateTimeOriginal"
      )
  from RKMaster_dataNote
  where
      json_extract(
          bplist_to_json(value),
          "$.{Exif}.UserComment"
      ) = "Screenshot"
  </code></pre></div>
  <p>And if you install the <a href="https://github.com/simonw/datasette-cluster-map">datasette-cluster-map</a> plugin, this query will show you a map of your most recent 1000 photos:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="select
      *, 
      json_extract(
          bplist_to_json(value),
          &quot;$.{GPS}.Latitude&quot;
      ) as latitude,
      -json_extract(
          bplist_to_json(value),
          &quot;$.{GPS}.Longitude&quot;
      ) as longitude,
      json_extract(
          bplist_to_json(value),
          &quot;$.{Exif}.DateTimeOriginal&quot;
      ) as datetime
  from
      RKMaster_dataNote
  where
      latitude is not null
  order by
      attachedToId desc
  "><pre><code>select
      *, 
      json_extract(
          bplist_to_json(value),
          "$.{GPS}.Latitude"
      ) as latitude,
      -json_extract(
          bplist_to_json(value),
          "$.{GPS}.Longitude"
      ) as longitude,
      json_extract(
          bplist_to_json(value),
          "$.{Exif}.DateTimeOriginal"
      ) as datetime
  from
      RKMaster_dataNote
  where
      latitude is not null
  order by
      attachedToId desc
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxOTEwMjI5Mjg=
name: datasette-render-binary
full_name: simonw/datasette-render-binary
html_url: https://github.com/simonw/datasette-render-binary
description: Datasette plugin for rendering binary data
created_at: 2019-06-09T15:25:52Z
updated_at: 2021-06-02T09:29:20Z
pushed_at: 2019-06-13T16:14:31Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin", "datasette-io"]
default_branch: master
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # datasette-render-binary

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-binary.svg)](https://pypi.org/project/datasette-render-binary/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-render-binary.svg?style=svg)](https://circleci.com/gh/simonw/datasette-render-binary)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-binary/blob/master/LICENSE)

  Datasette plugin for rendering binary data.

  Install this plugin in the same environment as Datasette to enable this new functionality:

      pip install datasette-render-binary

  Binary data in cells will now be rendered as a mixture of characters and octets.

  ![Screenshot](https://raw.githubusercontent.com/simonw/datasette-render-binary/master/example.png)

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-render-binary" class="anchor" aria-hidden="true" href="#user-content-datasette-render-binary"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-render-binary</h1>
  <p><a href="https://pypi.org/project/datasette-render-binary/" rel="nofollow"><img src="https://camo.githubusercontent.com/a83615e99f9534c53bfc77b437ad7e5b60431113905c2dcc6a1832df12ce734a/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d72656e6465722d62696e6172792e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-render-binary.svg" style="max-width:100%;"></a>
  <a href="https://circleci.com/gh/simonw/datasette-render-binary" rel="nofollow"><img src="https://camo.githubusercontent.com/969d0483aca477a61ba4afc0f0aa6ed38f54509ea1376031d455864f01ed217a/68747470733a2f2f636972636c6563692e636f6d2f67682f73696d6f6e772f6461746173657474652d72656e6465722d62696e6172792e7376673f7374796c653d737667" alt="CircleCI" data-canonical-src="https://circleci.com/gh/simonw/datasette-render-binary.svg?style=svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-render-binary/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Datasette plugin for rendering binary data.</p>
  <p>Install this plugin in the same environment as Datasette to enable this new functionality:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install datasette-render-binary
  "><pre><code>pip install datasette-render-binary
  </code></pre></div>
  <p>Binary data in cells will now be rendered as a mixture of characters and octets.</p>
  <p><a target="_blank" rel="noopener noreferrer" href="https://raw.githubusercontent.com/simonw/datasette-render-binary/master/example.png"><img src="https://raw.githubusercontent.com/simonw/datasette-render-binary/master/example.png" alt="Screenshot" style="max-width:100%;"></a></p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxOTUwODcxMzc=
name: datasette-auth-github
full_name: simonw/datasette-auth-github
html_url: https://github.com/simonw/datasette-auth-github
description: Datasette plugin that authenticates users against GitHub
created_at: 2019-07-03T16:02:53Z
updated_at: 2021-06-03T11:42:54Z
pushed_at: 2021-02-25T06:40:17Z
homepage: https://datasette-auth-github-demo.datasette.io/
language: Python
license: apache-2.0
topics: ["asgi", "datasette", "datasette-plugin", "datasette-io"]
default_branch: main
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # datasette-auth-github

  [![PyPI](https://img.shields.io/pypi/v/datasette-auth-github.svg)](https://pypi.org/project/datasette-auth-github/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-auth-github?include_prereleases&label=changelog)](https://github.com/simonw/datasette-auth-github/releases)
  [![Tests](https://github.com/simonw/datasette-auth-github/workflows/Test/badge.svg)](https://github.com/simonw/datasette-auth-github/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-auth-github/blob/main/LICENSE)

  Datasette plugin that authenticates users against GitHub.

  <!-- toc -->

  - [Setup instructions](#setup-instructions)
  - [The authenticated actor](#the-authenticated-actor)
  - [Restricting access to specific users](#restricting-access-to-specific-users)
  - [Restricting access to specific GitHub organizations or teams](#restricting-access-to-specific-github-organizations-or-teams)
  - [What to do if a user is removed from an organization or team](#what-to-do-if-a-user-is-removed-from-an-organization-or-team)

  <!-- tocstop -->

  ## Setup instructions

  * Install the plugin: `datasette install datasette-auth-github`
  * Create a GitHub OAuth app: https://github.com/settings/applications/new
  * Set the Authorization callback URL to `http://127.0.0.1:8001/-/github-auth-callback`
  * Create a `metadata.json` file with the following structure:

  ```json
  {
      "title": "datasette-auth-github demo",
      "plugins": {
          "datasette-auth-github": {
              "client_id": {"$env": "GITHUB_CLIENT_ID"},
              "client_secret": {"$env": "GITHUB_CLIENT_SECRET"}
          }
      }
  }
  ```

  Now you can start Datasette like this, passing in the secrets as environment variables:

      $ GITHUB_CLIENT_ID=XXX GITHUB_CLIENT_SECRET=YYY datasette \
          fixtures.db -m metadata.json

  Note that hard-coding secrets in `metadata.json` is a bad idea as they will be visible to anyone who can navigate to `/-/metadata`. Instead, we use Datasette's mechanism for [adding secret plugin configuration options](https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values).

  By default anonymous users will still be able to interact with Datasette. If you wish all users to have to sign in with a GitHub account first, add this to your ``metadata.json``:

  ```json
  {
      "allow": {
          "id": "*"
      },
      "plugins": {
          "datasette-auth-github": {
              "...": "..."
          }
      }
  }
  ```
  ## The authenticated actor

  Visit `/-/actor` when signed in to see the shape of the authenticated actor. It should look something like this:

  ```json
  {
      "actor": {
          "display": "simonw",
          "gh_id": "9599",
          "gh_name": "Simon Willison",
          "gh_login": "simonw",
          "gh_email": "...",
          "gh_orgs": [
              "dogsheep",
              "datasette-project"
          ],
          "gh_teams": [
              "dogsheep/test"
          ]
      }
  }
  ```

  The `gh_orgs` and `gh_teams` properties will only be present if you used `load_teams` or `load_orgs`, documented below.

  ## Restricting access to specific users

  You can use Datasette's [permissions mechanism](https://docs.datasette.io/en/stable/authentication.html) to specify which user or users are allowed to access your instance. Here's how to restrict access to just GitHub user `simonw`:

  ```json
  {
      "allow": {
          "gh_login": "simonw"
      },
      "plugins": {
          "datasette-auth-github": {
              "...": "..."
          }
      }
  }
  ```

  This `"allow"` block can be positioned at the database, table or query level instead: see [Configuring permissions in metadata.json](https://docs.datasette.io/en/stable/authentication.html#configuring-permissions-in-metadata-json) for details.

  Note that GitHub allows users to change their username, and it is possible for other people to claim old usernames. If you are concerned that your users may change their usernames you can key the allow blocks against GitHub user IDs instead, which do not change:

  ```json
  {
      "allow": {
          "gh_id": "9599"
      }
  }
  ```

  ## Restricting access to specific GitHub organizations or teams

  You can also restrict access to users who are members of a specific GitHub organization.

  You'll need to configure the plugin to check if the user is a member of that organization when they first sign in. You can do that using the `"load_orgs"` plugin configuration option.

  Then you can use `"allow": {"gh_orgs": [...]}` to specify which organizations are allowed access.

  ```json
  {
      "plugins": {
          "datasette-auth-github": {
              "...": "...",
              "load_orgs": ["your-organization"]
          }
      },
      "allow": {
          "gh_orgs": "your-organization"
      }
  }
  ```

  If your organization is [arranged into teams](https://help.github.com/en/articles/organizing-members-into-teams) you can restrict access to a specific team like this:

  ```json
  {
      "plugins": {
          "datasette-auth-github": {
              "...": "...",
              "load_teams": [
                  "your-organization/staff",
                  "your-organization/engineering",
              ]
          }
      },
      "allows": {
          "gh_team": "your-organization/engineering"
      }
  }
  ```

  ## What to do if a user is removed from an organization or team

  A user's organization and team memberships are checked once, when they first sign in. Those teams and organizations are then persisted in the user's signed `ds_actor` cookie.

  This means that if a user is removed from an organization or team but still has a Datasette cookie, they will still be able to access that Datasette instance.

  You can remedy this by rotating the `DATASETTE_SECRET` environment variable any time you make changes to your GitHub organization members.

  Changing this value will cause all of your existing users to be signed out, by invalidating their cookies. When they sign back in again their new memberships will be recorded in a new cookie.

  See [Configuring the secret](https://docs.datasette.io/en/stable/settings.html?highlight=secret#configuring-the-secret) in the Datasette documentation for more details.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-auth-github" class="anchor" aria-hidden="true" href="#user-content-datasette-auth-github"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-auth-github</h1>
  <p><a href="https://pypi.org/project/datasette-auth-github/" rel="nofollow"><img src="https://camo.githubusercontent.com/a3e596637d6128f29e3fcfeb8e50ecbe5c7e1c328e94c5d338238fa0f70a2a86/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d617574682d6769746875622e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-auth-github.svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-auth-github/releases"><img src="https://camo.githubusercontent.com/86aa92461a20542b5de5e6d64b24503b5b58f83bf66a98a86bc33ea443ed93b8/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f6461746173657474652d617574682d6769746875623f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/datasette-auth-github?include_prereleases&amp;label=changelog" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-auth-github/actions?query=workflow%3ATest"><img src="https://github.com/simonw/datasette-auth-github/workflows/Test/badge.svg" alt="Tests" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-auth-github/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Datasette plugin that authenticates users against GitHub.</p>

  <ul>
  <li><a href="#user-content-setup-instructions">Setup instructions</a></li>
  <li><a href="#user-content-the-authenticated-actor">The authenticated actor</a></li>
  <li><a href="#user-content-restricting-access-to-specific-users">Restricting access to specific users</a></li>
  <li><a href="#user-content-restricting-access-to-specific-github-organizations-or-teams">Restricting access to specific GitHub organizations or teams</a></li>
  <li><a href="#user-content-what-to-do-if-a-user-is-removed-from-an-organization-or-team">What to do if a user is removed from an organization or team</a></li>
  </ul>

  <h2><a id="user-content-setup-instructions" class="anchor" aria-hidden="true" href="#user-content-setup-instructions"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Setup instructions</h2>
  <ul>
  <li>Install the plugin: <code>datasette install datasette-auth-github</code></li>
  <li>Create a GitHub OAuth app: <a href="https://github.com/settings/applications/new">https://github.com/settings/applications/new</a></li>
  <li>Set the Authorization callback URL to <code>http://127.0.0.1:8001/-/github-auth-callback</code></li>
  <li>Create a <code>metadata.json</code> file with the following structure:</li>
  </ul>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
      &quot;title&quot;: &quot;datasette-auth-github demo&quot;,
      &quot;plugins&quot;: {
          &quot;datasette-auth-github&quot;: {
              &quot;client_id&quot;: {&quot;$env&quot;: &quot;GITHUB_CLIENT_ID&quot;},
              &quot;client_secret&quot;: {&quot;$env&quot;: &quot;GITHUB_CLIENT_SECRET&quot;}
          }
      }
  }
  "><pre>{
      <span class="pl-s"><span class="pl-pds">"</span>title<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>datasette-auth-github demo<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>plugins<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>datasette-auth-github<span class="pl-pds">"</span></span>: {
              <span class="pl-s"><span class="pl-pds">"</span>client_id<span class="pl-pds">"</span></span>: {<span class="pl-s"><span class="pl-pds">"</span>$env<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>GITHUB_CLIENT_ID<span class="pl-pds">"</span></span>},
              <span class="pl-s"><span class="pl-pds">"</span>client_secret<span class="pl-pds">"</span></span>: {<span class="pl-s"><span class="pl-pds">"</span>$env<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>GITHUB_CLIENT_SECRET<span class="pl-pds">"</span></span>}
          }
      }
  }</pre></div>
  <p>Now you can start Datasette like this, passing in the secrets as environment variables:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ GITHUB_CLIENT_ID=XXX GITHUB_CLIENT_SECRET=YYY datasette \
      fixtures.db -m metadata.json
  "><pre><code>$ GITHUB_CLIENT_ID=XXX GITHUB_CLIENT_SECRET=YYY datasette \
      fixtures.db -m metadata.json
  </code></pre></div>
  <p>Note that hard-coding secrets in <code>metadata.json</code> is a bad idea as they will be visible to anyone who can navigate to <code>/-/metadata</code>. Instead, we use Datasette's mechanism for <a href="https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values" rel="nofollow">adding secret plugin configuration options</a>.</p>
  <p>By default anonymous users will still be able to interact with Datasette. If you wish all users to have to sign in with a GitHub account first, add this to your <code>metadata.json</code>:</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
      &quot;allow&quot;: {
          &quot;id&quot;: &quot;*&quot;
      },
      &quot;plugins&quot;: {
          &quot;datasette-auth-github&quot;: {
              &quot;...&quot;: &quot;...&quot;
          }
      }
  }
  "><pre>{
      <span class="pl-s"><span class="pl-pds">"</span>allow<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>*<span class="pl-pds">"</span></span>
      },
      <span class="pl-s"><span class="pl-pds">"</span>plugins<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>datasette-auth-github<span class="pl-pds">"</span></span>: {
              <span class="pl-s"><span class="pl-pds">"</span>...<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>...<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <h2><a id="user-content-the-authenticated-actor" class="anchor" aria-hidden="true" href="#user-content-the-authenticated-actor"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>The authenticated actor</h2>
  <p>Visit <code>/-/actor</code> when signed in to see the shape of the authenticated actor. It should look something like this:</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
      &quot;actor&quot;: {
          &quot;display&quot;: &quot;simonw&quot;,
          &quot;gh_id&quot;: &quot;9599&quot;,
          &quot;gh_name&quot;: &quot;Simon Willison&quot;,
          &quot;gh_login&quot;: &quot;simonw&quot;,
          &quot;gh_email&quot;: &quot;...&quot;,
          &quot;gh_orgs&quot;: [
              &quot;dogsheep&quot;,
              &quot;datasette-project&quot;
          ],
          &quot;gh_teams&quot;: [
              &quot;dogsheep/test&quot;
          ]
      }
  }
  "><pre>{
      <span class="pl-s"><span class="pl-pds">"</span>actor<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>display<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>simonw<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>gh_id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>9599<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>gh_name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Simon Willison<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>gh_login<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>simonw<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>gh_email<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>...<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>gh_orgs<span class="pl-pds">"</span></span>: [
              <span class="pl-s"><span class="pl-pds">"</span>dogsheep<span class="pl-pds">"</span></span>,
              <span class="pl-s"><span class="pl-pds">"</span>datasette-project<span class="pl-pds">"</span></span>
          ],
          <span class="pl-s"><span class="pl-pds">"</span>gh_teams<span class="pl-pds">"</span></span>: [
              <span class="pl-s"><span class="pl-pds">"</span>dogsheep/test<span class="pl-pds">"</span></span>
          ]
      }
  }</pre></div>
  <p>The <code>gh_orgs</code> and <code>gh_teams</code> properties will only be present if you used <code>load_teams</code> or <code>load_orgs</code>, documented below.</p>
  <h2><a id="user-content-restricting-access-to-specific-users" class="anchor" aria-hidden="true" href="#user-content-restricting-access-to-specific-users"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Restricting access to specific users</h2>
  <p>You can use Datasette's <a href="https://docs.datasette.io/en/stable/authentication.html" rel="nofollow">permissions mechanism</a> to specify which user or users are allowed to access your instance. Here's how to restrict access to just GitHub user <code>simonw</code>:</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
      &quot;allow&quot;: {
          &quot;gh_login&quot;: &quot;simonw&quot;
      },
      &quot;plugins&quot;: {
          &quot;datasette-auth-github&quot;: {
              &quot;...&quot;: &quot;...&quot;
          }
      }
  }
  "><pre>{
      <span class="pl-s"><span class="pl-pds">"</span>allow<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>gh_login<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>simonw<span class="pl-pds">"</span></span>
      },
      <span class="pl-s"><span class="pl-pds">"</span>plugins<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>datasette-auth-github<span class="pl-pds">"</span></span>: {
              <span class="pl-s"><span class="pl-pds">"</span>...<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>...<span class="pl-pds">"</span></span>
          }
      }
  }</pre></div>
  <p>This <code>"allow"</code> block can be positioned at the database, table or query level instead: see <a href="https://docs.datasette.io/en/stable/authentication.html#configuring-permissions-in-metadata-json" rel="nofollow">Configuring permissions in metadata.json</a> for details.</p>
  <p>Note that GitHub allows users to change their username, and it is possible for other people to claim old usernames. If you are concerned that your users may change their usernames you can key the allow blocks against GitHub user IDs instead, which do not change:</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
      &quot;allow&quot;: {
          &quot;gh_id&quot;: &quot;9599&quot;
      }
  }
  "><pre>{
      <span class="pl-s"><span class="pl-pds">"</span>allow<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>gh_id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>9599<span class="pl-pds">"</span></span>
      }
  }</pre></div>
  <h2><a id="user-content-restricting-access-to-specific-github-organizations-or-teams" class="anchor" aria-hidden="true" href="#user-content-restricting-access-to-specific-github-organizations-or-teams"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Restricting access to specific GitHub organizations or teams</h2>
  <p>You can also restrict access to users who are members of a specific GitHub organization.</p>
  <p>You'll need to configure the plugin to check if the user is a member of that organization when they first sign in. You can do that using the <code>"load_orgs"</code> plugin configuration option.</p>
  <p>Then you can use <code>"allow": {"gh_orgs": [...]}</code> to specify which organizations are allowed access.</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-auth-github&quot;: {
              &quot;...&quot;: &quot;...&quot;,
              &quot;load_orgs&quot;: [&quot;your-organization&quot;]
          }
      },
      &quot;allow&quot;: {
          &quot;gh_orgs&quot;: &quot;your-organization&quot;
      }
  }
  "><pre>{
      <span class="pl-s"><span class="pl-pds">"</span>plugins<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>datasette-auth-github<span class="pl-pds">"</span></span>: {
              <span class="pl-s"><span class="pl-pds">"</span>...<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>...<span class="pl-pds">"</span></span>,
              <span class="pl-s"><span class="pl-pds">"</span>load_orgs<span class="pl-pds">"</span></span>: [<span class="pl-s"><span class="pl-pds">"</span>your-organization<span class="pl-pds">"</span></span>]
          }
      },
      <span class="pl-s"><span class="pl-pds">"</span>allow<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>gh_orgs<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>your-organization<span class="pl-pds">"</span></span>
      }
  }</pre></div>
  <p>If your organization is <a href="https://help.github.com/en/articles/organizing-members-into-teams">arranged into teams</a> you can restrict access to a specific team like this:</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-auth-github&quot;: {
              &quot;...&quot;: &quot;...&quot;,
              &quot;load_teams&quot;: [
                  &quot;your-organization/staff&quot;,
                  &quot;your-organization/engineering&quot;,
              ]
          }
      },
      &quot;allows&quot;: {
          &quot;gh_team&quot;: &quot;your-organization/engineering&quot;
      }
  }
  "><pre>{
      <span class="pl-s"><span class="pl-pds">"</span>plugins<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>datasette-auth-github<span class="pl-pds">"</span></span>: {
              <span class="pl-s"><span class="pl-pds">"</span>...<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>...<span class="pl-pds">"</span></span>,
              <span class="pl-s"><span class="pl-pds">"</span>load_teams<span class="pl-pds">"</span></span>: [
                  <span class="pl-s"><span class="pl-pds">"</span>your-organization/staff<span class="pl-pds">"</span></span>,
                  <span class="pl-s"><span class="pl-pds">"</span>your-organization/engineering<span class="pl-pds">"</span></span>,
              ]
          }
      },
      <span class="pl-s"><span class="pl-pds">"</span>allows<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>gh_team<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>your-organization/engineering<span class="pl-pds">"</span></span>
      }
  }</pre></div>
  <h2><a id="user-content-what-to-do-if-a-user-is-removed-from-an-organization-or-team" class="anchor" aria-hidden="true" href="#user-content-what-to-do-if-a-user-is-removed-from-an-organization-or-team"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>What to do if a user is removed from an organization or team</h2>
  <p>A user's organization and team memberships are checked once, when they first sign in. Those teams and organizations are then persisted in the user's signed <code>ds_actor</code> cookie.</p>
  <p>This means that if a user is removed from an organization or team but still has a Datasette cookie, they will still be able to access that Datasette instance.</p>
  <p>You can remedy this by rotating the <code>DATASETTE_SECRET</code> environment variable any time you make changes to your GitHub organization members.</p>
  <p>Changing this value will cause all of your existing users to be signed out, by invalidating their cookies. When they sign back in again their new memberships will be recorded in a new cookie.</p>
  <p>See <a href="https://docs.datasette.io/en/stable/settings.html?highlight=secret#configuring-the-secret" rel="nofollow">Configuring the secret</a> in the Datasette documentation for more details.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxOTUxNDU2Nzg=
name: sqlite-diffable
full_name: simonw/sqlite-diffable
html_url: https://github.com/simonw/sqlite-diffable
description: Tools for dumping/loading a SQLite database to diffable directory structure
created_at: 2019-07-04T00:58:46Z
updated_at: 2022-07-12T17:00:19Z
pushed_at: 2022-08-18T22:49:29Z
language: Python
license: apache-2.0
topics: ["datasette-io", "datasette-tool", "sqlite"]
default_branch: main
permissions: {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false}
visibility: public
readme:
  # sqlite-diffable

  [![PyPI](https://img.shields.io/pypi/v/sqlite-diffable.svg)](https://pypi.org/project/sqlite-diffable/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-diffable?include_prereleases&label=changelog)](https://github.com/simonw/sqlite-diffable/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sqlite-diffable/blob/main/LICENSE)

  Tools for dumping/loading a SQLite database to diffable directory structure

  ## Installation

      pip install sqlite-diffable

  ## Demo

  The repository at [simonw/simonwillisonblog-backup](https://github.com/simonw/simonwillisonblog-backup) contains a backup of the database on my blog, https://simonwillison.net/ - created using this tool.

  ## Dumping a database

  Given a SQLite database called `fixtures.db` containing a table `facetable`, the following will dump out that table to the `dump/` directory:

      sqlite-diffable dump fixtures.db dump/ facetable

  To dump out every table in that database, use `--all`:

      sqlite-diffable dump fixtures.db dump/ --all

  ## Loading a database

  To load a previously dumped database, run the following:

      sqlite-diffable load restored.db dump/

  This will show an error if any of the tables that are being restored already exist in the database file.

  You can replace those tables (dropping them before restoring them) using the `--replace` option:

      sqlite-diffable load restored.db dump/ --replace

  ## Converting to JSON objects

  Table rows are stored in the `.ndjson` files as newline-delimited JSON arrays, like this:

  ```
  ["a", "a", "a-a", 63, null, 0.7364712141640124, "$null"]
  ["a", "b", "a-b", 51, null, 0.6020187290499803, "$null"]
  ```

  Sometimes it can be more convenient to work with a list of JSON objects.

  The `sqlite-diffable objects` command can read a `.ndjson` file and its accompanying `.metadata.json` file and output JSON objects to standard output:

      sqlite-diffable objects fixtures.db dump/sortable.ndjson

  The output of that command looks something like this:
  ```
  {"pk1": "a", "pk2": "a", "content": "a-a", "sortable": 63, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.7364712141640124, "text": "$null"}
  {"pk1": "a", "pk2": "b", "content": "a-b", "sortable": 51, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.6020187290499803, "text": "$null"}
  ```

  Add `-o` to write that output to a file:

      sqlite-diffable objects fixtures.db dump/sortable.ndjson -o output.txt

  Add `--array` to output a JSON array of objects, as opposed to a newline-delimited file:

      sqlite-diffable objects fixtures.db dump/sortable.ndjson --array
  Output:
  ```
  [
  {"pk1": "a", "pk2": "a", "content": "a-a", "sortable": 63, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.7364712141640124, "text": "$null"},
  {"pk1": "a", "pk2": "b", "content": "a-b", "sortable": 51, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.6020187290499803, "text": "$null"}
  ]
  ```

  ## Storage format

  Each table is represented as two files. The first, `table_name.metadata.json`, contains metadata describing the structure of the table. For a table called `redirects_redirect` that file might look like this:

  ```json
  {
      "name": "redirects_redirect",
      "columns": [
          "id",
          "domain",
          "path",
          "target",
          "created"
      ],
      "schema": "CREATE TABLE [redirects_redirect] (\n   [id] INTEGER PRIMARY KEY,\n   [domain] TEXT,\n   [path] TEXT,\n   [target] TEXT,\n   [created] TEXT\n)"
  }
  ```

  It is an object with three keys: `name` is the name of the table, `columns` is an array of column strings and `schema` is the SQL schema text used for tha table.

  The second file, `table_name.ndjson`, contains [newline-delimited JSON](http://ndjson.org/) for every row in the table. Each row is represented as a JSON array with items corresponding to each of the columns defined in the metadata.

  That file for the `redirects_redirect.ndjson` table might look like this:

  ```
  [1, "feeds.simonwillison.net", "swn-everything", "https://simonwillison.net/atom/everything/", "2017-10-01T21:11:36.440537+00:00"]
  [2, "feeds.simonwillison.net", "swn-entries", "https://simonwillison.net/atom/entries/", "2017-10-01T21:12:32.478849+00:00"]
  [3, "feeds.simonwillison.net", "swn-links", "https://simonwillison.net/atom/links/", "2017-10-01T21:12:54.820729+00:00"]
  ```

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-sqlite-diffable" class="anchor" aria-hidden="true" href="#user-content-sqlite-diffable"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>sqlite-diffable</h1>
  <p dir="auto"><a href="https://pypi.org/project/sqlite-diffable/" rel="nofollow"><img src="https://camo.githubusercontent.com/f397aed437dd402178061cc28cd230ddaa228d7a19d4e6e33dd7d9d6b9aa7016/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f73716c6974652d6469666661626c652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/sqlite-diffable.svg" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/sqlite-diffable/releases"><img src="https://camo.githubusercontent.com/ad89df075a7b5a43a53000aaf22188af53405e2b5177102cb1b550954996dac4/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f73716c6974652d6469666661626c653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/sqlite-diffable?include_prereleases&amp;label=changelog" style="max-width: 100%;"></a>
  <a href="https://github.com/simonw/sqlite-diffable/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p>
  <p dir="auto">Tools for dumping/loading a SQLite database to diffable directory structure</p>
  <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install sqlite-diffable"><pre class="notranslate"><code>pip install sqlite-diffable
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2>
  <p dir="auto">The repository at <a href="https://github.com/simonw/simonwillisonblog-backup">simonw/simonwillisonblog-backup</a> contains a backup of the database on my blog, <a href="https://simonwillison.net/" rel="nofollow">https://simonwillison.net/</a> - created using this tool.</p>
  <h2 dir="auto"><a id="user-content-dumping-a-database" class="anchor" aria-hidden="true" href="#user-content-dumping-a-database"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Dumping a database</h2>
  <p dir="auto">Given a SQLite database called <code>fixtures.db</code> containing a table <code>facetable</code>, the following will dump out that table to the <code>dump/</code> directory:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-diffable dump fixtures.db dump/ facetable"><pre class="notranslate"><code>sqlite-diffable dump fixtures.db dump/ facetable
  </code></pre></div>
  <p dir="auto">To dump out every table in that database, use <code>--all</code>:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-diffable dump fixtures.db dump/ --all"><pre class="notranslate"><code>sqlite-diffable dump fixtures.db dump/ --all
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-loading-a-database" class="anchor" aria-hidden="true" href="#user-content-loading-a-database"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Loading a database</h2>
  <p dir="auto">To load a previously dumped database, run the following:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-diffable load restored.db dump/"><pre class="notranslate"><code>sqlite-diffable load restored.db dump/
  </code></pre></div>
  <p dir="auto">This will show an error if any of the tables that are being restored already exist in the database file.</p>
  <p dir="auto">You can replace those tables (dropping them before restoring them) using the <code>--replace</code> option:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-diffable load restored.db dump/ --replace"><pre class="notranslate"><code>sqlite-diffable load restored.db dump/ --replace
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-converting-to-json-objects" class="anchor" aria-hidden="true" href="#user-content-converting-to-json-objects"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Converting to JSON objects</h2>
  <p dir="auto">Table rows are stored in the <code>.ndjson</code> files as newline-delimited JSON arrays, like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[&quot;a&quot;, &quot;a&quot;, &quot;a-a&quot;, 63, null, 0.7364712141640124, &quot;$null&quot;]
  [&quot;a&quot;, &quot;b&quot;, &quot;a-b&quot;, 51, null, 0.6020187290499803, &quot;$null&quot;]"><pre class="notranslate"><code>["a", "a", "a-a", 63, null, 0.7364712141640124, "$null"]
  ["a", "b", "a-b", 51, null, 0.6020187290499803, "$null"]
  </code></pre></div>
  <p dir="auto">Sometimes it can be more convenient to work with a list of JSON objects.</p>
  <p dir="auto">The <code>sqlite-diffable objects</code> command can read a <code>.ndjson</code> file and its accompanying <code>.metadata.json</code> file and output JSON objects to standard output:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-diffable objects fixtures.db dump/sortable.ndjson"><pre class="notranslate"><code>sqlite-diffable objects fixtures.db dump/sortable.ndjson
  </code></pre></div>
  <p dir="auto">The output of that command looks something like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{&quot;pk1&quot;: &quot;a&quot;, &quot;pk2&quot;: &quot;a&quot;, &quot;content&quot;: &quot;a-a&quot;, &quot;sortable&quot;: 63, &quot;sortable_with_nulls&quot;: null, &quot;sortable_with_nulls_2&quot;: 0.7364712141640124, &quot;text&quot;: &quot;$null&quot;}
  {&quot;pk1&quot;: &quot;a&quot;, &quot;pk2&quot;: &quot;b&quot;, &quot;content&quot;: &quot;a-b&quot;, &quot;sortable&quot;: 51, &quot;sortable_with_nulls&quot;: null, &quot;sortable_with_nulls_2&quot;: 0.6020187290499803, &quot;text&quot;: &quot;$null&quot;}"><pre class="notranslate"><code>{"pk1": "a", "pk2": "a", "content": "a-a", "sortable": 63, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.7364712141640124, "text": "$null"}
  {"pk1": "a", "pk2": "b", "content": "a-b", "sortable": 51, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.6020187290499803, "text": "$null"}
  </code></pre></div>
  <p dir="auto">Add <code>-o</code> to write that output to a file:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-diffable objects fixtures.db dump/sortable.ndjson -o output.txt"><pre class="notranslate"><code>sqlite-diffable objects fixtures.db dump/sortable.ndjson -o output.txt
  </code></pre></div>
  <p dir="auto">Add <code>--array</code> to output a JSON array of objects, as opposed to a newline-delimited file:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="sqlite-diffable objects fixtures.db dump/sortable.ndjson --array"><pre class="notranslate"><code>sqlite-diffable objects fixtures.db dump/sortable.ndjson --array
  </code></pre></div>
  <p dir="auto">Output:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[
  {&quot;pk1&quot;: &quot;a&quot;, &quot;pk2&quot;: &quot;a&quot;, &quot;content&quot;: &quot;a-a&quot;, &quot;sortable&quot;: 63, &quot;sortable_with_nulls&quot;: null, &quot;sortable_with_nulls_2&quot;: 0.7364712141640124, &quot;text&quot;: &quot;$null&quot;},
  {&quot;pk1&quot;: &quot;a&quot;, &quot;pk2&quot;: &quot;b&quot;, &quot;content&quot;: &quot;a-b&quot;, &quot;sortable&quot;: 51, &quot;sortable_with_nulls&quot;: null, &quot;sortable_with_nulls_2&quot;: 0.6020187290499803, &quot;text&quot;: &quot;$null&quot;}
  ]"><pre class="notranslate"><code>[
  {"pk1": "a", "pk2": "a", "content": "a-a", "sortable": 63, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.7364712141640124, "text": "$null"},
  {"pk1": "a", "pk2": "b", "content": "a-b", "sortable": 51, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.6020187290499803, "text": "$null"}
  ]
  </code></pre></div>
  <h2 dir="auto"><a id="user-content-storage-format" class="anchor" aria-hidden="true" href="#user-content-storage-format"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Storage format</h2>
  <p dir="auto">Each table is represented as two files. The first, <code>table_name.metadata.json</code>, contains metadata describing the structure of the table. For a table called <code>redirects_redirect</code> that file might look like this:</p>
  <div class="highlight highlight-source-json notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
      &quot;name&quot;: &quot;redirects_redirect&quot;,
      &quot;columns&quot;: [
          &quot;id&quot;,
          &quot;domain&quot;,
          &quot;path&quot;,
          &quot;target&quot;,
          &quot;created&quot;
      ],
      &quot;schema&quot;: &quot;CREATE TABLE [redirects_redirect] (\n   [id] INTEGER PRIMARY KEY,\n   [domain] TEXT,\n   [path] TEXT,\n   [target] TEXT,\n   [created] TEXT\n)&quot;
  }"><pre>{
      <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>redirects_redirect<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"columns"</span>: [
          <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>domain<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>path<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>target<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>created<span class="pl-pds">"</span></span>
      ],
      <span class="pl-ent">"schema"</span>: <span class="pl-s"><span class="pl-pds">"</span>CREATE TABLE [redirects_redirect] (<span class="pl-cce">\n</span>   [id] INTEGER PRIMARY KEY,<span class="pl-cce">\n</span>   [domain] TEXT,<span class="pl-cce">\n</span>   [path] TEXT,<span class="pl-cce">\n</span>   [target] TEXT,<span class="pl-cce">\n</span>   [created] TEXT<span class="pl-cce">\n</span>)<span class="pl-pds">"</span></span>
  }</pre></div>
  <p dir="auto">It is an object with three keys: <code>name</code> is the name of the table, <code>columns</code> is an array of column strings and <code>schema</code> is the SQL schema text used for tha table.</p>
  <p dir="auto">The second file, <code>table_name.ndjson</code>, contains <a href="http://ndjson.org/" rel="nofollow">newline-delimited JSON</a> for every row in the table. Each row is represented as a JSON array with items corresponding to each of the columns defined in the metadata.</p>
  <p dir="auto">That file for the <code>redirects_redirect.ndjson</code> table might look like this:</p>
  <div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[1, &quot;feeds.simonwillison.net&quot;, &quot;swn-everything&quot;, &quot;https://simonwillison.net/atom/everything/&quot;, &quot;2017-10-01T21:11:36.440537+00:00&quot;]
  [2, &quot;feeds.simonwillison.net&quot;, &quot;swn-entries&quot;, &quot;https://simonwillison.net/atom/entries/&quot;, &quot;2017-10-01T21:12:32.478849+00:00&quot;]
  [3, &quot;feeds.simonwillison.net&quot;, &quot;swn-links&quot;, &quot;https://simonwillison.net/atom/links/&quot;, &quot;2017-10-01T21:12:54.820729+00:00&quot;]"><pre class="notranslate"><code>[1, "feeds.simonwillison.net", "swn-everything", "https://simonwillison.net/atom/everything/", "2017-10-01T21:11:36.440537+00:00"]
  [2, "feeds.simonwillison.net", "swn-entries", "https://simonwillison.net/atom/entries/", "2017-10-01T21:12:32.478849+00:00"]
  [3, "feeds.simonwillison.net", "swn-links", "https://simonwillison.net/atom/links/", "2017-10-01T21:12:54.820729+00:00"]
  </code></pre></div>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxOTU2OTY4MDQ=
name: datasette-cors
full_name: simonw/datasette-cors
html_url: https://github.com/simonw/datasette-cors
description: Datasette plugin for configuring CORS headers
created_at: 2019-07-07T21:03:11Z
updated_at: 2021-02-27T00:31:13Z
pushed_at: 2019-07-11T04:40:57Z
language: Python
license: apache-2.0
topics: ["datasette", "datasette-plugin", "datasette-io"]
default_branch: master
permissions: {"admin": false, "push": false, "pull": false}
readme:
  # datasette-cors

  [![PyPI](https://img.shields.io/pypi/v/datasette-cors.svg)](https://pypi.org/project/datasette-cors/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-cors.svg?style=svg)](https://circleci.com/gh/simonw/datasette-cors)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-cors/blob/master/LICENSE)

  Datasette plugin for configuring CORS headers, based on https://github.com/simonw/asgi-cors

  You can use this plugin to allow JavaScript running on a whitelisted set of domains to make `fetch()` calls to the JSON API provided by your Datasette instance.

  ## Installation

      pip install datasette-cors

  ## Configuration

  You need to add some configuration to your Datasette `metadata.json` file for this plugin to take effect.

  To whitelist specific domains, use this:

  ```json
  {
      "plugins": {
          "datasette-cors": {
              "hosts": ["https://www.example.com"]
          }
      }
  }
  ```

  You can also whitelist patterns like this:

  ```json
  {
      "plugins": {
          "datasette-cors": {
              "host_wildcards": ["https://*.example.com"]
          }
      }
  }
  ```

  ## Testing it

  To test this plugin out, run it locally by saving one of the above examples as `metadata.json` and running this:

      $ datasette --memory -m metadata.json

  Now visit https://www.example.com/ in your browser, open the browser developer console and paste in the following:

  ```javascript
  fetch("http://127.0.0.1:8001/:memory:.json?sql=select+sqlite_version%28%29").then(r => r.json()).then(console.log)
  ```

  If the plugin is running correctly, you will see the JSON response output to the console.

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-datasette-cors" class="anchor" aria-hidden="true" href="#user-content-datasette-cors"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>datasette-cors</h1>
  <p><a href="https://pypi.org/project/datasette-cors/" rel="nofollow"><img src="https://camo.githubusercontent.com/3f4dc5b1725858ca83723674cadd1a59373f3f4265fb01eb43c91ca9380b24bb/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f6461746173657474652d636f72732e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/datasette-cors.svg" style="max-width:100%;"></a>
  <a href="https://circleci.com/gh/simonw/datasette-cors" rel="nofollow"><img src="https://camo.githubusercontent.com/831507d824fb545803ebf51bf318c316eb5f9405e6cc9dabcb4835f356f4053d/68747470733a2f2f636972636c6563692e636f6d2f67682f73696d6f6e772f6461746173657474652d636f72732e7376673f7374796c653d737667" alt="CircleCI" data-canonical-src="https://circleci.com/gh/simonw/datasette-cors.svg?style=svg" style="max-width:100%;"></a>
  <a href="https://github.com/simonw/datasette-cors/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Datasette plugin for configuring CORS headers, based on <a href="https://github.com/simonw/asgi-cors">https://github.com/simonw/asgi-cors</a></p>
  <p>You can use this plugin to allow JavaScript running on a whitelisted set of domains to make <code>fetch()</code> calls to the JSON API provided by your Datasette instance.</p>
  <h2><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install datasette-cors
  "><pre><code>pip install datasette-cors
  </code></pre></div>
  <h2><a id="user-content-configuration" class="anchor" aria-hidden="true" href="#user-content-configuration"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Configuration</h2>
  <p>You need to add some configuration to your Datasette <code>metadata.json</code> file for this plugin to take effect.</p>
  <p>To whitelist specific domains, use this:</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-cors&quot;: {
              &quot;hosts&quot;: [&quot;https://www.example.com&quot;]
          }
      }
  }
  "><pre>{
      <span class="pl-s"><span class="pl-pds">"</span>plugins<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>datasette-cors<span class="pl-pds">"</span></span>: {
              <span class="pl-s"><span class="pl-pds">"</span>hosts<span class="pl-pds">"</span></span>: [<span class="pl-s"><span class="pl-pds">"</span>https://www.example.com<span class="pl-pds">"</span></span>]
          }
      }
  }</pre></div>
  <p>You can also whitelist patterns like this:</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;datasette-cors&quot;: {
              &quot;host_wildcards&quot;: [&quot;https://*.example.com&quot;]
          }
      }
  }
  "><pre>{
      <span class="pl-s"><span class="pl-pds">"</span>plugins<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>datasette-cors<span class="pl-pds">"</span></span>: {
              <span class="pl-s"><span class="pl-pds">"</span>host_wildcards<span class="pl-pds">"</span></span>: [<span class="pl-s"><span class="pl-pds">"</span>https://*.example.com<span class="pl-pds">"</span></span>]
          }
      }
  }</pre></div>
  <h2><a id="user-content-testing-it" class="anchor" aria-hidden="true" href="#user-content-testing-it"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Testing it</h2>
  <p>To test this plugin out, run it locally by saving one of the above examples as <code>metadata.json</code> and running this:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ datasette --memory -m metadata.json
  "><pre><code>$ datasette --memory -m metadata.json
  </code></pre></div>
  <p>Now visit <a href="https://www.example.com/" rel="nofollow">https://www.example.com/</a> in your browser, open the browser developer console and paste in the following:</p>
  <div class="highlight highlight-source-js position-relative" data-snippet-clipboard-copy-content="fetch(&quot;http://127.0.0.1:8001/:memory:.json?sql=select+sqlite_version%28%29&quot;).then(r =&gt; r.json()).then(console.log)
  "><pre><span class="pl-en">fetch</span><span class="pl-kos">(</span><span class="pl-s">"http://127.0.0.1:8001/:memory:.json?sql=select+sqlite_version%28%29"</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">then</span><span class="pl-kos">(</span><span class="pl-s1">r</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">r</span><span class="pl-kos">.</span><span class="pl-en">json</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">then</span><span class="pl-kos">(</span><span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-c1">log</span><span class="pl-kos">)</span></pre></div>
  <p>If the plugin is running correctly, you will see the JSON response output to the console.</p>
  </article></div>


node_id: MDEwOlJlcG9zaXRvcnkxOTc0MzExMDk=
name: dogsheep-beta
full_name: dogsheep/dogsheep-beta
html_url: https://github.com/dogsheep/dogsheep-beta
created_at: 2019-07-17T17:07:26Z
updated_at: 2021-06-13T14:39:01Z
pushed_at: 2021-06-13T14:38:59Z
homepage: https://dogsheep.github.io/
language: Python
topics: ["search", "datasette", "datasette-plugin", "dogsheep", "datasette-io", "datasette-tool"]
default_branch: main
permissions: {"admin": false, "push": false, "pull": false}
description:
  Build a search index across content from multiple SQLite database tables and run faceted searches against it using Datasette

readme:
  # dogsheep-beta

  [![PyPI](https://img.shields.io/pypi/v/dogsheep-beta.svg)](https://pypi.org/project/dogsheep-beta/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/beta?include_prereleases&label=changelog)](https://github.com/dogsheep/beta/releases)
  [![Tests](https://github.com/dogsheep/beta/workflows/Test/badge.svg)](https://github.com/dogsheep/beta/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/beta/blob/main/LICENSE)

  Build a search index across content from multiple SQLite database tables and run faceted searches against it using Datasette

  ## Example

  A live example of this plugin is running at https://datasette.io/-/beta - configured using [this YAML file](https://github.com/simonw/datasette.io/blob/main/templates/dogsheep-beta.yml).

  Read more about how this example works in [Building a search engine for datasette.io](https://simonwillison.net/2020/Dec/19/dogsheep-beta/).

  ## Installation

  Install this tool like so:

      $ pip install dogsheep-beta

  ## Usage

  Run the indexer using the `dogsheep-beta` command-line tool:

      $ dogsheep-beta index dogsheep.db config.yml

  The `config.yml` file contains details of the databases and document types that should be indexed:

  ```yaml
  twitter.db:
      tweets:
          sql: |-
              select
                  tweets.id as key,
                  'Tweet by @' || users.screen_name as title,
                  tweets.created_at as timestamp,
                  tweets.full_text as search_1
              from tweets join users on tweets.user = users.id
      users:
          sql: |-
              select
                  id as key,
                  name || ' @' || screen_name as title,
                  created_at as timestamp,
                  description as search_1
              from users
  ```

  This will create a `search_index` table in the `dogsheep.db` database populated by data from those SQL queries.

  By default the search index that this tool creates will be configured for Porter stemming. This means that searches for words like `run` will match documents containing `runs` or `running`.

  If you don't want to use Porter stemming, use the `--tokenize none` option:

      $ dogsheep-beta index dogsheep.db config.yml --tokenize none

  You can pass other SQLite tokenize argumenst here, see [the SQLite FTS tokenizers documentation](https://www.sqlite.org/fts5.html#tokenizers).

  ## Columns

  The columns that can be returned by our query are:

  - `key` - a unique (within that type) primary key
  - `title` - the title for the item
  - `timestamp` - an ISO8601 timestamp, e.g. `2020-09-02T21:00:21`
  - `search_1` - a larger chunk of text to be included in the search index
  - `category` - an integer category ID, see below
  - `is_public` - an integer (0 or 1, defaults to 0 if not set) specifying if this is public or not

  Public records are things like your public tweets, blog posts and GitHub commits.

  ## Categories

  Indexed items can be assigned a category. Categories are integers that correspond to records in the `categories` table, which defaults to containing the following:

  |   id | name       |
  |------|------------|
  |    1 | created    |
  |    2 | saved      |
  |    3 | received   |

  `created` is for items that have been created by the Dogsheep instance owner.

  `saved` is for items that they have saved, liked or favourited.

  `received` is for items that have been specifically sent to them by other people - incoming emails or direct messages for example.

  ## Datasette plugin

  Run `datasette install dogsheep-beta` (or use `pip install dogsheep-beta` in the same environment as Datasette) to install the Dogsheep Beta Datasette plugin.

  Once installed, a custom search interface will be made available at `/-/beta`. You can use this interface to execute searches.

  The Datasette plugin has some configuration options. You can set these by adding the following to your `metadata.json` configuration file:

  ```json
  {
      "plugins": {
          "dogsheep-beta": {
              "database": "beta",
              "config_file": "dogsheep-beta.yml",
              "template_debug": true
          }
      }
  }
  ```
  The configuration settings for the plugin are:
  - `database` - the database file that contains your search index. If the file is `beta.db` you should set `database` to `beta`.
  - `config_file` - the YAML file containing your Dogsheep Beta configuration.
  - `template_debug` - set this to `true` to enable debugging output if errors occur in your custom templates, see below.

  ## Custom results display

  Each indexed item type can define custom display HTML as part of the `config.yml` file. It can do this using a `display` key containing a fragment of Jinja template, and optionally a `display_sql` key with extra SQL to execute to fetch the data to display.

  Here's how to define a custom display template for a tweet:

  ```yaml
  twitter.db:
      tweets:
          sql: |-
              select
                  tweets.id as key,
                  'Tweet by @' || users.screen_name as title,
                  tweets.created_at as timestamp,
                  tweets.full_text as search_1
              from tweets join users on tweets.user = users.id
          display: |-
              <p>{{ title }} - tweeted at {{ timestamp }}</p>
              <blockquote>{{ search_1 }}</blockquote>
  ```
  This example reuses the value that were stored in the `search_index` table when the indexing query was run.

  To load in extra values to display in the template, use a `display_sql` query like this:

  ```yaml
  twitter.db:
      tweets:
          sql: |-
              select
                  tweets.id as key,
                  'Tweet by @' || users.screen_name as title,
                  tweets.created_at as timestamp,
                  tweets.full_text as search_1
              from tweets join users on tweets.user = users.id
          display_sql: |-
              select
                  users.screen_name,
                  tweets.full_text,
                  tweets.created_at
              from
                  tweets join users on tweets.user = users.id
              where
                  tweets.id = :key
          display: |-
              <p>{{ display.screen_name }} - tweeted at {{ display.created_at }}</p>
              <blockquote>{{ display.full_text }}</blockquote>
  ```
  The `display_sql` query will be executed for every search result, passing the key value from the `search_index` table as the `:key` parameter and the user's search term as the `:q` parameter.

  This performs well because [many small queries are efficient in SQLite](https://www.sqlite.org/np1queryprob.html).

  If an error occurs while rendering one of your templates the search results page will return a 500 error. You can use the `template_debug` configuration setting described above to instead output debugging information for the search results item that experienced the error.

  ## Displaying maps

  This plugin will eventually include a number of useful shortcuts for rendering interesting content.

  The first available shortcut is for displaying maps. Make your custom content output something like this:

  ```html
  <div
      data-map-latitude="{{ display.latitude }}"
      data-map-longitude="{{ display.longitude }}"
      style="display: none; float: right; width: 250px; height: 200px; background-color: #ccc;"
  ></div>
  ```
  JavaScript on the page will look for any elements with `data-map-latitude` and `data-map-longitude` and, if it finds any, will load Leaflet and convert those elements into maps centered on that location. The default zoom level will be 12, or you can set a `data-map-zoom` attribute to customize this.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd dogsheep-beta
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

readme_html:
  <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1><a id="user-content-dogsheep-beta" class="anchor" aria-hidden="true" href="#user-content-dogsheep-beta"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>dogsheep-beta</h1>
  <p><a href="https://pypi.org/project/dogsheep-beta/" rel="nofollow"><img src="https://camo.githubusercontent.com/f8d17c422fbd622ab366c57086226b7f90b6ab9057aa2f3b9f6844dd29bac733/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f646f6773686565702d626574612e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/dogsheep-beta.svg" style="max-width:100%;"></a>
  <a href="https://github.com/dogsheep/beta/releases"><img src="https://camo.githubusercontent.com/341e696d2cba7197bea24e27cc6db041753c20587a00ced21e77516d26a54730/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f646f6773686565702f626574613f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/dogsheep/beta?include_prereleases&amp;label=changelog" style="max-width:100%;"></a>
  <a href="https://github.com/dogsheep/beta/actions?query=workflow%3ATest"><img src="https://github.com/dogsheep/beta/workflows/Test/badge.svg" alt="Tests" style="max-width:100%;"></a>
  <a href="https://github.com/dogsheep/beta/blob/main/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width:100%;"></a></p>
  <p>Build a search index across content from multiple SQLite database tables and run faceted searches against it using Datasette</p>
  <h2><a id="user-content-example" class="anchor" aria-hidden="true" href="#user-content-example"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Example</h2>
  <p>A live example of this plugin is running at <a href="https://datasette.io/-/beta" rel="nofollow">https://datasette.io/-/beta</a> - configured using <a href="https://github.com/simonw/datasette.io/blob/main/templates/dogsheep-beta.yml">this YAML file</a>.</p>
  <p>Read more about how this example works in <a href="https://simonwillison.net/2020/Dec/19/dogsheep-beta/" rel="nofollow">Building a search engine for datasette.io</a>.</p>
  <h2><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2>
  <p>Install this tool like so:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ pip install dogsheep-beta
  "><pre><code>$ pip install dogsheep-beta
  </code></pre></div>
  <h2><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2>
  <p>Run the indexer using the <code>dogsheep-beta</code> command-line tool:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ dogsheep-beta index dogsheep.db config.yml
  "><pre><code>$ dogsheep-beta index dogsheep.db config.yml
  </code></pre></div>
  <p>The <code>config.yml</code> file contains details of the databases and document types that should be indexed:</p>
  <div class="highlight highlight-source-yaml position-relative" data-snippet-clipboard-copy-content="twitter.db:
      tweets:
          sql: |-
              select
                  tweets.id as key,
                  'Tweet by @' || users.screen_name as title,
                  tweets.created_at as timestamp,
                  tweets.full_text as search_1
              from tweets join users on tweets.user = users.id
      users:
          sql: |-
              select
                  id as key,
                  name || ' @' || screen_name as title,
                  created_at as timestamp,
                  description as search_1
              from users
  "><pre><span class="pl-ent">twitter.db</span>:
      <span class="pl-ent">tweets</span>:
          <span class="pl-ent">sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">            select</span>
  <span class="pl-s">                tweets.id as key,</span>
  <span class="pl-s">                'Tweet by @' || users.screen_name as title,</span>
  <span class="pl-s">                tweets.created_at as timestamp,</span>
  <span class="pl-s">                tweets.full_text as search_1</span>
  <span class="pl-s">            from tweets join users on tweets.user = users.id</span>
  <span class="pl-s"></span>    <span class="pl-ent">users</span>:
          <span class="pl-ent">sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">            select</span>
  <span class="pl-s">                id as key,</span>
  <span class="pl-s">                name || ' @' || screen_name as title,</span>
  <span class="pl-s">                created_at as timestamp,</span>
  <span class="pl-s">                description as search_1</span>
  <span class="pl-s">            from users</span></pre></div>
  <p>This will create a <code>search_index</code> table in the <code>dogsheep.db</code> database populated by data from those SQL queries.</p>
  <p>By default the search index that this tool creates will be configured for Porter stemming. This means that searches for words like <code>run</code> will match documents containing <code>runs</code> or <code>running</code>.</p>
  <p>If you don't want to use Porter stemming, use the <code>--tokenize none</code> option:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="$ dogsheep-beta index dogsheep.db config.yml --tokenize none
  "><pre><code>$ dogsheep-beta index dogsheep.db config.yml --tokenize none
  </code></pre></div>
  <p>You can pass other SQLite tokenize argumenst here, see <a href="https://www.sqlite.org/fts5.html#tokenizers" rel="nofollow">the SQLite FTS tokenizers documentation</a>.</p>
  <h2><a id="user-content-columns" class="anchor" aria-hidden="true" href="#user-content-columns"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Columns</h2>
  <p>The columns that can be returned by our query are:</p>
  <ul>
  <li><code>key</code> - a unique (within that type) primary key</li>
  <li><code>title</code> - the title for the item</li>
  <li><code>timestamp</code> - an ISO8601 timestamp, e.g. <code>2020-09-02T21:00:21</code></li>
  <li><code>search_1</code> - a larger chunk of text to be included in the search index</li>
  <li><code>category</code> - an integer category ID, see below</li>
  <li><code>is_public</code> - an integer (0 or 1, defaults to 0 if not set) specifying if this is public or not</li>
  </ul>
  <p>Public records are things like your public tweets, blog posts and GitHub commits.</p>
  <h2><a id="user-content-categories" class="anchor" aria-hidden="true" href="#user-content-categories"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Categories</h2>
  <p>Indexed items can be assigned a category. Categories are integers that correspond to records in the <code>categories</code> table, which defaults to containing the following:</p>
  <table>
  <thead>
  <tr>
  <th>id</th>
  <th>name</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1</td>
  <td>created</td>
  </tr>
  <tr>
  <td>2</td>
  <td>saved</td>
  </tr>
  <tr>
  <td>3</td>
  <td>received</td>
  </tr>
  </tbody>
  </table>
  <p><code>created</code> is for items that have been created by the Dogsheep instance owner.</p>
  <p><code>saved</code> is for items that they have saved, liked or favourited.</p>
  <p><code>received</code> is for items that have been specifically sent to them by other people - incoming emails or direct messages for example.</p>
  <h2><a id="user-content-datasette-plugin" class="anchor" aria-hidden="true" href="#user-content-datasette-plugin"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Datasette plugin</h2>
  <p>Run <code>datasette install dogsheep-beta</code> (or use <code>pip install dogsheep-beta</code> in the same environment as Datasette) to install the Dogsheep Beta Datasette plugin.</p>
  <p>Once installed, a custom search interface will be made available at <code>/-/beta</code>. You can use this interface to execute searches.</p>
  <p>The Datasette plugin has some configuration options. You can set these by adding the following to your <code>metadata.json</code> configuration file:</p>
  <div class="highlight highlight-source-json position-relative" data-snippet-clipboard-copy-content="{
      &quot;plugins&quot;: {
          &quot;dogsheep-beta&quot;: {
              &quot;database&quot;: &quot;beta&quot;,
              &quot;config_file&quot;: &quot;dogsheep-beta.yml&quot;,
              &quot;template_debug&quot;: true
          }
      }
  }
  "><pre>{
      <span class="pl-s"><span class="pl-pds">"</span>plugins<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>dogsheep-beta<span class="pl-pds">"</span></span>: {
              <span class="pl-s"><span class="pl-pds">"</span>database<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>beta<span class="pl-pds">"</span></span>,
              <span class="pl-s"><span class="pl-pds">"</span>config_file<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>dogsheep-beta.yml<span class="pl-pds">"</span></span>,
              <span class="pl-s"><span class="pl-pds">"</span>template_debug<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>
          }
      }
  }</pre></div>
  <p>The configuration settings for the plugin are:</p>
  <ul>
  <li><code>database</code> - the database file that contains your search index. If the file is <code>beta.db</code> you should set <code>database</code> to <code>beta</code>.</li>
  <li><code>config_file</code> - the YAML file containing your Dogsheep Beta configuration.</li>
  <li><code>template_debug</code> - set this to <code>true</code> to enable debugging output if errors occur in your custom templates, see below.</li>
  </ul>
  <h2><a id="user-content-custom-results-display" class="anchor" aria-hidden="true" href="#user-content-custom-results-display"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Custom results display</h2>
  <p>Each indexed item type can define custom display HTML as part of the <code>config.yml</code> file. It can do this using a <code>display</code> key containing a fragment of Jinja template, and optionally a <code>display_sql</code> key with extra SQL to execute to fetch the data to display.</p>
  <p>Here's how to define a custom display template for a tweet:</p>
  <div class="highlight highlight-source-yaml position-relative" data-snippet-clipboard-copy-content="twitter.db:
      tweets:
          sql: |-
              select
                  tweets.id as key,
                  'Tweet by @' || users.screen_name as title,
                  tweets.created_at as timestamp,
                  tweets.full_text as search_1
              from tweets join users on tweets.user = users.id
          display: |-
              &lt;p&gt;{{ title }} - tweeted at {{ timestamp }}&lt;/p&gt;
              &lt;blockquote&gt;{{ search_1 }}&lt;/blockquote&gt;
  "><pre><span class="pl-ent">twitter.db</span>:
      <span class="pl-ent">tweets</span>:
          <span class="pl-ent">sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">            select</span>
  <span class="pl-s">                tweets.id as key,</span>
  <span class="pl-s">                'Tweet by @' || users.screen_name as title,</span>
  <span class="pl-s">                tweets.created_at as timestamp,</span>
  <span class="pl-s">                tweets.full_text as search_1</span>
  <span class="pl-s">            from tweets join users on tweets.user = users.id</span>
  <span class="pl-s"></span>        <span class="pl-ent">display</span>: <span class="pl-s">|-</span>
  <span class="pl-s">            &lt;p&gt;{{ title }} - tweeted at {{ timestamp }}&lt;/p&gt;</span>
  <span class="pl-s">            &lt;blockquote&gt;{{ search_1 }}&lt;/blockquote&gt;</span></pre></div>
  <p>This example reuses the value that were stored in the <code>search_index</code> table when the indexing query was run.</p>
  <p>To load in extra values to display in the template, use a <code>display_sql</code> query like this:</p>
  <div class="highlight highlight-source-yaml position-relative" data-snippet-clipboard-copy-content="twitter.db:
      tweets:
          sql: |-
              select
                  tweets.id as key,
                  'Tweet by @' || users.screen_name as title,
                  tweets.created_at as timestamp,
                  tweets.full_text as search_1
              from tweets join users on tweets.user = users.id
          display_sql: |-
              select
                  users.screen_name,
                  tweets.full_text,
                  tweets.created_at
              from
                  tweets join users on tweets.user = users.id
              where
                  tweets.id = :key
          display: |-
              &lt;p&gt;{{ display.screen_name }} - tweeted at {{ display.created_at }}&lt;/p&gt;
              &lt;blockquote&gt;{{ display.full_text }}&lt;/blockquote&gt;
  "><pre><span class="pl-ent">twitter.db</span>:
      <span class="pl-ent">tweets</span>:
          <span class="pl-ent">sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">            select</span>
  <span class="pl-s">                tweets.id as key,</span>
  <span class="pl-s">                'Tweet by @' || users.screen_name as title,</span>
  <span class="pl-s">                tweets.created_at as timestamp,</span>
  <span class="pl-s">                tweets.full_text as search_1</span>
  <span class="pl-s">            from tweets join users on tweets.user = users.id</span>
  <span class="pl-s"></span>        <span class="pl-ent">display_sql</span>: <span class="pl-s">|-</span>
  <span class="pl-s">            select</span>
  <span class="pl-s">                users.screen_name,</span>
  <span class="pl-s">                tweets.full_text,</span>
  <span class="pl-s">                tweets.created_at</span>
  <span class="pl-s">            from</span>
  <span class="pl-s">                tweets join users on tweets.user = users.id</span>
  <span class="pl-s">            where</span>
  <span class="pl-s">                tweets.id = :key</span>
  <span class="pl-s"></span>        <span class="pl-ent">display</span>: <span class="pl-s">|-</span>
  <span class="pl-s">            &lt;p&gt;{{ display.screen_name }} - tweeted at {{ display.created_at }}&lt;/p&gt;</span>
  <span class="pl-s">            &lt;blockquote&gt;{{ display.full_text }}&lt;/blockquote&gt;</span></pre></div>
  <p>The <code>display_sql</code> query will be executed for every search result, passing the key value from the <code>search_index</code> table as the <code>:key</code> parameter and the user's search term as the <code>:q</code> parameter.</p>
  <p>This performs well because <a href="https://www.sqlite.org/np1queryprob.html" rel="nofollow">many small queries are efficient in SQLite</a>.</p>
  <p>If an error occurs while rendering one of your templates the search results page will return a 500 error. You can use the <code>template_debug</code> configuration setting described above to instead output debugging information for the search results item that experienced the error.</p>
  <h2><a id="user-content-displaying-maps" class="anchor" aria-hidden="true" href="#user-content-displaying-maps"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Displaying maps</h2>
  <p>This plugin will eventually include a number of useful shortcuts for rendering interesting content.</p>
  <p>The first available shortcut is for displaying maps. Make your custom content output something like this:</p>
  <div class="highlight highlight-text-html-basic position-relative" data-snippet-clipboard-copy-content="&lt;div
      data-map-latitude=&quot;{{ display.latitude }}&quot;
      data-map-longitude=&quot;{{ display.longitude }}&quot;
      style=&quot;display: none; float: right; width: 250px; height: 200px; background-color: #ccc;&quot;
  &gt;&lt;/div&gt;
  "><pre><span class="pl-kos">&lt;</span><span class="pl-ent">div</span>
      <span class="pl-c1">data-map-latitude</span>="<span class="pl-s">{{ display.latitude }}</span>"
      <span class="pl-c1">data-map-longitude</span>="<span class="pl-s">{{ display.longitude }}</span>"
      <span class="pl-c1">style</span>="<span class="pl-s">display: none; float: right; width: 250px; height: 200px; background-color: #ccc;</span>"
  <span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">div</span><span class="pl-kos">&gt;</span></pre></div>
  <p>JavaScript on the page will look for any elements with <code>data-map-latitude</code> and <code>data-map-longitude</code> and, if it finds any, will load Leaflet and convert those elements into maps centered on that location. The default zoom level will be 12, or you can set a <code>data-map-zoom</code> attribute to customize this.</p>
  <h2><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2>
  <p>To set up this plugin locally, first checkout the code. Then create a new virtual environment:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="cd dogsheep-beta
  python3 -mvenv venv
  source venv/bin/activate
  "><pre><code>cd dogsheep-beta
  python3 -mvenv venv
  source venv/bin/activate
  </code></pre></div>
  <p>Or if you are using <code>pipenv</code>:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pipenv shell
  "><pre><code>pipenv shell
  </code></pre></div>
  <p>Now install the dependencies and tests:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pip install -e '.[test]'
  "><pre><code>pip install -e '.[test]'
  </code></pre></div>
  <p>To run the tests:</p>
  <div class="snippet-clipboard-content position-relative" data-snippet-clipboard-copy-content="pytest
  "><pre><code>pytest
  </code></pre></div>
  </article></div>


