# pypi_packages

name: datasette-mask-columns
summary: Datasette plugin that masks specified database columns
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-mask-columns
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-mask-columns/
project_url: https://pypi.org/project/datasette-mask-columns/
project_urls: {"Homepage": "https://github.com/simonw/datasette-mask-columns"}
release_url: https://pypi.org/project/datasette-mask-columns/0.2.1/
version: 0.2.1
description:
  # datasette-mask-columns

  [![PyPI](https://img.shields.io/pypi/v/datasette-mask-columns.svg)](https://pypi.org/project/datasette-mask-columns/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-mask-columns?include_prereleases&label=changelog)](https://github.com/simonw/datasette-mask-columns/releases)
  [![Tests](https://github.com/simonw/datasette-mask-columns/workflows/Test/badge.svg)](https://github.com/simonw/datasette-mask-columns/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-mask-columns/blob/main/LICENSE)

  Datasette plugin that masks specified database columns

  ## Installation

      pip install datasette-mask-columns

  This depends on plugin hook changes in a not-yet released branch of Datasette. See [issue #678](https://github.com/simonw/datasette/issues/678) for details.

  ## Usage

  In your `metadata.json` file add a section like this describing the database and table in which you wish to mask columns:

  ```json
  {
      "databases": {
          "my-database": {
              "plugins": {
                  "datasette-mask-columns": {
                      "users": ["password"]
                  }
              }
          }
      }
  }
  ```
  All SQL queries against the `users` table in `my-database.db` will now return `null` for the `password` column, no matter what value that column actually holds.

  The table page for `users` will display the text `REDACTED` in the masked column. This visual hint will only be available on the table page; it will not display his text for arbitrary queries against the table.

requires_dist:
  ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "httpx ; extra == 'test'", "sqlite-utils ; extra == 'test'"]


name: datasette-media
summary: Datasette plugin for serving media based on a SQL query
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-media
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-media/
project_url: https://pypi.org/project/datasette-media/
release_url: https://pypi.org/project/datasette-media/0.5/
version: 0.5
description:
  # datasette-media

  [![PyPI](https://img.shields.io/pypi/v/datasette-media.svg)](https://pypi.org/project/datasette-media/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-media?include_prereleases&label=changelog)](https://github.com/simonw/datasette-media/releases)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-media.svg?style=svg)](https://circleci.com/gh/simonw/datasette-media)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-media/blob/master/LICENSE)

  Datasette plugin for serving media based on a SQL query.

  Use this when you have a database table containing references to files on disk - or binary content stored in BLOB columns - that you would like to be able to serve to your users.

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-media

  ### HEIC image support

  Modern iPhones save their photos using the [HEIC image format](https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format). Processing these images requires an additional dependency, [pyheif](https://pypi.org/project/pyheif/). You can include this dependency by running:

      $ pip install datasette-media[heif]

  ## Usage

  You can use this plugin to configure Datasette to serve static media based on SQL queries to an underlying database table.

  Media will be served from URLs that start with `/-/media/`. The full URL to each media asset will look like this:

      /-/media/type-of-media/media-key

  `type-of-media` will correspond to a configured SQL query, and might be something like `photo`. `media-key` will be an identifier that is used as part of the underlying SQL query to find which file should be served.

  ### Serving static files from disk

  The following ``metadata.json`` configuration will cause this plugin to serve files from disk, based on queries to a database table called `apple_photos`.

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath from apple_photos where uuid=:key"
              }
          }
      }
  }
  ```

  A request to `/-/media/photo/CF972D33-5324-44F2-8DAE-22CB3182CD31` will execute the following SQL query:

  ```sql
  select filepath from apple_photos where uuid=:key
  ```

  The value from the URL -  in this case `CF972D33-5324-44F2-8DAE-22CB3182CD31` - will be passed as the `:key` parameter to the query.

  The query returns a `filepath` value that has been read from the table. The plugin will then read that file from disk and serve it in response to the request.

  SQL queries default to running against the first connected database. You can specify a different database to execute the query against using `"database": "name_of_db"`. To execute against `photos.db`, use this:

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath from apple_photos where uuid=:key",
                  "database": "photos"
              }
          }
      }
  }
  ```

  See [dogsheep-photos](https://github.com/dogsheep/dogsheep-photos) for an example of an application that can benefit from this plugin.

  ### Serving binary content from BLOB columns

  If your SQL query returns a `content` column, this will be served directly to the user:

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select thumbnail as content from photos where uuid=:key",
                  "database": "thumbs"
              }
          }
      }
  }
  ```

  You can also return a `content_type` column which will be used as the `Content-Type` header served to the user:

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select body as content, 'text/html;charset=utf-8' as content_type from documents where id=:key",
                  "database": "documents"
              }
          }
      }
  }
  ```

  If you do not specify a `content_type` the default of `application/octet-stream` will be used.

  ### Serving content proxied from a URL

  To serve content that is itself fetched from elsewhere, return a `content_url` column. This can be particularly useful when combined with the ability to resize images (described in the next section).

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photos": {
                  "sql": "select photo_url as content_url from photos where id=:key",
                  "database": "photos",
                  "enable_transform": true
              }
          }
      }
  }
  ```

  Now you can access resized versions of images from that URL like so:

      /-/media/photos/13?w=200

  ### Setting a download file name

  The `content_filename` column can be returned to force browsers to download the content using a specific file name.

  ```json
  {
      "plugins": {
          "datasette-media": {
              "hello": {
                  "sql": "select 'Hello ' || :key as content, 'hello.txt' as content_filename"
              }
          }
      }
  }
  ```

  Visiting `/-/media/hello/Groot` will cause your browser to download a file called `hello.txt` containing the text `Hello Groot`.

  ### Resizing or transforming images

  Your SQL query can specify that an image should be resized and/or converted to another format by returning additional columns. All three are optional.

  * `resize_width` - the width to resize the image to
  * `resize_width` - the height to resize the image to
  * `output_format` - the output format to use (e.g. `jpeg` or `png`) - any output format [supported by Pillow](https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html) is allowed here.

  If you specify one but not the other of `resize_width` or `resize_height` the unspecified one will be calculated automatically to maintain the aspect ratio of the image.

  Here's an example configuration that will resize all images to be JPEGs that are 200 pixels in height:

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath, 200 as resize_height, 'jpeg' as output_format from apple_photos where uuid=:key",
                  "database": "photos"
              }
          }
      }
  }
  ```

  If you enable the `enable_transform` configuration option you can instead specify transform parameters at runtime using querystring parameters. For example:

  - `/-/media/photo/CF972D33?w=200` to resize to a fixed width
  - `/-/media/photo/CF972D33?h=200` to resize to a fixed height
  - `/-/media/photo/CF972D33?format=jpeg` to convert to JPEG

  That option is added like so:

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath from apple_photos where uuid=:key",
                  "database": "photos",
                  "enable_transform": true
              }
          }
      }
  }
  ```

  The maximum allowed height or width is 4000 pixels. You can change this limit using the `"max_width_height"` option:

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath from apple_photos where uuid=:key",
                  "database": "photos",
                  "enable_transform": true,
                  "max_width_height": 1000
              }
          }
      }
  }
  ```

  ## Configuration

  In addition to the different named content types, the following special plugin configuration setting is available:

  - `transform_threads` - number of threads to use for running transformations (e.g. resizing). Defaults to 4.

  This can be used like this:

  ```json
  {
      "plugins": {
          "datasette-media": {
              "photo": {
                  "sql": "select filepath from apple_photos where uuid=:key",
                  "database": "photos"
              },
              "transform_threads": 8
          }
      }
  }
  ```

project_urls:
  {"CI": "https://app.circleci.com/pipelines/github/simonw/datasette-media", "Changelog": "https://github.com/simonw/datasette-media/releases", "Homepage": "https://github.com/simonw/datasette-media", "Issues": "https://gitlab.com/simonw/datasette-media/issues"}

requires_dist:
  ["datasette (>=0.44)", "Pillow (>=7.1.2)", "httpx (>=0.13.3)", "pyheif (>=0.4) ; extra == 'heif'", "asgiref ; extra == 'test'", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "sqlite-utils ; extra == 'test'", "pytest-httpx (>=0.4.0) ; extra == 'test'"]


name: datasette-mp3-audio
summary: Turn .mp3 URLs into an audio player in the Datasette interface
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-mp3-audio
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-mp3-audio/
project_url: https://pypi.org/project/datasette-mp3-audio/
release_url: https://pypi.org/project/datasette-mp3-audio/0.2/
requires_python: >=3.7
version: 0.2
description:
  # datasette-mp3-audio

  [![PyPI](https://img.shields.io/pypi/v/datasette-mp3-audio.svg)](https://pypi.org/project/datasette-mp3-audio/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-mp3-audio?include_prereleases&label=changelog)](https://github.com/simonw/datasette-mp3-audio/releases)
  [![Tests](https://github.com/simonw/datasette-mp3-audio/workflows/Test/badge.svg)](https://github.com/simonw/datasette-mp3-audio/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-mp3-audio/blob/main/LICENSE)

  Turn .mp3 URLs into an audio player in the Datasette interface

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-mp3-audio

  ## Demo

  Try this plugin at [https://scotrail.datasette.io/scotrail/announcements](https://scotrail.datasette.io/scotrail/announcements)

  The demo uses ScotRail train announcements from [matteason/scotrail-announcements-june-2022](https://github.com/matteason/scotrail-announcements-june-2022).

  ## Usage

  Once installed, any cells with a value that ends in `.mp3` and starts with either `http://` or `/` or `https://` will be turned into an embedded HTML audio element like this:

  ```html
  <audio controls src="... value ...">Audio not supported</audio>
  ```

  A "Play X MP3s on this page" button will be added to athe top of any table page listing more than one MP3.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-mp3-audio
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-mp3-audio/actions", "Changelog": "https://github.com/simonw/datasette-mp3-audio/releases", "Homepage": "https://github.com/simonw/datasette-mp3-audio", "Issues": "https://github.com/simonw/datasette-mp3-audio/issues"}

requires_dist:
  ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "sqlite-utils ; extra == 'test'"]


name: datasette-multiline-links
summary: Make multiple newline separated URLs clickable in Datasette
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-multiline-links
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-multiline-links/
project_url: https://pypi.org/project/datasette-multiline-links/
release_url: https://pypi.org/project/datasette-multiline-links/0.1/
requires_dist: ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.7
version: 0.1
description:
  # datasette-multiline-links

  [![PyPI](https://img.shields.io/pypi/v/datasette-multiline-links.svg)](https://pypi.org/project/datasette-multiline-links/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-multiline-links?include_prereleases&label=changelog)](https://github.com/simonw/datasette-multiline-links/releases)
  [![Tests](https://github.com/simonw/datasette-multiline-links/workflows/Test/badge.svg)](https://github.com/simonw/datasette-multiline-links/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-multiline-links/blob/main/LICENSE)

  Make multiple newline separated URLs clickable in Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-multiline-links

  ## Usage

  Once installed, if a cell has contents like this:
  ```
  https://example.com
  Not a link
  https://google.com
  ```
  It will be rendered as:
  ```html
  <a href="https://example.com">https://example.com</a>
  Not a link
  <a href="https://google.com">https://google.com</a>
  ```
  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-multiline-links
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-multiline-links/actions", "Changelog": "https://github.com/simonw/datasette-multiline-links/releases", "Homepage": "https://github.com/simonw/datasette-multiline-links", "Issues": "https://github.com/simonw/datasette-multiline-links/issues"}


name: datasette-nteract-data-explorer
summary: automatic visual data explorer for datasette
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Cameron Yick
description_content_type: text/markdown
home_page: https://github.com/hydrosquall/datasette-nteract-data-explorer
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-nteract-data-explorer/
project_url: https://pypi.org/project/datasette-nteract-data-explorer/
release_url: https://pypi.org/project/datasette-nteract-data-explorer/0.5.1/
requires_dist: ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.7
version: 0.5.1
description:
  # datasette-nteract-data-explorer

  [![PyPI](https://img.shields.io/pypi/v/datasette-nteract-data-explorer.svg)](https://pypi.org/project/datasette-nteract-data-explorer/)
  [![Changelog](https://img.shields.io/github/v/release/hydrosquall/datasette-nteract-data-explorer?include_prereleases&label=changelog)](https://github.com/hydrosquall/datasette-nteract-data-explorer/releases)
  [![Tests](https://github.com/hydrosquall/datasette-nteract-data-explorer/workflows/Test/badge.svg)](https://github.com/hydrosquall/datasette-nteract-data-explorer/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/hydrosquall/datasette-nteract-data-explorer/blob/main/LICENSE)

  An automatic data visualization plugin for the [Datasette](https://datasette.io/) ecosystem. See your dataset from multiple views with an easy-to-use, customizable menu-based interface.

  ## Demo

  Try the [live demo](https://datasette-nteract-data-explorer.vercel.app/happy_planet_index/hpi_cleaned?_size=137)

  ![screenshot](https://p-qkfgo2.t2.n0.cdn.getcloudapp.com/items/yAuK9LRE/6802f849-315d-4a21-93b4-61c94d066bdc.jpg?v=f1ceee5ed70832d74e745b6508baeffb)

  _Running Datasette with the Happy Planet Index dataset_

  ## Installation

  Install this plugin in the same Python environment as Datasette.

  ```bash
  datasette install datasette-nteract-data-explorer
  ```

  ## Usage

  - Click "View in Data Explorer" to expand the visualization panel
  - Click the icons on the right side to change the visualization type.
  - Use the menus underneath the graphing area to configure your graph (e.g. change which columns to graph, colors to use, etc)
  - Use "advanced settings" mode to override the inferred column types. For example, you may want to treat a number as a "string" to be able to use it as a category.
  - See a [live demo](https://data-explorer.nteract.io/) of the original Nteract data-explorer component used in isolation.

  You can run a minimal demo after the installation step

  ```bash
  datasette -i demo/happy_planet_index.db
  ```

  If you're interested in improving the demo site, you can run a copy of the site the extra metadata/plugins used in the [published demo](https://datasette-nteract-data-explorer.vercel.app).

  ```bash
  make run-demo
  ```

  Thank you for reading this far! If you use the Data Explorer in your own site and would like others to find it, you can [mention it here](https://github.com/hydrosquall/datasette-nteract-data-explorer/discussions/10).

  ## Development

  See [contributing docs](./docs/CONTRIBUTING.md).

  ## Acknowledgements

  - The [Data Explorer](https://github.com/nteract/data-explorer) was designed by Elijah Meeks. I co-maintain this project as part of the [Nteract](https://nteract.io/) open-source team. You can read about the design behind this tool [here](https://blog.nteract.io/designing-the-nteract-data-explorer-f4476d53f897)
  - The data model is based on the [Frictionless Data Spec](https://specs.frictionlessdata.io/).
  - This plugin was bootstrapped by Simon Willison's [Datasette plugin template](https://simonwillison.net/2020/Jun/20/cookiecutter-plugins/)
  - Demo dataset from the [Happy Planet Index](https://happyplanetindex.org/) was cleaned by Doris Lee. This dataset was chosen because of its global appeal, modest size, and variety in column datatypes (numbers, low cardinality and high cardinality strings, booleans).
  - Hosting for the demo site is provided by Vercel.

  [![site hosted by vercel](https://www.datocms-assets.com/31049/1618983297-powered-by-vercel.svg)](https://vercel.com/?utm_source=datasette-visualization-plugin-demos&utm_campaign=oss)

project_urls:
  {"CI": "https://github.com/hydrosquall/datasette-nteract-data-explorer/actions", "Changelog": "https://github.com/hydrosquall/datasette-nteract-data-explorer/releases", "Homepage": "https://github.com/hydrosquall/datasette-nteract-data-explorer", "Issues": "https://github.com/hydrosquall/datasette-nteract-data-explorer/issues"}


name: datasette-packages
summary: Show a list of currently installed Python packages
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-packages
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-packages/
project_url: https://pypi.org/project/datasette-packages/
release_url: https://pypi.org/project/datasette-packages/0.2/
requires_python: >=3.7
version: 0.2
description:
  # datasette-packages

  [![PyPI](https://img.shields.io/pypi/v/datasette-packages.svg)](https://pypi.org/project/datasette-packages/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-packages?include_prereleases&label=changelog)](https://github.com/simonw/datasette-packages/releases)
  [![Tests](https://github.com/simonw/datasette-packages/workflows/Test/badge.svg)](https://github.com/simonw/datasette-packages/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-packages/blob/main/LICENSE)

  Show a list of currently installed Python packages

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-packages

  ## Usage

  Visit `/-/packages` to see a list of installed Python packages.

  Visit `/-/packages.json` to get that back as JSON.

  ## Demo

  The output of this plugin can be seen here:

  - https://latest-with-plugins.datasette.io/-/packages
  - https://latest-with-plugins.datasette.io/-/packages.json

  ## With datasette-graphql

  if you have version 2.1 or higher of the [datasette-graphql](https://datasette.io/plugins/datasette-graphql) plugin installed you can also query the list of packages using this GraphQL query:

  ```graphql
  {
      packages {
          name
          version
      }
  }
  ```
  [Demo of this query](https://latest-with-plugins.datasette.io/graphql?query=%7B%0A%20%20%20%20packages%20%7B%0A%20%20%20%20%20%20%20%20name%0A%20%20%20%20%20%20%20%20version%0A%20%20%20%20%7D%0A%7D).

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-packages
      python3 -mvenv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-packages/actions", "Changelog": "https://github.com/simonw/datasette-packages/releases", "Homepage": "https://github.com/simonw/datasette-packages", "Issues": "https://github.com/simonw/datasette-packages/issues"}

requires_dist:
  ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "datasette-graphql (>=2.1) ; extra == 'test'"]


name: datasette-permissions-sql
summary: Datasette plugin for configuring permission checks using SQL queries
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-permissions-sql
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-permissions-sql/
project_url: https://pypi.org/project/datasette-permissions-sql/
project_urls: {"Homepage": "https://github.com/simonw/datasette-permissions-sql"}
release_url: https://pypi.org/project/datasette-permissions-sql/0.3a0/
version: 0.3a0
description:
  # datasette-permissions-sql

  [![PyPI](https://img.shields.io/pypi/v/datasette-permissions-sql.svg)](https://pypi.org/project/datasette-permissions-sql/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-permissions-sql.svg?style=svg)](https://circleci.com/gh/simonw/datasette-permissions-sql)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-permissions-sql/blob/master/LICENSE)

  Datasette plugin for configuring permission checks using SQL queries

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-permissions-sql

  ## Usage

  First, read up on how Datasette's [authentication and permissions system](https://datasette.readthedocs.io/en/latest/authentication.html) works.

  This plugin lets you define rules containing SQL queries that are executed to see if the currently authenticated actor has permission to perform certain actions.

  Consider a canned query which authenticated users should only be able to execute if a row in the `users` table says that they are a member of staff.

  That `users` table in the `mydatabase.db` database could look like this:

  | id | username | is_staff |
  |--|--------|--------|
  | 1 | cleopaws | 0 |
  | 2 | simon | 1 |

  Authenticated users have an `actor` that looks like this:

  ```json
  {
      "id": 2,
      "username": "simon"
  }
  ```

  To configure the canned query to only be executable by staff users, add the following to your `metadata.json`:

  ```json
  {
      "plugins": {
          "datasette-permissions-sql": [
              {
                  "action": "view-query",
                  "resource": ["mydatabase", "promote_to_staff"],
                  "sql": "SELECT * FROM users WHERE is_staff = 1 AND id = :actor_id"
              }
          ]
      },
      "databases": {
          "mydatabase": {
              "queries": {
                  "promote_to_staff": {
                      "sql": "UPDATE users SET is is_staff=1 WHERE id=:id",
                      "write": true
                  }
              }
          }
      }
  }
  ```

  The `"datasette-permissions-sql"` key is a list of rules. Each of those rules has the following shape:

  ```json
  {
      "action": "name-of-action",
      "resource": ["resource identifier to run this on"],
      "sql": "SQL query to execute",
      "database": "mydatabase"
  }
  ```

  Both `"action"` and `"resource"` are optional. If present, the SQL query will only be executed on permission checks that match the action and, if present, the resource indicators.

  `"database"` is also optional: it specifies the named database that the query should be executed against. If it is not present the first connected database will be used.

  The Datasette documentation includes a [list of built-in permissions](https://datasette.readthedocs.io/en/stable/authentication.html#built-in-permissions) that you might want to use here.

  ### The SQL query

  If the SQL query returns any rows the action will be allowed. If it returns no rows, the plugin hook will return `False` and deny access to that action.

  The SQL query is called with a number of named parameters. You can use any of these as part of the query.

  The list of parameters is as follows:

  * `action` - the action, e.g. `"view-database"`
  * `resource_1` - the first component of the resource, if one was passed
  * `resource_2` - the second component of the resource, if available
  * `actor_*` - a parameter for every key on the actor. Usually `actor_id` is present.

  If any rows are returned, the permission check passes. If no rows are returned the check fails.

  Another example table, this time granting explicit access to individual tables. Consider a table called `table_access` that looks like this:

  | user_id | database | table |
  | - | - | - |
  | 1 | mydb | dogs |
  | 2 | mydb | dogs |
  | 1 | mydb | cats |

  The following SQL query would grant access to the `dogs` ttable in the `mydb.db` database to users 1 and 2 - but would forbid access for user 2 to the `cats` table:

  ```sql
  SELECT
      *
  FROM
      table_access
  WHERE
      user_id = :actor_id
      AND "database" = :resource_1
      AND "table" = :resource_2
  ```
  In a `metadata.yaml` configuration file that would look like this:

  ```yaml
  databases:
    mydb:
      allow_sql: {}
  plugins:
    datasette-permissions-sql:
    - action: view-table
      sql: |-
        SELECT
          *
        FROM
          table_access
        WHERE
          user_id = :actor_id
          AND "database" = :resource_1
          AND "table" = :resource_2
  ```
  We're using `allow_sql: {}` here to disable arbitrary SQL queries. This prevents users from running `select * from cats` directly to work around the permissions limits.

  ### Fallback mode

  The default behaviour of this plugin is to take full control of specified permissions. The SQL query will directly control if the user is allowed or denied access to the permission.

  This means that the default policy for each permission (which in Datasette core is "allow" for `view-database` and friends) will be ignored. It also means that any other `permission_allowed` plugins will not get their turn once this plugin has executed.

  You can change this on a per-rule basis using ``"fallback": true``:

  ```json
  {
      "action": "view-table",
      "resource": ["mydatabase", "mytable"],
      "sql": "select * from admins where user_id = :actor_id",
      "fallback": true
  }
  ```

  When running in fallback mode, a query result returning no rows will cause the plugin hook to return ``None`` - which means "I have no opinion on this permission, fall back to other plugins or the default".

  In this mode you can still return `False` (for "deny access") by returning a single row with a single value of `-1`. For example:

  ```json
  {
      "action": "view-table",
      "resource": ["mydatabase", "mytable"],
      "sql": "select -1 from banned where user_id = :actor_id",
      "fallback": true
  }
  ```

requires_dist:
  ["datasette (>=0.44)", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "httpx ; extra == 'test'", "sqlite-utils (~=2.0) ; extra == 'test'"]


name: datasette-placekey
summary: SQL functions for working with placekeys
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-placekey
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-placekey/
project_url: https://pypi.org/project/datasette-placekey/
release_url: https://pypi.org/project/datasette-placekey/0.1/
requires_dist: ["datasette", "placekey", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.6
version: 0.1
description:
  # datasette-placekey

  [![PyPI](https://img.shields.io/pypi/v/datasette-placekey.svg)](https://pypi.org/project/datasette-placekey/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-placekey?include_prereleases&label=changelog)](https://github.com/simonw/datasette-placekey/releases)
  [![Tests](https://github.com/simonw/datasette-placekey/workflows/Test/badge.svg)](https://github.com/simonw/datasette-placekey/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-placekey/blob/main/LICENSE)

  SQL functions for working with [placekeys](https://www.placekey.io/).

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-placekey

  ## Usage

  The following SQL functions are exposed - [documentation here](https://placekey.github.io/placekey-py/placekey.html#module-placekey.placekey).

  ```sql
  select
    geo_to_placekey(33.0896104,129.7900839),
    placekey_to_geo('@6nh-nhh-kvf'),
    placekey_to_geo_latitude('@6nh-nhh-kvf'),
    placekey_to_geo_longitude('@6nh-nhh-kvf'),
    placekey_to_h3('@6nh-nhh-kvf'),
    h3_to_placekey('8a30d94e4c87fff'),
    placekey_to_geojson('@6nh-nhh-kvf'),
    placekey_to_wkt('@6nh-nhh-kvf'),
    placekey_format_is_valid('@6nh-nhh-kvf');
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-placekey
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-placekey/actions", "Changelog": "https://github.com/simonw/datasette-placekey/releases", "Homepage": "https://github.com/simonw/datasette-placekey", "Issues": "https://github.com/simonw/datasette-placekey/issues"}


name: datasette-pretty-json
summary: Datasette plugin that pretty-prints any column values that are valid JSON objects or arrays
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-pretty-json
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-pretty-json/
project_url: https://pypi.org/project/datasette-pretty-json/
project_urls: {"Homepage": "https://github.com/simonw/datasette-pretty-json"}
release_url: https://pypi.org/project/datasette-pretty-json/0.2.2/
requires_dist: ["datasette", "pytest ; extra == 'test'"]
version: 0.2.2
description:
  # datasette-pretty-json

  [![PyPI](https://img.shields.io/pypi/v/datasette-pretty-json.svg)](https://pypi.org/project/datasette-pretty-json/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-pretty-json?include_prereleases&label=changelog)](https://github.com/simonw/datasette-pretty-json/releases)
  [![Tests](https://github.com/simonw/datasette-pretty-json/workflows/Test/badge.svg)](https://github.com/simonw/datasette-pretty-json/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-pretty-json/blob/main/LICENSE)


  [Datasette](https://github.com/simonw/datasette) plugin that pretty-prints any column values that are valid JSON objects or arrays.

  You may also be interested in [datasette-json-html](https://github.com/simonw/datasette-json-html).


name: datasette-pretty-traces
summary: Prettier formatting for ?_trace=1 traces
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-pretty-traces
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-pretty-traces/
project_url: https://pypi.org/project/datasette-pretty-traces/
release_url: https://pypi.org/project/datasette-pretty-traces/0.4/
requires_dist: ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.6
version: 0.4
description:
  # datasette-pretty-traces

  [![PyPI](https://img.shields.io/pypi/v/datasette-pretty-traces.svg)](https://pypi.org/project/datasette-pretty-traces/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-pretty-traces?include_prereleases&label=changelog)](https://github.com/simonw/datasette-pretty-traces/releases)
  [![Tests](https://github.com/simonw/datasette-pretty-traces/workflows/Test/badge.svg)](https://github.com/simonw/datasette-pretty-traces/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-pretty-traces/blob/main/LICENSE)

  Prettier formatting for `?_trace=1` traces

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-pretty-traces

  ## Usage

  Once installed, run Datasette using `--setting trace_debug 1`:

      datasette fixtures.db --setting trace_debug 1

  Then navigate to any page and add `?_trace=` to the URL:

      http://localhost:8001/?_trace=1

  The plugin will scroll you down the page to the visualized trace information.

  ## Demo

  You can try out the demo here:

  - [/?_trace=1](https://latest-with-plugins.datasette.io/?_trace=1) tracing the homepage
  - [/github/commits?_trace=1](https://latest-with-plugins.datasette.io/github/commits?_trace=1) tracing a table page

  ## Screenshot

  ![Screenshot showing the visualization produced by the plugin](https://user-images.githubusercontent.com/9599/145883732-a53accdd-5feb-4629-94cd-f73407c7943d.png)

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-pretty-traces
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-pretty-traces/actions", "Changelog": "https://github.com/simonw/datasette-pretty-traces/releases", "Homepage": "https://github.com/simonw/datasette-pretty-traces", "Issues": "https://github.com/simonw/datasette-pretty-traces/issues"}


name: datasette-psutil
summary: Datasette plugin adding a /-/psutil debugging endpoint
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-psutil
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-psutil/
project_url: https://pypi.org/project/datasette-psutil/
project_urls: {"Homepage": "https://github.com/simonw/datasette-psutil"}
release_url: https://pypi.org/project/datasette-psutil/0.2/
version: 0.2
description:
  # datasette-psutil

  [![PyPI](https://img.shields.io/pypi/v/datasette-psutil.svg)](https://pypi.org/project/datasette-psutil/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-psutil.svg?style=svg)](https://circleci.com/gh/simonw/datasette-psutil)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-psutil/blob/master/LICENSE)

  Datasette plugin adding a `/-/psutil` debugging endpoint

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-psutil

  ## Usage

  Visit `/-/psutil` on your Datasette instance to see various information provided by [psutil](https://psutil.readthedocs.io/).

  ## Example

  You can visit a live demo here: https://datasette-psutil-demo-j7hipcg4aq-uw.a.run.app/-/psutil

  Sample output:
  ```
  process.open_files()
    popenfile(path='/tmp/fixtures.db', fd=6)

  process.connections()
    pconn(fd=5, family=<AddressFamily.AF_INET: 2>, type=<SocketKind.SOCK_STREAM: 1>, laddr=addr(ip='169.254.8.130', port=8080), raddr=addr(ip='169.254.8.129', port=52621), status='ESTABLISHED')
    pconn(fd=3, family=<AddressFamily.AF_INET: 2>, type=<SocketKind.SOCK_STREAM: 1>, laddr=addr(ip='0.0.0.0', port=8080), raddr=(), status='LISTEN')

  process.memory_info()
    pmem(rss=56827904, vms=242540544, shared=0, text=0, lib=0, data=0, dirty=0)

  process.cmdline()
    '/usr/local/bin/python'
    '/usr/local/bin/datasette'
    'serve'
    '--host'
    '0.0.0.0'
    '-i'
    'fixtures.db'
    '--cors'
    '--inspect-file'
    'inspect-data.json'
    '--port'
    '8080'

  process.parents()
    psutil.Process(pid=1, name='sh', started='23:19:29')

  process.threads()
    pthread(id=2, user_time=7.27, system_time=3.99)
    pthread(id=4, user_time=0.01, system_time=0.0)
    pthread(id=5, user_time=0.0, system_time=0.0)
    pthread(id=6, user_time=0.0, system_time=0.0)
    pthread(id=7, user_time=0.0, system_time=0.0)
    pthread(id=8, user_time=0.0, system_time=0.0)

  psutil.getloadavg()
    (0.0, 0.0, 0.0)

  psutil.cpu_times(True)
    scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)
    scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)

  psutil.virtual_memory()
    svmem(total=2147483648, available=2093080576, percent=2.5, used=31113216, free=2093080576, active=42860544, inactive=11513856, buffers=0, cached=23289856, shared=262144, slab=0)

  list(psutil.process_iter())
    psutil.Process(pid=1, name='sh', started='23:19:29')
    psutil.Process(pid=2, name='datasette', started='23:19:29')
  ```

requires_dist:
  ["datasette (>=0.44)", "psutil", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "httpx ; extra == 'test'"]


name: datasette-public
summary: Make specific Datasette tables visible to the public
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-public
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-public/
project_url: https://pypi.org/project/datasette-public/
release_url: https://pypi.org/project/datasette-public/0.2/
requires_dist: ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.7
version: 0.2
description:
  # datasette-public

  [![PyPI](https://img.shields.io/pypi/v/datasette-public.svg)](https://pypi.org/project/datasette-public/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-public?include_prereleases&label=changelog)](https://github.com/simonw/datasette-public/releases)
  [![Tests](https://github.com/simonw/datasette-public/workflows/Test/badge.svg)](https://github.com/simonw/datasette-public/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-public/blob/main/LICENSE)

  Make specific Datasette tables visible to the public

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-public

  ## Usage

  Any tables listed in the `_public_tables` table will be visible to the public, even if the rest of the Datasette instance does not allow anonymous access.

  The root user (and any user with the new `public-tables` permission) will get a new option in the table action menu allowing them to toggle a table between public and private.

  Installing this plugin also causes `allow-sql` permission checks to fall back to checking if the user has access to the entire database. This is to avoid users with access to a single public table being able to access data from other tables using the `?_where=` query string parameter.

  ## Configuration

  This plugin creates a new table in one of your databases called `_public_tables`.

  This table defaults to being created in the first database passed to Datasette.

  To create it in a different named database, use this plugin configuration:

  ```json
  {
    "plugins": {
      "datasette-public": {
        "database": "database_to_create_table_in"
      }
    }
  }
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-public
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-public/actions", "Changelog": "https://github.com/simonw/datasette-public/releases", "Homepage": "https://github.com/simonw/datasette-public", "Issues": "https://github.com/simonw/datasette-public/issues"}


name: datasette-publish-fly
summary: Datasette plugin for publishing data using Fly
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-publish-fly
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-publish-fly/
project_url: https://pypi.org/project/datasette-publish-fly/
release_url: https://pypi.org/project/datasette-publish-fly/1.2/
version: 1.2
description:
  # datasette-publish-fly

  [![PyPI](https://img.shields.io/pypi/v/datasette-publish-fly.svg)](https://pypi.org/project/datasette-publish-fly/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-publish-fly?include_prereleases&label=changelog)](https://github.com/simonw/datasette-publish-fly/releases)
  [![Tests](https://github.com/simonw/datasette-publish-fly/workflows/Test/badge.svg)](https://github.com/simonw/datasette-publish-fly/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-publish-fly/blob/main/LICENSE)

  [Datasette](https://datasette.io/) plugin for deploying Datasette instances to [Fly.io](https://fly.io/).

  Project background: [Using SQLite and Datasette with Fly Volumes](https://simonwillison.net/2022/Feb/15/fly-volumes/)

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-publish-fly

  ## Deploying read-only data

  First, install the `flyctl` command-line tool by [following their instructions](https://fly.io/docs/getting-started/installing-flyctl/).

  Run `flyctl auth signup` to create an account there, or `flyctl auth login` if you already have one.

  You can now use `datasette publish fly` to publish one or more SQLite database files:

      datasette publish fly my-database.db --app="my-data-app"

  The argument you pass to `--app` will be used for the URL of your application: `my-data-app.fly.dev`.

  To update an application, run the publish command passing the same application name to the `--app` option.

  Fly have [a free tier](https://fly.io/docs/about/pricing/#free-allowances), beyond which they will charge you monthly for each application you have live.  Details of their pricing can be [found on their site](https://fly.io/docs/pricing/).

  Your application will be deployed at `https://your-app-name.fly.io/` - be aware that it may take several minutes to start working the first time you deploy it.

  ## Using Fly volumes for writable databases

  Fly [Volumes](https://fly.io/docs/reference/volumes/) provide persistant disk storage for Fly applications. Volumes can be 1GB or more in size and the Fly free tier includes 3GB of volume space.

  Datasette plugins such as [datasette-uploads-csvs](https://datasette.io/plugins/datasette-upload-csvs) and [datasette-tiddlywiki](https://datasette.io/plugins/datasette-tiddlywiki) can be deployed to Fly and store their mutable data in a volume.

  > :warning: **You should only run a single instance of your application** if your database accepts writes. Fly has excellent support for running multiple instances in different geographical regions, but `datasette-publish-fly` with volumes is not yet compatible with that model. You should probably [use Fly PostgreSQL instead](https://fly.io/blog/globally-distributed-postgres/).

  Here's how to deploy `datasette-tiddlywiki` with authentication provided by `datasette-auth-passwords`.

  First, you'll need to create a root password hash to use to sign into the instance.

  You can do that by installing the plugin and running the `datasette hash-password` command, or by using [this hosted tool](https://datasette-auth-passwords-demo.datasette.io/-/password-tool).

  The hash should look like `pbkdf2_sha256$...` - you'll need this for the next step.

  In this example we're also deploying a read-only database called `content.db`.

  Pick a name for your new application, then run the following:

      datasette publish fly \
      content.db \
      --app your-application-name \
      --create-volume 1 \
      --create-db tiddlywiki \
      --install datasette-auth-passwords \
      --install datasette-tiddlywiki \
      --plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$...'

  This will create the new application, deploy the `content.db` read-only database, create a 1GB volume for that application, create a new database in that volume called `tiddlywiki.db`, then install the two plugins and configure the password you specified.

  ### Updating applications that use a volume

  Once you have deployed an application using a volume, you can update that application without needing the `--create-volume` or `--create-db` options. To add the [datasette-graphq](https://datasette.io/plugins/datasette-graphql) plugin to your deployed application you would run the following:

      datasette publish fly \
      content.db \
      --app your-application-name \
      --install datasette-auth-passwords \
      --install datasette-tiddlywiki \
      --install datasette-graphql \
      --plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$...' \

  Since the application name is the same you don't need the `--create-volume` or `--create-db` options - these are persisted automatically between deploys.

  You do need to specify the full list of plugins that you want to have installed, and any plugin secrets.

  You also need to include any read-only database files that are part of the instance - `content.db` in this example - otherwise the new deployment will not include them.

  ### Advanced volume usage

  `datasette publish fly` will add a volume called `datasette` to your Fly application. You can customize the name using the `--volume name custom_name` option.

  Fly can be used to scale applications to run multiple instances in multiple regions around the world. This works well with read-only Datasette but is not currently recommended using Datasette with volumes, since each Fly replica would need its own volume and data stored in one instance would not be visible in others.

  If you want to use multiple instances with volumes you will need to switch to using the `flyctl` command directly. The `--generate-dir` option, described below, can help with this.

  ## Generating without deploying

  Use the `--generate-dir` option to generate a directory that can be deployed to Fly rather than deploying directly:

      datasette publish fly my-database.db \
        --app="my-generated-app" \
        --generate-dir /tmp/deploy-this

  You can then manually deploy your generated application using the following:

      cd /tmp/deploy-this
      flyctl apps create my-generated-app
      flyctl deploy

  ## datasette publish fly --help

  <!-- [[[cog
  import cog
  from datasette import cli
  from click.testing import CliRunner
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["publish", "fly", "--help"])
  help = result.output.replace("Usage: cli", "Usage: datasette")
  cog.out(
      "```\n{}```".format(help)
  )
  ]]] -->
  ```
  Usage: datasette publish fly [OPTIONS] [FILES]...

    Deploy an application to Fly that runs Datasette against the provided database
    files.

    Usage example:

        datasette publish fly my-database.db --app="my-data-app"

    Full documentation: https://datasette.io/plugins/datasette-publish-fly

  Options:
    -m, --metadata FILENAME         Path to JSON/YAML file containing metadata to
                                    publish
    --extra-options TEXT            Extra options to pass to datasette serve
    --branch TEXT                   Install datasette from a GitHub branch e.g.
                                    main
    --template-dir DIRECTORY        Path to directory containing custom templates
    --plugins-dir DIRECTORY         Path to directory containing custom plugins
    --static MOUNT:DIRECTORY        Serve static files from this directory at
                                    /MOUNT/...
    --install TEXT                  Additional packages (e.g. plugins) to install
    --plugin-secret <TEXT TEXT TEXT>...
                                    Secrets to pass to plugins, e.g. --plugin-
                                    secret datasette-auth-github client_id xxx
    --version-note TEXT             Additional note to show on /-/versions
    --secret TEXT                   Secret used for signing secure values, such as
                                    signed cookies
    --title TEXT                    Title for metadata
    --license TEXT                  License label for metadata
    --license_url TEXT              License URL for metadata
    --source TEXT                   Source label for metadata
    --source_url TEXT               Source URL for metadata
    --about TEXT                    About label for metadata
    --about_url TEXT                About URL for metadata
    --spatialite                    Enable SpatialLite extension
    --region TEXT                   Fly region to deploy to, e.g sjc - see
                                    https://fly.io/docs/reference/regions/
    --create-volume INTEGER RANGE   Create and attach volume of this size in GB
                                    [x>=1]
    --create-db TEXT                Names of read-write database files to create
    --volume-name TEXT              Volume name to use
    -a, --app TEXT                  Name of Fly app to deploy  [required]
    -o, --org TEXT                  Name of Fly org to deploy to
    --generate-dir DIRECTORY        Output generated application files and stop
                                    without deploying
    --show-files                    Output the generated Dockerfile, metadata.json
                                    and fly.toml
    --help                          Show this message and exit.
  ```
  <!-- [[[end]]] -->

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd datasette-publish-fly
      python -m venv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

  ### Integration tests

  The tests in `tests/test_integration.py` make actual calls to Fly to deploy a test application.

  These tests are skipped by default. If you have `flyctl` installed and configured, you can run the integration tests like this:

      pytest --integration -s

  The `-s` option here ensures that output from the deploys will be visible to you - otherwise it can look like the tests have hung.

  The tests will create applications on Fly that start with the prefix `publish-fly-temp-` and then delete them at the end of the run.

project_urls:
  {"CI": "https://github.com/simonw/datasette-publish-fly/actions", "Changelog": "https://github.com/simonw/datasette-publish-fly/releases", "Homepage": "https://github.com/simonw/datasette-publish-fly", "Issues": "https://github.com/simonw/datasette-publish-fly/issues"}

requires_dist:
  ["datasette (>=0.60.2)", "pytest ; extra == 'test'", "pytest-mock ; extra == 'test'", "cogapp ; extra == 'test'"]


name: datasette-publish-vercel
summary: Datasette plugin for publishing data using Vercel
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-publish-vercel
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-publish-vercel/
project_url: https://pypi.org/project/datasette-publish-vercel/
release_url: https://pypi.org/project/datasette-publish-vercel/0.14.2/
requires_dist: ["datasette (>=0.59)", "pytest ; extra == 'test'"]
version: 0.14.2
description:
  # datasette-publish-vercel

  [![PyPI](https://img.shields.io/pypi/v/datasette-publish-vercel.svg)](https://pypi.org/project/datasette-publish-vercel/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-publish-vercel?include_prereleases&label=changelog)](https://github.com/simonw/datasette-publish-vercel/releases)
  [![Tests](https://github.com/simonw/datasette-publish-vercel/workflows/Test/badge.svg)](https://github.com/simonw/datasette-publish-vercel/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-publish-vercel/blob/main/LICENSE)

  Datasette plugin for publishing data using [Vercel](https://vercel.com/).

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-publish-vercel

  ## Usage

  First, install the Vercel CLI tool by [following their instructions](https://vercel.com/download).

  Run `vercel login` to login to (or create) an account.

  Now you can use `datasette publish vercel` to publish your data:

      datasette publish vercel my-database.db --project=my-database

  The `--project` argument is required - it specifies the project name that should be used for your deployment. This will be used as part of the deployment's URL.

  ### Other options

  * `--no-prod` deploys to the project without updating the "production" URL alias to point to that new deployment. Without that option all deploys go directly to production.
  * `--debug` enables the Vercel CLI debug output.
  * `--token` allows you to pass a Now authentication token, rather than needing to first run `now login` to configure the tool. Tokens can be created in the Vercel web dashboard under Account Settings -> Tokens.
  * `--public` runs `vercel --public` to publish the application source code at `/_src` e.g. https://datasette-public.now.sh/_src and make recent logs visible at `/_logs` e.g. https://datasette-public.now.sh/_logs
  * `--generate-dir` - by default this tool generates a new Vercel app in a temporary directory, deploys it and then deletes the directory. Use `--generate-dir=my-app` to output the generated application files to a new directory of your choice instead. You can then deploy it by running `vercel` in that directory.
  * `--setting default_page_size 10` - use this to set Datasette settings, as described in [the documentation](https://docs.datasette.io/en/stable/settings.html). This is a replacement for the unsupported `--extra-options` option.

  ### Full help

  **Warning:** Some of these options are not yet implemented by this plugin. In particular, the following do not yet work:

  * `--extra-options` - use `--setting` described above instead.
  * `--plugin-secret`
  * `--version-note`

  ```
  $ datasette publish vercel --help

  Usage: datasette publish vercel [OPTIONS] [FILES]...

    Publish to https://vercel.com/

  Options:
    -m, --metadata FILENAME         Path to JSON/YAML file containing metadata to publish
    --extra-options TEXT            Extra options to pass to datasette serve
    --branch TEXT                   Install datasette from a GitHub branch e.g. main
    --template-dir DIRECTORY        Path to directory containing custom templates
    --plugins-dir DIRECTORY         Path to directory containing custom plugins
    --static MOUNT:DIRECTORY        Serve static files from this directory at /MOUNT/...
    --install TEXT                  Additional packages (e.g. plugins) to install
    --plugin-secret <TEXT TEXT TEXT>...
                                    Secrets to pass to plugins, e.g. --plugin-secret
                                    datasette-auth-github client_id xxx
    --version-note TEXT             Additional note to show on /-/versions
    --secret TEXT                   Secret used for signing secure values, such as signed
                                    cookies
    --title TEXT                    Title for metadata
    --license TEXT                  License label for metadata
    --license_url TEXT              License URL for metadata
    --source TEXT                   Source label for metadata
    --source_url TEXT               Source URL for metadata
    --about TEXT                    About label for metadata
    --about_url TEXT                About URL for metadata
    --token TEXT                    Auth token to use for deploy
    --project PROJECT               Vercel project name to use  [required]
    --scope TEXT                    Optional Vercel scope (e.g. a team name)
    --no-prod                       Don't deploy directly to production
    --debug                         Enable Vercel CLI debug output
    --public                        Publish source with Vercel CLI --public
    --generate-dir DIRECTORY        Output generated application files and stop without
                                    deploying
    --generate-vercel-json          Output generated vercel.json file and stop without
                                    deploying
    --vercel-json FILENAME          Custom vercel.json file to use instead of generating
                                    one
    --setting SETTING...            Setting, see docs.datasette.io/en/stable/settings.html
    --crossdb                       Enable cross-database SQL queries
    --help                          Show this message and exit.
  ```
  ## Using a custom `vercel.json` file

  If you want to add additional redirects or similar to your Vercel configuration you may want to provide a custom `vercel.json` file.

  To do this, first generate a configuration file (without running a deploy) using the `--generate-vercel-json` option:

      datasette publish vercel my-database.db \
        --project=my-database \
        --generate-vercel-json > vercel.json

  You can now edit the `vercel.json` file that this creates to add your custom options.

  Then run the deploy using:

      datasette publish vercel my-database.db \
        --project=my-database \
        --vercel-json=vercel.json

  ## Setting a `DATASETTE_SECRET`

  Datasette uses [a secret string](https://docs.datasette.io/en/stable/settings.html#configuring-the-secret) for purposes such as signing authentication cookies. This secret is reset when the server restarts, which will sign out any users who are authenticated using a signed cookie.

  You can avoid this by generating a `DATASETTE_SECRET` secret string and setting that as a [Vercel environment variable](https://vercel.com/docs/concepts/projects/environment-variables). If you do this the secret will stay consistent and your users will not be signed out.

  ## Using this with GitHub Actions

  This plugin can be used together with [GitHub Actions](https://github.com/features/actions) to deploy Datasette instances automatically on new pushes to a repo, or on a schedule.

  The GitHub Actions runners already have the Vercel deployment tool installed. You'll need to create an API token for your account at [vercel.com/account/tokens](https://vercel.com/account/tokens), and store that as a secret in your GitHub repository called `VERCEL_TOKEN`.

  Make sure your workflow has installed `datasette` and `datasette-publish-vercel` using `pip`, then add the following step to your GitHub Actions workflow:
  ```
      - name: Deploy Datasette using Vercel
        env:
          VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
        run: |-
          datasette publish vercel mydb.db \
            --token $VERCEL_TOKEN \
            --project my-vercel-project
  ```
  You can see a full example of a workflow that uses Vercel in this way [in the simonw/til repository](https://github.com/simonw/til/blob/12b3f0d3679320cbeafa5df164bbc08ba703625d/.github/workflows/build.yml).

project_urls:
  {"CI": "https://github.com/simonw/datasette-publish-vercel/actions", "Changelog": "https://github.com/simonw/datasette-publish-vercel/releases", "Homepage": "https://github.com/simonw/datasette-publish-vercel", "Issues": "https://github.com/simonw/datasette-publish-vercel/issues"}


name: datasette-pyinstrument
summary: Use pyinstrument to analyze Datasette page performance
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-pyinstrument
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-pyinstrument/
project_url: https://pypi.org/project/datasette-pyinstrument/
release_url: https://pypi.org/project/datasette-pyinstrument/0.1/
requires_dist: ["datasette", "pyinstrument", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.7
version: 0.1
description:
  # datasette-pyinstrument

  [![PyPI](https://img.shields.io/pypi/v/datasette-pyinstrument.svg)](https://pypi.org/project/datasette-pyinstrument/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-pyinstrument?include_prereleases&label=changelog)](https://github.com/simonw/datasette-pyinstrument/releases)
  [![Tests](https://github.com/simonw/datasette-pyinstrument/workflows/Test/badge.svg)](https://github.com/simonw/datasette-pyinstrument/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-pyinstrument/blob/main/LICENSE)

  Use pyinstrument to analyze Datasette page performance

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-pyinstrument

  ## Usage

  Once installed, adding `?_pyinstrument=1` to any URL within Datasette will replace the output of that page with the pyinstrument profiler results for it.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-pyinstrument
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-pyinstrument/actions", "Changelog": "https://github.com/simonw/datasette-pyinstrument/releases", "Homepage": "https://github.com/simonw/datasette-pyinstrument", "Issues": "https://github.com/simonw/datasette-pyinstrument/issues"}


name: datasette-query-files
summary: Write Datasette canned queries as plain SQL files
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Chris Amico
description_content_type: text/markdown
home_page: https://github.com/eyeseast/datasette-query-files
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-query-files/
project_url: https://pypi.org/project/datasette-query-files/
release_url: https://pypi.org/project/datasette-query-files/0.1.1/
requires_dist: ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.7
version: 0.1.1
description:
  # datasette-query-files

  [![PyPI](https://img.shields.io/pypi/v/datasette-query-files.svg)](https://pypi.org/project/datasette-query-files/)
  [![Changelog](https://img.shields.io/github/v/release/eyeseast/datasette-query-files?include_prereleases&label=changelog)](https://github.com/eyeseast/datasette-query-files/releases)
  [![Tests](https://github.com/eyeseast/datasette-query-files/workflows/Test/badge.svg)](https://github.com/eyeseast/datasette-query-files/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/eyeseast/datasette-query-files/blob/main/LICENSE)

  Write Datasette canned queries as plain SQL files.

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-query-files

  Or using `pip` or `pipenv`:

      pip install datasette-query-files
      pipenv install datasette-query-files

  ## Usage

  This plugin will look for [canned queries](https://docs.datasette.io/en/stable/sql_queries.html#canned-queries) in the filesystem, in addition any defined in metadata.

  Let's say you're working in a directory called `project-directory`, with a database file called `my-project.db`. Start by creating a `queries` directory with a `my-project` directory inside it. Any SQL file inside that `my-project` folder will become a canned query that can be run on the `my-project` database. If you have a `query-name.sql` file and a `query-name.json` (or `query-name.yml`) file in the same directory, the JSON file will be used as query metadata.

  ```
  project-directory/
    my-project.db
    queries/
      my-project/
        query-name.sql # a query
        query-name.yml # query metadata
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-query-files
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/eyeseast/datasette-query-files/actions", "Changelog": "https://github.com/eyeseast/datasette-query-files/releases", "Homepage": "https://github.com/eyeseast/datasette-query-files", "Issues": "https://github.com/eyeseast/datasette-query-files/issues"}


name: datasette-query-history
summary: Datasette plugin that keeps a list of the queries you've run and lets you rerun them.
classifiers: []
author: Bret Walker
description_content_type: text/markdown
home_page: https://github.com/bretwalker/datasette-query-history
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-query-history/
project_url: https://pypi.org/project/datasette-query-history/
release_url: https://pypi.org/project/datasette-query-history/0.2.3/
requires_dist: ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.6
version: 0.2.3
description:
  # datasette-query-history

  [![PyPI](https://img.shields.io/pypi/v/datasette-query-history.svg)](https://pypi.org/project/datasette-query-history/)
  [![Changelog](https://img.shields.io/github/v/release/bretwalker/datasette-query-history?include_prereleases&label=changelog)](https://github.com/bretwalker/datasette-query-history/releases)
  [![Tests](https://github.com/bretwalker/datasette-query-history/workflows/Test/badge.svg)](https://github.com/bretwalker/datasette-query-history/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/bretwalker/datasette-query-history/blob/main/LICENSE)

  Datasette plugin that keeps a list of the queries you've run and lets you rerun them.

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-query-history

  ## Usage

  Click the `Query History` button on the SQL editor page to see previous queries.  
  Click the ⬆︎ button to replace the current query with a previous query.  
  Click the `Clear Query History` button to clear the list previous queries.

  <img src="https://raw.githubusercontent.com/bretwalker/datasette-query-history/main/docs/datasette-query-history-example1.png" width="350px" alt="Screenshot of plugin">

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-query-history
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/bretwalker/datasette-query-history/actions", "Changelog": "https://github.com/bretwalker/datasette-query-history/releases", "Homepage": "https://github.com/bretwalker/datasette-query-history", "Issues": "https://github.com/bretwalker/datasette-query-history/issues"}


name: datasette-query-links
summary: Turn SELECT queries returned by a query into links to execute them
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-query-links
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-query-links/
project_url: https://pypi.org/project/datasette-query-links/
release_url: https://pypi.org/project/datasette-query-links/0.1.2/
requires_python: >=3.6
version: 0.1.2
description:
  # datasette-query-links

  [![PyPI](https://img.shields.io/pypi/v/datasette-query-links.svg)](https://pypi.org/project/datasette-query-links/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-query-links?include_prereleases&label=changelog)](https://github.com/simonw/datasette-query-links/releases)
  [![Tests](https://github.com/simonw/datasette-query-links/workflows/Test/badge.svg)](https://github.com/simonw/datasette-query-links/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-query-links/blob/main/LICENSE)

  Turn SELECT queries returned by a query into links to execute them

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-query-links

  ## Usage

  This is an experimental plugin, requiring Datasette  0.59a1 or higher.

  Any SQL query that returns a value that itself looks like a valid SQL query will be converted into a link to execute that SQL query when it is displayed in the Datasette interface.

  These links will only show for valid SQL query - if a SQL query would return an error it will not be turned into a link.

  ## Demo

  * [Here's an example query](https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++%27select+*+from+%5Bfacetable%5D%27+as+query%0D%0Aunion%0D%0Aselect%0D%0A++%27select+sqlite_version()%27%0D%0Aunion%0D%0Aselect%0D%0A++%27select+this+is+invalid+SQL+so+will+not+be+linked%27) showing the plugin in action.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-query-links
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-query-links/actions", "Changelog": "https://github.com/simonw/datasette-query-links/releases", "Homepage": "https://github.com/simonw/datasette-query-links", "Issues": "https://github.com/simonw/datasette-query-links/issues"}

requires_dist:
  ["datasette (>=0.59a1)", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "beautifulsoup4 ; extra == 'test'"]


name: datasette-redirect-forbidden
summary: Redirect forbidden requests to a login page
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-redirect-forbidden
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-redirect-forbidden/
project_url: https://pypi.org/project/datasette-redirect-forbidden/
release_url: https://pypi.org/project/datasette-redirect-forbidden/0.1/
requires_dist: ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.6
version: 0.1
description:
  # datasette-redirect-forbidden

  [![PyPI](https://img.shields.io/pypi/v/datasette-redirect-forbidden.svg)](https://pypi.org/project/datasette-redirect-forbidden/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-redirect-forbidden?include_prereleases&label=changelog)](https://github.com/simonw/datasette-redirect-forbidden/releases)
  [![Tests](https://github.com/simonw/datasette-redirect-forbidden/workflows/Test/badge.svg)](https://github.com/simonw/datasette-redirect-forbidden/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-redirect-forbidden/blob/main/LICENSE)

  Redirect forbidden requests to a login page

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-redirect-forbidden

  ## Usage

  Add the following to your `metadata.yml` (or `metadata.json`) file to configure the plugin:

  ```yaml
  plugins:
    datasette-redirect-forbidden:
      redirect_to: /-/login
  ```
  Any 403 forbidden pages will redirect to the specified page.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-redirect-forbidden
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-redirect-forbidden/actions", "Changelog": "https://github.com/simonw/datasette-redirect-forbidden/releases", "Homepage": "https://github.com/simonw/datasette-redirect-forbidden", "Issues": "https://github.com/simonw/datasette-redirect-forbidden/issues"}


name: datasette-redirect-to-https
summary: Datasette plugin that redirects all non-https requests to https
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-redirect-to-https
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-redirect-to-https/
project_url: https://pypi.org/project/datasette-redirect-to-https/
release_url: https://pypi.org/project/datasette-redirect-to-https/0.2/
requires_python: >=3.6
version: 0.2
description:
  # datasette-redirect-to-https

  [![PyPI](https://img.shields.io/pypi/v/datasette-redirect-to-https.svg)](https://pypi.org/project/datasette-redirect-to-https/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-redirect-to-https?include_prereleases&label=changelog)](https://github.com/simonw/datasette-redirect-to-https/releases)
  [![Tests](https://github.com/simonw/datasette-redirect-to-https/workflows/Test/badge.svg)](https://github.com/simonw/datasette-redirect-to-https/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-redirect-to-https/blob/main/LICENSE)

  Datasette plugin that redirects all non-https requests to https

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-redirect-to-https

  ## Usage

  Once installed, incoming GET requests to the `http://` protocol will be 301 redirected to the `https://` equivalent page.

  HTTP verbs other than GET will get a 405 Method Not Allowed HTTP error.

  ## Configuration

  Some hosting providers handle HTTPS for you, passing requests back to your application server over HTTP.

  For this plugin to work correctly, you need to detect that they the original incoming request came in over HTTP.

  Hosting providers like this often set an additional HTTP header such as `x-forwarded-proto: http` to let you know the original protocol.

  You can configure `datasette-redirect-to-https` to respect this header using the following plugin configuration in `metadata.json`:

  ```json
  {
    "plugins": {
      "datasette-redirect-to-https": {
        "if_headers": {
          "x-forwarded-proto": "http"
        }
      }
    }
  }
  ```
  The above example will redirect to `https://` if the incoming request has a `x-forwarded-proto: http` request header.

  If multiple `if_headers` are listed, the redirect will occur if any of them match.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-redirect-to-https
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-redirect-to-https/actions", "Changelog": "https://github.com/simonw/datasette-redirect-to-https/releases", "Homepage": "https://github.com/simonw/datasette-redirect-to-https", "Issues": "https://github.com/simonw/datasette-redirect-to-https/issues"}

requires_dist:
  ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "asgi-lifespan ; extra == 'test'"]


name: datasette-remote-metadata
summary: Periodically refresh Datasette metadata from a remote URL
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-remote-metadata
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-remote-metadata/
project_url: https://pypi.org/project/datasette-remote-metadata/
release_url: https://pypi.org/project/datasette-remote-metadata/0.1/
requires_python: >=3.6
version: 0.1
description:
  # datasette-remote-metadata

  [![PyPI](https://img.shields.io/pypi/v/datasette-remote-metadata.svg)](https://pypi.org/project/datasette-remote-metadata/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-remote-metadata?include_prereleases&label=changelog)](https://github.com/simonw/datasette-remote-metadata/releases)
  [![Tests](https://github.com/simonw/datasette-remote-metadata/workflows/Test/badge.svg)](https://github.com/simonw/datasette-remote-metadata/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-remote-metadata/blob/main/LICENSE)

  Periodically refresh Datasette metadata from a remote URL

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-remote-metadata

  ## Usage

  Add the following to your `metadata.json`:

  ```json
  {
      "plugins": {
          "datasette-remote-metadata": {
              "url": "https://example.com/remote-metadata.yml"
          }
      }
  }
  ```
  The plugin will fetch the specified metadata  from that URL at startup and combine it with any existing metadata. You can use a URL to either a JSON file or a YAML file.

  It will periodically refresh that metadata - by default every 30 seconds, unless you specify an alternative `"ttl"` value in the plugin configuration.

  ## Configuration

  Available configuration options are as follows:

  - `"url"` - the URL to retrieve remote metadata from. Can link to a JSON or a YAML file.
  - `"ttl"` - integer value in secords: how frequently should the script check for fresh metadata. Defaults to 30 seconds.
  - `"headers"` - a dictionary of additional request headers to send.
  - `"cachebust"` - if true, a random `?0.29508` value will be added to the query string of the remote metadata to bust any intermediary caches.

  This example `metadata.json` configuration refreshes every 10 seconds, uses cache busting and sends an `Authorization: Bearer xyz` header with the request:

  ```json
  {
      "plugins": {
          "datasette-remote-metadata": {
              "url": "https://example.com/remote-metadata.yml",
              "ttl": 10,
              "cachebust": true,
              "headers": {
                  "Authorization": "Bearer xyz"
              }
          }
      }
  }
  ```
  This example if you are using `metadata.yaml` for configuration:
  ```yaml
  plugins:
    datasette-remote-metadata:
      url: https://example.com/remote-metadata.yml
      ttl: 10
      cachebust: true
      headers:
        Authorization: Bearer xyz
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-remote-metadata
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-remote-metadata/actions", "Changelog": "https://github.com/simonw/datasette-remote-metadata/releases", "Homepage": "https://github.com/simonw/datasette-remote-metadata", "Issues": "https://github.com/simonw/datasette-remote-metadata/issues"}

requires_dist:
  ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "pytest-httpx ; extra == 'test'"]


name: datasette-render-binary
summary: Datasette plugin for rendering binary data
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-render-binary
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-render-binary/
project_url: https://pypi.org/project/datasette-render-binary/
project_urls: {"Homepage": "https://github.com/simonw/datasette-render-binary"}
release_url: https://pypi.org/project/datasette-render-binary/0.3/
requires_dist: ["datasette", "filetype", "pytest ; extra == 'test'"]
version: 0.3
description:
  # datasette-render-binary

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-binary.svg)](https://pypi.org/project/datasette-render-binary/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-render-binary.svg?style=svg)](https://circleci.com/gh/simonw/datasette-render-binary)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-binary/blob/master/LICENSE)

  Datasette plugin for rendering binary data.

  Install this plugin in the same environment as Datasette to enable this new functionality:

      pip install datasette-render-binary

  Binary data in cells will now be rendered as a mixture of characters and octets.

  ![Screenshot](https://raw.githubusercontent.com/simonw/datasette-render-binary/master/example.png)


name: datasette-render-html
summary: Datasette plugin that renders specified cells as HTML
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-render-html
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-render-html/
project_url: https://pypi.org/project/datasette-render-html/
project_urls: {"Homepage": "https://github.com/simonw/datasette-render-html"}
release_url: https://pypi.org/project/datasette-render-html/0.1.2/
requires_dist: ["datasette", "pytest ; extra == 'test'"]
version: 0.1.2
description:
  # datasette-render-html

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-html.svg)](https://pypi.org/project/datasette-render-html/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-render-html.svg?style=svg)](https://circleci.com/gh/simonw/datasette-render-html)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-html/blob/master/LICENSE)

  This Datasette plugin lets you configure Datasette to render specific columns as HTML in the table and row interfaces.

  This means you can store HTML in those columns and have it rendered as such on those pages.

  If you have a database called `docs.db` containing a `glossary` table and you want the `definition` column in that table to be rendered as HTML, you would use a `metadata.json` file that looks like this:

      {
          "databases": {
              "docs": {
                  "tables": {
                      "glossary": {
                          "plugins": {
                              "datasette-render-html": {
                                  "columns": ["definition"]
                              }
                          }
                      }
                  }
              }
          }
      }

  ## Security

  This plugin allows HTML to be rendered exactly as it is stored in the database. As such, you should be sure only to use this against columns with content that you trust - otherwise you could open yourself up to an [XSS attack](https://owasp.org/www-community/attacks/xss/).

  It's possible to configure this plugin to apply to columns with specific names across whole databases or the full Datasette instance, but doing so is not safe. It could open you up to XSS vulnerabilities where an attacker composes a SQL query that results in a column containing unsafe HTML.

  As such, you should only use this plugin against specific columns in specific tables, as shown in the example above.


name: datasette-render-image-tags
summary: Turn any URLs ending in .jpg/.png/.gif into img tags with width 200
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-render-image-tags
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-render-image-tags/
project_url: https://pypi.org/project/datasette-render-image-tags/
release_url: https://pypi.org/project/datasette-render-image-tags/0.1/
requires_dist: ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.7
version: 0.1
description:
  # datasette-render-image-tags

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-image-tags.svg)](https://pypi.org/project/datasette-render-image-tags/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-render-image-tags?include_prereleases&label=changelog)](https://github.com/simonw/datasette-render-image-tags/releases)
  [![Tests](https://github.com/simonw/datasette-render-image-tags/workflows/Test/badge.svg)](https://github.com/simonw/datasette-render-image-tags/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-image-tags/blob/main/LICENSE)

  Turn any URLs ending in .jpg/.png/.gif into img tags with width 200

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-render-image-tags

  ## Usage

  Once installed, any cells contaning a URL that ends with `.png` or `.jpg` or `.jpeg` or `.gif` will be rendered using an image tag, with a width of 200px.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-render-image-tags
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-render-image-tags/actions", "Changelog": "https://github.com/simonw/datasette-render-image-tags/releases", "Homepage": "https://github.com/simonw/datasette-render-image-tags", "Issues": "https://github.com/simonw/datasette-render-image-tags/issues"}


name: datasette-render-images
summary: Datasette plugin that renders binary blob images using data-uris
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-render-images
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-render-images/
project_url: https://pypi.org/project/datasette-render-images/
project_urls: {"Homepage": "https://github.com/simonw/datasette-render-images"}
release_url: https://pypi.org/project/datasette-render-images/0.3.2/
requires_dist: ["datasette", "pytest ; extra == 'test'"]
version: 0.3.2
description:
  # datasette-render-images

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-images.svg)](https://pypi.org/project/datasette-render-images/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-render-images?include_prereleases&label=changelog)](https://github.com/simonw/datasette-render-images/releases)
  [![Tests](https://github.com/simonw/datasette-render-images/workflows/Test/badge.svg)](https://github.com/simonw/datasette-render-images/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-images/blob/main/LICENSE)

  A Datasette plugin that renders binary blob images with data-uris, using the [render_cell() plugin hook](https://datasette.readthedocs.io/en/stable/plugins.html#render-cell-value-column-table-database-datasette).

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-render-images

  ## Usage

  If a database row contains binary image data (PNG, GIF or JPEG), this plugin will detect that it is an image (using the [imghdr module](https://docs.python.org/3/library/imghdr.html) and render that cell using an `<img src="data:image/png;base64,...">` element.

  Here's a [demo of the plugin in action](https://datasette-render-images-demo.datasette.io/favicons/favicons).

  ## Creating a compatible database table

  You can use the [sqlite-utils insert-files](https://sqlite-utils.readthedocs.io/en/stable/cli.html#inserting-binary-data-from-files) command to insert image files into a database table:

      $ pip install sqlite-utils
      $ sqlite-utils insert-files gifs.db images *.gif

  See [Fun with binary data and SQLite](https://simonwillison.net/2020/Jul/30/fun-binary-data-and-sqlite/) for more on this tool.

  ## Configuration

  By default the plugin will only render images that are smaller than 100KB. You can adjust this limit using the `size_limit` plugin configuration option - for example, to increase the limit to 1MB (1000000 bytes) use the following in `metadata.json`:

  ```json
  {
      "plugins": {
          "datasette-render-images": {
              "size_limit": 1000000
          }
      }
  }
  ```


name: datasette-render-markdown
summary: Datasette plugin for rendering Markdown
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-render-markdown
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-render-markdown/
project_url: https://pypi.org/project/datasette-render-markdown/
project_urls: {"Homepage": "https://github.com/simonw/datasette-render-markdown"}
release_url: https://pypi.org/project/datasette-render-markdown/2.1/
requires_dist: ["datasette", "markdown", "bleach", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
version: 2.1
description:
  # datasette-render-markdown

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-markdown.svg)](https://pypi.org/project/datasette-render-markdown/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-render-markdown?include_prereleases&label=changelog)](https://github.com/simonw/datasette-render-markdown/releases)
  [![Tests](https://github.com/simonw/datasette-render-markdown/workflows/Test/badge.svg)](https://github.com/simonw/datasette-render-markdown/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-markdown/blob/main/LICENSE)

  Datasette plugin for rendering Markdown.

  ## Installation

  Install this plugin in the same environment as Datasette to enable this new functionality:

      $ pip install datasette-render-markdown

  ## Usage

  You can explicitly list the columns you would like to treat as Markdown using [plugin configuration](https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration) in a `metadata.json` file.

  Add a `"datasette-render-markdown"` configuration block and use a `"columns"` key to list the columns you would like to treat as Markdown values:

  ```json
  {
      "plugins": {
          "datasette-render-markdown": {
              "columns": ["body"]
          }
      }
  }
  ```

  This will cause any `body` column in any table to be treated as markdown and safely rendered using [Python-Markdown](https://python-markdown.github.io/). The resulting HTML is then run through [Bleach](https://bleach.readthedocs.io/) to avoid the risk of XSS security problems.

  Save this to `metadata.json` and run Datasette with the `--metadata` flag to load this configuration:

      $ datasette serve mydata.db --metadata metadata.json

  The configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the `entries` table in the `news.db` database:

  ```json
  {
      "databases": {
          "news": {
              "tables": {
                  "entries": {
                      "plugins": {
                          "datasette-render-markdown": {
                              "columns": ["body"]
                          }
                      }
                  }
              }
          }
      }
  }
  ```

  And here's how to apply it to every `body` column in every table in the `news.db` database:

  ```json
  {
      "databases": {
          "news": {
              "plugins": {
                  "datasette-render-markdown": {
                      "columns": ["body"]
                  }
              }
          }
      }
  }
  ```

  ## Columns that match a naming convention

  This plugin can also render markdown in any columns that match a specific naming convention.

  By default, columns that have a name ending in `_markdown` will be rendered.

  You can try this out using the following query:

  ```sql
  select '# Hello there

  * This is a list
  * of items

  [And a link](https://github.com/simonw/datasette-render-markdown).'
  as demo_markdown
  ```

  You can configure a different list of wildcard patterns using the `"patterns"` configuration key. Here's how to render columns that end in either `_markdown` or `_md`:

  ```json
  {
      "plugins": {
          "datasette-render-markdown": {
              "patterns": ["*_markdown", "*_md"]
          }
      }
  }
  ```

  To disable wildcard column matching entirely, set `"patterns": []` in your plugin metadata configuration.

  ## Markdown extensions

  The [Python-Markdown library](https://python-markdown.github.io/) that powers this plugin supports extensions, both [bundled](https://python-markdown.github.io/extensions/) and [third-party](https://github.com/Python-Markdown/markdown/wiki/Third-Party-Extensions). These can be used to enable additional Markdown features such as [table support](https://python-markdown.github.io/extensions/tables/).

  You can configure support for extensions using the `"extensions"` key in your plugin metadata configuration.

  Since extensions may introduce new HTML tags, you will also need to add those tags to the list of tags that are allowed by the [Bleach](https://bleach.readthedocs.io/) sanitizer. You can do that using the `"extra_tags"` key, and you can whitelist additional HTML attributes using `"extra_attrs"`. See [the Bleach documentation](https://bleach.readthedocs.io/en/latest/clean.html#allowed-tags-tags) for more information on this.

  Here's how to enable support for [Markdown tables](https://python-markdown.github.io/extensions/tables/):

  ```json
  {
      "plugins": {
          "datasette-render-markdown": {
              "extensions": ["tables"],
              "extra_tags": ["table", "thead", "tr", "th", "td", "tbody"]
          }
      }
  }
  ```

  ### GitHub-Flavored Markdown

  Enabling [GitHub-Flavored Markdown](https://help.github.com/en/github/writing-on-github) (useful for if you are working with data imported from GitHub using [github-to-sqlite](https://github.com/dogsheep/github-to-sqlite)) is a little more complicated.

  First, you will need to install the [py-gfm](https://py-gfm.readthedocs.io) package:

      $ pip install py-gfm

  Note that `py-gfm` has [a bug](https://github.com/Zopieux/py-gfm/issues/13) that causes it to pin to `Markdown<3.0` - so if you are using it you should install it _before_ installing `datasette-render-markdown` to ensure you get a compatibly version of that dependency.

  Now you can configure it like this. Note that the extension name is `mdx_gfm:GithubFlavoredMarkdownExtension` and you need to whitelist several extra HTML tags and attributes:

  ```json
  {
      "plugins": {
          "datasette-render-markdown": {
              "extra_tags": [
                  "hr",
                  "br",
                  "details",
                  "summary",
                  "input"
              ],
              "extra_attrs": {
                  "input": [
                      "type",
                      "disabled",
                      "checked"
                  ],
              },
              "extensions": [
                  "mdx_gfm:GithubFlavoredMarkdownExtension"
              ]
          }
      }
  }
  ```

  The `<input type="" checked disabled>` attributes are needed to support rendering checkboxes in issue descriptions.

  ## Markdown in templates

  The plugin also adds a new template function: `render_markdown(value)`. You can use this in your templates like so:

  ```html+jinja
  {{ render_markdown("""
  # This is markdown

  * One
  * Two
  * Three
  """) }}
  ```

  You can load additional extensions and whitelist tags by passing extra arguments to the function like this:

  ```html+jinja
  {{ render_markdown("""
  ## Markdown table

  First Header  | Second Header
  ------------- | -------------
  Content Cell  | Content Cell
  Content Cell  | Content Cell
  """, extensions=["tables"],
      extra_tags=["table", "thead", "tr", "th", "td", "tbody"])) }}
  ```


name: datasette-render-timestamps
summary: Datasette plugin for rendering timestamps
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-render-timestamps
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-render-timestamps/
project_url: https://pypi.org/project/datasette-render-timestamps/
project_urls: {"Homepage": "https://github.com/simonw/datasette-render-timestamps"}
release_url: https://pypi.org/project/datasette-render-timestamps/1.0.1/
requires_dist: ["datasette", "pytest ; extra == 'test'"]
version: 1.0.1
description:
  # datasette-render-timestamps

  [![PyPI](https://img.shields.io/pypi/v/datasette-render-timestamps.svg)](https://pypi.org/project/datasette-render-timestamps/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-render-timestamps.svg?style=svg)](https://circleci.com/gh/simonw/datasette-render-timestamps)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-render-timestamps/blob/master/LICENSE)

  Datasette plugin for rendering timestamps.

  ## Installation

  Install this plugin in the same environment as Datasette to enable this new functionality:

      pip install datasette-render-timestamps

  The plugin will then look out for integer numbers that are likely to be timestamps - anything that would be a number of seconds from 5 years ago to 5 years in the future.

  These will then be rendered in a more readable format.

  ## Configuration

  You can disable automatic column detection in favour of explicitly listing the columns that you would like to render using [plugin configuration](https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration) in a `metadata.json` file.

  Add a `"datasette-render-timestamps"` configuration block and use a `"columns"` key to list the columns you would like to treat as timestamp values:

  ```json
  {
      "plugins": {
          "datasette-render-timestamps": {
              "columns": ["created", "updated"]
          }
      }
  }
  ```
  This will cause any `created` or `updated` columns in any table to be treated as timestamps and rendered.

  Save this to `metadata.json` and run datasette with the `--metadata` flag to load this configuration:

      datasette serve mydata.db --metadata metadata.json

  To disable automatic timestamp detection entirely, you can use `"columnns": []`.

  This configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the `entries` table in the `news.db` database:

  ```json
  {
      "databases": {
          "news": {
              "tables": {
                  "entries": {
                      "plugins": {
                          "datasette-render-timestamps": {
                              "columns": ["created", "updated"]
                          }
                      }
                  }
              }
          }
      }
  }
  ```

  ### Customizing the date format

  The default format is `%B %d, %Y - %H:%M:%S UTC` which renders for example: `October 10, 2019 - 07:18:29 UTC`. If you want another format, the date format can be customized using plugin configuration. Any format string supported by [strftime](http://strftime.org/) may be used. For example:

  ```json
  {
      "plugins": {
          "datasette-render-timestamps": {
              "format": "%Y-%m-%d-%H:%M:%S"
          }
      }
  }
  ```


name: datasette-ripgrep
summary: Web interface for searching your code using ripgrep, built as a Datasette plugin
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-ripgrep
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-ripgrep/
project_url: https://pypi.org/project/datasette-ripgrep/
release_url: https://pypi.org/project/datasette-ripgrep/0.7/
requires_python: >=3.6
version: 0.7
description:
  # datasette-ripgrep

  [![PyPI](https://img.shields.io/pypi/v/datasette-ripgrep.svg)](https://pypi.org/project/datasette-ripgrep/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-ripgrep?include_prereleases&label=changelog)](https://github.com/simonw/datasette-ripgrep/releases)
  [![Tests](https://github.com/simonw/datasette-ripgrep/workflows/Test/badge.svg)](https://github.com/simonw/datasette-ripgrep/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-ripgrep/blob/main/LICENSE)

  Web interface for searching your code using [ripgrep](https://github.com/BurntSushi/ripgrep), built as a [Datasette](https://datasette.io/) plugin

  ## Demo

  Try this plugin out at https://ripgrep.datasette.io/-/ripgrep - where you can run regular expression searches across the source code of Datasette and all of the `datasette-*` plugins belonging to the [simonw GitHub user](https://github.com/simonw).

  Some example searches:

  - [with.\*AsyncClient](https://ripgrep.datasette.io/-/ripgrep?pattern=with.*AsyncClient) - regular expression search for `with.*AsyncClient`
  - [.plugin_config, literal=on](https://ripgrep.datasette.io/-/ripgrep?pattern=.plugin_config\(&literal=on) - a non-regular expression search for `.plugin_config(`
  - [with.\*AsyncClient glob=datasette/\*\*](https://ripgrep.datasette.io/-/ripgrep?pattern=with.*AsyncClient&glob=datasette%2F%2A%2A) - search for that pattern only within the `datasette/` top folder
  - ["sqlite-utils\[">\] glob=setup.py](https://ripgrep.datasette.io/-/ripgrep?pattern=%22sqlite-utils%5B%22%3E%5D&glob=setup.py) - a regular expression search for packages that depend on either `sqlite-utils` or `sqlite-utils>=some-version`
  - [test glob=!\*.html](https://ripgrep.datasette.io/-/ripgrep?pattern=test&glob=%21*.html) - search for the string `test` but exclude results in HTML files

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-ripgrep

  The `rg` executable needs to be [installed](https://github.com/BurntSushi/ripgrep/blob/master/README.md#installation) such that it can be run by this tool.

  ## Usage

  This plugin requires configuration: it needs to a `path` setting so that it knows where to run searches.

  Create a `metadata.json` file that looks like this:

  ```json
  {
      "plugins": {
          "datasette-ripgrep": {
              "path": "/path/to/your/files"
          }
      }
  }
  ```

  Now run Datasette using `datasette -m metadata.json`. The plugin will add an interface at `/-/ripgrep` for running searches.

  ## Plugin configuration

  The `"path"` configuration is required. Optional extra configuration options are:

  - `time_limit` - floating point number. The `rg` process will be terminated if it takes longer than this limit. The default is one second, `1.0`.
  - `max_lines` - integer. The `rg` process will be terminated if it returns more than this number of lines. The default is `2000`.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-ripgrep
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-ripgrep/actions", "Changelog": "https://github.com/simonw/datasette-ripgrep/releases", "Homepage": "https://github.com/simonw/datasette-ripgrep", "Issues": "https://github.com/simonw/datasette-ripgrep/issues"}

requires_dist:
  ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "httpx ; extra == 'test'"]


name: datasette-rure
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-rure
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-rure/
project_url: https://pypi.org/project/datasette-rure/
project_urls: {"Homepage": "https://github.com/simonw/datasette-rure"}
release_url: https://pypi.org/project/datasette-rure/0.3/
requires_dist: ["datasette", "rure", "pytest ; extra == 'test'"]
version: 0.3
summary:
  Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine

description:
  # datasette-rure

  [![PyPI](https://img.shields.io/pypi/v/datasette-rure.svg)](https://pypi.org/project/datasette-rure/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-rure.svg?style=svg)](https://circleci.com/gh/simonw/datasette-rure)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-rure/blob/master/LICENSE)

  Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine

  Install this plugin in the same environment as Datasette to enable the `regexp()` SQL function.

      $ pip install datasette-rure

  The plugin is built on top of the [rure-python](https://github.com/davidblewett/rure-python) library by David Blewett.

  ## regexp() to test regular expressions

  You can test if a value matches a regular expression like this:

      select regexp('hi.*there', 'hi there')
      -- returns 1
      select regexp('not.*there', 'hi there')
      -- returns 0

  You can also use SQLite's custom syntax to run matches:

      select 'hi there' REGEXP 'hi.*there'
      -- returns 1

  This means you can select rows based on regular expression matches - for example, to select every article where the title begins with an E or an F:

      select * from articles where title REGEXP '^[EF]'

  Try this out: [REGEXP interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+*+from+articles+where+title+REGEXP+%27%5E%5BEF%5D%27)

  ## regexp_match() to extract groups

  You can extract captured subsets of a pattern using `regexp_match()`.

      select regexp_match('.*( and .*)', title) as n from articles where n is not null
      -- Returns the ' and X' component of any matching titles, e.g.
      --     and Recognition
      --     and Transitions Their Place
      -- etc

  This will return the first parenthesis match when called with two arguments. You can call it with three arguments to indicate which match you would like to extract:

      select regexp_match('.*(and)(.*)', title, 2) as n from articles where n is not null

  The function will return `null` for invalid inputs e.g. a pattern without capture groups.

  Try this out: [regexp_match() interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+%27WHY+%27+%7C%7C+regexp_match%28%27Why+%28.*%29%27%2C+title%29+as+t+from+articles+where+t+is+not+null)

  ## regexp_matches() to extract multiple matches at once

  The `regexp_matches()` function can be used to extract multiple patterns from a single string. The result is returned as a JSON array, which can then be further processed using SQLite's [JSON functions](https://www.sqlite.org/json1.html).

  The first argument is a regular expression with named capture groups. The second argument is the string to be matched.

      select regexp_matches(
          'hello (?P<name>\w+) the (?P<species>\w+)',
          'hello bob the dog, hello maggie the cat, hello tarquin the otter'
      )

  This will return a list of JSON objects, each one representing the named captures from the original regular expression:

      [
          {"name": "bob", "species": "dog"},
          {"name": "maggie", "species": "cat"},
          {"name": "tarquin", "species": "otter"}
      ]

  Try this out: [regexp_matches() interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+regexp_matches%28%0D%0A++++%27hello+%28%3FP%3Cname%3E%5Cw%2B%29+the+%28%3FP%3Cspecies%3E%5Cw%2B%29%27%2C%0D%0A++++%27hello+bob+the+dog%2C+hello+maggie+the+cat%2C+hello+tarquin+the+otter%27%0D%0A%29)


name: datasette-sandstorm-support
summary: Authentication and permissions for Datasette on Sandstorm
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-sandstorm-support
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-sandstorm-support/
project_url: https://pypi.org/project/datasette-sandstorm-support/
release_url: https://pypi.org/project/datasette-sandstorm-support/0.2/
requires_dist: ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.7
version: 0.2
description:
  # datasette-sandstorm-support

  [![PyPI](https://img.shields.io/pypi/v/datasette-sandstorm-support.svg)](https://pypi.org/project/datasette-sandstorm-support/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-sandstorm-support?include_prereleases&label=changelog)](https://github.com/simonw/datasette-sandstorm-support/releases)
  [![Tests](https://github.com/simonw/datasette-sandstorm-support/workflows/Test/badge.svg)](https://github.com/simonw/datasette-sandstorm-support/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-sandstorm-support/blob/main/LICENSE)

  Authentication and permissions for Datasette on Sandstorm

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-sandstorm-support

  ## Usage

  This plugin is part of [datasette-sandstorm](https://github.com/ocdtrekkie/datasette-sandstorm).

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-sandstorm-support
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-sandstorm-support/actions", "Changelog": "https://github.com/simonw/datasette-sandstorm-support/releases", "Homepage": "https://github.com/simonw/datasette-sandstorm-support", "Issues": "https://github.com/simonw/datasette-sandstorm-support/issues"}


name: datasette-saved-queries
summary: Datasette plugin that lets users save and execute queries
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-saved-queries
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-saved-queries/
project_url: https://pypi.org/project/datasette-saved-queries/
release_url: https://pypi.org/project/datasette-saved-queries/0.2.1/
version: 0.2.1
description:
  # datasette-saved-queries

  [![PyPI](https://img.shields.io/pypi/v/datasette-saved-queries.svg)](https://pypi.org/project/datasette-saved-queries/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-saved-queries?label=changelog)](https://github.com/simonw/datasette-saved-queries/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-saved-queries/blob/master/LICENSE)

  Datasette plugin that lets users save and execute queries

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-saved-queries

  ## Usage

  When the plugin is installed Datasette will automatically create a `saved_queries` table in the first connected database when it starts up.

  It also creates a `save_query` writable canned query which you can use to save new queries.

  Queries that you save will be added to the query list on the database page.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-saved-queries
      python -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-saved-queries/actions", "Changelog": "https://github.com/simonw/datasette-saved-queries/releases", "Homepage": "https://github.com/simonw/datasette-saved-queries", "Issues": "https://github.com/simonw/datasette-saved-queries/issues"}

requires_dist:
  ["datasette (>=0.45)", "sqlite-utils", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "httpx ; extra == 'test'"]


name: datasette-scale-to-zero
summary: Quit Datasette if it has not received traffic for a specified time period
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-scale-to-zero
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-scale-to-zero/
project_url: https://pypi.org/project/datasette-scale-to-zero/
release_url: https://pypi.org/project/datasette-scale-to-zero/0.2/
requires_dist: ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.7
version: 0.2
description:
  # datasette-scale-to-zero

  [![PyPI](https://img.shields.io/pypi/v/datasette-scale-to-zero.svg)](https://pypi.org/project/datasette-scale-to-zero/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-scale-to-zero?include_prereleases&label=changelog)](https://github.com/simonw/datasette-scale-to-zero/releases)
  [![Tests](https://github.com/simonw/datasette-scale-to-zero/workflows/Test/badge.svg)](https://github.com/simonw/datasette-scale-to-zero/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-scale-to-zero/blob/main/LICENSE)

  Quit Datasette if it has not received traffic for a specified time period

  Some hosting providers such as [Fly](https://fly.io/) offer a scale to zero mechanism, where servers can shut down and will be automatically started when new traffic arrives.

  This plugin can be used to configure Datasette to quit X minutes (or seconds, or hours) after the last request it received. It can also cause the Datasette server to exit after a configured maximum time whether or not it is receiving traffic.

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-scale-to-zero

  ## Configuration

  This plugin will only take effect if it has been configured.

  Add the following to your ``metadata.json`` or ``metadata.yml`` configuration file:

  ```json
  {
      "plugins": {
          "datasette-scale-to-zero": {
              "duration": "10m"
          }
      }
  }
  ```
  This will cause Datasette to quit if it has not received any HTTP traffic for 10 minutes.

  You can set this value using a suffix of `m` for minutes, `h` for hours or `s` for seconds.

  To cause Datasette to exit if the server has been running for longer than a specific time, use `"max-age"`:

  ```json
  {
      "plugins": {
          "datasette-scale-to-zero": {
              "max-age": "10h"
          }
      }
  }
  ```
  This example will exit the Datasette server if it has been running for more than ten hours.

  You can use `"duration"` and `"max-age"` together in the same configuration file:

  ```json
  {
      "plugins": {
          "datasette-scale-to-zero": {
              "max-age": "10h",
              "duration": "5m"
          }
      }
  }
  ```
  This example will quit if no traffic has been received in five minutes, or if the server has been running for ten hours.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-scale-to-zero
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-scale-to-zero/actions", "Changelog": "https://github.com/simonw/datasette-scale-to-zero/releases", "Homepage": "https://github.com/simonw/datasette-scale-to-zero", "Issues": "https://github.com/simonw/datasette-scale-to-zero/issues"}


name: datasette-schema-versions
summary: Datasette plugin that shows the schema version of every attached database
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-schema-versions
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-schema-versions/
project_url: https://pypi.org/project/datasette-schema-versions/
release_url: https://pypi.org/project/datasette-schema-versions/0.2/
version: 0.2
description:
  # datasette-schema-versions

  [![PyPI](https://img.shields.io/pypi/v/datasette-schema-versions.svg)](https://pypi.org/project/datasette-schema-versions/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-schema-versions?include_prereleases&label=changelog)](https://github.com/simonw/datasette-schema-versions/releases)
  [![Tests](https://github.com/simonw/datasette-schema-versions/workflows/Test/badge.svg)](https://github.com/simonw/datasette-schema-versions/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-schema-versions/blob/main/LICENSE)

  Datasette plugin that shows the schema version of every attached database

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-schema-versions

  ## Usage

  Visit `/-/schema-versions` on your Datasette instance to see a numeric version for the schema for each of your databases.

  Any changes you make to the schema will increase this version number.

project_urls:
  {"CI": "https://github.com/simonw/datasette-schema-versions/actions", "Changelog": "https://github.com/simonw/datasette-schema-versions/releases", "Homepage": "https://github.com/simonw/datasette-schema-versions", "Issues": "https://github.com/simonw/datasette-schema-versions/issues"}

requires_dist:
  ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "httpx ; extra == 'test'", "sqlite-utils ; extra == 'test'"]


name: datasette-seaborn
summary: Statistical visualizations for Datasette using Seaborn
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-seaborn
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-seaborn/
project_url: https://pypi.org/project/datasette-seaborn/
release_url: https://pypi.org/project/datasette-seaborn/0.2a0/
version: 0.2a0
description:
  # datasette-seaborn

  [![PyPI](https://img.shields.io/pypi/v/datasette-seaborn.svg)](https://pypi.org/project/datasette-seaborn/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-seaborn?include_prereleases&label=changelog)](https://github.com/simonw/datasette-seaborn/releases)
  [![Tests](https://github.com/simonw/datasette-seaborn/workflows/Test/badge.svg)](https://github.com/simonw/datasette-seaborn/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-seaborn/blob/main/LICENSE)

  Statistical visualizations for Datasette using Seaborn

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-seaborn

  ## Usage

  Navigate to the new `.seaborn` extension for any Datasette table.

  The `_seaborn` argument specifies a method on `sns` to execute, e.g. `?_seaborn=relplot`.

  Extra arguments to those methods can be specified using e.g. `&_seaborn_x=column_name`.

  ## Configuration

  The plugin implements a default rendering time limit of five seconds. You can customize this limit using the `render_time_limit` setting, which accepts a floating point number of seconds. Add this to your `metadata.json`:

  ```json
  {
      "plugins": {
          "datasette-seaborn": {
              "render_time_limit": 1.0
          }
      }
  }
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-seaborn
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-seaborn/actions", "Changelog": "https://github.com/simonw/datasette-seaborn/releases", "Homepage": "https://github.com/simonw/datasette-seaborn", "Issues": "https://github.com/simonw/datasette-seaborn/issues"}

requires_dist:
  ["datasette (>=0.50)", "seaborn (>=0.11.0)", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]


name: datasette-search-all
summary: Datasette plugin for searching all searchable tables at once
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-search-all
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-search-all/
project_url: https://pypi.org/project/datasette-search-all/
release_url: https://pypi.org/project/datasette-search-all/1.1/
requires_python: >=3.7
version: 1.1
description:
  # datasette-search-all

  [![PyPI](https://img.shields.io/pypi/v/datasette-search-all.svg)](https://pypi.org/project/datasette-search-all/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-search-all?include_prereleases&label=changelog)](https://github.com/simonw/datasette-search-all/releases)
  [![Tests](https://github.com/simonw/datasette-search-all/workflows/Test/badge.svg)](https://github.com/simonw/datasette-search-all/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-search-all/blob/main/LICENSE)

  Datasette plugin for searching all searchable tables at once.

  ## Installation

  Install the plugin in the same Python environment as Datasette:

      pip install datasette-search-all

  ## Background

  See [datasette-search-all: a new plugin for searching multiple Datasette tables at once](https://simonwillison.net/2020/Mar/9/datasette-search-all/) for background on this project. You can try the plugin out at https://fara.datasettes.com/

  ## Usage

  This plugin only works if at least one of the tables connected to your Datasette instance has been configured for SQLite's full-text search.

  The [Datasette search documentation](https://docs.datasette.io/en/stable/full_text_search.html) includes details on how to enable full-text search for a table.

  You can also use the following tools:

  * [sqlite-utils](https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search) includes a command-line tool for enabling full-text search.
  * [datasette-enable-fts](https://github.com/simonw/datasette-enable-fts) is a Datasette plugin that adds a web interface for enabling search for specific columns.

  If the plugin detects at least one searchable table it will add a search form to the homepage.

  You can also navigate to `/-/search` on your Datasette instance to use the search interface directly.

  ## Screenshot

  ![Animated screenshot showing the plugin in action](https://raw.githubusercontent.com/simonw/datasette-search-all/main/animated-screenshot.gif)

project_urls:
  {"CI": "https://github.com/simonw/datasette-search-all/actions", "Changelog": "https://github.com/simonw/datasette-search-all/releases", "Homepage": "https://github.com/simonw/datasette-search-all", "Issues": "https://github.com/simonw/datasette-search-all/issues"}

requires_dist:
  ["datasette (>=0.61)", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "sqlite-utils ; extra == 'test'"]


name: datasette-sentry
summary: Datasette plugin for configuring Sentry
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-sentry
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-sentry/
project_url: https://pypi.org/project/datasette-sentry/
project_urls: {"Homepage": "https://github.com/simonw/datasette-sentry"}
release_url: https://pypi.org/project/datasette-sentry/0.3/
version: 0.3
classifiers:
  ["License :: OSI Approved :: Apache Software License", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8"]

description:
  # datasette-sentry

  [![PyPI](https://img.shields.io/pypi/v/datasette-sentry.svg)](https://pypi.org/project/datasette-sentry/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-sentry?include_prereleases&label=changelog)](https://github.com/simonw/datasette-sentry/releases)
  [![Tests](https://github.com/simonw/datasette-sentry/workflows/Test/badge.svg)](https://github.com/simonw/datasette-sentry/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-sentry/blob/main/LICENSE)

  Datasette plugin for configuring Sentry for error reporting
   
  ## Installation

      pip install datasette-sentry

  ## Usage

  This plugin only takes effect if your `metadata.json` file contains relevant top-level plugin configuration in a `"datasette-sentry"` configuration key.

  You will need a Sentry DSN - see their [Getting Started instructions](https://docs.sentry.io/error-reporting/quickstart/?platform=python).

  Add it to `metadata.json` like this:

  ```json
  {
      "plugins": {
          "datasette-sentry": {
              "dsn": "https://KEY@sentry.io/PROJECTID"
          }
      }
  }
  ```
  Settings in `metadata.json` are visible to anyone who visits the `/-/metadata` URL so this is a good place to take advantage of Datasette's [secret configuration values](https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values), in which case your configuration will look more like this:
  ```json
  {
      "plugins": {
          "datasette-sentry": {
              "dsn": {
                  "$env": "SENTRY_DSN"
              }
          }
      }
  }
  ```
  Then make a `SENTRY_DSN` environment variable available to Datasette.

  ## Configuration

  In addition to the `dsn` setting, you can also configure the Sentry [sample rate](https://docs.sentry.io/platforms/python/configuration/sampling/) by setting  `sample_rate` to a floating point number between 0 and 1.

  For example, to capture 25% of errors you would do this:

  ```json
  {
      "plugins": {
          "datasette-sentry": {
              "dsn": {
                  "$env": "SENTRY_DSN"
              },
              "sample_rate": 0.25
          }
      }
  }
  ```

requires_dist:
  ["sentry-sdk", "datasette (>=0.62)", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]


name: datasette-show-errors
summary: Datasette plugin for displaying error tracebacks
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-show-errors
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-show-errors/
project_url: https://pypi.org/project/datasette-show-errors/
project_urls: {"Homepage": "https://github.com/simonw/datasette-show-errors"}
release_url: https://pypi.org/project/datasette-show-errors/0.2/
requires_dist: ["starlette", "datasette", "pytest ; extra == 'test'"]
version: 0.2
description:
  # datasette-show-errors

  [![PyPI](https://img.shields.io/pypi/v/datasette-show-errors.svg)](https://pypi.org/project/datasette-show-errors/)
  [![CircleCI](https://circleci.com/gh/simonw/datasette-show-errors.svg?style=svg)](https://circleci.com/gh/simonw/datasette-show-errors)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-show-errors/blob/master/LICENSE)

  Datasette plugin for displaying error tracebacks.

  ## Installation

      pip install datasette-show-errors

  ## Usage

  Installing the plugin will cause any internal error to be displayed with a full traceback, rather than just a generic 500 page.

  Be careful not to use this in a context that might expose sensitive information.


name: datasette-sitemap
summary: Generate sitemap.xml for Datasette sites
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-sitemap
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-sitemap/
project_url: https://pypi.org/project/datasette-sitemap/
release_url: https://pypi.org/project/datasette-sitemap/1.0/
requires_python: >=3.7
version: 1.0
description:
  # datasette-sitemap

  [![PyPI](https://img.shields.io/pypi/v/datasette-sitemap.svg)](https://pypi.org/project/datasette-sitemap/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-sitemap?include_prereleases&label=changelog)](https://github.com/simonw/datasette-sitemap/releases)
  [![Tests](https://github.com/simonw/datasette-sitemap/workflows/Test/badge.svg)](https://github.com/simonw/datasette-sitemap/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-sitemap/blob/main/LICENSE)

  Generate sitemap.xml for Datasette sites

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-sitemap

  ## Demo

  This plugin is used for the sitemap on [til.simonwillison.net](https://til.simonwillison.net/):

  - https://til.simonwillison.net/sitemap.xml

  Here's [the configuration](https://github.com/simonw/til/blob/d4f67743a90a67100b46145986b2dec6f8d96583/metadata.yaml#L14-L16) used for that sitemap.

  ## Usage

  Once configured, this plugin adds a sitemap at `/sitemap.xml` with a list of URLs.

  This list is defined using a SQL query in `metadata.json` (or `.yml`) that looks like this:

  ```json
  {
    "plugins": {
      "datasette-sitemap": {
        "query": "select '/' || id as path from my_table"
      }
    }
  }
  ```

  Using `metadata.yml` allows for multi-line SQL queries which can be easier to maintain:

  ```yaml
  plugins:
    datasette-sitemap:
      query: |
        select
          '/' || id as path
        from
          my_table
  ```
  The SQL query must return a column called `path`. The values in this column must begin with a `/`. They will be used to generate a sitemap that looks like this:

  ```xml
  <?xml version="1.0" encoding="UTF-8"?>
  <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <url><loc>https://example.com/1</loc></url>
    <url><loc>https://example.com/2</loc></url>
  </urlset>
  ```
  You can use ``UNION`` in your SQL query to combine results from multiple tables, or include literal paths that you want to include in the index:

  ```sql
  select
    '/data/table1/' || id as path
    from table1
  union
  select
    '/data/table2/' || id as path
    from table2
  union
  select
    '/about' as path
  ```
  If your Datasette instance has multiple databases you can configure the database to query using the `database` configuration property.

  By default the domain name for the genearted URLs in the sitemap will be detected from the incoming request.

  You can set `base_url` instead to override this. This should not include a trailing slash.

  This example shows both of those settings, running the query against the `content` database and setting a custom base URL:

  ```yaml
  plugins:
    datasette-sitemap:
      query: |
        select '/plugins/' || name as path from plugins
        union
        select '/tools/' || name as path from tools
        union
        select '/news' as path
      database: content
      base_url: https://datasette.io
  ```
  [Try that query](https://datasette.io/content?sql=select+%27%2Fplugins%2F%27+||+name+as+path+from+plugins%0D%0Aunion%0D%0Aselect+%27%2Ftools%2F%27+||+name+as+path+from+tools%0D%0Aunion%0D%0Aselect+%27%2Fnews%27+as+path%0D%0A).

  ## robots.txt

  This plugin adds a `robots.txt` file pointing to the sitemap:

  ```
  Sitemap: http://example.com/sitemap.xml
  ```

  You can take full control of the sitemap by installing and configuring the [datasette-block-robots](https://datasette.io/plugins/datasette-block-robots) plugin.

  This plugin will add the `Sitemap:` line even if you are using `datasette-block-robots` for the rest of your `robots.txt` file.

  ## Adding paths to the sitemap from other plugins

  This plugin adds a new [plugin hook](https://docs.datasette.io/en/stable/plugin_hooks.html) to Datasete called `sitemap_extra_paths()` which can be used by other plugins to add their own additional lines to the `sitemap.xml` file.

  The hook accepts these optional parameters:

  - `datasette`: The current [Datasette instance](https://docs.datasette.io/en/stable/internals.html#datasette-class). You can use this to execute SQL queries or read plugin configuration settings.
  - `request`: The [Request object](https://docs.datasette.io/en/stable/internals.html#request-object) representing the incoming request to `/sitemap.xml`.

  The hook should return a list of strings, each representing a path to be added to the sitemap. Each path must begin with a `/`.

  It can also return an `async def` function, which will be awaited and used to generate a list of lines. Use this option if you need to make `await` calls inside you hook implementation.

  This example uses the hook to add two extra paths, one of which came from a SQL query:

  ```python
  from datasette import hookimpl

  @hookimpl
  def sitemap_extra_paths(datasette):
      async def inner():
          db = datasette.get_database()
          path_from_db = (await db.execute("select '/example'")).single_value()
          return ["/about", path_from_db]
      return inner
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-sitemap
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-sitemap/actions", "Changelog": "https://github.com/simonw/datasette-sitemap/releases", "Homepage": "https://github.com/simonw/datasette-sitemap", "Issues": "https://github.com/simonw/datasette-sitemap/issues"}

requires_dist:
  ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "datasette-block-robots ; extra == 'test'"]


name: datasette-socrata
summary: Import data from Socrata into Datasette
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-socrata
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-socrata/
project_url: https://pypi.org/project/datasette-socrata/
release_url: https://pypi.org/project/datasette-socrata/0.3/
requires_python: >=3.7
version: 0.3
description:
  # datasette-socrata

  [![PyPI](https://img.shields.io/pypi/v/datasette-socrata.svg)](https://pypi.org/project/datasette-socrata/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-socrata?include_prereleases&label=changelog)](https://github.com/simonw/datasette-socrata/releases)
  [![Tests](https://github.com/simonw/datasette-socrata/workflows/Test/badge.svg)](https://github.com/simonw/datasette-socrata/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-socrata/blob/main/LICENSE)

  Import data from Socrata into Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-socrata

  ## Usage

  Make sure you have [enabled WAL mode](https://til.simonwillison.net/sqlite/enabling-wal-mode) on your database files before using this plugin.

  Once installed, an interface for importing data from Socrata will become available at this URL:

      /-/import-socrata

  Users will be able to paste in a URL to a dataset on Socrata in order to initialize an import.

  You can also pre-fill the form by passing a `?url=` parameter, for example:

      /-/import-socrata?url=https://data.sfgov.org/City-Infrastructure/Street-Tree-List/tkzw-k3nq

  Any database that is attached to Datasette, is NOT loaded as immutable (with the `-i` option) and that has WAL mode enabled will be available for users to import data into.

  The `import-socrata` permission governs access. By default the `root` actor (accessible using `datasette --root` to start Datasette) is granted that permission.

  You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to other users.

  ## Configuration

  If you only want Socrata imports to be allowed to a specific database, you can configure that using plugin configration in `metadata.yml`:

  ```yaml
  plugins:
    datasette-socrata:
      database: socrata
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-socrata
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-socrata/actions", "Changelog": "https://github.com/simonw/datasette-socrata/releases", "Homepage": "https://github.com/simonw/datasette-socrata", "Issues": "https://github.com/simonw/datasette-socrata/issues"}

requires_dist:
  ["datasette", "sqlite-utils (>=3.27)", "datasette-low-disk-space-hook", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "pytest-httpx ; extra == 'test'"]


name: datasette-sqlite-fts4
summary: Datasette plugin exposing SQL functions from sqlite-fts4
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-sqlite-fts4
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-sqlite-fts4/
project_url: https://pypi.org/project/datasette-sqlite-fts4/
release_url: https://pypi.org/project/datasette-sqlite-fts4/0.3.2/
requires_python: >=3.7
version: 0.3.2
description:
  # datasette-sqlite-fts4

  [![PyPI](https://img.shields.io/pypi/v/datasette-sqlite-fts4.svg)](https://pypi.org/project/datasette-sqlite-fts4/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-sqlite-fts4?include_prereleases&label=changelog)](https://github.com/simonw/datasette-sqlite-fts4/releases)
  [![Tests](https://github.com/simonw/datasette-sqlite-fts4/workflows/Test/badge.svg)](https://github.com/simonw/datasette-sqlite-fts4/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-sqlite-fts4/blob/main/LICENSE)

  Datasette plugin that exposes the custom SQL functions from [sqlite-fts4](https://github.com/simonw/sqlite-fts4).

  [Interactive demo](https://datasette-sqlite-fts4.datasette.io/24ways-fts4?sql=select%0D%0A++++json_object%28%0D%0A++++++++"label"%2C+articles.title%2C+"href"%2C+articles.url%0D%0A++++%29+as+article%2C%0D%0A++++articles.author%2C%0D%0A++++rank_score%28matchinfo%28articles_fts%2C+"pcx"%29%29+as+score%2C%0D%0A++++rank_bm25%28matchinfo%28articles_fts%2C+"pcnalx"%29%29+as+bm25%2C%0D%0A++++json_object%28%0D%0A++++++++"pre"%2C+annotate_matchinfo%28matchinfo%28articles_fts%2C+"pcxnalyb"%29%2C+"pcxnalyb"%29%0D%0A++++%29+as+annotated_matchinfo%2C%0D%0A++++matchinfo%28articles_fts%2C+"pcxnalyb"%29+as+matchinfo%2C%0D%0A++++decode_matchinfo%28matchinfo%28articles_fts%2C+"pcxnalyb"%29%29+as+decoded_matchinfo%0D%0Afrom%0D%0A++++articles_fts+join+articles+on+articles.rowid+%3D+articles_fts.rowid%0D%0Awhere%0D%0A++++articles_fts+match+%3Asearch%0D%0Aorder+by+bm25&search=jquery+maps). Read [Exploring search relevance algorithms with SQLite](https://simonwillison.net/2019/Jan/7/exploring-search-relevance-algorithms-sqlite/) for further details on this project.

  ## Installation

      pip install datasette-sqlite-fts4

  If you are deploying a database using `datasette publish` you can include this plugin using the `--install` option:

      datasette publish now mydb.db --install=datasette-sqlite-fts4

project_urls:
  {"CI": "https://github.com/simonw/datasette-sqlite-fts4/actions", "Changelog": "https://github.com/simonw/datasette-sqlite-fts4/releases", "Homepage": "https://github.com/simonw/datasette-sqlite-fts4", "Issues": "https://github.com/simonw/datasette-sqlite-fts4/issues"}

requires_dist:
  ["datasette", "sqlite-fts4 (>=1.0.3)", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]


name: datasette-template-request
summary: Expose the Datasette request object to custom templates
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-template-request
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-template-request/
project_url: https://pypi.org/project/datasette-template-request/
release_url: https://pypi.org/project/datasette-template-request/0.1/
requires_dist: ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.6
version: 0.1
description:
  # datasette-template-request

  [![PyPI](https://img.shields.io/pypi/v/datasette-template-request.svg)](https://pypi.org/project/datasette-template-request/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-template-request?include_prereleases&label=changelog)](https://github.com/simonw/datasette-template-request/releases)
  [![Tests](https://github.com/simonw/datasette-template-request/workflows/Test/badge.svg)](https://github.com/simonw/datasette-template-request/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-template-request/blob/main/LICENSE)

  Expose the Datasette request object to custom templates

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-template-request

  ## Usage

  Once this plugin is installed, Datasette [custom templates](https://docs.datasette.io/en/stable/custom_templates.html) can use `{{ request }}` to access the current [request object](https://docs.datasette.io/en/stable/internals.html#request-object). For example, to access `?name=Cleo` in the query string a template could use this:

      Name: {{ request.args.name }}

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-template-request
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-template-request/actions", "Changelog": "https://github.com/simonw/datasette-template-request/releases", "Homepage": "https://github.com/simonw/datasette-template-request", "Issues": "https://github.com/simonw/datasette-template-request/issues"}


name: datasette-template-sql
summary: Datasette plugin for executing SQL queries from templates
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-template-sql
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-template-sql/
project_url: https://pypi.org/project/datasette-template-sql/
release_url: https://pypi.org/project/datasette-template-sql/1.0.2/
requires_python: >=3.6
version: 1.0.2
description:
  # datasette-template-sql

  [![PyPI](https://img.shields.io/pypi/v/datasette-template-sql.svg)](https://pypi.org/project/datasette-template-sql/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-template-sql?include_prereleases&label=changelog)](https://github.com/simonw/datasette-template-sql/releases)
  [![Tests](https://github.com/simonw/datasette-template-sql/workflows/Test/badge.svg)](https://github.com/simonw/datasette-template-sql/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-template-sql/blob/main/LICENSE)

  Datasette plugin for executing SQL queries from templates.

  ## Examples

  [www.niche-museums.com](https://www.niche-museums.com/) uses this plugin to run a custom themed website on top of Datasette. The full source code for the site [is here](https://github.com/simonw/museums) - see also [niche-museums.com, powered by Datasette](https://simonwillison.net/2019/Nov/25/niche-museums/).

  [simonw/til](https://github.com/simonw/til) is another simple example, described in [Using a self-rewriting README powered by GitHub Actions to track TILs](https://simonwillison.net/2020/Apr/20/self-rewriting-readme/).

  ## Installation

  Run this command to install the plugin in the same environment as Datasette:

      $ pip install datasette-template-sql

  ## Usage

  This plugin makes a new function, `sql(sql_query)`, available to your Datasette templates.

  You can use it like this:

  ```html+jinja
  {% for row in sql("select 1 + 1 as two, 2 * 4 as eight") %}
      {% for key in row.keys() %}
          {{ key }}: {{ row[key] }}<br>
      {% endfor %}
  {% endfor %}
  ```

  The plugin will execute SQL against the current database for the page in  `database.html`, `table.html` and `row.html` templates. If a template does not have a current database (`index.html` for example) the query will execute against the first attached database.

  ### Queries with arguments

  You can construct a SQL query using `?` or `:name` parameter syntax by passing a list or dictionary as a second argument:

  ```html+jinja
  {% for row in sql("select distinct topic from til order by topic") %}
      <h2>{{ row.topic }}</h2>
      <ul>
          {% for til in sql("select * from til where topic = ?", [row.topic]) %}
              <li><a href="{{ til.url }}">{{ til.title }}</a> - {{ til.created[:10] }}</li>
          {% endfor %}
      </ul>
  {% endfor %}
  ```

  Here's the same example using the `:topic` style of parameters:

  ```html+jinja
  {% for row in sql("select distinct topic from til order by topic") %}
      <h2>{{ row.topic }}</h2>
      <ul>
          {% for til in sql("select * from til where topic = :topic", {"topic": row.topic}) %}
              <li><a href="{{ til.url }}">{{ til.title }}</a> - {{ til.created[:10] }}</li>
          {% endfor %}
      </ul>
  {% endfor %}
  ```

  ### Querying a different database

  You can pass an optional `database=` argument to specify a named database to use for the query. For example, if you have attached a `news.db` database you could use this:

  ```html+jinja
  {% for article in sql(
      "select headline, date, summary from articles order by date desc limit 5",
      database="news"
  ) %}
      <h3>{{ article.headline }}</h2>
      <p class="date">{{ article.date }}</p>
      <p>{{ article.summary }}</p>
  {% endfor %}
  ```

project_urls:
  {"CI": "https://github.com/simonw/datasette-template-sql/actions", "Changelog": "https://github.com/simonw/datasette-template-sql/releases", "Homepage": "https://github.com/simonw/datasette-template-sql", "Issues": "https://github.com/simonw/datasette-template-sql/issues"}

requires_dist:
  ["datasette (>=0.54)", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "sqlite-utils ; extra == 'test'"]


name: datasette-tiddlywiki
summary: Run TiddlyWiki in Datasette and save Tiddlers to a SQLite database
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-tiddlywiki
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-tiddlywiki/
project_url: https://pypi.org/project/datasette-tiddlywiki/
release_url: https://pypi.org/project/datasette-tiddlywiki/0.2/
requires_dist: ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.6
version: 0.2
description:
  # datasette-tiddlywiki

  [![PyPI](https://img.shields.io/pypi/v/datasette-tiddlywiki.svg)](https://pypi.org/project/datasette-tiddlywiki/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-tiddlywiki?include_prereleases&label=changelog)](https://github.com/simonw/datasette-tiddlywiki/releases)
  [![Tests](https://github.com/simonw/datasette-tiddlywiki/workflows/Test/badge.svg)](https://github.com/simonw/datasette-tiddlywiki/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-tiddlywiki/blob/main/LICENSE)

  Run [TiddlyWiki](https://tiddlywiki.com/) in Datasette and save Tiddlers to a SQLite database

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-tiddlywiki

  ## Usage

  Start Datasette with a `tiddlywiki.db` database. You can create it if it does not yet exist using `--create`.

  You need to be signed in as the `root` user to write to the wiki, so use the `--root` option and click on the link it provides:

      % datasette tiddlywiki.db --create --root
      http://127.0.0.1:8001/-/auth-token?token=456670f1e8d01a8a33b71e17653130de17387336e29afcdfb4ab3d18261e6630
      # ...

  Navigate to `/-/tiddlywiki` on your instance to interact with TiddlyWiki.

  ## Authentication and permissions

  By default, the wiki can be read by anyone who has permission to read the `tiddlywiki.db` database. Only the signed in `root` user can write to it.

  You can sign in using the `--root` option described above, or you can set a password for that user using the [datasette-auth-passwords](https://datasette.io/plugins/datasette-auth-passwords) plugin and sign in using the `/-/login` page.

  You can use the `edit-tiddlywiki` permission to grant edit permisions to other users, using another plugin such as [datasette-permissions-sql](https://datasette.io/plugins/datasette-permissions-sql).

  You can use the `view-database` permission against the `tiddlywiki` database to control who can view the wiki.

  Datasette's permissions mechanism is described in full in [the Datasette documentation](https://docs.datasette.io/en/stable/authentication.html).

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-tiddlywiki
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-tiddlywiki/actions", "Changelog": "https://github.com/simonw/datasette-tiddlywiki/releases", "Homepage": "https://github.com/simonw/datasette-tiddlywiki", "Issues": "https://github.com/simonw/datasette-tiddlywiki/issues"}


name: datasette-tiles
summary: Mapping tile server for Datasette, serving tiles from MBTiles packages
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-tiles
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-tiles/
project_url: https://pypi.org/project/datasette-tiles/
release_url: https://pypi.org/project/datasette-tiles/0.6.1/
requires_python: >=3.6
version: 0.6.1
description:
  # datasette-tiles

  [![PyPI](https://img.shields.io/pypi/v/datasette-tiles.svg)](https://pypi.org/project/datasette-tiles/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-tiles?include_prereleases&label=changelog)](https://github.com/simonw/datasette-tiles/releases)
  [![Tests](https://github.com/simonw/datasette-tiles/workflows/Test/badge.svg)](https://github.com/simonw/datasette-tiles/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-tiles/blob/main/LICENSE)

  Datasette plugin for serving MBTiles map tiles

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-tiles

  ## Demo

  You can try this plugin out at https://datasette-tiles-demo.datasette.io/-/tiles

  ## Usage

  This plugin scans all database files connected to Datasette to see if any of them are valid MBTiles databases.

  It can then serve tiles from those databases at the following URL:

      /-/tiles/db-name/zoom/x/y.png

  An example map for each database demonstrating the configured minimum and maximum zoom for that database can be found at `/-/tiles/db-name` - this can also be accessed via the table and database action menus for that database.

  Visit `/-/tiles` for an index page of attached valid databases.

  You can install the [datasette-basemap](https://datasette.io/plugins/datasette-basemap) plugin to get a `basemap` default set of tiles, handling zoom levels 0 to 6 using OpenStreetMap.

  ### Tile coordinate systems

  There are two tile coordinate systems in common use for online maps. The first is used by OpenStreetMap and Google Maps, the second is from a specification called [Tile Map Service](https://en.wikipedia.org/wiki/Tile_Map_Service), or TMS.

  Both systems use three components: `z/x/y` - where `z` is the zoom level, `x` is the column and `y` is the row.

  The difference is in the way the `y` value is counted. OpenStreetMap has y=0 at the top. TMS has y=0 at the bottom.

  An illustrative example: at zoom level 2 the map is divided into 16 total tiles. The OpenStreetMap scheme numbers them like so:

      0/0  1/0  2/0  3/0
      0/1  1/1  2/1  3/1
      0/2  1/2  2/2  3/2
      0/3  1/3  2/3  3/3

  The TMS scheme looks like this:

      0/3  1/3  2/3  3/3
      0/2  1/2  2/2  3/2
      0/1  1/1  2/1  3/1
      0/0  1/0  2/0  3/0

  `datasette-tiles` can serve tiles using either of these standards. For the OpenStreetMap / Google Maps 0-at-the-top system, use the following URL:

      /-/tiles/database-name/{z}/{x}/{y}.png

  For the TMS 0-at-the-bottom system, use this:

      /-/tiles-tms/database-name/{z}/{x}/{y}.png

  ### Configuring a Leaflet tile layer

  The following JavaScript will configure a [Leaflet TileLayer](https://leafletjs.com/reference-1.7.1.html#tilelayer) for use with this plugin:

  ```javascript
  var tiles = leaflet.tileLayer("/-/tiles/basemap/{z}/{x}/{y}.png", {
    minZoom: 0,
    maxZoom: 6,
    attribution: "\u00a9 OpenStreetMap contributors"
  });
  ```

  ### Tile stacks

  `datasette-tiles` can be configured to serve tiles from multiple attached MBTiles files, searching each database in order for a tile and falling back to the next in line if that tile is not found.

  For a demo of this in action, visit https://datasette-tiles-demo.datasette.io/-/tiles-stack and zoom in on Japan. It should start showing [Stamen's Toner map](maps.stamen.com) of Japan once you get to zoom level 6 and 7.

  The `/-/tiles-stack/{z}/{x}/{y}.png` endpoint provides this feature.

  If you start Datasette like this:

      datasette world.mbtiles country.mbtiles city1.mbtiles city2.mbtiles

  Any requests for a tile from the `/-/tiles-stack` path will first check the `city2` database, than `city1`, then `country`, then `world`.

  If you have the [datasette-basemap](https://datasette.io/plugins/datasette-basemap) plugin installed it will be given special treatment: the `basemap` database will always be the last database checked for a tile.

  Rather than rely on the order in which databases were attached, you can instead configure an explicit order using the `tiles-stack-order` plugin setting. Add the following to your `metadata.json` file:

  ```json
  {
      "plugins": {
          "datasette-tiles": {
              "tiles-stack-order": ["world", "country"]
          }
      }
  }
  ```

  You can then run Datasette like this:

      datasette -m metadata.json country.mbtiles world.mbtiles

  This endpoint serves tiles using the OpenStreetMap / Google Maps coordinate system. To load tiles using the TMS coordinate system use this endpoint instead:

      /-/tiles-stack-tms/{z}/{x}/{y}.png

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-tiles
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-tiles/actions", "Changelpog": "https://github.com/simonw/datasette-tiles/releases", "Homepage": "https://github.com/simonw/datasette-tiles", "Issues": "https://github.com/simonw/datasette-tiles/issues"}

requires_dist:
  ["datasette", "datasette-leaflet (>=0.2.2)", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "datasette-basemap (>=0.2) ; extra == 'test'"]


name: datasette-total-page-time
summary: Add a note to the Datasette footer measuring the total page load time
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-total-page-time
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-total-page-time/
project_url: https://pypi.org/project/datasette-total-page-time/
release_url: https://pypi.org/project/datasette-total-page-time/0.1/
requires_dist: ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.7
version: 0.1
description:
  # datasette-total-page-time

  [![PyPI](https://img.shields.io/pypi/v/datasette-total-page-time.svg)](https://pypi.org/project/datasette-total-page-time/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-total-page-time?include_prereleases&label=changelog)](https://github.com/simonw/datasette-total-page-time/releases)
  [![Tests](https://github.com/simonw/datasette-total-page-time/workflows/Test/badge.svg)](https://github.com/simonw/datasette-total-page-time/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-total-page-time/blob/main/LICENSE)

  Add a note to the Datasette footer measuring the total page load time

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-total-page-time

  ## Usage

  Once this plugin is installed, a note will appear in the footer of every page showing how long the page took to generate.

  > Queries took 326.74ms · Page took 386.310ms

  ## How it works

  Measuring how long a page takes to load and then injecting that note into the page is tricky, because you need to finish generating the page before you know how long it took to load it!

  This plugin uses the [asgi_wrapper](https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette) plugin hook to measure the time taken by Datasette and then inject the following JavaScript at the bottom of the response, after the closing `</html>` tag but with the correct measured value:

  ```html
  <script>
  let footer = document.querySelector("footer");
  if (footer) {
      let ms = 37.224;
      let s = ` &middot; Page took ${ms.toFixed(3)}ms`;
      footer.innerHTML += s;
  }
  </script>
  ```
  This script is injected only on pages with the `text/html` content type - so it should not affect JSON or CSV returned by Datasette.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-total-page-time
      python3 -mvenv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-total-page-time/actions", "Changelog": "https://github.com/simonw/datasette-total-page-time/releases", "Homepage": "https://github.com/simonw/datasette-total-page-time", "Issues": "https://github.com/simonw/datasette-total-page-time/issues"}


name: datasette-upload-csvs
summary: Datasette plugin for uploading CSV files and converting them to database tables
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://datasette.io/plugins/datasette-upload-csvs
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-upload-csvs/
project_url: https://pypi.org/project/datasette-upload-csvs/
release_url: https://pypi.org/project/datasette-upload-csvs/0.8.2/
requires_python: >=3.7
version: 0.8.2
description:
  # datasette-upload-csvs

  [![PyPI](https://img.shields.io/pypi/v/datasette-upload-csvs.svg)](https://pypi.org/project/datasette-upload-csvs/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-upload-csvs?include_prereleases&label=changelog)](https://github.com/simonw/datasette-upload-csvs/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-upload-csvs/blob/main/LICENSE)

  Datasette plugin for uploading CSV files and converting them to database tables

  ## Installation

      datasette install datasette-upload-csvs

  ## Usage

  The plugin adds an interface at `/-/upload-csvs` for uploading a CSV file and using it to create a new database table.

  By default only [the root actor](https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor) can access the page - so you'll need to run Datasette with the `--root` option and click on the link shown in the terminal to sign in and access the page.

  The `upload-csvs` permission governs access. You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to the write interface.

project_urls:
  {"CI": "https://github.com/simonw/datasette-upload-csvs/actions", "Changelog": "https://github.com/simonw/datasette-upload-csvs/releases", "Homepage": "https://datasette.io/plugins/datasette-upload-csvs", "Issues": "https://github.com/simonw/datasette-upload-csvs/issues"}

requires_dist:
  ["datasette (>=0.61)", "asgi-csrf (>=0.7)", "starlette", "aiofiles", "python-multipart", "charset-normalizer", "sqlite-utils", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "asgiref ; extra == 'test'", "httpx ; extra == 'test'", "asgi-lifespan ; extra == 'test'"]


name: datasette-upload-dbs
summary: Upload SQLite database files to Datasette
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-upload-dbs
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-upload-dbs/
project_url: https://pypi.org/project/datasette-upload-dbs/
release_url: https://pypi.org/project/datasette-upload-dbs/0.1.2/
requires_dist: ["datasette", "starlette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.7
version: 0.1.2
description:
  # datasette-upload-dbs

  [![PyPI](https://img.shields.io/pypi/v/datasette-upload-dbs.svg)](https://pypi.org/project/datasette-upload-dbs/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-upload-dbs?include_prereleases&label=changelog)](https://github.com/simonw/datasette-upload-dbs/releases)
  [![Tests](https://github.com/simonw/datasette-upload-dbs/workflows/Test/badge.svg)](https://github.com/simonw/datasette-upload-dbs/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-upload-dbs/blob/main/LICENSE)

  Upload SQLite database files to Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      datasette install datasette-upload-dbs

  ## Configuration

  This plugin requires you to configure a directory in which uploaded files will be stored.

  On startup, Datasette will automatically load any SQLite files that it finds in that directory. This means it is safe to restart your server in between file uploads.

  To configure the directory as `/home/datasette/uploads`, add this to a `metadata.yml` configuration file:

  ```yaml
  plugins:
    datasette-upload-dbs:
      directory: /home/datasette/uploads
  ```

  Or if you are using `metadata.json`:

  ```json
  {
    "plugins": {
      "datasette-upload-dbs": {
        "directory": "/home/datasette/uploads"
      }
    }
  }
  ```
  You can use `"."` for the current folder when the server starts, or `"uploads"` for a folder relative to that folder. The folder will be created on startup if it does not already exist.

  Then start Datasette like this:

      datasette -m metadata.yml

  ## Usage

  Only users with the `upload-dbs` permission will be able to upload files. The `root` user has this permission by default - other users can be granted access using permission plugins, see the [Permissions](https://docs.datasette.io/en/stable/authentication.html#permissions) documentation for details.

  To start Datasette as the root user, run this:

      datasette -m metadata.yml --root

  And follow the link that is displayd on the console.

  If a user has that permission they will see an "Upload database" link in the navigation menu.

  This will take them to `/-/upload-dbs` where they will be able to upload database files, by selecting them or by dragging them onto the drop area.

  ![Animated demo showing a file being dropped onto a box, then uploading and redirecting to the database page](https://github.com/simonw/datasette-upload-dbs/raw/main/upload-demo.gif)

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-upload-dbs
      python3 -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-upload-dbs/actions", "Changelog": "https://github.com/simonw/datasette-upload-dbs/releases", "Homepage": "https://github.com/simonw/datasette-upload-dbs", "Issues": "https://github.com/simonw/datasette-upload-dbs/issues"}


name: datasette-vega
summary: A Datasette plugin that provides tools for generating charts using Vega
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-vega
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-vega/
project_url: https://pypi.org/project/datasette-vega/
project_urls: {"Homepage": "https://github.com/simonw/datasette-vega"}
release_url: https://pypi.org/project/datasette-vega/0.6.2/
requires_dist: ["datasette"]
version: 0.6.2
description:
  # datasette-vega

  [![PyPI](https://img.shields.io/pypi/v/datasette-vega.svg)](https://pypi.org/project/datasette-vega/)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-vega/blob/master/LICENSE)

  A [Datasette](https://github.com/simonw/datasette) plugin that provides tools
  for generating charts using [Vega](https://vega.github.io/).

  ![Datasette Vega interface](https://raw.githubusercontent.com/simonw/datasette-vega/master/datasette-vega.png)

  Try out the latest master build as a live demo at https://datasette-vega-latest.datasette.io/ or try the latest release installed as a plugin at https://fivethirtyeight.datasettes.com/

  To add this to your Datasette installation, install the plugin like so:

      pip install datasette-vega

  The plugin will then add itself to every Datasette table view.

  If you are publishing data using the `datasette publish` command, you can
  include this plugin like so:

      datasette publish now mydatabase.db --install=datasette-vega

  If you don't want to install any software at all, you can publish your CSV files with Datasette and activate this plugin using the web application at https://publish.datasettes.com/


name: datasette-verify
summary: Verify that SQLite files can be opened using Datasette
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-verify
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-verify/
project_url: https://pypi.org/project/datasette-verify/
release_url: https://pypi.org/project/datasette-verify/0.1/
requires_dist: ["datasette (>=0.59a2)", "pytest ; extra == 'test'"]
requires_python: >=3.6
version: 0.1
description:
  # datasette-verify

  [![PyPI](https://img.shields.io/pypi/v/datasette-verify.svg)](https://pypi.org/project/datasette-verify/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-verify?include_prereleases&label=changelog)](https://github.com/simonw/datasette-verify/releases)
  [![Tests](https://github.com/simonw/datasette-verify/workflows/Test/badge.svg)](https://github.com/simonw/datasette-verify/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-verify/blob/main/LICENSE)

  Verify that SQLite files can be opened using Datasette

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-verify

  This plugin depends on [Datasette 0.59a2](https://github.com/simonw/datasette/releases/tag/0.59a2) or higher, as it uses the [register_commands()](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hook-register-commands) plugin hook.

  ## Usage

  To confirm that files can be opened by Datasette, run the following:

      datasette verify file1.db file2.db

  You can pass one or more file paths.

  The command will exit silently with a 0 exit code if the files are all valid SQLite databases that Datasette can open.

  It will exit with a 1 exit code and display an error for the first file it finds that is not valid.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-verify
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-verify/actions", "Changelog": "https://github.com/simonw/datasette-verify/releases", "Homepage": "https://github.com/simonw/datasette-verify", "Issues": "https://github.com/simonw/datasette-verify/issues"}


name: datasette-write
summary: Datasette plugin providing a UI for writing to a database
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-write
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-write/
project_url: https://pypi.org/project/datasette-write/
release_url: https://pypi.org/project/datasette-write/0.2/
version: 0.2
description:
  # datasette-write

  [![PyPI](https://img.shields.io/pypi/v/datasette-write.svg)](https://pypi.org/project/datasette-write/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-write?label=changelog)](https://github.com/simonw/datasette-write/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-write/blob/master/LICENSE)

  Datasette plugin providing a UI for writing to a database

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ pip install datasette-write

  ## Usage

  Having installed the plugin, visit `/-/write` on your Datasette instance to submit SQL queries that will be executed against a write connection to the specified database.

  By default only the `root` user can access the page - so you'll need to run Datasette with the `--root` option and click on the link shown in the terminal to sign in and access the page.

  The `datasette-write` permission governs access. You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to the write interface.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-write
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-write/actions", "Changelog": "https://github.com/simonw/datasette-write/releases", "Homepage": "https://github.com/simonw/datasette-write", "Issues": "https://github.com/simonw/datasette-write/issues"}

requires_dist:
  ["datasette (>=0.45)", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "httpx ; extra == 'test'"]


name: datasette-x-forwarded-host
summary: Treat the X-Forwarded-Host header as the Host header
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-x-forwarded-host
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-x-forwarded-host/
project_url: https://pypi.org/project/datasette-x-forwarded-host/
release_url: https://pypi.org/project/datasette-x-forwarded-host/0.1/
requires_dist: ["datasette", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.6
version: 0.1
description:
  # datasette-x-forwarded-host

  [![PyPI](https://img.shields.io/pypi/v/datasette-x-forwarded-host.svg)](https://pypi.org/project/datasette-x-forwarded-host/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-x-forwarded-host?include_prereleases&label=changelog)](https://github.com/simonw/datasette-x-forwarded-host/releases)
  [![Tests](https://github.com/simonw/datasette-x-forwarded-host/workflows/Test/badge.svg)](https://github.com/simonw/datasette-x-forwarded-host/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-x-forwarded-host/blob/main/LICENSE)

  Treat the X-Forwarded-Host header as the Host header

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-x-forwarded-host

  ## Usage

  Once installed, Datasette will replace the `host` header with the content of the incoming `x-forwarded-host` header.

  This helps Datasette generate links to new pages that work when hosted behind a proxy that rewrites the `host` header.

  Only use this plugin in deployment environmens where you know the `x-forwarded-host` header can be trusted!

  This has been tested on GitHub Codespaces.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-x-forwarded-host
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-x-forwarded-host/actions", "Changelog": "https://github.com/simonw/datasette-x-forwarded-host/releases", "Homepage": "https://github.com/simonw/datasette-x-forwarded-host", "Issues": "https://github.com/simonw/datasette-x-forwarded-host/issues"}


name: datasette-yaml
summary: Export Datasette records as YAML
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/datasette-yaml
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette-yaml/
project_url: https://pypi.org/project/datasette-yaml/
release_url: https://pypi.org/project/datasette-yaml/0.1.1/
version: 0.1.1
description:
  # datasette-yaml

  [![PyPI](https://img.shields.io/pypi/v/datasette-yaml.svg)](https://pypi.org/project/datasette-yaml/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette-yaml?include_prereleases&label=changelog)](https://github.com/simonw/datasette-yaml/releases)
  [![Tests](https://github.com/simonw/datasette-yaml/workflows/Test/badge.svg)](https://github.com/simonw/datasette-yaml/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette-yaml/blob/main/LICENSE)

  Export Datasette records as YAML

  ## Installation

  Install this plugin in the same environment as Datasette.

      $ datasette install datasette-yaml

  ## Usage

  Having installed this plugin, every table and query will gain a new `.yaml` export link.

  You can also construct these URLs directly: `/dbname/tablename.yaml`

  ## Demo

  The plugin is running on [covid-19.datasettes.com](https://covid-19.datasettes.co/) - for example [/covid/latest_ny_times_counties_with_populations.yaml](https://covid-19.datasettes.com/covid/latest_ny_times_counties_with_populations.yaml)

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd datasette-yaml
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/datasette-yaml/actions", "Changelog": "https://github.com/simonw/datasette-yaml/releases", "Homepage": "https://github.com/simonw/datasette-yaml", "Issues": "https://github.com/simonw/datasette-yaml/issues"}

requires_dist:
[cell repeats an earlier one in this file: 138 characters]


name: datasette
summary: An open source multi-tool for exploring and publishing data
author: Simon Willison
description_content_type: text/markdown
home_page: https://datasette.io/
license: Apache License, Version 2.0
package_url: https://pypi.org/project/datasette/
project_url: https://pypi.org/project/datasette/
release_url: https://pypi.org/project/datasette/0.63.1/
requires_python: >=3.7
version: 0.63.1
classifiers:
  ["Development Status :: 4 - Beta", "Framework :: Datasette", "Intended Audience :: Developers", "Intended Audience :: End Users/Desktop", "Intended Audience :: Science/Research", "License :: OSI Approved :: Apache Software License", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Topic :: Database"]

description:
  <img src="https://datasette.io/static/datasette-logo.svg" alt="Datasette">

  [![PyPI](https://img.shields.io/pypi/v/datasette.svg)](https://pypi.org/project/datasette/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/datasette?label=changelog)](https://docs.datasette.io/en/stable/changelog.html)
  [![Python 3.x](https://img.shields.io/pypi/pyversions/datasette.svg?logo=python&logoColor=white)](https://pypi.org/project/datasette/)
  [![Tests](https://github.com/simonw/datasette/workflows/Test/badge.svg)](https://github.com/simonw/datasette/actions?query=workflow%3ATest)
  [![Documentation Status](https://readthedocs.org/projects/datasette/badge/?version=latest)](https://docs.datasette.io/en/latest/?badge=latest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/datasette/blob/main/LICENSE)
  [![docker: datasette](https://img.shields.io/badge/docker-datasette-blue)](https://hub.docker.com/r/datasetteproject/datasette)
  [![discord](https://img.shields.io/discord/823971286308356157?label=discord)](https://discord.gg/ktd74dm5mw)

  *An open source multi-tool for exploring and publishing data*

  Datasette is a tool for exploring and publishing data. It helps people take data of any shape or size and publish that as an interactive, explorable website and accompanying API.

  Datasette is aimed at data journalists, museum curators, archivists, local governments, scientists, researchers and anyone else who has data that they wish to share with the world.

  [Explore a demo](https://global-power-plants.datasettes.com/global-power-plants/global-power-plants), watch [a video about the project](https://simonwillison.net/2021/Feb/7/video/) or try it out by [uploading and publishing your own CSV data](https://docs.datasette.io/en/stable/getting_started.html#try-datasette-without-installing-anything-using-glitch).

  * [datasette.io](https://datasette.io/) is the official project website
  * Latest [Datasette News](https://datasette.io/news)
  * Comprehensive documentation: https://docs.datasette.io/
  * Examples: https://datasette.io/examples
  * Live demo of current `main` branch: https://latest.datasette.io/
  * Questions, feedback or want to talk about the project? Join our [Discord](https://discord.gg/ktd74dm5mw)

  Want to stay up-to-date with the project? Subscribe to the [Datasette newsletter](https://datasette.substack.com/) for tips, tricks and news on what's new in the Datasette ecosystem.

  ## Installation

  If you are on a Mac, [Homebrew](https://brew.sh/) is the easiest way to install Datasette:

      brew install datasette

  You can also install it using `pip` or `pipx`:

      pip install datasette

  Datasette requires Python 3.7 or higher. We also have [detailed installation instructions](https://docs.datasette.io/en/stable/installation.html) covering other options such as Docker.

  ## Basic usage

      datasette serve path/to/database.db

  This will start a web server on port 8001 - visit http://localhost:8001/ to access the web interface.

  `serve` is the default subcommand, you can omit it if you like.

  Use Chrome on OS X? You can run datasette against your browser history like so:

       datasette ~/Library/Application\ Support/Google/Chrome/Default/History --nolock

  Now visiting http://localhost:8001/History/downloads will show you a web interface to browse your downloads data:

  ![Downloads table rendered by datasette](https://static.simonwillison.net/static/2017/datasette-downloads.png)

  ## metadata.json

  If you want to include licensing and source information in the generated datasette website you can do so using a JSON file that looks something like this:

      {
          "title": "Five Thirty Eight",
          "license": "CC Attribution 4.0 License",
          "license_url": "http://creativecommons.org/licenses/by/4.0/",
          "source": "fivethirtyeight/data on GitHub",
          "source_url": "https://github.com/fivethirtyeight/data"
      }

  Save this in `metadata.json` and run Datasette like so:

      datasette serve fivethirtyeight.db -m metadata.json

  The license and source information will be displayed on the index page and in the footer. They will also be included in the JSON produced by the API.

  ## datasette publish

  If you have [Heroku](https://heroku.com/) or [Google Cloud Run](https://cloud.google.com/run/) configured, Datasette can deploy one or more SQLite databases to the internet with a single command:

      datasette publish heroku database.db

  Or:

      datasette publish cloudrun database.db

  This will create a docker image containing both the datasette application and the specified SQLite database files. It will then deploy that image to Heroku or Cloud Run and give you a URL to access the resulting website and API.

  See [Publishing data](https://docs.datasette.io/en/stable/publish.html) in the documentation for more details.

  ## Datasette Lite

  [Datasette Lite](https://lite.datasette.io/) is Datasette packaged using WebAssembly so that it runs entirely in your browser, no Python web application server required. Read more about that in the [Datasette Lite documentation](https://github.com/simonw/datasette-lite/blob/main/README.md).

project_urls:
  {"CI": "https://github.com/simonw/datasette/actions?query=workflow%3ATest", "Changelog": "https://docs.datasette.io/en/stable/changelog.html", "Documentation": "https://docs.datasette.io/en/stable/", "Homepage": "https://datasette.io/", "Issues": "https://github.com/simonw/datasette/issues", "Live demo": "https://latest.datasette.io/", "Source code": "https://github.com/simonw/datasette"}

requires_dist:
  ["asgiref (>=3.2.10)", "click (>=7.1.1)", "click-default-group-wheel (>=1.2.2)", "Jinja2 (>=2.10.3)", "hupper (>=1.9)", "httpx (>=0.20)", "pint (>=0.9)", "pluggy (>=1.0)", "uvicorn (>=0.11)", "aiofiles (>=0.4)", "janus (>=0.6.2)", "asgi-csrf (>=0.9)", "PyYAML (>=5.3)", "mergedeep (>=1.1.1)", "itsdangerous (>=1.1)", "furo (==2022.9.29) ; extra == 'docs'", "sphinx-autobuild ; extra == 'docs'", "codespell ; extra == 'docs'", "blacken-docs ; extra == 'docs'", "sphinx-copybutton ; extra == 'docs'", "rich ; extra == 'rich'", "pytest (>=5.2.2) ; extra == 'test'", "pytest-xdist (>=2.2.1) ; extra == 'test'", "pytest-asyncio (>=0.17) ; extra == 'test'", "beautifulsoup4 (>=4.8.1) ; extra == 'test'", "black (==22.10.0) ; extra == 'test'", "blacken-docs (==1.12.1) ; extra == 'test'", "pytest-timeout (>=1.4.2) ; extra == 'test'", "trustme (>=0.7) ; extra == 'test'", "cogapp (>=3.3.0) ; extra == 'test'"]


name: db-to-sqlite
summary: CLI tool for exporting tables or queries from any SQL database to a SQLite file
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/db-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/db-to-sqlite/
project_url: https://pypi.org/project/db-to-sqlite/
release_url: https://pypi.org/project/db-to-sqlite/1.4/
version: 1.4
classifiers:
  ["Development Status :: 3 - Alpha", "Intended Audience :: Developers", "Intended Audience :: End Users/Desktop", "Intended Audience :: Science/Research", "License :: OSI Approved :: Apache Software License", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Topic :: Database"]

description:
  # db-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/db-to-sqlite.svg)](https://pypi.python.org/pypi/db-to-sqlite)
  [![Changelog](https://img.shields.io/github/v/release/simonw/db-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/db-to-sqlite/releases)
  [![Tests](https://github.com/simonw/db-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/db-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/db-to-sqlite/blob/main/LICENSE)

  CLI tool for exporting tables or queries from any SQL database to a SQLite file.

  ## Installation

  Install from PyPI like so:

      pip install db-to-sqlite

  If you want to use it with MySQL, you can install the extra dependency like this:

      pip install 'db-to-sqlite[mysql]'

  Installing the `mysqlclient` library on OS X can be tricky - I've found [this recipe](https://gist.github.com/simonw/90ac0afd204cd0d6d9c3135c3888d116) to work (run that before installing `db-to-sqlite`).

  For PostgreSQL, use this:

      pip install 'db-to-sqlite[postgresql]'

  ## Usage

      Usage: db-to-sqlite [OPTIONS] CONNECTION PATH

        Load data from any database into SQLite.

        PATH is a path to the SQLite file to create, e.c. /tmp/my_database.db

        CONNECTION is a SQLAlchemy connection string, for example:

            postgresql://localhost/my_database
            postgresql://username:passwd@localhost/my_database

            mysql://root@localhost/my_database
            mysql://username:passwd@localhost/my_database

        More: https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls

      Options:
        --version                     Show the version and exit.
        --all                         Detect and copy all tables
        --table TEXT                  Specific tables to copy
        --skip TEXT                   When using --all skip these tables
        --redact TEXT...              (table, column) pairs to redact with ***
        --sql TEXT                    Optional SQL query to run
        --output TEXT                 Table in which to save --sql query results
        --pk TEXT                     Optional column to use as a primary key
        --index-fks / --no-index-fks  Should foreign keys have indexes? Default on
        -p, --progress                Show progress bar
        --postgres-schema TEXT        PostgreSQL schema to use
        --help                        Show this message and exit.

  For example, to save the content of the `blog_entry` table from a PostgreSQL database to a local file called `blog.db` you could do this:

      db-to-sqlite "postgresql://localhost/myblog" blog.db \
          --table=blog_entry

  You can specify `--table` more than once.

  You can also save the data from all of your tables, effectively creating a SQLite copy of your entire database. Any foreign key relationships will be detected and added to the SQLite database. For example:

      db-to-sqlite "postgresql://localhost/myblog" blog.db \
          --all

  When running `--all` you can specify tables to skip using `--skip`:

      db-to-sqlite "postgresql://localhost/myblog" blog.db \
          --all \
          --skip=django_migrations

  If you want to save the results of a custom SQL query, do this:

      db-to-sqlite "postgresql://localhost/myblog" output.db \
          --output=query_results \
          --sql="select id, title, created from blog_entry" \
          --pk=id

  The `--output` option specifies the table that should contain the results of the query.

  ## Using db-to-sqlite with PostgreSQL schemas

  If the tables you want to copy from your PostgreSQL database aren't in the default schema, you can specify an alternate one with the `--postgres-schema` option:

      db-to-sqlite "postgresql://localhost/myblog" blog.db \
          --all \
          --postgres-schema my_schema

  ## Using db-to-sqlite with Heroku Postgres

  If you run an application on [Heroku](https://www.heroku.com/) using their [Postgres database product](https://www.heroku.com/postgres), you can use the `heroku config` command to access a compatible connection string:

      $ heroku config --app myappname | grep HEROKU_POSTG
      HEROKU_POSTGRESQL_OLIVE_URL: postgres://username:password@ec2-xxx-xxx-xxx-x.compute-1.amazonaws.com:5432/dbname

  You can pass this to `db-to-sqlite` to create a local SQLite database with the data from your Heroku instance.

  You can even do this using a bash one-liner:

      $ db-to-sqlite $(heroku config --app myappname | grep HEROKU_POSTG | cut -d: -f 2-) \
          /tmp/heroku.db --all -p
      1/23: django_migrations
      ...
      17/23: blog_blogmark
      [####################################]  100%
      ...

  ## Related projects

  * [Datasette](https://github.com/simonw/datasette): A tool for exploring and publishing data. Works great with SQLite files generated using `db-to-sqlite`.
  * [sqlite-utils](https://github.com/simonw/sqlite-utils): Python CLI utility and library for manipulating SQLite databases.
  * [csvs-to-sqlite](https://github.com/simonw/csvs-to-sqlite): Convert CSV files into a SQLite database.

  ## Development

  To set up this tool locally, first checkout the code. Then create a new virtual environment:

      cd db-to-sqlite
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

  This will skip tests against MySQL or PostgreSQL if you do not have their additional dependencies installed.

  You can install those extra dependencies like so:

      pip install -e '.[test_mysql,test_postgresql]'

  You can alternative use `pip install psycopg2-binary` if you cannot install the `psycopg2` dependency used by the `test_postgresql` extra.

  See [Running a MySQL server using Homebrew](https://til.simonwillison.net/homebrew/mysql-homebrew) for tips on running the tests against MySQL on macOS, including how to install the `mysqlclient` dependency.

  The PostgreSQL and MySQL tests default to expecting to run against servers on localhost. You can use environment variables to point them at different test database servers:

  - `MYSQL_TEST_DB_CONNECTION` - defaults to `mysql://root@localhost/test_db_to_sqlite`
  - `POSTGRESQL_TEST_DB_CONNECTION` - defaults to `postgresql://localhost/test_db_to_sqlite`

  The database you indicate in the environment variable - `test_db_to_sqlite` by default - will be deleted and recreated on every test run.

project_urls:
  {"CI": "https://travis-ci.com/simonw/db-to-sqlite", "Changelog": "https://github.com/simonw/db-to-sqlite/releases", "Documentation": "https://github.com/simonw/db-to-sqlite/blob/main/README.md", "Homepage": "https://github.com/simonw/db-to-sqlite", "Issues": "https://github.com/simonw/db-to-sqlite/issues", "Source code": "https://github.com/simonw/db-to-sqlite"}

requires_dist:
  ["sqlalchemy", "sqlite-utils (>=2.9.1)", "click", "mysqlclient ; extra == 'mysql'", "psycopg2 ; extra == 'postgresql'", "pytest ; extra == 'test'", "pytest ; extra == 'test_mysql'", "mysqlclient ; extra == 'test_mysql'", "pytest ; extra == 'test_postgresql'", "psycopg2 ; extra == 'test_postgresql'"]


name: dbf-to-sqlite
summary: CLCLI tool for converting DBF files (dBase, FoxPro etc) to SQLite
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/dbf-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/dbf-to-sqlite/
project_url: https://pypi.org/project/dbf-to-sqlite/
project_urls: {"Homepage": "https://github.com/simonw/dbf-to-sqlite"}
release_url: https://pypi.org/project/dbf-to-sqlite/0.1/
requires_dist: ["dbf (==0.97.11)", "click", "sqlite-utils"]
version: 0.1
classifiers:
  ["Development Status :: 3 - Alpha", "Intended Audience :: Developers", "Intended Audience :: End Users/Desktop", "Intended Audience :: Science/Research", "License :: OSI Approved :: Apache Software License", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Topic :: Database"]

description:
  # dbf-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/dbf-to-sqlite.svg)](https://pypi.python.org/pypi/dbf-to-sqlite)
  [![Travis CI](https://travis-ci.com/simonw/dbf-to-sqlite.svg?branch=master)](https://travis-ci.com/simonw/dbf-to-sqlite)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/dbf-to-sqlite/blob/master/LICENSE)


  CLI tool for converting DBF files (dBase, FoxPro etc) to SQLite.

      $ dbf-to-sqlite --help
      Usage: dbf-to-sqlite [OPTIONS] DBF_PATHS... SQLITE_DB

        Convert DBF files (dBase, FoxPro etc) to SQLite

        https://github.com/simonw/dbf-to-sqlite

      Options:
        --version      Show the version and exit.
        --table TEXT   Table name to use (only valid for single files)
        -v, --verbose  Show what's going on
        --help         Show this message and exit.

  Example usage:

      $ dbf-to-sqlite *.DBF database.db

  This will create a new SQLite database called `database.db` containing one table for each of the `DBF` files in the current directory.

  Looking for DBF files to try this out on? Try downloading the [Himalayan Database](http://himalayandatabase.com/) of all expeditions that have climbed in the Nepal Himalaya.


name: django-sql-dashboard
summary: Django app for building dashboards using raw SQL queries
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/django-sql-dashboard
license: Apache License, Version 2.0
package_url: https://pypi.org/project/django-sql-dashboard/
project_url: https://pypi.org/project/django-sql-dashboard/
release_url: https://pypi.org/project/django-sql-dashboard/1.1/
requires_python: >=3.6
version: 1.1
description:
  # django-sql-dashboard

  [![PyPI](https://img.shields.io/pypi/v/django-sql-dashboard.svg)](https://pypi.org/project/django-sql-dashboard/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/django-sql-dashboard?include_prereleases&label=changelog)](https://github.com/simonw/django-sql-dashboard/releases)
  [![Tests](https://github.com/simonw/django-sql-dashboard/workflows/Test/badge.svg)](https://github.com/simonw/django-sql-dashboard/actions?query=workflow%3ATest)
  [![Documentation Status](https://readthedocs.org/projects/django-sql-dashboard/badge/?version=latest)](http://django-sql-dashboard.datasette.io/en/latest/?badge=latest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/django-sql-dashboard/blob/main/LICENSE)

  Django SQL Dashboard provides an authenticated interface for executing read-only SQL queries directly against your PostgreSQL database, bringing a useful subset of [Datasette](https://datasette.io/) to Django.

  Applications include ad-hoc analysis and debugging, plus the creation of reporting dashboards that can be shared with team members or published online.

  See my blog for [more about this project](https://simonwillison.net/2021/May/10/django-sql-dashboard/), including [a video demo](https://www.youtube.com/watch?v=ausrmMZkPEY).

  Features include:

  - Safely run read-only one or more SQL queries against your database and view the results in your browser
  - Bookmark queries and share those links with other members of your team
  - Create [saved dashboards](https://django-sql-dashboard.datasette.io/en/latest/saved-dashboards.html) from your queries, with full control over who can view and edit them
  - [Named parameters](https://django-sql-dashboard.datasette.io/en/latest/sql.html#sql-parameters) such as `select * from entries where id = %(id)s` will be turned into form fields, allowing quick creation of interactive dashboards
  - Produce [bar charts](https://django-sql-dashboard.datasette.io/en/latest/widgets.html#bar-label-bar-quantity), [progress bars](https://django-sql-dashboard.datasette.io/en/latest/widgets.html#total-count-completed-count) and more from SQL queries, with the ability to easily create new [custom dashboard widgets](https://django-sql-dashboard.datasette.io/en/latest/widgets.html#custom-widgets) using the Django template system
  - Write SQL queries that safely construct and render [markdown](https://django-sql-dashboard.datasette.io/en/latest/widgets.html#markdown) and [HTML](https://django-sql-dashboard.datasette.io/en/latest/widgets.html#html)
  - Export the full results of a SQL query as a downloadable CSV or TSV file, using a combination of Django's [streaming HTTP response](https://docs.djangoproject.com/en/3.2/ref/request-response/#django.http.StreamingHttpResponse) mechanism and PostgreSQL [server-side cursors](https://www.psycopg.org/docs/usage.html#server-side-cursors) to efficiently stream large amounts of data without running out of resources
  - Copy and paste the results of SQL queries directly into tools such as Google Sheets or Excel
  - Uses Django's authentication system, so dashboard accounts can be granted using Django's Admin tools

  ## Documentation

  Full documentation is at [django-sql-dashboard.datasette.io](https://django-sql-dashboard.datasette.io/)

  ## Screenshot

  <img width="1018" alt="Screenshot showing a SQL query that produces a table and one that produces a bar chart" src="https://user-images.githubusercontent.com/9599/124050883-42ad2300-d9d0-11eb-83e6-44ad85f7ef64.png">

  ## Alternatives

  - [django-sql-explorer](https://github.com/groveco/django-sql-explorer) provides a related set of functionality that also works against database backends other than PostgreSQL

project_urls:
  {"CI": "https://github.com/simonw/django-sql-dashboard/actions", "Changelog": "https://github.com/simonw/django-sql-dashboard/releases", "Documentation": "https://django-sql-dashboard.datasette.io/", "Homepage": "https://github.com/simonw/django-sql-dashboard", "Issues": "https://github.com/simonw/django-sql-dashboard/issues"}

requires_dist:
  ["Django (>=3.0)", "markdown", "bleach", "black (>=22.3.0) ; extra == 'test'", "psycopg2 ; extra == 'test'", "pytest ; extra == 'test'", "pytest-django (==4.2.0) ; extra == 'test'", "pytest-pythonpath ; extra == 'test'", "dj-database-url ; extra == 'test'", "testing.postgresql ; extra == 'test'", "beautifulsoup4 ; extra == 'test'", "html5lib ; extra == 'test'"]


name: dogsheep-beta
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/dogsheep/beta
license: Apache License, Version 2.0
package_url: https://pypi.org/project/dogsheep-beta/
project_url: https://pypi.org/project/dogsheep-beta/
release_url: https://pypi.org/project/dogsheep-beta/0.10.2/
version: 0.10.2
summary:
  Build a search index across content from multiple SQLite database tables and run faceted searches against it using Datasette

description:
  # dogsheep-beta

  [![PyPI](https://img.shields.io/pypi/v/dogsheep-beta.svg)](https://pypi.org/project/dogsheep-beta/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/beta?include_prereleases&label=changelog)](https://github.com/dogsheep/beta/releases)
  [![Tests](https://github.com/dogsheep/beta/workflows/Test/badge.svg)](https://github.com/dogsheep/beta/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/beta/blob/main/LICENSE)

  Build a search index across content from multiple SQLite database tables and run faceted searches against it using Datasette

  ## Example

  A live example of this plugin is running at https://datasette.io/-/beta - configured using [this YAML file](https://github.com/simonw/datasette.io/blob/main/templates/dogsheep-beta.yml).

  Read more about how this example works in [Building a search engine for datasette.io](https://simonwillison.net/2020/Dec/19/dogsheep-beta/).

  ## Installation

  Install this tool like so:

      $ pip install dogsheep-beta

  ## Usage

  Run the indexer using the `dogsheep-beta` command-line tool:

      $ dogsheep-beta index dogsheep.db config.yml

  The `config.yml` file contains details of the databases and document types that should be indexed:

  ```yaml
  twitter.db:
      tweets:
          sql: |-
              select
                  tweets.id as key,
                  'Tweet by @' || users.screen_name as title,
                  tweets.created_at as timestamp,
                  tweets.full_text as search_1
              from tweets join users on tweets.user = users.id
      users:
          sql: |-
              select
                  id as key,
                  name || ' @' || screen_name as title,
                  created_at as timestamp,
                  description as search_1
              from users
  ```

  This will create a `search_index` table in the `dogsheep.db` database populated by data from those SQL queries.

  By default the search index that this tool creates will be configured for Porter stemming. This means that searches for words like `run` will match documents containing `runs` or `running`.

  If you don't want to use Porter stemming, use the `--tokenize none` option:

      $ dogsheep-beta index dogsheep.db config.yml --tokenize none

  You can pass other SQLite tokenize argumenst here, see [the SQLite FTS tokenizers documentation](https://www.sqlite.org/fts5.html#tokenizers).

  ## Columns

  The columns that can be returned by our query are:

  - `key` - a unique (within that type) primary key
  - `title` - the title for the item
  - `timestamp` - an ISO8601 timestamp, e.g. `2020-09-02T21:00:21`
  - `search_1` - a larger chunk of text to be included in the search index
  - `category` - an integer category ID, see below
  - `is_public` - an integer (0 or 1, defaults to 0 if not set) specifying if this is public or not

  Public records are things like your public tweets, blog posts and GitHub commits.

  ## Categories

  Indexed items can be assigned a category. Categories are integers that correspond to records in the `categories` table, which defaults to containing the following:

  |   id | name       |
  |------|------------|
  |    1 | created    |
  |    2 | saved      |
  |    3 | received   |

  `created` is for items that have been created by the Dogsheep instance owner.

  `saved` is for items that they have saved, liked or favourited.

  `received` is for items that have been specifically sent to them by other people - incoming emails or direct messages for example.

  ## Datasette plugin

  Run `datasette install dogsheep-beta` (or use `pip install dogsheep-beta` in the same environment as Datasette) to install the Dogsheep Beta Datasette plugin.

  Once installed, a custom search interface will be made available at `/-/beta`. You can use this interface to execute searches.

  The Datasette plugin has some configuration options. You can set these by adding the following to your `metadata.json` configuration file:

  ```json
  {
      "plugins": {
          "dogsheep-beta": {
              "database": "beta",
              "config_file": "dogsheep-beta.yml",
              "template_debug": true
          }
      }
  }
  ```
  The configuration settings for the plugin are:
  - `database` - the database file that contains your search index. If the file is `beta.db` you should set `database` to `beta`.
  - `config_file` - the YAML file containing your Dogsheep Beta configuration.
  - `template_debug` - set this to `true` to enable debugging output if errors occur in your custom templates, see below.

  ## Custom results display

  Each indexed item type can define custom display HTML as part of the `config.yml` file. It can do this using a `display` key containing a fragment of Jinja template, and optionally a `display_sql` key with extra SQL to execute to fetch the data to display.

  Here's how to define a custom display template for a tweet:

  ```yaml
  twitter.db:
      tweets:
          sql: |-
              select
                  tweets.id as key,
                  'Tweet by @' || users.screen_name as title,
                  tweets.created_at as timestamp,
                  tweets.full_text as search_1
              from tweets join users on tweets.user = users.id
          display: |-
              <p>{{ title }} - tweeted at {{ timestamp }}</p>
              <blockquote>{{ search_1 }}</blockquote>
  ```
  This example reuses the value that were stored in the `search_index` table when the indexing query was run.

  To load in extra values to display in the template, use a `display_sql` query like this:

  ```yaml
  twitter.db:
      tweets:
          sql: |-
              select
                  tweets.id as key,
                  'Tweet by @' || users.screen_name as title,
                  tweets.created_at as timestamp,
                  tweets.full_text as search_1
              from tweets join users on tweets.user = users.id
          display_sql: |-
              select
                  users.screen_name,
                  tweets.full_text,
                  tweets.created_at
              from
                  tweets join users on tweets.user = users.id
              where
                  tweets.id = :key
          display: |-
              <p>{{ display.screen_name }} - tweeted at {{ display.created_at }}</p>
              <blockquote>{{ display.full_text }}</blockquote>
  ```
  The `display_sql` query will be executed for every search result, passing the key value from the `search_index` table as the `:key` parameter and the user's search term as the `:q` parameter.

  This performs well because [many small queries are efficient in SQLite](https://www.sqlite.org/np1queryprob.html).

  If an error occurs while rendering one of your templates the search results page will return a 500 error. You can use the `template_debug` configuration setting described above to instead output debugging information for the search results item that experienced the error.

  ## Displaying maps

  This plugin will eventually include a number of useful shortcuts for rendering interesting content.

  The first available shortcut is for displaying maps. Make your custom content output something like this:

  ```html
  <div
      data-map-latitude="{{ display.latitude }}"
      data-map-longitude="{{ display.longitude }}"
      style="display: none; float: right; width: 250px; height: 200px; background-color: #ccc;"
  ></div>
  ```
  JavaScript on the page will look for any elements with `data-map-latitude` and `data-map-longitude` and, if it finds any, will load Leaflet and convert those elements into maps centered on that location. The default zoom level will be 12, or you can set a `data-map-zoom` attribute to customize this.

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd dogsheep-beta
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/dogsheep/beta/actions", "Changelog": "https://github.com/dogsheep/beta/releases", "Homepage": "https://github.com/dogsheep/beta", "Issues": "https://github.com/dogsheep/beta/issues"}

requires_dist:
  ["datasette (>=0.50.2)", "click", "PyYAML", "sqlite-utils (>=3.0)", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'", "httpx ; extra == 'test'", "beautifulsoup4 ; extra == 'test'", "html5lib ; extra == 'test'"]


name: dogsheep-photos
summary: Save details of your photos to a SQLite database and upload them to S3
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/dogsheep/dogsheep-photos
license: Apache License, Version 2.0
package_url: https://pypi.org/project/dogsheep-photos/
project_url: https://pypi.org/project/dogsheep-photos/
project_urls: {"Homepage": "https://github.com/dogsheep/dogsheep-photos"}
release_url: https://pypi.org/project/dogsheep-photos/0.4.1/
version: 0.4.1
description:
  # dogsheep-photos

  [![PyPI](https://img.shields.io/pypi/v/dogsheep-photos.svg)](https://pypi.org/project/dogsheep-photos/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/dogsheep-photos?include_prereleases&label=changelog)](https://github.com/dogsheep/dogsheep-photos/releases)
  [![CircleCI](https://circleci.com/gh/dogsheep/dogsheep-photos.svg?style=svg)](https://circleci.com/gh/dogsheep/dogsheep-photos)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/dogsheep-photos/blob/master/LICENSE)

  Save details of your photos to a SQLite database and upload them to S3.

  See [Using SQL to find my best photo of a pelican according to Apple Photos](https://simonwillison.net/2020/May/21/apple-photos-sqlite/) for background information on this project.

  ## What these tools do

  These tools are a work-in-progress mechanism for taking full ownership of your photos. The core idea is to help implement the following:

  * Every photo you have taken lives in a single, private Amazon S3 bucket
  * You have a single SQLite database file which stores metadata about those photos - potentially pulled from multiple different places. This may include EXIF data, Apple Photos, the results of running machine learning APIs against photos and much more besides.
  * You can then use [Datasette](https://github.com/simonw/datasette) to explore your own photos.

  I'm a heavy user of Apple Photos so the initial releases of this tool will have a bias towards that, but ideally I would like a subset of these tools to be useful to people no matter which core photo solution they are using.

  ## Installation

      $ pip install dogsheep-photos

  ## Authentication (if using S3)

  If you want to use S3 to store your photos, you will need to first create S3 credentials for a new, dedicated bucket.

  This is a big pain. Here's [how I did it](https://github.com/dogsheep/dogsheep-photos/issues/4).

  Run this command and paste in your credentials. You will need three values: the name of your S3 bucket, your Access key ID and your Secret access key.

      $ dogsheep-photos s3-auth

  This will create a file called `auth.json` in your current directory containing the required values. To save the file at a different path or filename, use the `--auth=myauth.json` option.

  ## Uploading photos

  Run this command to upload every photo in a specific directory to your S3 bucket:

      $ dogsheep-photos upload photos.db \
          ~/Pictures/Photos\ Library.photoslibrary/original

  The command will only upload photos that have not yet been uploaded, based on their sha256 hash.

  `photos.db` will be created with an `uploads` table containing details of which files were uploaded.

  To see what the command would do without uploading any files, use the `--dry-run` option.

  The sha256 hash of the photo contents will be used as the name of the file in the bucket, with an extension matching the type of file. This is an implementation of the [Content addressable storage](https://en.wikipedia.org/wiki/Content-addressable_storage) pattern.

  ## Importing Apple Photos metadata

  The `apple-photos` command imports metadata from your Apple Photos library.

      $ photo-to-sqlite apple-photos photos.db

  Imported metadata includes places, people, albums, quality scores and machine learning labels for the photo contents.

  ## Creating a subset database

  You can create a new, subset database of photos using the `create-subset` command.

  This is useful for creating a shareable SQLite database that only contains metadata for a selected set of photos.

  Since photo metadata contains latitude and longitude you may not want to share a database that includes photos taken at your home address.

  `create-subset` takes three arguments: an existing database file created using the `apple-photos` command, the name of the new, shareable database file you would like to create and a SQL query that returns the `sha256` hash values of the photos you would like to include in that database.

  For example, here's how to create a shareable database of just the photos that have been added to albums containing the word "Public":

      $ dogsheep-photos create-subset \
          photos.db \
          public.db \
          "select sha256 from apple_photos where albums like '%Public%'"

  ## Serving photos locally with datasette-media

  If you don't want to upload your photos to S3 but you still want to browse them using Datasette you can do so using the [datasette-media](https://github.com/simonw/datasette-media) plugin. This plugin adds the ability to serve images and other static files directly from disk, configured using a SQL query.

  To use it, first install Datasette and the plugin:

      $ pip install datasette datasette-media

  If any of your photos are `.HEIC` images taken by an iPhone you should also install the optional `pyheif` dependency:

      $ pip install pyheif

  Now create a `metadata.yaml` file configuring the plugin:

  ```yaml
  plugins:
    datasette-media:
      thumbnail:
        sql: |-
          select path as filepath, 200 as resize_height from apple_photos where uuid = :key
      large:
        sql: |-
          select path as filepath, 1024 as resize_height from apple_photos where uuid = :key
  ```
  This will configure two URL endpoins - one for 200 pixel high thumbnails and one for 1024 pixel hight larger images.

  Create your `photos.db` database using the `apple-photos` command, then run Datasette like this:

      $ datasette -m metadata.yaml

  Your photos will be served on URLs that look like this:

      http://127.0.0.1:8001/-/media/thumbnail/F4469918-13F3-43D8-9EC1-734C0E6B60AD
      http://127.0.0.1:8001/-/media/large/F4469918-13F3-43D8-9EC1-734C0E6B60AD

  You can find the UUIDs for use in these URLs by running `select uuid from photos_with_apple_metadata`.

  ### Displaying images using datasette-json-html

  If you are using `datasette-media` to serve photos you can include images directly in Datasette query results using the [datasette-json-html](https://github.com/simonw/datasette-json-html) plugin.

  Run `pip install datasette-json-html` to install the plugin, then use queries like this to view your images:

  ```sql
  select
      json_object(
          'img_src',
          '/-/media/thumbnail/' || uuid
      ) as photo,
      uuid,
      date
  from
      apple_photos
  order by
      date desc
  limit 10;
  ```
  The `photo` column returned by this query should render as image tags that display the correct images.

  ### Displaying images using custom template pages

  Datasette's [custom pages](https://datasette.readthedocs.io/en/stable/custom_templates.html#custom-pages) feature lets you create custom pages for a Datasette instance by dropping HTML templates into a `templates/pages` directory and then running Datasette using `datasette --template-dir=templates/`.

  You can combine that ability with the [datasette-template-sql](https://github.com/simonw/datasette-template-sql) plugin to create custom template pages that directly display photos served by `datasette-media`.

  Install the plugin using `pip install datasette-template-sql`.

  Create a `templates/pages` folder and add the following files:

  `recent-photos.html`
  ```html+jinja
  <h1>Recent photos</h1>

  <div>
  {% for photo in sql("select * from apple_photos order by date desc limit 20") %}
      <img src="/-/media/photo/{{ photo['uuid'] }}">
  {% endfor %}
  </div>
  ```
  `random-photos.html`
  ```html+jinja
  <h1>Random photos</h1>

  <div>
  {% for photo in sql("with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 20") %}
      <img src="/-/media/photo/{{ photo['uuid'] }}">
  {% endfor %}
  </div>
  ```
  Now run Datasette like this:

      $ datasette photos.db -m metadata.yaml --template-dir=templates/

  Visiting `http://localhost:8001/recent-photos` will display 20 recent photos. Visiting `http://localhost:8001/random-photos` will display 20 photos randomly selected from your 5,000 most recent.

requires_dist:
  ["sqlite-utils (>=2.7)", "boto3 (>=1.12.41)", "osxphotos (>=0.28.13) ; sys_platform == \"darwin\"", "pytest ; extra == 'test'"]


name: download-tiles
summary: Download map tiles and store them in an MBTiles database
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/download-tiles
license: Apache License, Version 2.0
package_url: https://pypi.org/project/download-tiles/
project_url: https://pypi.org/project/download-tiles/
release_url: https://pypi.org/project/download-tiles/0.4.1/
requires_python: >=3.6
version: 0.4.1
description:
  # download-tiles

  [![PyPI](https://img.shields.io/pypi/v/download-tiles.svg)](https://pypi.org/project/download-tiles/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/download-tiles?include_prereleases&label=changelog)](https://github.com/simonw/download-tiles/releases)
  [![Tests](https://github.com/simonw/download-tiles/workflows/Test/badge.svg)](https://github.com/simonw/download-tiles/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/download-tiles/blob/master/LICENSE)

  Download map tiles and store them in an MBTiles database

  ## Installation

  Install this tool using `pip`:

      $ pip install download-tiles

  ## Usage

  This tool downloads tiles from a specified [TMS (Tile Map Server)](https://wiki.openstreetmap.org/wiki/TMS) server for a specified bounding box and range of zoom levels and stores those tiles in a MBTiles SQLite database. It is a command-line wrapper around the [Landez](https://github.com/makinacorpus/landez) Python libary.

  **Please use this tool responsibly**. Consult the usage policies of the tile servers you are interacting with, for example the [OpenStreetMap Tile Usage Policy](https://operations.osmfoundation.org/policies/tiles/).

  Running the following will download zoom levels 0-3 of OpenStreetMap, 85 tiles total, and store them in a SQLite database called `world.mbtiles`:

      download-tiles world.mbtiles

  You can customize which tile and zoom levels are downloaded using command options:

  `--zoom-levels=0-3` or `-z=0-3`

  The different zoom levels to download. Specify a single number, e.g. `15`, or a range of numbers e.g. `0-4`. Be careful with this setting as you can easily go over the limits requested by the underlying tile server.

  `--bbox=3.9,-6.3,14.5,10.2` or `-b=3.9,-6.3,14.5,10.2`

  The bounding box to fetch. Should be specified as `min-lon,min-lat,max-lon,max-lat`. You can use [bboxfinder.com](http://bboxfinder.com/) to find these for different areas.

  `--city=london` or `--country=madagascar`

  These options can be used instead of `--bbox`. The city or country specified will be looked up using the [Nominatum API](https://nominatim.org/release-docs/latest/api/Search/) and used to derive a bounding box.

  `--show-bbox`

  Use this option to output the bounding box that was retrieved for the `--city` or `--country` without downloading any tiles.

  `--name=Name`

  A name for this tile collection, used for the `name` field in the `metadata` table. If not specified a UUID will be used, or if you used `--city` or `--country` the name will be set to the full name of that place.

  `--attribution="Attribution string"`

  Attribution string to bake into the `metadata` table. This will default to `© OpenStreetMap contributors` unless you use `--tiles-url` to specify an alternative tile server, in which case you should specify a custom attribution string.

  You can use the `--attribution=osm` shortcut to specify the `© OpenStreetMap contributors` value without having to type it out in full.

  `--tiles-url=https://...`

  The tile server URL to use. This should include `{z}` and `{x}` and `{y}` specifiers, and can optionally include `{s}` for subdomains.

  The default URL used here is for OpenStreetMap, `http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png`

  `--tiles-subdomains=a,b,c`

  A comma-separated list of subdomains to use for the `{s}` parameter.

  `--verbose`

  Use this option to turn on verbose logging.

  `--cache-dir=/tmp/tiles`

  Provide a directory to cache downloaded tiles between runs. This can be useful if you are worried you might not have used the correct options for the bounding box or zoom levels.

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd download-tiles
      python -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/download-tiles/actions", "Changelog": "https://github.com/simonw/download-tiles/releases", "Homepage": "https://github.com/simonw/download-tiles", "Issues": "https://github.com/simonw/download-tiles/issues"}

requires_dist:
  ["click", "requests", "landez (==2.5.0)", "pytest ; extra == 'test'", "requests-mock ; extra == 'test'"]


name: evernote-to-sqlite
summary: Tools for converting Evernote content to SQLite
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/dogsheep/evernote-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/evernote-to-sqlite/
project_url: https://pypi.org/project/evernote-to-sqlite/
release_url: https://pypi.org/project/evernote-to-sqlite/0.3.2/
requires_dist: ["click", "sqlite-utils (>=3.0)", "pytest ; extra == 'test'"]
version: 0.3.2
description:
  # evernote-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/evernote-to-sqlite.svg)](https://pypi.org/project/evernote-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/evernote-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/evernote-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/evernote-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/evernote-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/evernote-to-sqlite/blob/master/LICENSE)

  Tools for converting Evernote content to SQLite. See [Building an Evernote to SQLite exporter](https://simonwillison.net/2020/Oct/16/building-evernote-sqlite-exporter/) for background on this project.

  ## Installation

  Install this tool using `pip`:

      $ pip install evernote-to-sqlite

  ## Usage

  Currently the only available command is `evernote-to-sqlite enex`, which converts Evernote's ENEX export files into a SQLite database.

  You can create [an ENEX export](https://help.evernote.com/hc/en-us/articles/209005557-Export-notes-and-notebooks-as-ENEX-or-HTML) in the Evernote desktop application by selecting some notes (or all of your notes) and using the `File -> Export Notes...` menu option.

  This used to be able to export everything in one go, but it looks like more recent Evernote versions only allow exporting up to fifty notes at a time, or let you export an entire notebook by right-clicking on the notebook and selecting "Export notebook...".

  You can convert that file to SQLite like so:

      $ evernote-to-sqlite enex evernote.db MyNotes.enex

  This will display a progress bar and create a SQLite database file called `evernote.db`.

  ### Limitations

  Unfortunately the ENEX export format does not include a unique identifier for each note. This means you cannot use this tool to re-import notes after they have been updated - you should consider this tool to be a one-time transformation of an ENEX file into an equivalent SQLite database.

  ENEX exports also do not include details of which notebook a note belongs to.

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd evernote-to-sqlite
      python -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/dogsheep/evernote-to-sqlite/actions", "Changelog": "https://github.com/dogsheep/evernote-to-sqlite/releases", "Homepage": "https://github.com/dogsheep/evernote-to-sqlite", "Issues": "https://github.com/dogsheep/evernote-to-sqlite/issues"}


name: fec-to-sqlite
summary: Save FEC campaign finance data to a SQLite database
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/dogsheep/fec-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/fec-to-sqlite/
project_url: https://pypi.org/project/fec-to-sqlite/
project_urls: {"Homepage": "https://github.com/dogsheep/fec-to-sqlite"}
release_url: https://pypi.org/project/fec-to-sqlite/0.2/
requires_dist: ["sqlite-utils", "click", "requests", "fecfile", "tqdm", "pytest ; extra == 'test'"]
version: 0.2
description:
  # fec-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/fec-to-sqlite.svg)](https://pypi.org/project/fec-to-sqlite/)
  [![CircleCI](https://circleci.com/gh/simonw/fec-to-sqlite.svg?style=svg)](https://circleci.com/gh/simonw/fec-to-sqlite)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/fec-to-sqlite/blob/master/LICENSE)

  Create a SQLite database using FEC campaign contributions data.

  This tool builds on [fecfile](https://github.com/esonderegger/) by Evan Sonderegger.

  ## How to install

      $ pip install fec-to-sqlite

  ## Usage

      $ fec-to-sqlite filings filings.db 1146148

  This fetches the filing with ID `1146148` and stores it in tables in a SQLite database called `filings.db`. It will create any tables it needs.

  You can pass more than one filing ID, separated by spaces.


name: genome-to-sqlite
summary: Import your genome into a SQLite database
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/dogsheep/genome-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/genome-to-sqlite/
project_url: https://pypi.org/project/genome-to-sqlite/
project_urls: {"Homepage": "https://github.com/dogsheep/genome-to-sqlite"}
release_url: https://pypi.org/project/genome-to-sqlite/0.1/
requires_dist: ["sqlite-utils", "pytest ; extra == 'test'"]
version: 0.1
description:
  # genome-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/genome-to-sqlite.svg)](https://pypi.org/project/genome-to-sqlite/)
  [![CircleCI](https://circleci.com/gh/dogsheep/genome-to-sqlite.svg?style=svg)](https://circleci.com/gh/dogsheep/genome-to-sqlite)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/genome-to-sqlite/blob/master/LICENSE)

  Import your genome into a SQLite database.

  ## How to install

      $ pip install genome-to-sqlite

  ## How to use

  First, export your genome. This tool has only been tested against 23andMe so far. You can request an export of your genome from https://you.23andme.com/tools/data/download/

  Now you can convert the resulting `export.zip` file to SQLite like so:

      $ genome-to-sqlite export.zip genome.db

  A progress bar will be displayed. You can disable this using `--silent`.

  ```
  Importing genome  [#----------------]    5%  00:01:33
  ```

  You can explore the resulting data using [Datasette](https://datasette.readthedocs.io/) like this:

      $ datasette genome.db --config facet_time_limit_ms:1000

  Bumping up the facet time limit is useful in order to enable faceting by chromosome:

  http://127.0.0.1:8001/genome/genome?_facet=chromosome&_sort=position


name: geocode-sqlite
summary: Geocode rows from a SQLite table
classifiers: []
author: Chris Amico
description_content_type: text/markdown
home_page: https://github.com/eyeseast/geocode-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/geocode-sqlite/
project_url: https://pypi.org/project/geocode-sqlite/
release_url: https://pypi.org/project/geocode-sqlite/0.8.2/
version: 0.8.2
description:
  # geocode-sqlite

  [![PyPI](https://img.shields.io/pypi/v/geocode-sqlite.svg)](https://pypi.org/project/geocode-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/eyeseast/geocode-sqlite?include_prereleases&label=changelog)](https://github.com/eyeseast/geocode-sqlite/releases)
  [![Tests](https://github.com/eyeseast/geocode-sqlite/workflows/Test/badge.svg)](https://github.com/eyeseast/geocode-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/eyeseast/geocode-sqlite/blob/master/LICENSE)

  Geocode rows from a SQLite table

  ## Installation

  Install this tool using `pip` or `pipx`:

  ```sh
  # install inside a virtualenv
  pip install geocode-sqlite

  # install globally
  pipx install geocode-sqlite
  ```

  ## Usage

  Let's say you have a spreadsheet with addresses in it, and you'd like to map those locations.
  First, create a SQLite database and insert rows from that spreadsheet using `sqlite-utils`.

  ```sh
  sqlite-utils insert data.db data data.csv --csv
  ```

  Now, geocode it using OpenStreetMap's Nominatim geocoder.

  ```sh
  geocode-sqlite nominatim data.db data \
   --location="{address}, {city}, {state} {zip}" \
   --delay=1 \
   --user-agent="this-is-me"
  ```

  In the command above, you're using Nominatim, which is free and only asks for a unique user agent (`--user-agent`).

  This will connect to a database (`data.db`) and read all rows from the table `data` (skipping any that already
  have both a `latitude` and `longitude` column filled).

  You're also telling the geocoder how to extract a location query (`--location`) from a row of data, using Python's
  built-in string formatting, and setting a rate limit (`--delay`) of one request per second.

  For each row where geocoding succeeds, `latitude` and `longitude` will be populated. If you hit an error, or a rate limit,
  run the same query and pick up where you left off.

  The resulting table layout can be visualized with [datasette-cluster-map](https://datasette.io/plugins/datasette-cluster-map).

  Under the hood, this package uses the excellent [geopy](https://geopy.readthedocs.io/en/latest/) library, which is stable and thoroughly road-tested. If you need help understanding a particular geocoder's options, consult [geopy's documentation](https://geopy.readthedocs.io/en/latest/#module-geopy.geocoders).

  ### Supported Geocoders

  The CLI currently supports these geocoders:

  - `bing`
  - `googlev3`
  - `mapquest` (and `open-mapquest`)
  - `mapbox`
  - `nominatim`
  - `opencage`

  #### Adding new geocoders

  1. Open an issue with the name of the geocoding service as the ticket title ([example](https://github.com/eyeseast/geocode-sqlite/issues/35)). Put any noteworthy implementation details in the ticket body, like where to get an API key if one is required.
  2. Fork the repo and add a geocoder.
  3. Add an example to the `Makefile`. Add tests if there's new shared functionality.

  ### Common arguments and options

  Each geocoder needs to know where to find the data it's working with. These are the first two arguments:

  - `database`: a path to a SQLite file, which must already exist
  - `table`: the name of a table, in that database, which exists and has data to geocode

  From there, we have a set of options passed to every geocoder:

  - `location`: a [string format](https://docs.python.org/3/library/stdtypes.html#str.format) that will be expanded with each row to build a full query, to be geocoded
  - `delay`: a delay between each call (some services require this)
  - `latitude`: latitude column name
  - `longitude`: longitude column name
  - `geojson`: store results as GeoJSON, instead of in latitude and longitude columns
  - `spatialite`: store results in a SpatiaLite geometry column, instead of in latitude and longitude columns
  - `raw`: store raw geocoding results in a JSON column

  Each geocoder takes additional, specific arguments beyond these, such as API keys. Again, [geopy's documentation](https://geopy.readthedocs.io/en/latest/#module-geopy.geocoders) is an excellent resource.

  ## Using SpatiaLite

  The `--spatialite` flag will store results in a [geometry column](https://www.gaia-gis.it/gaia-sins/spatialite-cookbook-5/cookbook_topics.adminstration.html#topic_TABLE_to_SpatialTable), instead of `latitude` and `longitude` columns. This is useful if you're doing other GIS operations, such as using a [spatial index](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=SpatialIndex). See the [SpatiaLite cookbook](https://www.gaia-gis.it/gaia-sins/spatialite-cookbook-5/index.html) and [functions list](https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html) for more of what's possible.

  ## Capturing additional geocoding data

  Geocoding services typically return more data than just coordinates. This might include accuracy, normalized addresses or other context. This can be captured using the `--raw` flag. By default, this will add a `raw` column and store the full geocoding response as JSON. If you want to rename that column, pass a value, like `--raw custom_raw`.

  The shape of this response object will vary between services. You can query specific values using [SQLite's built-in JSON functions](https://www.sqlite.org/json1.html). For example, this will work with Google's geocoder:

  ```sql
  select
    json_extract(raw, '$.formatted_address') as address,
    json_extract(raw, '$.geometry.location_type') as location_type
  from
    innout_test
  ```

  Check each geocoding service's documentation for what's included in the response.

  ## Python API

  The command line interface aims to support the most common options for each geocoder. For more fine-grained control, use the Python API.

  As with the CLI, this assumes you already have a SQLite database and a table of location data.

  ```python
  from geocode_sqlite import geocode_table
  from geopy.geocoders import Nominatim

  # create a geocoder instance, with some extra options
  nominatim = Nominatim(user_agent="this-is-me", domain="nominatim.local.dev", scheme="http")

  # assuming our database is in the same directory
  count = geocode_table("data.db", "data", query_template="{address}, {city}, {state} {zip}")

  # when it's done
  print(f"Geocoded {count} rows")
  ```

  Any [geopy geocoder](https://geopy.readthedocs.io/en/latest/#module-geopy.geocoders) can be used with the Python API.

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

  ```sh
  cd geocode-sqlite
  python -m venv .venv
  source .venv/bin/activate
  ```

  Or if you are using `pipenv`:

  ```sh
  pipenv shell
  ```

  Now install the dependencies and tests:

  ```sh
  pip install -e '.[test]'
  ```

  To run the tests:

  ```sh
  pytest
  ```

  Please remember that this library is mainly glue code between other well-tested projects, specifically: [click](https://click.palletsprojects.com/), [geopy](https://geopy.readthedocs.io/en/stable/) and [sqlite-utils](https://sqlite-utils.datasette.io/en/stable/). Tests should focus on making sure those parts fit together correctly. We can assume the parts themselves already work.

  To that end, there is a test geocoder included: `geocode_sqlite.testing.DummyGeocoder`. That geocoder works with an included dataset of In-N-Out Burger locations provided by [AllThePlaces](https://www.alltheplaces.xyz/). It works like a normal GeoPy geocoder, except it will only return results for In-N-Out locations using the included database.

project_urls:
  {"CI": "https://github.com/eyeseast/geocode-sqlite/actions", "Changelog": "https://github.com/eyeseast/geocode-sqlite/releases", "Homepage": "https://github.com/eyeseast/geocode-sqlite", "Issues": "https://github.com/eyeseast/geocode-sqlite/issues"}

requires_dist:
  ["click (>=7.0)", "sqlite-utils", "geopy", "requests", "pytest ; extra == 'test'", "geojson-to-sqlite ; extra == 'test'"]


name: geojson-to-sqlite
summary: CLI tool for converting GeoJSON to SQLite (optionally with SpatiaLite)
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/geojson-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/geojson-to-sqlite/
project_url: https://pypi.org/project/geojson-to-sqlite/
project_urls: {"Homepage": "https://github.com/simonw/geojson-to-sqlite"}
release_url: https://pypi.org/project/geojson-to-sqlite/1.1.1/
requires_dist: ["sqlite-utils (>=3.23)", "shapely", "pytest ; extra == 'test'", "dirty-equals ; extra == 'test'"]
version: 1.1.1
description:
  # geojson-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/geojson-to-sqlite.svg)](https://pypi.org/project/geojson-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/geojson-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/geojson-to-sqlite/releases)
  [![Tests](https://github.com/simonw/geojson-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/geojson-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/geojson-to-sqlite/blob/main/LICENSE)

  CLI tool for converting GeoJSON to SQLite (optionally with SpatiaLite)

  [RFC 7946: The GeoJSON Format](https://tools.ietf.org/html/rfc7946)

  ## How to install

      $ pip install geojson-to-sqlite

  ## How to use

  You can run this tool against a GeoJSON file like so:

      $ geojson-to-sqlite my.db features features.geojson

  This will load all of the features from the `features.geojson` file into a table called `features`.

  Each row will have a `geometry` column containing the feature geometry, and columns for each of the keys found in any `properties` attached to those features. (To bundle all properties into a single JSON object, use the `--properties` flag.)

  The table will be created the first time you run the command.

  On subsequent runs you can use the `--alter` option to add any new columns that are missing from the table.

  You can pass more than one GeoJSON file, in which case the contents of all of the files will be inserted into the same table.

  If your features have an `"id"` property it will be used as the primary key for the table. You can also use `--pk=PROPERTY` with the name of a different property to use that as the primary key instead. If you don't want to use the `"id"` as the primary key (maybe it contains duplicate values) you can use `--pk ''` to specify no primary key.

  Specifying a primary key also will allow you to upsert data into the rows instead of insert data into new rows.

  If no primary key is specified, a SQLite `rowid` column will be used.

  You can use `-` as the filename to import from standard input. For example:

      $ curl https://eric.clst.org/assets/wiki/uploads/Stuff/gz_2010_us_040_00_20m.json \
          | geojson-to-sqlite my.db states - --pk GEO_ID

  ## Using with SpatiaLite

  By default, the `geometry` column will contain JSON.

  If you have installed the [SpatiaLite](https://www.gaia-gis.it/fossil/libspatialite/index) module for SQLite you can instead import the geometry into a geospatially indexed column.

  You can do this using the `--spatialite` option, like so:

      $ geojson-to-sqlite my.db features features.geojson --spatialite

  The tool will search for the SpatiaLite module in the following locations:

  - `/usr/lib/x86_64-linux-gnu/mod_spatialite.so`
  - `/usr/local/lib/mod_spatialite.dylib`

  If you have installed the module in another location, you can use the `--spatialite_mod=xxx` option to specify where:

      $ geojson-to-sqlite my.db features features.geojson \
          --spatialite_mod=/usr/lib/mod_spatialite.dylib

  You can create a SpatiaLite spatial index on the `geometry` column using the `--spatial-index` option:

      $ geojson-to-sqlite my.db features features.geojson --spatial-index

  Using this option implies `--spatialite` so you do not need to add that.

  ## Streaming large datasets

  For large datasets, consider using newline-delimited JSON to stream features into the database without loading the entire feature collection into memory.

  For example, to load a day of earthquake reports from USGS:

      $ geojson-to-sqlite quakes.db quakes tests/quakes.ndjson \
        --nl --pk=id --spatialite

  When using newline-delimited JSON, tables will also be created from the first feature, instead of guessing types based on the first 100 features.

  If you want to use a larger subset of your data to guess column types (for example, if some fields are inconsistent) you can use [fiona](https://fiona.readthedocs.io/en/latest/cli.html) to collect features into a single collection.

      $ head tests/quakes.ndjson | fio collect | \
        geojson-to-sqlite quakes.db quakes - --spatialite

  This will take the first 10 lines from `tests/quakes.ndjson`, pass them to `fio collect`, which turns them into a single feature collection, and pass that, in turn, to `geojson-to-sqlite`.

  ## Using this with Datasette

  Databases created using this tool can be explored and published using [Datasette](https://datasette.readthedocs.io/).

  The Datasette documentation includes a section on [how to use it to browse SpatiaLite databases](https://datasette.readthedocs.io/en/stable/spatialite.html).

  The [datasette-leaflet-geojson](https://datasette.io/plugins/datasette-leaflet-geojson) plugin can be used to visualize columns containing GeoJSON geometries on a [Leaflet](https://leafletjs.com/) map.

  If you are using SpatiaLite you will need to output the geometry as GeoJSON in order for that plugin to work. You can do that using the SpaitaLite `AsGeoJSON()` function - something like this:

  ```sql
  select rowid, AsGeoJSON(geometry) from mytable limit 10
  ```

  The [datasette-geojson-map](https://datasette.io/plugins/datasette-geojson-map) is an alternative plugin which will automatically render SpatiaLite geometries as a Leaflet map on the corresponding table page, without needing you to call `AsGeoJSON(geometry)`.


name: git-history
summary: Tools for analyzing Git history using SQLite
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/git-history
license: Apache License, Version 2.0
package_url: https://pypi.org/project/git-history/
project_url: https://pypi.org/project/git-history/
release_url: https://pypi.org/project/git-history/0.6.1/
requires_python: >=3.6
version: 0.6.1
description:
  # git-history

  [![PyPI](https://img.shields.io/pypi/v/git-history.svg)](https://pypi.org/project/git-history/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/git-history?include_prereleases&label=changelog)](https://github.com/simonw/git-history/releases)
  [![Tests](https://github.com/simonw/git-history/workflows/Test/badge.svg)](https://github.com/simonw/git-history/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/git-history/blob/master/LICENSE)

  Tools for analyzing Git history using SQLite

  For background on this project see [git-history: a tool for analyzing scraped data collected using Git and SQLite](https://simonwillison.net/2021/Dec/7/git-history/)

  ## Installation

  Install this tool using `pip`:

      $ pip install git-history

  ## Demos

  [git-history-demos.datasette.io](http://git-history-demos.datasette.io/) hosts three example databases created using this tool:

  - [pge-outages](https://git-history-demos.datasette.io/pge-outages) shows a history of PG&E (the electricity supplier) [outages](https://pgealerts.alerts.pge.com/outagecenter/), using data collected in [simonw/pge-outages](https://github.com/simonw/pge-outages) converted using [pge-outages.sh](https://github.com/simonw/git-history/blob/main/demos/pge-outages.sh)
  - [ca-fires](https://git-history-demos.datasette.io/ca-fires) shows a history of fires in California reported on [fire.ca.gov/incidents](https://www.fire.ca.gov/incidents/), from data in [simonw/ca-fires-history](https://github.com/simonw/ca-fires-history) converted using [ca-fires.sh](https://github.com/simonw/git-history/blob/main/demos/ca-fires.sh)
  - [sf-bay-511](https://git-history-demos.datasette.io/sf-bay-511) has records of San Francisco Bay Area traffic and transit incident data from [511.org](https://511.org/), collected in [dbreunig/511-events-history](https://github.com/dbreunig/511-events-history) converted using [sf-bay-511.sh](https://github.com/simonw/git-history/blob/main/demos/sf-bay-511.sh)

  The demos are deployed using [Datasette](https://datasette.io/) on [Google Cloud Run](https://cloud.google.com/run/) by [this GitHub Actions workflow](https://github.com/simonw/git-history/blob/main/.github/workflows/deploy-demos.yml).

  ## Usage

  This tool can be run against a Git repository that holds a file that contains JSON, CSV/TSV or some other format and which has multiple versions tracked in the Git history. Read [Git scraping: track changes over time by scraping to a Git repository](https://simonwillison.net/2020/Oct/9/git-scraping/) to understand how you might create such a repository.

  The `file` command analyzes the history of an individual file within the repository, and generates a SQLite database table that represents the different versions of that file over time.

  The file is assumed to contain multiple objects - for example, the results of scraping an electricity outage map or a CSV file full of records.

  Assuming you have a file called `incidents.json` that is a JSON array of objects, with multiple versions of that file recorded in a repository. Each version of that file might look something like this:

  ```json
  [
      {
          "IncidentID": "abc123",
          "Location": "Corner of 4th and Vermont",
          "Type": "fire"
      },
      {
          "IncidentID": "cde448",
          "Location": "555 West Example Drive",
          "Type": "medical"
      }
  ]
  ```

  Change directory into the GitHub repository in question and run the following:

      git-history file incidents.db incidents.json

  This will create a new SQLite database in the `incidents.db` file with three tables:

  - `commits` containing a row for every commit, with a `hash` column, the `commit_at` date and a foreign key to a `namespace`.
  - `item` containing a row for every item in every version of the `filename.json` file - with an extra `_commit` column that is a foreign key back to the `commit` table.
  - `namespaces` containing a single row. This allows you to build multiple tables for different files, using the `--namespace` option described below.

  The database schema for this example will look like this:

  <!-- [[[cog
  import cog, json
  from git_history import cli
  from click.testing import CliRunner
  from tests.test_git_history import make_repo
  import sqlite_utils
  import tempfile, pathlib
  tmpdir = pathlib.Path(tempfile.mkdtemp())
  db_path = str(tmpdir / "data.db")
  make_repo(tmpdir)
  runner = CliRunner()
  result = runner.invoke(cli.cli, [
      "file", db_path, str(tmpdir / "repo" / "incidents.json"), "--repo", str(tmpdir / "repo")
  ])
  cog.out("```sql\n")
  cog.out(sqlite_utils.Database(db_path).schema)
  cog.out("\n```")
  ]]] -->
  ```sql
  CREATE TABLE [namespaces] (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT
  );
  CREATE UNIQUE INDEX [idx_namespaces_name]
      ON [namespaces] ([name]);
  CREATE TABLE [commits] (
     [id] INTEGER PRIMARY KEY,
     [namespace] INTEGER REFERENCES [namespaces]([id]),
     [hash] TEXT,
     [commit_at] TEXT
  );
  CREATE UNIQUE INDEX [idx_commits_namespace_hash]
      ON [commits] ([namespace], [hash]);
  CREATE TABLE [item] (
     [IncidentID] TEXT,
     [Location] TEXT,
     [Type] TEXT
  );
  ```
  <!-- [[[end]]] -->

  If you have 10 historic versions of the `incidents.json` file and each one contains 30 incidents, you will end up with 10 * 30 = 300 rows in your `item` table.

  ### Track the history of individual items using IDs

  If your objects have a unique identifier - or multiple columns that together form a unique identifier - you can use the `--id` option to de-duplicate and track changes to each of those items over time.

  This provides a much more interesting way to apply this tool.

  If there is a unique identifier column called `IncidentID` you could run the following:

      git-history file incidents.db incidents.json --id IncidentID

  The database schema used here is very different from the one used without the `--id` option.

  If you have already imported history, the command will skip any commits that it has seen already and just process new ones. This means that even though an initial import could be slow subsequent imports should run a lot faster.

  This command will create six tables - `commits`, `item`, `item_version`, `columns`, `item_changed` and `namespaces`.

  Here's the full schema:

  <!-- [[[cog
  db_path2 = str(tmpdir / "data2.db")
  result = runner.invoke(cli.cli, [
      "file", db_path2, str(tmpdir / "repo" / "incidents.json"),
      "--repo", str(tmpdir / "repo"),
      "--id", "IncidentID"
  ])
  cog.out("```sql\n")
  cog.out(sqlite_utils.Database(db_path2).schema)
  cog.out("\n```")
  ]]] -->
  ```sql
  CREATE TABLE [namespaces] (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT
  );
  CREATE UNIQUE INDEX [idx_namespaces_name]
      ON [namespaces] ([name]);
  CREATE TABLE [commits] (
     [id] INTEGER PRIMARY KEY,
     [namespace] INTEGER REFERENCES [namespaces]([id]),
     [hash] TEXT,
     [commit_at] TEXT
  );
  CREATE UNIQUE INDEX [idx_commits_namespace_hash]
      ON [commits] ([namespace], [hash]);
  CREATE TABLE [item] (
     [_id] INTEGER PRIMARY KEY,
     [_item_id] TEXT
  , [IncidentID] TEXT, [Location] TEXT, [Type] TEXT, [_commit] INTEGER);
  CREATE UNIQUE INDEX [idx_item__item_id]
      ON [item] ([_item_id]);
  CREATE TABLE [item_version] (
     [_id] INTEGER PRIMARY KEY,
     [_item] INTEGER REFERENCES [item]([_id]),
     [_version] INTEGER,
     [_commit] INTEGER REFERENCES [commits]([id]),
     [IncidentID] TEXT,
     [Location] TEXT,
     [Type] TEXT,
     [_item_full_hash] TEXT
  );
  CREATE TABLE [columns] (
     [id] INTEGER PRIMARY KEY,
     [namespace] INTEGER REFERENCES [namespaces]([id]),
     [name] TEXT
  );
  CREATE UNIQUE INDEX [idx_columns_namespace_name]
      ON [columns] ([namespace], [name]);
  CREATE TABLE [item_changed] (
     [item_version] INTEGER REFERENCES [item_version]([_id]),
     [column] INTEGER REFERENCES [columns]([id]),
     PRIMARY KEY ([item_version], [column])
  );
  CREATE VIEW item_version_detail AS select
    commits.commit_at as _commit_at,
    commits.hash as _commit_hash,
    item_version.*,
    (
      select json_group_array(name) from columns
      where id in (
        select column from item_changed
        where item_version = item_version._id
      )
  ) as _changed_columns
  from item_version
    join commits on commits.id = item_version._commit;
  CREATE INDEX [idx_item_version__item]
      ON [item_version] ([_item]);
  ```
  <!-- [[[end]]] -->

  #### item table

  The `item` table will contain the most recent version of each row, de-duplicated by ID, plus the following additional columns:

  - `_id` - a numeric integer primary key, used as a foreign key from the `item_version` table.
  - `_item_id` - a hash of the values of the columns specified using the `--id` option to the command. This is used for de-duplication when processing new versions.
  - `_commit` - a foreign key to the `commit` table, representing the most recent commit to modify this item.

  #### item_version table

  The `item_version` table will contain a row for each captured differing version of that item, plus the following columns:

  - `_id` - a numeric ID for the item version record.
  - `_item` - a foreign key to the `item` table.
  - `_version` - the numeric version number, starting at 1 and incrementing for each captured version.
  - `_commit` - a foreign key to the `commit` table.
  - `_item_full_hash` - a hash of this version of the item. This is used internally by the tool to identify items that have changed between commits.

  The other columns in this table represent columns in the original data that have changed since the previous version. If the value has not changed, it will be represented by a `null`.

  If a value was previously set but has been changed back to `null` it will still be represented as `null` in the `item_version` row. You can identify these using the `item_changed` many-to-many table described below.

  You can use the `--full-versions` option to store full copies of the item at each version, rather than just storing the columns that have changed.

  #### item_version_detail view

  This SQL view joins `item_version` against `commits` to add three further columns: `_commit_at` with the date of the commit, and `_commit_hash` with the Git commit hash.

  #### item_changed

  This many-to-many table indicates exactly which columns were changed in an `item_version`.

  - `item_version` is a foreign key to a row in the `item_version` table.
  - `column` is a foreign key to a row in the `columns` table.

  This table with have the largest number of rows, which is why it stores just two integers in order to save space.

  #### columns

  The `columns` table stores column names. It is referenced by `item_changed`.

  - `id` - an integer ID.
  - `name` - the name of the column.
  - `namespace` - a foreign key to `namespaces`, for if multiple file histories are sharing the same database.

  #### Reserved column names

  <!-- [[[cog
  from git_history.utils import RESERVED
  cog.out("Note that ")
  cog.out(", ".join("`{}`".format(r) for r in RESERVED))
  cog.out(" are considered reserved column names for the purposes of this tool.")
  ]]] -->
  Note that `_id`, `_item_full_hash`, `_item`, `_item_id`, `_version`, `_commit`, `_item_id`, `_commit_at`, `_commit_hash`, `_changed_columns`, `rowid` are considered reserved column names for the purposes of this tool.
  <!-- [[[end]]] -->

  If your data contains any of these they will be renamed to add a trailing underscore, for example `_id_`, `_item_`, `_version_`, to avoid clashing with the reserved columns.

  If you have a column with a name such as `_commit_` it will be renamed too, adding an additional trailing underscore, so `_commit_` becomes `_commit__` and `_commit__` becomes `_commit___`.

  ### Additional options

  - `--repo DIRECTORY` - the path to the Git repository, if it is not the current working directory.
  - `--branch TEXT` - the Git branch to analyze - defaults to `main`.
  - `--id TEXT` - as described above: pass one or more columns that uniquely identify a record, so that changes to that record can be calculated over time.
  - `--full-versions` - instead of recording just the columns that have changed in the `item_version` table record a full copy of each version of theh item.
  - `--ignore TEXT` - one or more columns to ignore - they will not be included in the resulting database.
  - `--csv` - treat the data is CSV or TSV rather than JSON, and attempt to guess the correct dialect
  - `--dialect` - use a spcific CSV dialect. Options are `excel`, `excel-tab` and `unix` - see [the Python CSV documentation](https://docs.python.org/3/library/csv.html#csv.excel) for details.
  - `--skip TEXT` - one or more full Git commit hashes that should be skipped. You can use this if some of the data in your revision history is corrupted in a way that prevents this tool from working.
  - `--start-at TEXT` - skip commits prior to the specified commit hash.
  - `--start-after TEXT` - skip commits up to and including the specified commit hash, then start processing from the following commit.
  - `--convert TEXT` - custom Python code for a conversion, described below.
  - `--import TEXT` - additional Python modules to import for `--convert`.
  - `--ignore-duplicate-ids` - if a single version of a file has the same ID in it more than once, the tool will exit with an error. Use this option to ignore this and instead pick just the first of the two duplicates.
  - `--namespace TEXT` - use this if you wish to include the history of multiple different files in the same database. The default is `item` but you can set it to something else, which will produce tables with names like `yournamespace` and `yournamespace_version`.
  - `--wal` - Enable WAL mode on the created database file. Use this if you plan to run queries against the database while `git-history` is creating it.
  - `--silent` - don't show the progress bar.

  ### CSV and TSV data

  If the data in your repository is a CSV or TSV file you can process it by adding the `--csv` option. This will attempt to detect which delimiter is used by the file, so the same option works for both comma- and tab-separated values.

      git-history file trees.db trees.csv --id TreeID

  You can also specify the CSV dialect using the `--dialect` option.

  ### Custom conversions using --convert

  If your data is not already either CSV/TSV or a flat JSON array, you can reshape it using the `--convert` option.

  The format needed by this tool is an array of dictionaries, as demonstrated by the `incidents.json` example above.

  If your data does not fit this shape, you can provide a snippet of Python code to converts the on-disk content of each stored file into a Python list of dictionaries.

  For example, if your stored files each look like this:

  ```json
  {
      "incidents": [
          {
              "id": "552",
              "name": "Hawthorne Fire",
              "engines": 3
          },
          {
              "id": "556",
              "name": "Merlin Fire",
              "engines": 1
          }
      ]
  }
  ```
  You could use the following Python snippet to convert them to the required format:

  ```python
  json.loads(content)["incidents"]
  ```
  (The `json` module is exposed to your custom function by default.)

  You would then run the tool like this:

      git-history file database.db incidents.json \
        --id id \
        --convert 'json.loads(content)["incidents"]'

  The `content` variable is always a `bytes` object representing the content of the file at a specific moment in the repository's history.

  You can import additional modules using `--import`. This example shows how you could read a CSV file that uses `;` as the delimiter:

      git-history file trees.db ../sf-tree-history/Street_Tree_List.csv \
        --repo ../sf-tree-history \
        --import csv \
        --import io \
        --convert '
          fp = io.StringIO(content.decode("utf-8"))
          return list(csv.DictReader(fp, delimiter=";"))
          ' \
        --id TreeID

  You can import nested modules such as [ElementTree](https://docs.python.org/3/library/xml.etree.elementtree.html)  using `--import xml.etree.ElementTree`, then refer to them in your function body as `xml.etree.ElementTree`. For example, if your tracked data was in an `items.xml` file that looked like this:

  ```xml
  <items>
    <item id="1" name="One" />
    <item id="2" name="Two" />
    <item id="3" name="Three" />
  </item>
  ```
  You could load it using the following `--convert` script:
  ```
  git-history file items.xml --convert '
  tree = xml.etree.ElementTree.fromstring(content)
  return [el.attrib for el in tree.iter("item")]
  ' --import xml.etree.ElementTree --id id
  ```

  If your Python code spans more than one line it needs to include a `return` statement.

  You can also use Python generators in your `--convert` code, for example:

      git-history file stats.db package-stats/stats.json \
          --repo package-stats \
          --convert '
          data = json.loads(content)
          for key, counts in data.items():
              for date, count in counts.items():
                  yield {
                      "package": key,
                      "date": date,
                      "count": count
                  }
          ' --id package --id date

  This conversion function expects data that looks like this:

  ```json
  {
      "airtable-export": {
          "2021-05-18": 66,
          "2021-05-19": 60,
          "2021-05-20": 87
      }
  }
  ```

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd git-history
      python -m venv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

  To update the schema examples in this README file:

      cog -r README.md

project_urls:
  {"CI": "https://github.com/simonw/git-history/actions", "Changelog": "https://github.com/simonw/git-history/releases", "Homepage": "https://github.com/simonw/git-history", "Issues": "https://github.com/simonw/git-history/issues"}

requires_dist:
  ["click", "GitPython", "sqlite-utils (>=3.19)", "pytest ; extra == 'test'", "cogapp ; extra == 'test'"]


name: github-to-sqlite
summary: Save data from GitHub to a SQLite database
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/dogsheep/github-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/github-to-sqlite/
project_url: https://pypi.org/project/github-to-sqlite/
project_urls: {"Homepage": "https://github.com/dogsheep/github-to-sqlite"}
release_url: https://pypi.org/project/github-to-sqlite/2.8.3/
version: 2.8.3
description:
  # github-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/github-to-sqlite.svg)](https://pypi.org/project/github-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/github-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/github-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/github-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/github-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/github-to-sqlite/blob/main/LICENSE)

  Save data from GitHub to a SQLite database.

  <!-- toc -->

  - [Demo](#demo)
  - [How to install](#how-to-install)
  - [Authentication](#authentication)
  - [Fetching issues for a repository](#fetching-issues-for-a-repository)
  - [Fetching pull requests for a repository](#fetching-pull-requests-for-a-repository)
  - [Fetching issue comments for a repository](#fetching-issue-comments-for-a-repository)
  - [Fetching commits for a repository](#fetching-commits-for-a-repository)
  - [Fetching releases for a repository](#fetching-releases-for-a-repository)
  - [Fetching tags for a repository](#fetching-tags-for-a-repository)
  - [Fetching contributors to a repository](#fetching-contributors-to-a-repository)
  - [Fetching repos belonging to a user or organization](#fetching-repos-belonging-to-a-user-or-organization)
  - [Fetching specific repositories](#fetching-specific-repositories)
  - [Fetching repos that have been starred by a user](#fetching-repos-that-have-been-starred-by-a-user)
  - [Fetching users that have starred specific repos](#fetching-users-that-have-starred-specific-repos)
  - [Fetching GitHub Actions workflows](#fetching-github-actions-workflows)
  - [Scraping dependents for a repository](#scraping-dependents-for-a-repository)
  - [Fetching emojis](#fetching-emojis)
  - [Making authenticated API calls](#making-authenticated-api-calls)

  <!-- tocstop -->

  ## Demo

  https://github-to-sqlite.dogsheep.net/ hosts a [Datasette](https://datasette.io/) demo of a database created by [running this tool](https://github.com/dogsheep/github-to-sqlite/blob/main/.github/workflows/deploy-demo.yml#L40-L60) against all of the repositories in the [Dogsheep GitHub organization](https://github.com/dogsheep), plus the [datasette](https://github.com/simonw/datasette) and [sqlite-utils](https://github.com/simonw/sqlite-utils) repositories.

  ## How to install

      $ pip install github-to-sqlite

  ## Authentication

  Create a GitHub personal access token: https://github.com/settings/tokens

  Run this command and paste in your new token:

      $ github-to-sqlite auth

  This will create a file called `auth.json` in your current directory containing the required value. To save the file at a different path or filename, use the `--auth=myauth.json` option.

  As an alternative to using an `auth.json` file you can add your access token to an environment variable called `GITHUB_TOKEN`.

  ## Fetching issues for a repository

  The `issues` command retrieves all of the issues belonging to a specified repository.

      $ github-to-sqlite issues github.db simonw/datasette

  If an `auth.json` file is present it will use the token from that file. It works without authentication for public repositories but you should be aware that GitHub have strict IP-based rate limits for unauthenticated requests.

  You can point to a different location of `auth.json` using `-a`:

      $ github-to-sqlite issues github.db simonw/datasette -a /path/to/auth.json

  You can use the `--issue` option one or more times to load specific issues:

      $ github-to-sqlite issues github.db simonw/datasette --issue=1

  Example: [issues table](https://github-to-sqlite.dogsheep.net/github/issues)

  ## Fetching pull requests for a repository

  While pull requests are a type of issue, you will get more information on pull requests by pulling them separately. For example, whether a pull request has been merged and when.

  Following the API of issues, the `pull-requests` command retrieves all of the pull requests belonging to a specified repository.

      $ github-to-sqlite pull-requests github.db simonw/datasette

  You can use the `--pull-request` option one or more times to load specific pull request:

      $ github-to-sqlite pull-requests github.db simonw/datasette --pull-request=81

  Note that the `merged_by` column on the `pull_requests` table will only be populated for pull requests that are loaded using the `--pull-request` option - the GitHub API does not return this field for pull requests that are loaded in bulk.

  Example: [pull_requests table](https://github-to-sqlite.dogsheep.net/github/pull_requests)

  ## Fetching issue comments for a repository

  The `issue-comments` command retrieves all of the comments on all of the issues in a repository.

  It is recommended you run `issues` first, so that each imported comment can have a foreign key poining to its issue.

      $ github-to-sqlite issues github.db simonw/datasette
      $ github-to-sqlite issue-comments github.db simonw/datasette

  You can use the `--issue` option to only load comments for a specific issue within that repository, for example:

      $ github-to-sqlite issue-comments github.db simonw/datasette --issue=1

  Example: [issue_comments table](https://github-to-sqlite.dogsheep.net/github/issue_comments)

  ## Fetching commits for a repository

  The `commits` command retrieves details of all of the commits for one or more repositories. It currently fetches the sha, commit message and author and committer details - it does no retrieve the full commit body.

      $ github-to-sqlite commits github.db simonw/datasette simonw/sqlite-utils

  The command accepts one or more repositories.

  By default it will stop as soon as it sees a commit that has previously been retrieved. You can force it to retrieve all commits (including those that have been previously inserted) using `--all`.

  Example: [commits table](https://github-to-sqlite.dogsheep.net/github/commits)

  ## Fetching releases for a repository

  The `releases` command retrieves the releases for one or more repositories.

      $ github-to-sqlite releases github.db simonw/datasette simonw/sqlite-utils

  The command accepts one or more repositories.

  Example: [releases table](https://github-to-sqlite.dogsheep.net/github/releases)

  ## Fetching tags for a repository

  The `tags` command retrieves all of the tags for one or more repositories.

      $ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils

  Example: [tags table](https://github-to-sqlite.dogsheep.net/github/tags)

  ## Fetching contributors to a repository

  The `contributors` command retrieves details of all of the contributors for one or more repositories.

      $ github-to-sqlite contributors github.db simonw/datasette simonw/sqlite-utils

  The command accepts one or more repositories. It populates a `contributors` table, with foreign keys to `repos` and `users` and a `contributions` table listing the number of commits to that repository for each contributor.

  Example: [contributors table](https://github-to-sqlite.dogsheep.net/github/contributors)

  ## Fetching repos belonging to a user or organization

  The `repos` command fetches repos belonging to a user or organization.

  Without any other arguments, this command will fetch all repos that the currently authenticated user owns, collaborates on or can access via one of their organizations:

      $ github-to-sqlite repos github.db

  To fetch repos belonging to a specific user or organization, provide their username as an argument:

      $ github-to-sqlite repos github.db dogsheep # organization
      $ github-to-sqlite repos github.db simonw # user

  You can pass more than one username to fetch for multiple users or organizations at once:

      $ github-to-sqlite repos github.db simonw dogsheep

  Add the `--readme` option to save the README for the repo in a column called `readme`. Add `--readme-html` to save the HTML rendered version of the README into a collumn called `readme_html`.

  Example: [repos table](https://github-to-sqlite.dogsheep.net/github/repos)

  ## Fetching specific repositories

  You can use `-r` with the `repos` command one or more times to fetch just specific repositories.

      $ github-to-sqlite repos github.db -r simonw/datasette -r dogsheep/github-to-sqlite

  ## Fetching repos that have been starred by a user

  The `starred` command fetches the repos that have been starred by a user.

      $ github-to-sqlite starred github.db simonw

  If you are using an `auth.json` file you can omit the username to retrieve the starred repos for the authenticated user.

  Example: [stars table](https://github-to-sqlite.dogsheep.net/github/stars)

  ## Fetching users that have starred specific repos

  The `stargazers` command fetches the users that have starred the specified repos.

      $ github-to-sqlite stargazers github.db simonw/datasette dogsheep/github-to-sqlite

  You can specify one or more repository using `owner/repo` syntax.

  Users fetched using this command will be inserted into the `users` table. Many-to-many records showing which repository they starred will be added to the `stars` table.

  ## Fetching GitHub Actions workflows

  The `workflows` command fetches the YAML workflow configurations from each repository's `.github/workflows` directory and parses them to populate `workflows`, `jobs` and `steps` tables.

      $ github-to-sqlite workflows github.db simonw/datasette dogsheep/github-to-sqlite

  You can specify one or more repository using `owner/repo` syntax.

  Example: [workflows table](https://github-to-sqlite.dogsheep.net/github/workflows), [jobs table](https://github-to-sqlite.dogsheep.net/github/jobs), [steps table](https://github-to-sqlite.dogsheep.net/github/steps)

  ## Scraping dependents for a repository

  The GitHub dependency graph can show other GitHub projects that depend on a specific repo, for example [simonw/datasette/network/dependents](https://github.com/simonw/datasette/network/dependents).

  This data is not yet available through the GitHub API. The `scrape-dependents` command scrapes those pages and uses the GitHub API to load full versions of the dependent repositories.

      $ github-to-sqlite scrape-dependents github.db simonw/datasette

  The command accepts one or more repositories.

  Add `-v` for verbose output.

  Example: [dependents table](https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc)

  ## Fetching emojis

  You can fetch a list of every emoji supported by GitHub using the `emojis` command:

      $ github-to-sqlite emojis github.db

  This will create a table callad `emojis` with a primary key `name` and a `url` column.

  If you add the `--fetch` option the command will also fetch the binary content of the images and place them in an `image` column:

      $ github-to-sqlite emojis emojis.db -f
      [########----------------------------]  397/1799   22%  00:03:43

  You can then use the [datasette-render-images](https://github.com/simonw/datasette-render-images) plugin to browse them visually.

  Example: [emojis table](https://github-to-sqlite.dogsheep.net/github/emojis)

  ## Making authenticated API calls

  The `github-to-sqlite get` command provides a convenient shortcut for making authenticated calls to the API. Once you have created your `auth.json` file (or set a `GITHUB_TOKEN` environment variable) you can use it like this:

      $ github-to-sqlite get https://api.github.com/gists

  This will make an authenticated call to the URL you provide and pretty-print the resulting JSON to the console.

  You can ommit the `https://api.github.com/` prefix, for example:

      $ github-to-sqlite get /gists

  Many GitHub APIs are [paginated using the HTTP Link header](https://docs.github.com/en/rest/guides/traversing-with-pagination). You can follow this pagination and output a list of all of the resulting items using `--paginate`:

      $ github-to-sqlite get /users/simonw/repos --paginate

  You can outline newline-delimited JSON for each item using `--nl`. This can be useful for streaming items into another tool.

      $ github-to-sqlite get /users/simonw/repos --nl

requires_dist:
  ["sqlite-utils (>=2.7.2)", "requests", "PyYAML", "pytest ; extra == 'test'", "requests-mock ; extra == 'test'", "bs4 ; extra == 'test'"]


name: google-drive-to-sqlite
summary: Create a SQLite database containing metadata from Google Drive
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/google-drive-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/google-drive-to-sqlite/
project_url: https://pypi.org/project/google-drive-to-sqlite/
release_url: https://pypi.org/project/google-drive-to-sqlite/0.4/
requires_python: >=3.6
version: 0.4
description:
  # google-drive-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/google-drive-to-sqlite.svg)](https://pypi.org/project/google-drive-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/google-drive-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/google-drive-to-sqlite/releases)
  [![Tests](https://github.com/simonw/google-drive-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/google-drive-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/google-drive-to-sqlite/blob/master/LICENSE)

  Create a SQLite database containing metadata from [Google Drive](https://www.google.com/drive)

  If you use Google Drive, and especially if you have shared drives with other people there's a good chance you have hundreds or even thousands of files that you may not be fully aware of.

  This tool can download metadata about those files - their names, sizes, folders, content types, permissions, creation dates and more - and store them in a SQLite database.

  This lets you use SQL to analyze your Google Drive contents, using [Datasette](https://datasette.io/) or the SQLite command-line tool or any other SQLite database browsing software.

  ## Installation

  Install this tool using `pip`:

      pip install google-drive-to-sqlite

  ## Quickstart

  Authenticate with Google Drive by running:

      google-drive-to-sqlite auth

  Now create a SQLite database with metadata about all of the files you have starred using:

      google-drive-to-sqlite files starred.db --starred

  You can explore the resulting database using [Datasette](https://datasette.io/):

      $ pip install datasette
      $ datasette starred.db
      INFO:     Started server process [24661]
      INFO:     Uvicorn running on http://127.0.0.1:8001

  ## Authentication

  > :warning: **This application has not yet been verified by Google** - you may find you are unable to authenticate until that verification is complete. [#10](https://github.com/simonw/google-drive-to-sqlite/issues/10)
  >
  > You can work around this issue by [creating your own OAuth client ID key](https://til.simonwillison.net/googlecloud/google-oauth-cli-application) and passing it to the `auth` command using `--google-client-id` and `--google-client-secret`.

  First, authenticate with Google Drive using the `auth` command:

      $ google-drive-to-sqlite auth
      Visit the following URL to authenticate with Google Drive

      https://accounts.google.com/o/oauth2/v2/auth?...

      Then return here and paste in the resulting code:
      Paste code here: 

  Follow the link, sign in with Google Drive and then copy and paste the resulting code back into the tool.

  This will save an authentication token to the file called `auth.json` in the current directory.

  To specify a different location for that file, use the `--auth` option:

      google-drive-to-sqlite auth --auth ~/google-drive-auth.json

  The `auth` command also provides options for using a different scope, Google client ID and Google client secret. You can use these to create your own custom authentication tokens that can work with other Google APIs, see [issue #5](https://github.com/simonw/google-drive-to-sqlite/issues/5) for details.

  Full `--help`:

  <!-- [[[cog
  import cog
  from google_drive_to_sqlite import cli
  from click.testing import CliRunner
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["auth", "--help"])
  help = result.output.replace("Usage: cli", "Usage: google-drive-to-sqlite")
  cog.out(
      "```\n{}\n```\n".format(help)
  )
  ]]] -->
  ```
  Usage: google-drive-to-sqlite auth [OPTIONS]

    Authenticate user and save credentials

  Options:
    -a, --auth FILE              Path to save token, defaults to auth.json
    --google-client-id TEXT      Custom Google client ID
    --google-client-secret TEXT  Custom Google client secret
    --scope TEXT                 Custom token scope
    --help                       Show this message and exit.

  ```
  <!-- [[[end]]] -->

  To revoke the token that is stored in `auth.json`, such that it cannot be used to access Google Drive in the future, run the `revoke` command:

      google-drive-to-sqlite revoke

  Or if your token is stored in another location:

      google-drive-to-sqlite revoke -a ~/google-drive-auth.json

  You will need to obtain a fresh token using the `auth` command in order to continue using this tool.

  ## google-drive-to-sqlite files

  To retrieve metadata about the files in your Google Drive, or a folder or search within it, use the `google-drive-to-sqlite files` command.

  This will default to writing details about every file in your Google Drive to a SQLite database:

      google-drive-to-sqlite files files.db

  Files and folders will be written to databases tables, which will be created if they do not yet exist. The database schema is [shown below](#database-schema).

  If a file or folder already exists, based on a matching `id`, it will be replaced with fresh data.

  Instead of writing to SQLite you can use `--json` to output as JSON, or `--nl` to output as newline-delimited JSON:

      google-drive-to-sqlite files --nl

  Use `--folder ID` to retrieve everything in a specified folder and its sub-folders:

      google-drive-to-sqlite files files.db --folder 1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i

  Use `--q QUERY` to use a [custom search query](https://developers.google.com/drive/api/v3/reference/query-ref):

      google-drive-to-sqlite files files.db -q "viewedByMeTime > '2022-01-01'"

  The following shortcut options help build queries:

  - `--full-text TEXT` to search for files where the full text matches a search term
  - `--starred` for files and folders you have starred
  - `--trashed` for files and folders in the trash
  - `--shared-with-me` for files and folders that have been shared with you
  - `--apps` for Google Apps documents, spreadsheets, presentations and drawings (equivalent to setting all  of the next four options)
  - `--docs` for Google Apps documents
  - `--sheets` for Google Apps spreadsheets
  - `--presentations` for Google Apps presentations
  - `--drawings` for Google Apps drawings

  You can combine these - for example, this returns all files that you have starred and that were shared with you:

      google-drive-to-sqlite files highlights.db \
        --starred --shared-with-me

  Multiple options are treated as AND, with the exception of the Google Apps options which are treated as OR - so the following would retrieve all spreadsheets and presentations that have also been starred:

      google-drive-to-sqlite files highlights.db \
        --starred --sheets --presentations

  You can use `--stop-after X` to stop after retrieving X files, useful for trying out a new search pattern and seeing results straight away.

  The `--import-json` and `--import-nl` options are mainly useful for testing and developing this tool. They allow you to replay the JSON or newline-delimited JSON that was previously fetched using `--json` or `--nl` and use it to create a fresh SQLite database, without needing to make any outbound API calls:

      # Fetch all starred files from the API, write to starred.json
      google-drive-to-sqlite files -q 'starred = true' --json > starred.json
      # Now import that data into a new SQLite database file
      google-drive-to-sqlite files starred.db --import-json starred.json

  Full `--help`:

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["files", "--help"])
  help = result.output.replace("Usage: cli", "Usage: google-drive-to-sqlite")
  cog.out(
      "```\n{}\n```\n".format(help)
  )
  ]]] -->
  ```
  Usage: google-drive-to-sqlite files [OPTIONS] [DATABASE]

    Retrieve metadata for files in Google Drive, and write to a SQLite database or
    output as JSON.

        google-drive-to-sqlite files files.db

    Use --json to output JSON, --nl for newline-delimited JSON:

        google-drive-to-sqlite files files.db --json

    Use a folder ID to recursively fetch every file in that folder and its sub-
    folders:

        google-drive-to-sqlite files files.db --folder
        1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i

    Fetch files you have starred:

        google-drive-to-sqlite files starred.db --starred

  Options:
    -a, --auth FILE       Path to auth.json token file
    --folder TEXT         Files in this folder ID and its sub-folders
    -q TEXT               Files matching this query
    --full-text TEXT      Search for files with text match
    --starred             Files you have starred
    --trashed             Files in the trash
    --shared-with-me      Files that have been shared with you
    --apps                Google Apps docs, spreadsheets, presentations and
                          drawings
    --docs                Google Apps docs
    --sheets              Google Apps spreadsheets
    --presentations       Google Apps presentations
    --drawings            Google Apps drawings
    --json                Output JSON rather than write to DB
    --nl                  Output newline-delimited JSON rather than write to DB
    --stop-after INTEGER  Stop paginating after X results
    --import-json FILE    Import from this JSON file instead of the API
    --import-nl FILE      Import from this newline-delimited JSON file
    -v, --verbose         Send verbose output to stderr
    --help                Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## google-drive-to-sqlite download FILE_ID

  The `download` command can be used to download files from Google Drive.

  You'll need one or more file IDs, which look something like `0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB`.

  To download the file, run this:

      google-drive-to-sqlite download 0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB

  This will detect the content type of the file and use that as the extension - so if this file is a JPEG the file would be downloaded as:

      0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB.jpeg

  You can pass multiple file IDs to the command at once.

  To hide the progress bar and filename output, use `-s` or `--silent`.

  If you are downloading a single file you can use the `-o` output to specify a filename and location:

      google-drive-to-sqlite download 0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB \
        -o my-image.jpeg

  Use `-o -` to write the file contents to standard output:

      google-drive-to-sqlite download 0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB \
        -o - > my-image.jpeg

  Full `--help`:

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["download", "--help"])
  help = result.output.replace("Usage: cli", "Usage: google-drive-to-sqlite")
  cog.out(
      "```\n{}\n```\n".format(help)
  )
  ]]] -->
  ```
  Usage: google-drive-to-sqlite download [OPTIONS] FILE_IDS...

    Download one or more files to disk, based on their file IDs.

    The file content will be saved to a file with the name:

        FILE_ID.ext

    Where the extension is automatically picked based on the type of file.

    If you are downloading a single file you can specify a filename with -o:

        google-drive-to-sqlite download MY_FILE_ID -o myfile.txt

  Options:
    -a, --auth FILE    Path to auth.json token file
    -o, --output FILE  File to write to, or - for standard output
    -s, --silent       Hide progress bar and filename
    --help             Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## google-drive-to-sqlite export FORMAT FILE_ID

  The `export` command can be used to export Google Docs documents, spreadsheets and presentations in a number of different formats.

  You'll need one or more document IDs, which look something like `10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU`. You can find these by looking at the URL of your document on the Google Docs site.

  To export that document as PDF, run this:

      google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU

  The file will be exported as:

      10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU-export.pdf

  You can pass multiple file IDs to the command at once.

  For the `FORMAT` option you can use any of the mime type options listed [on this page](https://developers.google.com/drive/api/v3/ref-export-formats) - for example, to export as an Open Office document you could use:

      google-drive-to-sqlite export \
       application/vnd.oasis.opendocument.text \
       10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU

  For convenience the following shortcuts for common file formats are provided:

  - Google Docs: `html`, `txt`, `rtf`, `pdf`, `doc`, `zip`, `epub`
  - Google Sheets: `xls`, `pdf`, `csv`, `tsv`, `zip`
  - Presentations: `ppt`, `pdf`, `txt`
  - Drawings: `jpeg`, `png`, `svg`

  The `zip` option returns a zip file of HTML. `txt` returns plain text. The others should be self-evident.

  To hide the filename output, use `-s` or `--silent`.

  If you are exporting a single file you can use the `-o` output to specify a filename and location:

      google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU \
        -o my-document.pdf

  Use `-o -` to write the file contents to standard output:

      google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU \
        -o - > my-document.pdf

  Full `--help`:

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["export", "--help"])
  help = result.output.replace("Usage: cli", "Usage: google-drive-to-sqlite")
  cog.out(
      "```\n{}\n```\n".format(help)
  )
  ]]] -->
  ```
  Usage: google-drive-to-sqlite export [OPTIONS] FORMAT FILE_IDS...

    Export one or more files to the specified format.

    Usage:

        google-drive-to-sqlite export pdf FILE_ID_1 FILE_ID_2

    The file content will be saved to a file with the name:

        FILE_ID-export.ext

    Where the extension is based on the format you specified.

    Available export formats can be seen here:
    https://developers.google.com/drive/api/v3/ref-export-formats

    Or you can use one of the following shortcuts:

    - Google Docs: html, txt, rtf, pdf, doc, zip, epub
    - Google Sheets: xls, pdf, csv, tsv, zip
    - Presentations: ppt, pdf, txt
    - Drawings: jpeg, png, svg

    "zip" returns a zip file of HTML.

    If you are exporting a single file you can specify a filename with -o:

        google-drive-to-sqlite export zip MY_FILE_ID -o myfile.zip

  Options:
    -a, --auth FILE    Path to auth.json token file
    -o, --output FILE  File to write to, or - for standard output
    -s, --silent       Hide progress bar and filename
    --help             Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## google-drive-to-sqlite get URL

  The `get` command makes authenticated requests to the specified URL, using credentials derived from the `auth.json` file.

  For example:

      $ google-drive-to-sqlite get 'https://www.googleapis.com/drive/v3/about?fields=*'
      {
          "kind": "drive#about",
          "user": {
              "kind": "drive#user",
              "displayName": "Simon Willison",
      # ...

  If the resource you are fetching supports pagination you can use `--paginate key` to paginate through all of the rows in a specified key. For example, the following API has a `nextPageToken` key and a `files` list, suggesting it supports pagination:

      $ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files
      {
          "kind": "drive#fileList",
          "nextPageToken": "~!!~AI9...wogHHYlc=",
          "incompleteSearch": false,
          "files": [
              {
                  "kind": "drive#file",
                  "id": "1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg",
                  "name": "Title of a spreadsheet",
                  "mimeType": "application/vnd.google-apps.spreadsheet"
              },

  To paginate through everything in the `files` list you would use `--paginate files` like this:

      $ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files --paginate files
      [
        {
          "kind": "drive#file",
          "id": "1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg",
          "name": "Title of a spreadsheet",
          "mimeType": "application/vnd.google-apps.spreadsheet"
        },
        # ...

  Add `--nl` to stream paginated data as newline-delimited JSON:

      $ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files --paginate files --nl
      {"kind": "drive#file", "id": "1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg", "name": "Title of a spreadsheet", "mimeType": "application/vnd.google-apps.spreadsheet"}
      {"kind": "drive#file", "id": "1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i", "name": "Subfolder", "mimeType": "application/vnd.google-apps.folder"}

  Add `--stop-after 5` to stop after 5 records - useful for testing.

  Full `--help`:

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["get", "--help"])
  help = result.output.replace("Usage: cli", "Usage: google-drive-to-sqlite")
  cog.out(
      "```\n{}\n```\n".format(help)
  )
  ]]] -->
  ```
  Usage: google-drive-to-sqlite get [OPTIONS] URL

    Make an authenticated HTTP GET to the specified URL

  Options:
    -a, --auth FILE       Path to auth.json token file
    --paginate TEXT       Paginate through all results in this key
    --nl                  Output paginated data as newline-delimited JSON
    --stop-after INTEGER  Stop paginating after X results
    -v, --verbose         Send verbose output to stderr
    --help                Show this message and exit.

  ```
  <!-- [[[end]]] -->


  ## Database schema

  The database created by this tool has the following schema:

  <!-- [[[cog
  import tempfile, pathlib, sqlite_utils
  tmpdir = pathlib.Path(tempfile.mkdtemp())
  db_path = str(tmpdir / "docs.db")
  result = runner.invoke(cli.cli, [
      "files", db_path, "--import-json", "tests/folder-and-children.json"
  ])
  cog.out("```sql\n")
  schema = sqlite_utils.Database(db_path).schema
  # Tidy up some formatting
  schema = schema.replace(", [", ",\n   [")
  schema = schema.replace("\n,\n", ",\n")
  schema = schema.replace("TEXT);", "TEXT\n);")
  cog.out(schema)
  cog.out("\n```")
  ]]] -->
  ```sql
  CREATE TABLE [drive_users] (
     [permissionId] TEXT PRIMARY KEY,
     [kind] TEXT,
     [displayName] TEXT,
     [photoLink] TEXT,
     [me] INTEGER,
     [emailAddress] TEXT
  );
  CREATE TABLE [drive_folders] (
     [id] TEXT PRIMARY KEY,
     [_parent] TEXT,
     [_owner] TEXT,
     [lastModifyingUser] TEXT,
     [kind] TEXT,
     [name] TEXT,
     [mimeType] TEXT,
     [starred] INTEGER,
     [trashed] INTEGER,
     [explicitlyTrashed] INTEGER,
     [parents] TEXT,
     [spaces] TEXT,
     [version] TEXT,
     [webViewLink] TEXT,
     [iconLink] TEXT,
     [hasThumbnail] INTEGER,
     [thumbnailVersion] TEXT,
     [viewedByMe] INTEGER,
     [createdTime] TEXT,
     [modifiedTime] TEXT,
     [modifiedByMe] INTEGER,
     [shared] INTEGER,
     [ownedByMe] INTEGER,
     [viewersCanCopyContent] INTEGER,
     [copyRequiresWriterPermission] INTEGER,
     [writersCanShare] INTEGER,
     [folderColorRgb] TEXT,
     [quotaBytesUsed] TEXT,
     [isAppAuthorized] INTEGER,
     [linkShareMetadata] TEXT,
     FOREIGN KEY([_parent]) REFERENCES [drive_folders]([id]),
     FOREIGN KEY([_owner]) REFERENCES [drive_users]([permissionId]),
     FOREIGN KEY([lastModifyingUser]) REFERENCES [drive_users]([permissionId])
  );
  CREATE TABLE [drive_files] (
     [id] TEXT PRIMARY KEY,
     [_parent] TEXT,
     [_owner] TEXT,
     [lastModifyingUser] TEXT,
     [kind] TEXT,
     [name] TEXT,
     [mimeType] TEXT,
     [starred] INTEGER,
     [trashed] INTEGER,
     [explicitlyTrashed] INTEGER,
     [parents] TEXT,
     [spaces] TEXT,
     [version] TEXT,
     [webViewLink] TEXT,
     [iconLink] TEXT,
     [hasThumbnail] INTEGER,
     [thumbnailVersion] TEXT,
     [viewedByMe] INTEGER,
     [createdTime] TEXT,
     [modifiedTime] TEXT,
     [modifiedByMe] INTEGER,
     [shared] INTEGER,
     [ownedByMe] INTEGER,
     [viewersCanCopyContent] INTEGER,
     [copyRequiresWriterPermission] INTEGER,
     [writersCanShare] INTEGER,
     [quotaBytesUsed] TEXT,
     [isAppAuthorized] INTEGER,
     [linkShareMetadata] TEXT,
     FOREIGN KEY([_parent]) REFERENCES [drive_folders]([id]),
     FOREIGN KEY([_owner]) REFERENCES [drive_users]([permissionId]),
     FOREIGN KEY([lastModifyingUser]) REFERENCES [drive_users]([permissionId])
  );
  ```
  <!-- [[[end]]] -->

  ## Thumbnails

  You can construct a thumbnail image for a known file ID using the following URL:

      https://drive.google.com/thumbnail?sz=w800-h800&id=FILE_ID

  Users who are signed into Google Drive and have permission to view a file will be redirected to a thumbnail version of that file. You can tweak the `w800` and `h800` parameters to request different thumbnail sizes.

  ## Privacy policy

  This tool requests access to your Google Drive account in order to retrieve metadata about your files there. It also offers a feature that can download the content of those files.

  The credentials used to access your account are stored in the auth.json file on your computer. The metadata and content retrieved from Google Drive is also stored only on your own personal computer.

  At no point to the developers of this tool gain access to any of your data.

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd google-drive-to-sqlite
      python -m venv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/google-drive-to-sqlite/actions", "Changelog": "https://github.com/simonw/google-drive-to-sqlite/releases", "Homepage": "https://github.com/simonw/google-drive-to-sqlite", "Issues": "https://github.com/simonw/google-drive-to-sqlite/issues"}

requires_dist:
  ["click", "httpx", "sqlite-utils", "pytest ; extra == 'test'", "pytest-httpx ; extra == 'test'", "pytest-mock ; extra == 'test'", "cogapp ; extra == 'test'"]


name: google-takeout-to-sqlite
summary: Save data from Google Takeout to a SQLite database
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/dogsheep/google-takeout-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/google-takeout-to-sqlite/
project_url: https://pypi.org/project/google-takeout-to-sqlite/
project_urls: {"Homepage": "https://github.com/dogsheep/google-takeout-to-sqlite"}
release_url: https://pypi.org/project/google-takeout-to-sqlite/0.2/
requires_dist: ["sqlite-utils (~=1.11)", "pytest ; extra == 'test'"]
version: 0.2
description:
  # google-takeout-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/google-takeout-to-sqlite.svg)](https://pypi.org/project/google-takeout-to-sqlite/)
  [![CircleCI](https://circleci.com/gh/dogsheep/google-takeout-to-sqlite.svg?style=svg)](https://circleci.com/gh/dogsheep/google-takeout-to-sqlite)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/google-takeout-to-sqlite/blob/master/LICENSE)

  Save data from google-takeout to a SQLite database.

  ## How to install

      $ pip install google-takeout-to-sqlite

  Request your Google data from https://takeout.google.com/ - wait for the email and download the zip file.

  This tool only supports a subset of the available options. More will be added over time.

  ## My Activity

  You can request the "My Activity" export and then import it with the following command:

      $ google-takeout-to-sqlite my-activity takeout.db ~/Downloads/takeout-20190530.zip

  This will create a database file called `takeout.db` if one does not already exist.

  ## Location History

  Your location history records latitude, longitude and timestame for where Google has tracked your location. You can import it using this command:

      $ google-takeout-to-sqlite location-history takeout.db ~/Downloads/takeout-20190530.zip

  ## Browsing your data with Datasette

  Once you have imported Google data into a SQLite database file you can browse your data using [Datasette](https://github.com/simonw/datasette). Install Datasette like so:

      $ pip install datasette

  Now browse your data by running this and then visiting `http://localhost:8001/`

      $ datasette takeout.db

  Install the [datasette-cluster-map](https://github.com/simonw/datasette-cluster-map) plugin to see your location history on a map:

      $ pip install datasette-cluster-map


name: hacker-news-to-sqlite
summary: Create a SQLite database containing data pulled from Hacker News
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/dogsheep/hacker-news-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/hacker-news-to-sqlite/
project_url: https://pypi.org/project/hacker-news-to-sqlite/
project_urls: {"Homepage": "https://github.com/dogsheep/hacker-news-to-sqlite"}
release_url: https://pypi.org/project/hacker-news-to-sqlite/0.4/
version: 0.4
description:
  # hacker-news-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/hacker-news-to-sqlite.svg)](https://pypi.org/project/hacker-news-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/hacker-news-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/hacker-news-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/hacker-news-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/hacker-news-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/hacker-news-to-sqlite/blob/main/LICENSE)

  Create a SQLite database containing data fetched from [Hacker News](https://news.ycombinator.com/).

  ## How to install

      $ pip install hacker-news-to-sqlite

  ## Usage

      $ hacker-news-to-sqlite user hacker-news.db your-username
      Importing items:  37%|███████████                        | 845/2297 [05:09<11:02,  2.19it/s]

  Imports all of your Hacker News submissions and comments into a SQLite database called `hacker-news.db`.

      $ hacker-news-to-sqlite trees hacker-news.db 22640038 22643218

  Fetches the entire comments tree in which any of those content IDs appears.

  ## Browsing your data with Datasette

  You can use [Datasette](https://datasette.readthedocs.org/) to browse your data. Install Datasette like this:

      $ pip install datasette

  Now run it against your `hacker-news.db` file like so:

      $ datasette hacker-news.db

  Visit `https://localhost:8001/` to search and explore your data.

  You can improve the display of your data usinng the [datasette-render-timestamps](https://github.com/simonw/datasette-render-timestamps) and [datasette-render-html](https://github.com/simonw/datasette-render-html) plugins. Install them like this:

      $ pip install datasette-render-timestamps datasette-render-html

  Now save the following configuration in a file called `metadata.json`:

  ```json
  {
      "databases": {
          "hacker-news": {
              "tables": {
                  "items": {
                      "plugins": {
                          "datasette-render-html": {
                              "columns": [
                                  "text"
                              ]
                          },
                          "datasette-render-timestamps": {
                              "columns": [
                                  "time"
                              ]
                          }
                      }
                  },
                  "users": {
                      "plugins": {
                          "datasette-render-timestamps": {
                              "columns": [
                                  "created"
                              ]
                          }
                      }
                  }
              }
          }
      }
  }
  ```
  Run Datasette like this:

      $ datasette -m metadata.json hacker-news.db

  The timestamp columns will now be rendered as human-readable dates, and any HTML in your posts will be displayed as rendered HTML.

requires_dist:
  ["sqlite-utils", "click", "requests", "tqdm", "pytest ; extra == 'test'", "requests-mock ; extra == 'test'"]


name: healthkit-to-sqlite
summary: Convert an Apple Healthkit export zip to a SQLite database
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://datasette.io/tools/healthkit-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/healthkit-to-sqlite/
project_url: https://pypi.org/project/healthkit-to-sqlite/
release_url: https://pypi.org/project/healthkit-to-sqlite/1.0.1/
requires_dist: ["sqlite-utils (>=2.4.4)", "pytest ; extra == 'test'"]
version: 1.0.1
description:
  # healthkit-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/healthkit-to-sqlite.svg)](https://pypi.org/project/healthkit-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/healthkit-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/healthkit-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/healthkit-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/healthkit-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/healthkit-to-sqlite/blob/main/LICENSE)

  Convert an Apple Healthkit export zip to a SQLite database

  ## How to install

      $ pip install healthkit-to-sqlite

  ## How to use

  First you need to export your Apple HealthKit data.

  1. On your iPhone, open the "Health" app
  2. Click the profile icon in the top right
  3. Click "Export Health Data" at the bottom of that page
  4. Save the resulting file somewhere you can access it, or AirDrop it directly to your laptop.

  Now you can convert the resulting `export.zip` file to SQLite like so:

      $ healthkit-to-sqlite export.zip healthkit.db

  A progress bar will be displayed. You can disable this using `--silent`.

  ```
  Importing from HealthKit  [#-------------]    5%  00:01:33
  ```

  You can explore the resulting data using [Datasette](https://datasette.readthedocs.io/) like this:

      $ datasette healthkit.db

project_urls:
  {"CI": "https://github.com/dogsheep/healthkit-to-sqlite/actions", "Changelog": "https://github.com/dogsheep/healthkit-to-sqlite/releases", "Homepage": "https://datasette.io/tools/healthkit-to-sqlite", "Issues": "https://github.com/dogsheep/healthkit-to-sqlite/issues"}


name: inaturalist-to-sqlite
summary: Create a SQLite database containing your observation history from iNaturalist
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/dogsheep/inaturalist-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/inaturalist-to-sqlite/
project_url: https://pypi.org/project/inaturalist-to-sqlite/
project_urls: {"Homepage": "https://github.com/dogsheep/inaturalist-to-sqlite"}
release_url: https://pypi.org/project/inaturalist-to-sqlite/0.2.1/
requires_dist: ["sqlite-utils (>=2.0)", "click", "requests", "pytest ; extra == 'test'"]
version: 0.2.1
description:
  # inaturalist-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/inaturalist-to-sqlite.svg)](https://pypi.org/project/inaturalist-to-sqlite/)
  [![CircleCI](https://circleci.com/gh/dogsheep/inaturalist-to-sqlite.svg?style=svg)](https://circleci.com/gh/dogsheep/inaturalist-to-sqlite)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/inaturalist-to-sqlite/blob/master/LICENSE)

  Create a SQLite database containing your observation history from [iNaturalist](https://www.inaturalist.org/).

  ## How to install

      $ pip install inaturalist-to-sqlite

  ## Usage

      $ inaturalist-to-sqlite inaturalist.db yourusername

  (Or try `simonw` if you don't yet have an iNaturalist account)

  This will import all of your iNaturalist observations into a SQLite database called `inaturalist.db`.


name: markdown-to-sqlite
summary: CLI tool for loading markdown files into a SQLite database
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/markdown-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/markdown-to-sqlite/
project_url: https://pypi.org/project/markdown-to-sqlite/
release_url: https://pypi.org/project/markdown-to-sqlite/1.0/
requires_dist: ["yamldown", "markdown", "sqlite-utils", "click", "pytest ; extra == 'test'"]
requires_python: >=3.6
version: 1.0
classifiers:
  ["Intended Audience :: Developers", "Intended Audience :: End Users/Desktop", "Intended Audience :: Science/Research", "License :: OSI Approved :: Apache Software License", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Topic :: Database"]

description:
  # markdown-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/markdown-to-sqlite.svg)](https://pypi.python.org/pypi/markdown-to-sqlite)
  [![Changelog](https://img.shields.io/github/v/release/simonw/markdown-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/markdown-to-sqlite/releases)
  [![Tests](https://github.com/simonw/markdown-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/markdown-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/markdown-to-sqlite/blob/main/LICENSE)

  CLI tool for loading markdown files into a SQLite database.

  YAML embedded in the markdown files will be used to populate additional columns.

      Usage: markdown-to-sqlite [OPTIONS] DBNAME TABLE PATHS...

  For example:

      $ markdown-to-sqlite docs.db documents file1.md file2.md

  ## Breaking change

  Prior to version 1.0 this argument order was different - markdown files were listed before the database and table.

project_urls:
  {"CI": "https://github.com/simonw/markdown-to-sqlite/actions", "Changelog": "https://github.com/simonw/markdown-to-sqlite/releases", "Homepage": "https://github.com/simonw/markdown-to-sqlite", "Issues": "https://github.com/simonw/markdown-to-sqlite/issues"}


name: mbox-to-sqlite
summary: Load email from .mbox files into SQLite
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/mbox-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/mbox-to-sqlite/
project_url: https://pypi.org/project/mbox-to-sqlite/
release_url: https://pypi.org/project/mbox-to-sqlite/0.1a0/
requires_dist: ["click", "sqlite-utils", "pytest ; extra == 'test'"]
requires_python: >=3.7
version: 0.1a0
description:
  # mbox-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/mbox-to-sqlite.svg)](https://pypi.org/project/mbox-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/mbox-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/mbox-to-sqlite/releases)
  [![Tests](https://github.com/simonw/mbox-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/mbox-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/mbox-to-sqlite/blob/master/LICENSE)

  Load email from .mbox files into SQLite

  ## Installation

  Install this tool using `pip`:

      pip install mbox-to-sqlite

  ## Usage

  Use the `mbox` command to import a `.mbox` file into a SQLite database:

      mbox-to-sqlite mbox emails.db path/to/messages.mbox

  You can try this out against an example containing a sample of 3,266 emails from the [Enron corpus](https://en.wikipedia.org/wiki/Enron_Corpus) like this:

      curl -O https://raw.githubusercontent.com/ivanhb/EMA/master/server/data/mbox/enron/mbox-enron-white-s-all.mbox
      mbox-to-sqlite mbox enron.db mbox-enron-white-s-all.mbox

  You can then explore the resulting database using [Datasette](https://datasette.io/):

      datasette enron.db

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd mbox-to-sqlite
      python -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/mbox-to-sqlite/actions", "Changelog": "https://github.com/simonw/mbox-to-sqlite/releases", "Homepage": "https://github.com/simonw/mbox-to-sqlite", "Issues": "https://github.com/simonw/mbox-to-sqlite/issues"}


name: pocket-to-sqlite
summary: Create a SQLite database containing data from your Pocket account
classifiers: ["License :: OSI Approved :: Apache Software License"]
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/dogsheep/pocket-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/pocket-to-sqlite/
project_url: https://pypi.org/project/pocket-to-sqlite/
release_url: https://pypi.org/project/pocket-to-sqlite/0.2.2/
requires_dist: ["sqlite-utils (>=2.4.4)", "click", "requests", "pytest ; extra == 'test'"]
version: 0.2.2
description:
  # pocket-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/pocket-to-sqlite.svg)](https://pypi.org/project/pocket-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/pocket-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/pocket-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/pocket-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/pocket-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/pocket-to-sqlite/blob/main/LICENSE)

  Create a SQLite database containing data from your [Pocket](https://getpocket.com/) account.

  ## How to install

      $ pip install pocket-to-sqlite

  ## Usage

  You will need to first obtain a valid OAuth token for your Pocket account. You can do this by running the `auth` command and following the prompts:

      $ pocket-to-sqlite auth
      Visit this page and sign in with your Pocket account:

      https://getpocket.com/auth/author...

      Once you have signed in there, hit <enter> to continue
      Authentication tokens written to auth.json

  Now you can fetch all of your items from Pocket like this:

      $ pocket-to-sqlite fetch pocket.db

  The first time you run this command it will fetch all of your items, and display a progress bar while it does it.

  On subsequent runs it will only fetch new items.

  You can force it to fetch everything from the beginning again using `--all`. Use `--silent` to disable the progress bar.

  ## Using with Datasette

  The SQLite database produced by this tool is designed to be browsed using [Datasette](https://datasette.readthedocs.io/). Use the [datasette-render-timestamps](https://github.com/simonw/datasette-render-timestamps) plugin to improve the display of the timestamp values.

project_urls:
  {"CI": "https://github.com/dogsheep/pocket-to-sqlite/actions", "Changelog": "https://github.com/dogsheep/pocket-to-sqlite/releases", "Homepage": "https://github.com/dogsheep/pocket-to-sqlite", "Issues": "https://github.com/dogsheep/pocket-to-sqlite/issues"}


name: pypi-to-sqlite
summary: Load data about Python packages from PyPI into SQLite
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/pypi-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/pypi-to-sqlite/
project_url: https://pypi.org/project/pypi-to-sqlite/
release_url: https://pypi.org/project/pypi-to-sqlite/0.2.2/
requires_python: >=3.7
version: 0.2.2
description:
  # pypi-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/pypi-to-sqlite.svg)](https://pypi.org/project/pypi-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/pypi-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/pypi-to-sqlite/releases)
  [![Tests](https://github.com/simonw/pypi-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/pypi-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/pypi-to-sqlite/blob/master/LICENSE)

  Load data about Python packages from PyPI into SQLite

  ## Installation

  Install this tool using `pip`:

      pip install pypi-to-sqlite

  ## Usage

  To create a SQLite database with details of one or more packages, run:

      pypi-to-sqlite pypi.db datasette sqlite-utils

  You can also process JSON that you have previously saved to disk like so:

      curl -o datasette.json https://pypi.org/pypi/datasette/json
      pypi-to-sqlite pypi.db -f datasette.json

  The tool will create three tables: `packages`, `versions` and `releases`. The full table schema is shown below.

  To create the tables with a prefix, use `--prefix prefix`. For example:

      pypi-to-sqlite pypi.db datasette --prefix pypi_

  This will create tables called `pypi_packages`, `pypi_versions` and `pypi_releases`.

  ## Database schema

  <!-- [[[cog
  import cog, json
  from pypi_to_sqlite import cli
  from click.testing import CliRunner
  import sqlite_utils
  import tempfile, pathlib
  tmpdir = pathlib.Path(tempfile.mkdtemp())
  db_path = str(tmpdir / "pypi.db")
  runner = CliRunner()
  result = runner.invoke(cli.cli, [db_path, "-f", "tests/datasette-block.json"])
  cog.out("```sql\n")
  cog.out(sqlite_utils.Database(db_path).schema)
  cog.out("\n```")
  ]]] -->
  ```sql
  CREATE TABLE [packages] (
     [name] TEXT PRIMARY KEY,
     [summary] TEXT,
     [classifiers] TEXT,
     [description] TEXT,
     [author] TEXT,
     [author_email] TEXT,
     [description_content_type] TEXT,
     [home_page] TEXT,
     [keywords] TEXT,
     [license] TEXT,
     [maintainer] TEXT,
     [maintainer_email] TEXT,
     [package_url] TEXT,
     [platform] TEXT,
     [project_url] TEXT,
     [project_urls] TEXT,
     [release_url] TEXT,
     [requires_dist] TEXT,
     [requires_python] TEXT,
     [version] TEXT,
     [yanked] INTEGER,
     [yanked_reason] TEXT
  );
  CREATE TABLE [versions] (
     [id] TEXT PRIMARY KEY,
     [package] TEXT REFERENCES [packages]([name]),
     [name] TEXT
  );
  CREATE TABLE [releases] (
     [md5_digest] TEXT PRIMARY KEY,
     [package] TEXT REFERENCES [packages]([name]),
     [version] TEXT REFERENCES [versions]([id]),
     [packagetype] TEXT,
     [filename] TEXT,
     [comment_text] TEXT,
     [digests] TEXT,
     [has_sig] INTEGER,
     [python_version] TEXT,
     [requires_python] TEXT,
     [size] INTEGER,
     [upload_time] TEXT,
     [upload_time_iso_8601] TEXT,
     [url] TEXT,
     [yanked] INTEGER,
     [yanked_reason] TEXT
  );
  ```
  <!-- [[[end]]] -->

  ## pypi-to-sqlite --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["--help"])
  cog.out("```\n")
  cog.out(result.output.replace("Usage: cli", "Usage: pypi-to-sqlite"))
  cog.out("\n```")
  ]]] -->
  ```
  Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]...

    Load data about Python packages from PyPI into SQLite

    Usage example:

        pypi-to-sqlite pypy.db datasette sqlite-utils

    Use -f to load data from a JSON file instead:

        pypi-to-sqlite pypy.db -f datasette.json

    Created tables will be packages, versions and releases

    To create tables called pypi_packages, pypi_versions, pypi_releases use
    --prefix pypi_:

        pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_

  Options:
    --version            Show the version and exit.
    -f, --file FILENAME  Import JSON from this file
    -d, --delay FLOAT    Wait this many seconds between requests
    --prefix TEXT        Prefix to use for the created database tables
    --help               Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd pypi-to-sqlite
      python -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/pypi-to-sqlite/actions", "Changelog": "https://github.com/simonw/pypi-to-sqlite/releases", "Homepage": "https://github.com/simonw/pypi-to-sqlite", "Issues": "https://github.com/simonw/pypi-to-sqlite/issues"}

requires_dist:
  ["click", "sqlite-utils", "httpx", "pytest ; extra == 'test'", "pytest-httpx ; extra == 'test'", "cogapp ; extra == 'test'"]


name: s3-credentials
summary: A tool for creating credentials for accessing S3 buckets
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/s3-credentials
license: Apache License, Version 2.0
package_url: https://pypi.org/project/s3-credentials/
project_url: https://pypi.org/project/s3-credentials/
release_url: https://pypi.org/project/s3-credentials/0.14/
requires_python: >=3.6
version: 0.14
description:
  # s3-credentials

  [![PyPI](https://img.shields.io/pypi/v/s3-credentials.svg)](https://pypi.org/project/s3-credentials/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/s3-credentials?include_prereleases&label=changelog)](https://github.com/simonw/s3-credentials/releases)
  [![Tests](https://github.com/simonw/s3-credentials/workflows/Test/badge.svg)](https://github.com/simonw/s3-credentials/actions?query=workflow%3ATest)
  [![Documentation Status](https://readthedocs.org/projects/s3-credentials/badge/?version=latest)](https://s3-credentials.readthedocs.org/)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/s3-credentials/blob/master/LICENSE)

  A tool for creating credentials for accessing S3 buckets

  For project background, see [s3-credentials: a tool for creating credentials for S3 buckets](https://simonwillison.net/2021/Nov/3/s3-credentials/) on my blog.

  ## Installation

      pip install s3-credentials

  ## Basic usage

  To create a new S3 bucket and output credentials that can be used with only that bucket:
  ```
  % s3-credentials create my-new-s3-bucket --create-bucket
  Created bucket:  my-new-s3-bucket
  Created user: s3.read-write.my-new-s3-bucket with permissions boundary: arn:aws:iam::aws:policy/AmazonS3FullAccess
  Attached policy s3.read-write.my-new-s3-bucket to user s3.read-write.my-new-s3-bucket
  Created access key for user: s3.read-write.my-new-s3-bucket
  {
      "UserName": "s3.read-write.my-new-s3-bucket",
      "AccessKeyId": "AKIAWXFXAIOZOYLZAEW5",
      "Status": "Active",
      "SecretAccessKey": "...",
      "CreateDate": "2021-11-03 01:38:24+00:00"
  }
  ```
  The tool can do a lot more than this. See the [documentation](https://s3-credentials.readthedocs.io/) for details.

  ## Documentation

  - [Full documentation](https://s3-credentials.readthedocs.io/)
  - [Command help reference](https://s3-credentials.readthedocs.io/en/stable/help.html)
  - [Release notes](https://github.com/simonw/s3-credentials/releases)

project_urls:
  {"CI": "https://github.com/simonw/s3-credentials/actions", "Changelog": "https://github.com/simonw/s3-credentials/releases", "Homepage": "https://github.com/simonw/s3-credentials", "Issues": "https://github.com/simonw/s3-credentials/issues"}

requires_dist:
  ["click", "boto3", "pytest ; extra == 'test'", "pytest-mock ; extra == 'test'", "cogapp ; extra == 'test'", "moto[s3] ; extra == 'test'"]


name: s3-ocr
summary: Tools for running OCR against files stored in S3
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/s3-ocr
license: Apache License, Version 2.0
package_url: https://pypi.org/project/s3-ocr/
project_url: https://pypi.org/project/s3-ocr/
release_url: https://pypi.org/project/s3-ocr/0.6.3/
requires_python: >=3.7
version: 0.6.3
description:
  # s3-ocr

  [![PyPI](https://img.shields.io/pypi/v/s3-ocr.svg)](https://pypi.org/project/s3-ocr/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/s3-ocr?include_prereleases&label=changelog)](https://github.com/simonw/s3-ocr/releases)
  [![Tests](https://github.com/simonw/s3-ocr/workflows/Test/badge.svg)](https://github.com/simonw/s3-ocr/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/s3-ocr/blob/master/LICENSE)

  Tools for running OCR against files stored in S3

  Background on this project: [s3-ocr: Extract text from PDF files stored in an S3 bucket](https://simonwillison.net/2022/Jun/30/s3-ocr/)

  ## Installation

  Install this tool using `pip`:

      pip install s3-ocr

  ## Demo

  You can see the results of running this tool against three PDFs from the Internet Archive ([one](https://archive.org/details/unmaskingrobert00houdgoog), [two](https://archive.org/details/practicalmagicia00harr), [three](https://archive.org/details/latestmagicbeing00hoff)) in [this example table](https://s3-ocr-demo.datasette.io/pages/pages?_facet=path#facet-path) hosted using [Datasette](https://datasette.io/).

  ## Starting OCR against PDFs in a bucket

  The `start` command takes a list of keys and submits them to [Textract](https://aws.amazon.com/textract/) for OCR processing.

  You need to have AWS configured using environment variables, credentials file in your home directory or a JSON or INI file generated using [s3-credentials](https://datasette.io/tools/s3-credentials).

  You can start the process running like this:

      s3-ocr start name-of-your-bucket my-pdf-file.pdf

  The paths you specify should be paths within the bucket. If you stored your PDF files in folders inside the bucket it should look like this:

      s3-ocr start name-of-your-bucket path/to/one.pdf path/to/two.pdf

  OCR can take some time. The results of the OCR will be stored in `textract-output` in your bucket.

  To process every file in the bucket with a `.pdf` extension use `--all`:

      s3-ocr start name-of-bucket --all

  To process every file with a `.pdf` extension within a specific folder, use `--prefix`:

      s3-ocr start name-of-bucket --prefix path/to/folder

  ### s3-ocr start --help

  <!-- [[[cog
  import cog
  from s3_ocr import cli
  from click.testing import CliRunner
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["start", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help)
  )
  ]]] -->
  ```
  Usage: s3-ocr start [OPTIONS] BUCKET [KEYS]...

    Start OCR tasks for PDF files in an S3 bucket

        s3-ocr start name-of-bucket path/to/one.pdf path/to/two.pdf

    To process every file with a .pdf extension:

        s3-ocr start name-of-bucket --all

    To process every .pdf in the PUBLIC/ folder:

        s3-ocr start name-of-bucket --prefix PUBLIC/

  Options:
    --all                 Process all PDF files in the bucket
    --prefix TEXT         Process all PDF files within this prefix
    --dry-run             Show what this would do, but don't actually do it
    --no-retry            Don't retry failed requests
    --access-key TEXT     AWS access key ID
    --secret-key TEXT     AWS secret access key
    --session-token TEXT  AWS session token
    --endpoint-url TEXT   Custom endpoint URL
    -a, --auth FILENAME   Path to JSON/INI file containing credentials
    --help                Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## Checking status

  The `s3-ocr status <bucket-name>` command shows a rough indication of progress through the tasks:

  ```
  % s3-ocr status sfms-history
  153 complete out of 532 jobs
  ```
  It compares the jobs that have been submitted, based on `.s3-ocr.json` files, to the jobs that have their results written to the `textract-output/` folder.

  ### s3-ocr status --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["status", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr status [OPTIONS] BUCKET

    Show status of OCR jobs for a bucket

  Options:
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Inspecting a job

  The `s3-ocr inspect-job <job_id>` command can be used to check the status of a specific job ID:
  ```
  % s3-ocr inspect-job b267282745685226339b7e0d4366c4ff6887b7e293ed4b304dc8bb8b991c7864
  {
    "DocumentMetadata": {
      "Pages": 583
    },
    "JobStatus": "SUCCEEDED",
    "DetectDocumentTextModelVersion": "1.0"
  }
  ```

  ### s3-ocr inspect-job --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["inspect-job", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr inspect-job [OPTIONS] JOB_ID

    Show the current status of an OCR job

        s3-ocr inspect-job <job_id>

  Options:
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Fetching the results

  Once an OCR job has completed you can download the resulting JSON using the `fetch` command:

      s3-ocr fetch name-of-bucket path/to/file.pdf

  This will save files in the current directory with names like this:

  - `4d9b5cf580e761fdb16fd24edce14737ebc562972526ef6617554adfa11d6038-1.json`
  - `4d9b5cf580e761fdb16fd24edce14737ebc562972526ef6617554adfa11d6038-2.json`

  The number of files will vary depending on the length of the document.

  If you don't want separate files you can combine them together using the `-c/--combine` option:

      s3-ocr fetch name-of-bucket path/to/file.pdf --combine output.json

  The `output.json` file will then contain data that looks something like this:

  ```
  {
    "Blocks": [
      {
        "BlockType": "PAGE",
        "Geometry": {...}
        "Page": 1,
        ...
      },
      {
        "BlockType": "LINE",
        "Page": 1,
        ...
        "Text": "Barry",
      },
  ```
  ### s3-ocr fetch --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["fetch", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr fetch [OPTIONS] BUCKET KEY

    Fetch the OCR results for a specified file

        s3-ocr fetch name-of-bucket path/to/key.pdf

    This will save files in the current directory called things like

        a806e67e504fc15f...48314e-1.json     a806e67e504fc15f...48314e-2.json

    To combine these together into a single JSON file with a specified name, use:

        s3-ocr fetch name-of-bucket path/to/key.pdf --combine output.json

    Use "--output -" to print the combined JSON to standard output instead.

  Options:
    -c, --combine FILENAME  Write combined JSON to file
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Fetching just the text of a page

  If you don't want to deal with the JSON directly, you can use the `text` command to retrieve just the text extracted from a PDF:

      s3-ocr text name-of-bucket path/to/file.pdf

  This will output plain text to standard output.

  To save that to a file, use this:

      s3-ocr text name-of-bucket path/to/file.pdf > text.txt

  Separate pages will be separated by three newlines. To separate them using a `----` horizontal divider instead add `--divider`:

      s3-ocr text name-of-bucket path/to/file.pdf --divider

  ### s3-ocr text --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["text", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr text [OPTIONS] BUCKET KEY

    Retrieve the text from an OCRd PDF file

        s3-ocr text name-of-bucket path/to/key.pdf

  Options:
    --divider             Add ---- between pages
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Avoiding processing duplicates

  If you move files around within your S3 bucket `s3-ocr` can lose track of which files have already been processed. This can lead to additional Textract charges for processing should you run `s3-ocr start` against those new files.

  The `s3-ocr dedupe` command addresses this by scanning your bucket for files that have a new name but have previously been processed. It does this by looking at the `ETag` for each file, which represents the MD5 hash of the file contents.

  The command will write out new `.s3ocr.json` files for each detected duplicate. This will avoid those duplicates being run those duplicates through OCR a second time should yo run `s3-ocr start`.

      s3-ocr dedupe name-of-bucket

  Add `--dry-run` for a preview of the changes that will be made to your bucket.

  ### s3-ocr dedupe --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["dedupe", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr dedupe [OPTIONS] BUCKET

    Scan every file in the bucket checking for duplicates - files that have not
    yet been OCRd but that have the same contents (based on ETag) as a file that
    HAS been OCRd.

        s3-ocr dedupe name-of-bucket

  Options:
    --dry-run             Show output without writing anything to S3
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Changes made to your bucket

  To keep track of which files have been submitted for processing, `s3-ocr` will create a JSON file for every file that it adds to the OCR queue.

  This file will be called:

      path-to-file/name-of-file.pdf.s3-ocr.json

  Each of these JSON files contains data that looks like this:

  ```json
  {
    "job_id": "a34eb4e8dc7e70aa9668f7272aa403e85997364199a654422340bc5ada43affe",
    "etag": "\"b0c77472e15500347ebf46032a454e8e\""
  }
  ```
  The recorded `job_id` can be used later to associate the file with the results of the OCR task in `textract-output/`.

  The `etag` is the ETag of the S3 object at the time it was submitted. This can be used later to determine if a file has changed since it last had OCR run against it.

  This design for the tool, with the `.s3-ocr.json` files tracking jobs that have been submitted, means that it is safe to run `s3-ocr start` against the same bucket multiple times without the risk of starting duplicate OCR jobs.

  ## Creating a SQLite index of your OCR results

  The `s3-ocr index <bucket> <database_file>` command creates a SQLite database containing the results of the OCR, and configures SQLite full-text search against the text:

  ```
  % s3-ocr index sfms-history index.db
  Fetching job details  [####################################]  100%
  Populating pages table  [####################----------------]   55%  00:03:18
  ```
  The schema of the resulting database looks like this (excluding the FTS tables):
  ```sql
  CREATE TABLE [pages] (
     [path] TEXT,
     [page] INTEGER,
     [folder] TEXT,
     [text] TEXT,
     PRIMARY KEY ([path], [page])
  );
  CREATE TABLE [ocr_jobs] (
     [key] TEXT PRIMARY KEY,
     [job_id] TEXT,
     [etag] TEXT,
     [s3_ocr_etag] TEXT
  );
  CREATE TABLE [fetched_jobs] (
     [job_id] TEXT PRIMARY KEY
  );
  ```
  The database is designed to be used with [Datasette](https://datasette.io).

  ### s3-ocr index --help

  <!-- [[[cog
  result = runner.invoke(cli.cli, ["index", "--help"])
  help = result.output.replace("Usage: cli", "Usage: s3-ocr")
  cog.out(
      "```\n{}\n```".format(help.split("--access-key")[0] + "--access-key ...")
  )
  ]]] -->
  ```
  Usage: s3-ocr index [OPTIONS] BUCKET DATABASE

    Create a SQLite database with OCR results for files in a bucket

  Options:
    --access-key ...
  ```
  <!-- [[[end]]] -->

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd s3-ocr
      python -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

  To regenerate the README file with the latest `--help`:

      cog -r README.md

project_urls:
  {"CI": "https://github.com/simonw/s3-ocr/actions", "Changelog": "https://github.com/simonw/s3-ocr/releases", "Homepage": "https://github.com/simonw/s3-ocr", "Issues": "https://github.com/simonw/s3-ocr/issues"}

requires_dist:
  ["click (>=8.0)", "boto3", "sqlite-utils", "pytest ; extra == 'test'", "moto[s3,textract] ; extra == 'test'", "cogapp ; extra == 'test'", "pytest-mock ; extra == 'test'"]


name: shapefile-to-sqlite
summary: Load shapefiles into a SQLite (optionally SpatiaLite) database
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/shapefile-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/shapefile-to-sqlite/
project_url: https://pypi.org/project/shapefile-to-sqlite/
project_urls: {"Homepage": "https://github.com/simonw/shapefile-to-sqlite"}
release_url: https://pypi.org/project/shapefile-to-sqlite/0.4.1/
requires_dist: ["sqlite-utils (>=2.2)", "Shapely", "Fiona", "pyproj", "pytest ; extra == 'test'"]
version: 0.4.1
description:
  # shapefile-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/shapefile-to-sqlite.svg)](https://pypi.org/project/shapefile-to-sqlite/)
  [![CircleCI](https://circleci.com/gh/simonw/shapefile-to-sqlite.svg?style=svg)](https://circleci.com/gh/simonw/shapefile-to-sqlite)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/shapefile-to-sqlite/blob/main/LICENSE)

  Load shapefiles into a SQLite (optionally SpatiaLite) database.

  Project background: [Things I learned about shapefiles building shapefile-to-sqlite](https://simonwillison.net/2020/Feb/19/shapefile-to-sqlite/)

  ## How to install

      $ pip install shapefile-to-sqlite

  ## How to use

  You can run this tool against a shapefile file like so:

      $ shapefile-to-sqlite my.db features.shp

  This will load the geometries as GeoJSON in a text column.

  ## Using with SpatiaLite

  If you have [SpatiaLite](https://www.gaia-gis.it/fossil/libspatialite/index) available you can load them as SpatiaLite geometries like this:

      $ shapefile-to-sqlite my.db features.shp --spatialite

  The data will be loaded into a table called `features` - based on the name of the shapefile. You can specify an alternative table name using `--table`:

      $ shapefile-to-sqlite my.db features.shp --table=places --spatialite

  The tool will search for the SpatiaLite module in the following locations:

  - `/usr/lib/x86_64-linux-gnu/mod_spatialite.so`
  - `/usr/local/lib/mod_spatialite.dylib`

  If you have installed the module in another location, you can use the `--spatialite_mod=xxx` option to specify where:

      $ shapefile-to-sqlite my.db features.shp \
          --spatialite_mod=/usr/lib/mod_spatialite.dylib

  You can use the `--spatial-index` option to create a spatial index on the `geometry` column:

      $ shapefile-to-sqlite my.db features.shp --spatial-index

  You can omit `--spatialite` if you use either `--spatialite-mod` or `--spatial-index`.

  ## Projections

  By default, this tool will attempt to convert geometries in the shapefile to the WGS 84 projection, for best conformance with the [GeoJSON specification](https://tools.ietf.org/html/rfc7946).

  If you want it to leave the data in whatever projection was used by the shapefile, use the `--crs=keep` option.

  You can convert the data to another output projection by passing it to the `--crs` option. For example, to convert to [EPSG:2227](https://epsg.io/2227) (California zone 3) use `--crs=espg:2227`.

  The full list of formats accepted by the `--crs` option is [documented here](https://pyproj4.github.io/pyproj/stable/api/crs.html#pyproj.crs.CRS.__init__).

  ## Extracting columns

  If your data contains columns with a small number of heavily duplicated values - the names of specific agencies responsible for parcels of land for example - you can extract those columns into separate lookup tables referenced by foreign keys using the `-c` option:

      $ shapefile-to-sqlite my.db features.shp -c agency

  This will create a `agency` table with `id` and `name` columns, and will create the `agency` column in your main table as an integer foreign key reference to that table.

  The `-c` option can be used multiple times.


name: shot-scraper
summary: A command-line utility for taking automated screenshots of websites
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/shot-scraper
license: Apache License, Version 2.0
package_url: https://pypi.org/project/shot-scraper/
project_url: https://pypi.org/project/shot-scraper/
release_url: https://pypi.org/project/shot-scraper/1.0.1/
requires_python: >=3.7
version: 1.0.1
description:
  # shot-scraper

  [![PyPI](https://img.shields.io/pypi/v/shot-scraper.svg)](https://pypi.org/project/shot-scraper/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/shot-scraper?include_prereleases&label=changelog)](https://github.com/simonw/shot-scraper/releases)
  [![Tests](https://github.com/simonw/shot-scraper/workflows/Test/badge.svg)](https://github.com/simonw/shot-scraper/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/shot-scraper/blob/master/LICENSE)

  A command-line utility for taking automated screenshots of websites

  For background on this project see [shot-scraper: automated screenshots for documentation, built on Playwright](https://simonwillison.net/2022/Mar/10/shot-scraper/).

  ## Documentation

  - [Full documentation for shot-scraper](https://shot-scraper.datasette.io/)
  - [Tutorial: Automating screenshots for the Datasette documentation using shot-scraper](https://simonwillison.net/2022/Oct/14/automating-screenshots/)
  - [Release notes](https://github.com/simonw/shot-scraper/releases)

  ## Get started with GitHub Actions

  To get started without installing any software, use the [shot-scraper-template](https://github.com/simonw/shot-scraper-template) template to create your own GitHub repository which takes screenshots of a page using `shot-scraper`. See [Instantly create a GitHub repository to take screenshots of a web page](https://simonwillison.net/2022/Mar/14/shot-scraper-template/) for details.

  ## Quick installation

  You can install the `shot-scraper` CLI tool using [pip](https://pip.pypa.io/):

      pip install shot-scraper
      # Now install the browser it needs:
      shot-scraper install

  ## Taking your first screenshot

  You can take a screenshot of a web page like this:

      shot-scraper https://datasette.io/

  This will create a screenshot in a file called `datasette-io.png`.

  Many more options are available, see [Taking a screenshot](https://shot-scraper.datasette.io/en/stable/screenshots.html) for details.

  ## Examples

  - The [shot-scraper-demo](https://github.com/simonw/shot-scraper-demo) repository uses this tool to capture recently spotted owls in El Granada, CA according to [this page](https://www.owlsnearme.com/?place=127871), and to  generate an annotated screenshot illustrating a Datasette feature as described [in my blog](https://simonwillison.net/2022/Mar/10/shot-scraper/#a-complex-example).
  - The [Datasette Documentation](https://docs.datasette.io/en/latest/) uses screenshots taken by `shot-scraper` running in the [simonw/datasette-screenshots](https://github.com/simonw/datasette-screenshots) GitHub repository, described in detail in [Automating screenshots for the Datasette documentation using shot-scraper](https://simonwillison.net/2022/Oct/14/automating-screenshots/).
  - Ben Welsh built [@newshomepages](https://twitter.com/newshomepages), a Twitter bot that uses `shot-scraper` and GitHub Actions to take screenshots of news website homepages and publish them to Twitter. The code for that lives in [palewire/news-homepages](https://github.com/palewire/news-homepages).
  - [scrape-hacker-news-by-domain](https://github.com/simonw/scrape-hacker-news-by-domain) uses `shot-scraper javascript` to scrape a web page. See [Scraping web pages from the command-line with shot-scraper](https://simonwillison.net/2022/Mar/14/scraping-web-pages-shot-scraper/) for details of how this works.

project_urls:
  {"CI": "https://github.com/simonw/shot-scraper/actions", "Changelog": "https://github.com/simonw/shot-scraper/releases", "Homepage": "https://github.com/simonw/shot-scraper", "Issues": "https://github.com/simonw/shot-scraper/issues"}

requires_dist:
  ["click", "PyYAML", "playwright", "click-default-group", "pytest ; extra == 'test'", "cogapp ; extra == 'test'", "pytest-mock ; extra == 'test'"]


name: sphinx-to-sqlite
summary: Create a SQLite database from Sphinx documentation
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/sphinx-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/sphinx-to-sqlite/
project_url: https://pypi.org/project/sphinx-to-sqlite/
release_url: https://pypi.org/project/sphinx-to-sqlite/0.1a1/
requires_dist: ["click", "sqlite-utils", "pytest ; extra == 'test'"]
version: 0.1a1
description:
  # sphinx-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/sphinx-to-sqlite.svg)](https://pypi.org/project/sphinx-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/sphinx-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/sphinx-to-sqlite/releases)
  [![Tests](https://github.com/simonw/sphinx-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/sphinx-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sphinx-to-sqlite/blob/master/LICENSE)

  Create a SQLite database from Sphinx documentation.

  ## Demo

  You can see the results of running this tool against the [Datasette documentation](https://docs.datasette.io/) at https://latest-docs.datasette.io/docs/sections

  ## Installation

  Install this tool using `pip`:

      $ pip install sphinx-to-sqlite

  ## Usage

  First run `sphinx-build` with the `-b xml` option to create XML files in your `_build/` directory.

  Then run:

      $ sphinx-to-sqlite docs.db path/to/_build

  To build the SQLite database.

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd sphinx-to-sqlite
      python -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/sphinx-to-sqlite/actions", "Changelog": "https://github.com/simonw/sphinx-to-sqlite/releases", "Homepage": "https://github.com/simonw/sphinx-to-sqlite", "Issues": "https://github.com/simonw/sphinx-to-sqlite/issues"}


name: sqlite-colorbrewer
summary: A custom function to use ColorBrewer scales in SQLite queries
classifiers: ["Framework :: Datasette", "License :: OSI Approved :: Apache Software License"]
author: Chris Amico
description_content_type: text/markdown
home_page: https://github.com/eyeseast/sqlite-colorbrewer
license: Apache License, Version 2.0
package_url: https://pypi.org/project/sqlite-colorbrewer/
project_url: https://pypi.org/project/sqlite-colorbrewer/
release_url: https://pypi.org/project/sqlite-colorbrewer/0.2/
requires_dist: ["datasette ; extra == 'test'", "pytest ; extra == 'test'", "pytest-asyncio ; extra == 'test'"]
requires_python: >=3.6
version: 0.2
description:
  # sqlite-colorbrewer

  [![PyPI](https://img.shields.io/pypi/v/sqlite-colorbrewer.svg)](https://pypi.org/project/sqlite-colorbrewer/)
  [![Changelog](https://img.shields.io/github/v/release/eyeseast/sqlite-colorbrewer?include_prereleases&label=changelog)](https://github.com/eyeseast/sqlite-colorbrewer/releases)
  [![Tests](https://github.com/eyeseast/sqlite-colorbrewer/workflows/Test/badge.svg)](https://github.com/eyeseast/sqlite-colorbrewer/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/eyeseast/sqlite-colorbrewer/blob/main/LICENSE)

  A custom function to use [ColorBrewer](https://colorbrewer2.org/) scales in SQLite queries.

  Colors are exported from [here](https://colorbrewer2.org/export/colorbrewer.json).

  ## Installation

  To install as a Python library and use with the [standard SQLite3 module](https://docs.python.org/3/library/sqlite3.html):

      pip install sqlite-colorbrewer

  To install this plugin in the same environment as Datasette.

      datasette install sqlite-colorbrewer

  ## Usage

  If you're using this library with Datasette, it will be automatically registered as a plugin and available for use in SQL queries, like so:

  ```sql
  SELECT colorbrewer('Blues', 9, 0);
  ```

  That will return a single value: `"rgb(247,251,255)"`

  To use with a SQLite connection outside of Datasette, use the `register` function:

  ```python
  >>> import sqlite3
  >>> import sqlite_colorbrewer

  >>> conn = sqlite3.connect(':memory')
  >>> sqlite_colorbrewer.register(conn)

  >>> cursor = conn.execute("SELECT colorbrewer('Blues', 9, 0);")
  >>> result = next(cursor)
  >>> print(result)
  rgb(247,251,255)
  ```

  ## Development

  To set up this plugin locally, first checkout the code. Then create a new virtual environment:

      cd sqlite-colorbrewer
      python3 -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

  To build `sqlite_colorbrewer/colorbrewer.py`:

      ./json_to_python.py
      black . # to format the resulting file

  ## ColorBrewer

  Copyright (c) 2002 Cynthia Brewer, Mark Harrower, and The Pennsylvania State University.

  Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

  See the [ColorBrewer updates](http://www.personal.psu.edu/cab38/ColorBrewer/ColorBrewer_updates.html) for updates to copyright information.

project_urls:
  {"CI": "https://github.com/eyeseast/sqlite-colorbrewer/actions", "Changelog": "https://github.com/eyeseast/sqlite-colorbrewer/releases", "Homepage": "https://github.com/eyeseast/sqlite-colorbrewer", "Issues": "https://github.com/eyeseast/sqlite-colorbrewer/issues"}


name: sqlite-comprehend
summary: Tools for running data in a SQLite database through AWS Comprehend
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/sqlite-comprehend
license: Apache License, Version 2.0
package_url: https://pypi.org/project/sqlite-comprehend/
project_url: https://pypi.org/project/sqlite-comprehend/
release_url: https://pypi.org/project/sqlite-comprehend/0.2.2/
requires_python: >=3.7
version: 0.2.2
description:
  # sqlite-comprehend

  [![PyPI](https://img.shields.io/pypi/v/sqlite-comprehend.svg)](https://pypi.org/project/sqlite-comprehend/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-comprehend?include_prereleases&label=changelog)](https://github.com/simonw/sqlite-comprehend/releases)
  [![Tests](https://github.com/simonw/sqlite-comprehend/workflows/Test/badge.svg)](https://github.com/simonw/sqlite-comprehend/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sqlite-comprehend/blob/master/LICENSE)

  Tools for running data in a SQLite database through [AWS Comprehend](https://aws.amazon.com/comprehend/)

  See [sqlite-comprehend: run AWS entity extraction against content in a SQLite database](https://simonwillison.net/2022/Jul/11/sqlite-comprehend/) for background on this project.

  ## Installation

  Install this tool using `pip`:

      pip install sqlite-comprehend

  ## Demo

  You can see examples of tables generated using this command here:

  - [comprehend_entities](https://datasette.simonwillison.net/simonwillisonblog/comprehend_entities) - the extracted entities, classified by type
  - [blog_entry_comprehend_entities](https://datasette.simonwillison.net/simonwillisonblog/blog_entry_comprehend_entities) - a table relating entities to the entries that they appear in
  - [comprehend_entity_types](https://datasette.simonwillison.net/simonwillisonblog/comprehend_entity_types) - a small lookup table of entity types

  ## Configuration

  You will need AWS credentials with the `comprehend:BatchDetectEntities` [IAM permission](https://docs.aws.amazon.com/comprehend/latest/dg/access-control-managing-permissions.html).

  You can configure credentials [using these instructions](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html). You can also save them to a JSON or INI configuration file and pass them to the command using `-a credentials.ini`, or pass them using the `--access-key` and `--secret-key` options.

  ## Entity extraction

  The `sqlite-comprehend entities` command runs entity extraction against every row in the specified table and saves the results to your database.

  Specify the database, the table and one or more columns containing text in that table. The following runs against the `text` column in the `pages` table of the `sfms.db` SQLite database:

      sqlite-comprehend sfms.db pages text

  Results will be written into a `pages_comprehend_entities` table. Change the name of the output table by passing `-o other_table_name`.

  You can run against a subset of rows by adding a `--where` clause:

      sqlite-comprehend sfms.db pages text --where 'id < 10'

  You can also used named parameters in your `--where` clause:

      sqlite-comprehend sfms.db pages text --where 'id < :maxid' -p maxid 10

  Only the first 5,000 characters for each row will be considered. Be sure to review [Comprehend's pricing](https://aws.amazon.com/comprehend/pricing/) - which starts at $0.0001 per hundred characters.

  If your context includes HTML tags, you can strip them out before extracting entities by adding `--strip-tags`:

      sqlite-comprehend sfms.db pages text --strip-tags

  Rows that have been processed are recorded in the `pages_comprehend_entities_done` table. If you run the command more than once it will only process rows that have been newly added.

  You can delete records from that `_done` table to run them again.

  ### sqlite-comprehend entities --help

  <!-- [[[cog
  from click.testing import CliRunner
  from sqlite_comprehend import cli
  runner = CliRunner()
  result = runner.invoke(cli.cli, ["entities", "--help"])
  help = result.output.replace("Usage: cli", "Usage: sqlite-comprehend")
  cog.out(
      "```\n{}\n```".format(help)
  )
  ]]] -->
  ```
  Usage: sqlite-comprehend entities [OPTIONS] DATABASE TABLE COLUMNS...

    Detect entities in columns in a table

    To extract entities from columns text1 and text2 in mytable:

        sqlite-comprehend entities my.db mytable text1 text2

    To run against just a subset of the rows in the table, add:

        --where "id < :max_id" -p max_id 50

    Results will be written to a table called mytable_comprehend_entities

    To specify a different output table, use -o custom_table_name

  Options:
    --where TEXT                WHERE clause to filter table
    -p, --param <TEXT TEXT>...  Named :parameters for SQL query
    -o, --output TEXT           Custom output table
    -r, --reset                 Start from scratch, deleting previous results
    --strip-tags                Strip HTML tags before extracting entities
    --access-key TEXT           AWS access key ID
    --secret-key TEXT           AWS secret access key
    --session-token TEXT        AWS session token
    --endpoint-url TEXT         Custom endpoint URL
    -a, --auth FILENAME         Path to JSON/INI file containing credentials
    --help                      Show this message and exit.

  ```
  <!-- [[[end]]] -->

  ## Schema

  Assuming an input table called `pages` the tables created by this tool will have the following schema:

  <!-- [[[cog
  import cog, json
  from sqlite_comprehend import cli
  from unittest.mock import patch
  from click.testing import CliRunner
  import sqlite_utils
  import tempfile, pathlib
  tmpdir = pathlib.Path(tempfile.mkdtemp())
  db_path = str(tmpdir / "data.db")
  db = sqlite_utils.Database(db_path)
  db["pages"].insert_all(
      [
          {
              "id": 1,
              "text": "John Bob",
          },
          {
              "id": 2,
              "text": "Sandra X",
          },
      ],
      pk="id",
  )
  with patch('boto3.client') as client:
      client.return_value.batch_detect_entities.return_value = {
          "ResultList": [
              {
                  "Index": 0,
                  "Entities": [
                      {
                          "Score": 0.8,
                          "Type": "PERSON",
                          "Text": "John Bob",
                          "BeginOffset": 0,
                          "EndOffset": 5,
                      },
                  ],
              },
              {
                  "Index": 1,
                  "Entities": [
                      {
                          "Score": 0.8,
                          "Type": "PERSON",
                          "Text": "Sandra X",
                          "BeginOffset": 0,
                          "EndOffset": 5,
                      },
                  ],
              },
          ],
          "ErrorList": [],
      }
      runner = CliRunner()
      result = runner.invoke(cli.cli, [
          "entities", db_path, "pages", "text"
      ])
  cog.out("```sql\n")
  cog.out(db.schema)
  cog.out("\n```")
  ]]] -->
  ```sql
  CREATE TABLE [pages] (
     [id] INTEGER PRIMARY KEY,
     [text] TEXT
  );
  CREATE TABLE [comprehend_entity_types] (
     [id] INTEGER PRIMARY KEY,
     [value] TEXT
  );
  CREATE TABLE [comprehend_entities] (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT,
     [type] INTEGER REFERENCES [comprehend_entity_types]([id])
  );
  CREATE TABLE [pages_comprehend_entities] (
     [id] INTEGER REFERENCES [pages]([id]),
     [score] FLOAT,
     [entity] INTEGER REFERENCES [comprehend_entities]([id]),
     [begin_offset] INTEGER,
     [end_offset] INTEGER
  );
  CREATE UNIQUE INDEX [idx_comprehend_entity_types_value]
      ON [comprehend_entity_types] ([value]);
  CREATE UNIQUE INDEX [idx_comprehend_entities_type_name]
      ON [comprehend_entities] ([type], [name]);
  CREATE TABLE [pages_comprehend_entities_done] (
     [id] INTEGER PRIMARY KEY REFERENCES [pages]([id])
  );
  ```
  <!-- [[[end]]] -->

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd sqlite-comprehend
      python -m venv venv
      source venv/bin/activate

  Now install the dependencies and test dependencies:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/sqlite-comprehend/actions", "Changelog": "https://github.com/simonw/sqlite-comprehend/releases", "Homepage": "https://github.com/simonw/sqlite-comprehend", "Issues": "https://github.com/simonw/sqlite-comprehend/issues"}

requires_dist:
  ["click", "boto3", "sqlite-utils", "pytest ; extra == 'test'", "pytest-mock ; extra == 'test'", "cogapp ; extra == 'test'"]


name: sqlite-diffable
summary: Tools for dumping/loading a SQLite database to diffable directory structure
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/sqlite-diffable
license: Apache License, Version 2.0
package_url: https://pypi.org/project/sqlite-diffable/
project_url: https://pypi.org/project/sqlite-diffable/
release_url: https://pypi.org/project/sqlite-diffable/0.5/
requires_dist: ["click", "sqlite-utils", "pytest ; extra == 'test'", "black ; extra == 'test'"]
version: 0.5
classifiers:
  ["Development Status :: 3 - Alpha", "Intended Audience :: Developers", "Intended Audience :: End Users/Desktop", "Intended Audience :: Science/Research", "License :: OSI Approved :: Apache Software License", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Topic :: Database"]

description:
  # sqlite-diffable

  [![PyPI](https://img.shields.io/pypi/v/sqlite-diffable.svg)](https://pypi.org/project/sqlite-diffable/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-diffable?include_prereleases&label=changelog)](https://github.com/simonw/sqlite-diffable/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sqlite-diffable/blob/main/LICENSE)

  Tools for dumping/loading a SQLite database to diffable directory structure

  ## Installation

      pip install sqlite-diffable

  ## Demo

  The repository at [simonw/simonwillisonblog-backup](https://github.com/simonw/simonwillisonblog-backup) contains a backup of the database on my blog, https://simonwillison.net/ - created using this tool.

  ## Dumping a database

  Given a SQLite database called `fixtures.db` containing a table `facetable`, the following will dump out that table to the `dump/` directory:

      sqlite-diffable dump fixtures.db dump/ facetable

  To dump out every table in that database, use `--all`:

      sqlite-diffable dump fixtures.db dump/ --all

  ## Loading a database

  To load a previously dumped database, run the following:

      sqlite-diffable load restored.db dump/

  This will show an error if any of the tables that are being restored already exist in the database file.

  You can replace those tables (dropping them before restoring them) using the `--replace` option:

      sqlite-diffable load restored.db dump/ --replace

  ## Converting to JSON objects

  Table rows are stored in the `.ndjson` files as newline-delimited JSON arrays, like this:

  ```
  ["a", "a", "a-a", 63, null, 0.7364712141640124, "$null"]
  ["a", "b", "a-b", 51, null, 0.6020187290499803, "$null"]
  ```

  Sometimes it can be more convenient to work with a list of JSON objects.

  The `sqlite-diffable objects` command can read a `.ndjson` file and its accompanying `.metadata.json` file and output JSON objects to standard output:

      sqlite-diffable objects fixtures.db dump/sortable.ndjson

  The output of that command looks something like this:
  ```
  {"pk1": "a", "pk2": "a", "content": "a-a", "sortable": 63, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.7364712141640124, "text": "$null"}
  {"pk1": "a", "pk2": "b", "content": "a-b", "sortable": 51, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.6020187290499803, "text": "$null"}
  ```

  Add `-o` to write that output to a file:

      sqlite-diffable objects fixtures.db dump/sortable.ndjson -o output.txt

  Add `--array` to output a JSON array of objects, as opposed to a newline-delimited file:

      sqlite-diffable objects fixtures.db dump/sortable.ndjson --array
  Output:
  ```
  [
  {"pk1": "a", "pk2": "a", "content": "a-a", "sortable": 63, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.7364712141640124, "text": "$null"},
  {"pk1": "a", "pk2": "b", "content": "a-b", "sortable": 51, "sortable_with_nulls": null, "sortable_with_nulls_2": 0.6020187290499803, "text": "$null"}
  ]
  ```

  ## Storage format

  Each table is represented as two files. The first, `table_name.metadata.json`, contains metadata describing the structure of the table. For a table called `redirects_redirect` that file might look like this:

  ```json
  {
      "name": "redirects_redirect",
      "columns": [
          "id",
          "domain",
          "path",
          "target",
          "created"
      ],
      "schema": "CREATE TABLE [redirects_redirect] (\n   [id] INTEGER PRIMARY KEY,\n   [domain] TEXT,\n   [path] TEXT,\n   [target] TEXT,\n   [created] TEXT\n)"
  }
  ```

  It is an object with three keys: `name` is the name of the table, `columns` is an array of column strings and `schema` is the SQL schema text used for tha table.

  The second file, `table_name.ndjson`, contains [newline-delimited JSON](http://ndjson.org/) for every row in the table. Each row is represented as a JSON array with items corresponding to each of the columns defined in the metadata.

  That file for the `redirects_redirect.ndjson` table might look like this:

  ```
  [1, "feeds.simonwillison.net", "swn-everything", "https://simonwillison.net/atom/everything/", "2017-10-01T21:11:36.440537+00:00"]
  [2, "feeds.simonwillison.net", "swn-entries", "https://simonwillison.net/atom/entries/", "2017-10-01T21:12:32.478849+00:00"]
  [3, "feeds.simonwillison.net", "swn-links", "https://simonwillison.net/atom/links/", "2017-10-01T21:12:54.820729+00:00"]
  ```

project_urls:
  {"CI": "https://github.com/simonw/sqlite-diffable/actions", "Changelog": "https://github.com/simonw/sqlite-diffable/releases", "Homepage": "https://github.com/simonw/sqlite-diffable", "Issues": "https://github.com/simonw/sqlite-diffable/issues"}


name: sqlite-generate
summary: Tool for generating demo SQLite databases
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/sqlite-generate
license: Apache License, Version 2.0
package_url: https://pypi.org/project/sqlite-generate/
project_url: https://pypi.org/project/sqlite-generate/
release_url: https://pypi.org/project/sqlite-generate/1.1.1/
requires_dist: ["click", "Faker", "sqlite-utils", "pytest ; extra == 'test'"]
version: 1.1.1
description:
  # sqlite-generate

  [![PyPI](https://img.shields.io/pypi/v/sqlite-generate.svg)](https://pypi.org/project/sqlite-generate/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-generate?label=changelog)](https://github.com/simonw/sqlite-generate/releases)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sqlite-generate/blob/master/LICENSE)

  Tool for generating demo SQLite databases

  ## Installation

  Install this plugin using `pip`:

      $ pip install sqlite-generate

  ## Demo

  You can see a demo of the database generated using this command running in [Datasette](https://github.com/simonw/datasette) at https://sqlite-generate-demo.datasette.io/

  The demo is generated using the following command:

      sqlite-generate demo.db --seed seed --fts --columns=10 --fks=0,3 --pks=0,2

  ## Usage

  To generate a SQLite database file called `data.db` with 10 randomly named tables in it, run the following:

      sqlite-generate data.db

  You can use the `--tables` option to generate a different number of tables:

      sqlite-generate data.db --tables 20

  You can run the command against the same database file multiple times to keep adding new tables, using different settings for each batch of generated tables.

  By default each table will contain a random number of rows between 0 and 200. You can customize this with the `--rows` option:

      sqlite-generate data.db --rows 20

  This will insert 20 rows into each table.

      sqlite-generate data.db --rows 500,2000

  This inserts a random number of rows between 500 and 2000 into each table.

  Each table will have 5 columns. You can change this using `--columns`:

      sqlite-generate data.db --columns 10

  `--columns` can also accept a range:

      sqlite-generate data.db --columns 5,15

  You can control the random number seed used with the `--seed` option. This will result in the exact same database file being created by multiple runs of the tool:

      sqlite-generate data.db --seed=myseed

  By default each table will contain between 0 and 2 foreign key columns to other tables. You can control this using the `--fks` option, with either a single number or a range:

      sqlite-generate data.db --columns=20 --fks=5,15

  Each table will have a single primary key column called `id`. You can use the `--pks=` option to change the number of primary key columns on each table. Drop it to 0 to generate [rowid tables](https://www.sqlite.org/rowidtable.html). Increase it above 1 to generate tables with compound primary keys. Or use a range to get a random selection of different primary key layouts:

      sqlite-generate data.db --pks=0,2

  To configure [SQLite full-text search](https://www.sqlite.org/fts5.html) for all columns of type text, use `--fts`:

      sqlite-generate data.db --fts

  This will use FTS5 by default. To use [FTS4](https://www.sqlite.org/fts3.html) instead, use `--fts4`.

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd sqlite-generate
      python -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/sqlite-generate/actions", "Changelog": "https://github.com/simonw/sqlite-generate/releases", "Homepage": "https://github.com/simonw/sqlite-generate", "Issues": "https://github.com/simonw/sqlite-generate/issues"}


name: sqlite-transform
summary: Tool for running transformations on columns in a SQLite database.
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/sqlite-transform
license: Apache License, Version 2.0
package_url: https://pypi.org/project/sqlite-transform/
project_url: https://pypi.org/project/sqlite-transform/
project_urls: {"Homepage": "https://github.com/simonw/sqlite-transform"}
release_url: https://pypi.org/project/sqlite-transform/1.2.1/
requires_dist: ["dateutils", "tqdm", "click", "sqlite-utils", "pytest ; extra == 'test'"]
version: 1.2.1
description:
  # sqlite-transform

  ![No longer maintained](https://img.shields.io/badge/no%20longer-maintained-red)
  [![PyPI](https://img.shields.io/pypi/v/sqlite-transform.svg)](https://pypi.org/project/sqlite-transform/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-transform?include_prereleases&label=changelog)](https://github.com/simonw/sqlite-transform/releases)
  [![Tests](https://github.com/simonw/sqlite-transform/workflows/Test/badge.svg)](https://github.com/simonw/sqlite-transform/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/sqlite-transform/blob/main/LICENSE)

  Tool for running transformations on columns in a SQLite database.

  > **:warning: This tool is no longer maintained**
  >
  > I added a new tool to [sqlite-utils](https://sqlite-utils.datasette.io/) called [sqlite-utils convert](https://sqlite-utils.datasette.io/en/stable/cli.html#converting-data-in-columns) which provides a super-set of the functionality originally provided here. `sqlite-transform` is no longer maintained, and I recommend switching to using `sqlite-utils convert` instead.

  ## How to install

      pip install sqlite-transform

  ## parsedate and parsedatetime

  These subcommands will run all values in the specified column through `dateutils.parser.parse()` and replace them with the result, formatted as an ISO timestamp or ISO date.

  For example, if a row in the database has an `opened` column which contains `10/10/2019 08:10:00 PM`, running the following command:

      sqlite-transform parsedatetime my.db mytable opened

  Will result in that value being replaced by `2019-10-10T20:10:00`.

  Using the `parsedate` subcommand here would result in `2019-10-10` instead.

  In the case of ambiguous dates such as `03/04/05` these commands both default to assuming American-style `mm/dd/yy` format. You can pass `--dayfirst` to specify that the day should be assumed to be first, or `--yearfirst` for the year.

  ## jsonsplit

  The `jsonsplit` subcommand takes columns that contain a comma-separated list, for example a `tags` column containing records like `"trees,park,dogs"` and converts it into a JSON array `["trees", "park", "dogs"]`.

  This is useful for taking advantage of Datasette's [Facet by JSON array](https://docs.datasette.io/en/stable/facets.html#facet-by-json-array) feature.

      sqlite-transform jsonsplit my.db mytable tags

  It defaults to splitting on commas, but you can specify a different delimiter character using the `--delimiter` option, for example:

      sqlite-transform jsonsplit \
          my.db mytable tags --delimiter ';'

  Values within the array will be treated as strings, so a column containing `123,552,775` will be converted into the JSON array `["123", "552", "775"]`.

  You can specify a different type for these values using `--type int` or `--type float`, for example:

      sqlite-transform jsonsplit \
          my.db mytable tags --type int

  This will result in that column being converted into `[123, 552, 775]`.

  ## lambda for executing your own code

  The `lambda` subcommand lets you specify Python code which will be executed against the column.

  Here's how to convert a column to uppercase:

      sqlite-transform lambda my.db mytable mycolumn --code='str(value).upper()'

  The code you provide will be compiled into a function that takes `value` as a single argument. You can break your function body into multiple lines, provided the last line is a `return` statement:

      sqlite-transform lambda my.db mytable mycolumn --code='value = str(value)
      return value.upper()'

  You can also specify Python modules that should be imported and made available to your code using one or more `--import` options:

      sqlite-transform lambda my.db mytable mycolumn \
          --code='"\n".join(textwrap.wrap(value, 10))' \
          --import=textwrap

  The `--dry-run` option will output a preview of the transformation against the first ten rows, without modifying the database.

  ## Saving the result to a separate column

  Each of these commands accepts optional `--output` and `--output-type` options. These can be used to save the result of the transformation to a separate column, which will be created if the column does not already exist.

  To save the result of `jsonsplit` to a new column called `json_tags`, use the following:

      sqlite-transform jsonsplit my.db mytable tags \
        --output json_tags

  The type of the created column defaults to `text`, but a different column type can be specified using `--output-type`. This example will create a new floating point column called `float_id` with a copy of each item's ID increased by 0.5:

      sqlite-transform lambda my.db mytable id \
        --code 'float(value) + 0.5' \
        --output float_id \
        --output-type float

  You can drop the original column at the end of the operation by adding `--drop`.

  ## Splitting a column into multiple columns

  Sometimes you may wish to convert a single column into multiple derived columns. For example, you may have a `location` column containing `latitude,longitude` values which you wish to split out into separate `latitude` and `longitude` columns.

  You can achieve this using the `--multi` option to `sqlite-transform lambda`. This option expects your `--code` function to return a Python dictionary: new columns well be created and populated for each of the keys in that dictionary.

  For the `latitude,longitude` example you would use the following:

      sqlite-transform lambda demo.db places location \
        --code 'return {
          "latitude": float(value.split(",")[0]),
          "longitude": float(value.split(",")[1]),
        }' --multi

  The type of the returned values will be taken into account when creating the new columns. In this example, the resulting database schema will look like this:

  ```sql
  CREATE TABLE [places] (
      [location] TEXT,
      [latitude] FLOAT,
      [longitude] FLOAT
  );
  ```
  The code function can also return `None`, in which case its output will be ignored.

  You can drop the original column at the end of the operation by adding `--drop`.

  ## Disabling the progress bar

  By default each command will show a progress bar. Pass `-s` or `--silent` to hide that progress bar.


name: sqlite-utils
summary: CLI tool and Python utility functions for manipulating SQLite databases
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/sqlite-utils
license: Apache License, Version 2.0
package_url: https://pypi.org/project/sqlite-utils/
project_url: https://pypi.org/project/sqlite-utils/
release_url: https://pypi.org/project/sqlite-utils/3.30/
requires_python: >=3.6
version: 3.30
classifiers:
  ["Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "Intended Audience :: End Users/Desktop", "Intended Audience :: Science/Research", "License :: OSI Approved :: Apache Software License", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Topic :: Database"]

description:
  # sqlite-utils

  [![PyPI](https://img.shields.io/pypi/v/sqlite-utils.svg)](https://pypi.org/project/sqlite-utils/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/sqlite-utils?include_prereleases&label=changelog)](https://sqlite-utils.datasette.io/en/stable/changelog.html)
  [![Python 3.x](https://img.shields.io/pypi/pyversions/sqlite-utils.svg?logo=python&logoColor=white)](https://pypi.org/project/sqlite-utils/)
  [![Tests](https://github.com/simonw/sqlite-utils/workflows/Test/badge.svg)](https://github.com/simonw/sqlite-utils/actions?query=workflow%3ATest)
  [![Documentation Status](https://readthedocs.org/projects/sqlite-utils/badge/?version=stable)](http://sqlite-utils.datasette.io/en/stable/?badge=stable)
  [![codecov](https://codecov.io/gh/simonw/sqlite-utils/branch/main/graph/badge.svg)](https://codecov.io/gh/simonw/sqlite-utils)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/sqlite-utils/blob/main/LICENSE)
  [![discord](https://img.shields.io/discord/823971286308356157?label=discord)](https://discord.gg/Ass7bCAMDw)

  Python CLI utility and library for manipulating SQLite databases.

  ## Some feature highlights

  - [Pipe JSON](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data) (or [CSV or TSV](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-csv-or-tsv-data)) directly into a new SQLite database file, automatically creating a table with the appropriate schema
  - [Run in-memory SQL queries](https://sqlite-utils.datasette.io/en/stable/cli.html#querying-data-directly-using-an-in-memory-database), including joins, directly against data in CSV, TSV or JSON files and view the results
  - [Configure SQLite full-text search](https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search) against your database tables and run search queries against them, ordered by relevance
  - Run [transformations against your tables](https://sqlite-utils.datasette.io/en/stable/cli.html#transforming-tables) to make schema changes that SQLite `ALTER TABLE` does not directly support, such as changing the type of a column
  - [Extract columns](https://sqlite-utils.datasette.io/en/stable/cli.html#extracting-columns-into-a-separate-table) into separate tables to better normalize your existing data

  Read more on my blog, in this series of posts on [New features in sqlite-utils](https://simonwillison.net/series/sqlite-utils-features/) and other [entries tagged sqliteutils](https://simonwillison.net/tags/sqliteutils/).

  ## Installation

      pip install sqlite-utils

  Or if you use [Homebrew](https://brew.sh/) for macOS:

      brew install sqlite-utils

  ## Using as a CLI tool

  Now you can do things with the CLI utility like this:

      $ sqlite-utils memory dogs.csv "select * from t"
      [{"id": 1, "age": 4, "name": "Cleo"},
       {"id": 2, "age": 2, "name": "Pancakes"}]

      $ sqlite-utils insert dogs.db dogs dogs.csv --csv
      [####################################]  100%

      $ sqlite-utils tables dogs.db --counts
      [{"table": "dogs", "count": 2}]

      $ sqlite-utils dogs.db "select id, name from dogs"
      [{"id": 1, "name": "Cleo"},
       {"id": 2, "name": "Pancakes"}]

      $ sqlite-utils dogs.db "select * from dogs" --csv
      id,age,name
      1,4,Cleo
      2,2,Pancakes

      $ sqlite-utils dogs.db "select * from dogs" --table
        id    age  name
      ----  -----  --------
         1      4  Cleo
         2      2  Pancakes

  You can import JSON data into a new database table like this:

      $ curl https://api.github.com/repos/simonw/sqlite-utils/releases \
          | sqlite-utils insert releases.db releases - --pk id

  Or for data in a CSV file:

      $ sqlite-utils insert dogs.db dogs dogs.csv --csv

  `sqlite-utils memory` lets you import CSV or JSON data into an in-memory database and run SQL queries against it in a single command:

      $ cat dogs.csv | sqlite-utils memory - "select name, age from stdin"

  See the [full CLI documentation](https://sqlite-utils.datasette.io/en/stable/cli.html) for comprehensive coverage of many more commands.

  ## Using as a library

  You can also `import sqlite_utils` and use it as a Python library like this:

  ```python
  import sqlite_utils
  db = sqlite_utils.Database("demo_database.db")
  # This line creates a "dogs" table if one does not already exist:
  db["dogs"].insert_all([
      {"id": 1, "age": 4, "name": "Cleo"},
      {"id": 2, "age": 2, "name": "Pancakes"}
  ], pk="id")
  ```

  Check out the [full library documentation](https://sqlite-utils.datasette.io/en/stable/python-api.html) for everything else you can do with the Python library.

  ## Related projects

  * [Datasette](https://datasette.io/): A tool for exploring and publishing data
  * [csvs-to-sqlite](https://github.com/simonw/csvs-to-sqlite): Convert CSV files into a SQLite database
  * [db-to-sqlite](https://github.com/simonw/db-to-sqlite): CLI tool for exporting a MySQL or PostgreSQL database as a SQLite file
  * [dogsheep](https://dogsheep.github.io/): A family of tools for personal analytics, built on top of `sqlite-utils`

project_urls:
  {"CI": "https://github.com/simonw/sqlite-utils/actions", "Changelog": "https://sqlite-utils.datasette.io/en/stable/changelog.html", "Documentation": "https://sqlite-utils.datasette.io/en/stable/", "Homepage": "https://github.com/simonw/sqlite-utils", "Issues": "https://github.com/simonw/sqlite-utils/issues", "Source code": "https://github.com/simonw/sqlite-utils"}

requires_dist:
  ["sqlite-fts4", "click", "click-default-group-wheel", "tabulate", "python-dateutil", "furo ; extra == 'docs'", "sphinx-autobuild ; extra == 'docs'", "codespell ; extra == 'docs'", "sphinx-copybutton ; extra == 'docs'", "beanbag-docutils (>=2.0) ; extra == 'docs'", "flake8 ; extra == 'flake8'", "mypy ; extra == 'mypy'", "types-click ; extra == 'mypy'", "types-tabulate ; extra == 'mypy'", "types-python-dateutil ; extra == 'mypy'", "data-science-types ; extra == 'mypy'", "pytest ; extra == 'test'", "black ; extra == 'test'", "hypothesis ; extra == 'test'", "cogapp ; extra == 'test'"]


name: swarm-to-sqlite
summary: Create a SQLite database containing your checkin history from Foursquare Swarm
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/dogsheep/swarm-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/swarm-to-sqlite/
project_url: https://pypi.org/project/swarm-to-sqlite/
release_url: https://pypi.org/project/swarm-to-sqlite/0.3.3/
requires_dist: ["sqlite-utils (>=3.3)", "click", "requests", "pytest ; extra == 'test'"]
version: 0.3.3
description:
  # swarm-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/swarm-to-sqlite.svg)](https://pypi.org/project/swarm-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/dogsheep/swarm-to-sqlite?include_prereleases&label=changelog)](https://github.com/dogsheep/swarm-to-sqlite/releases)
  [![Tests](https://github.com/dogsheep/swarm-to-sqlite/workflows/Test/badge.svg)](https://github.com/dogsheep/swarm-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/dogsheep/swarm-to-sqlite/blob/main/LICENSE)

  Create a SQLite database containing your checkin history from Foursquare Swarm.

  ## How to install

      $ pip install swarm-to-sqlite

  ## Usage

  You will need to first obtain a valid OAuth token for your Foursquare account. You can do so using this tool: https://your-foursquare-oauth-token.glitch.me/

  Simplest usage is to simply provide the name of the database file you wish to write to. The tool will prompt you to paste in your token, and will then download your checkins and store them in the specified database file.

      $ swarm-to-sqlite checkins.db
      Please provide your Foursquare OAuth token:
      Importing 3699 checkins  [#########-----------------------] 27% 00:02:31

  You can also pass the token as a command-line option:

      $ swarm-to-sqlite checkins.db --token=XXX

  Or as an environment variable:

      $ export FOURSQUARE_TOKEN=XXX
      $ swarm-to-sqlite checkins.db

  To retrieve just checkins within the past X hours, days or weeks, use the `--since=` option. For example, to pull only checkins that happened within the last 10 days use:

      $ swarm-to-sqlite checkins.db --token=XXX --since=10d

  Use `2w` for two weeks, `10h` for ten hours, `3d` for three days.

  In addition to saving the checkins to a database, you can also write them to a JSON file using the `--save` option:

      $ swarm-to-sqlite checkins.db --save=checkins.json

  Having done this, you can re-import checkins directly from that file (rather than making API calls to fetch data from Foursquare) like this:

      $ swarm-to-sqlite checkins.db --load=checkins.json

  ## Using with Datasette

  The SQLite database produced by this tool is designed to be browsed using [Datasette](https://datasette.io/).

  You can install the [datasette-cluster-map](https://datasette.io/plugins/datasette-cluster-map) plugin to view your checkins on a map.

project_urls:
  {"CI": "https://github.com/dogsheep/swarm-to-sqlite/actions", "Changelog": "https://github.com/dogsheep/swarm-to-sqlite/releases", "Homepage": "https://github.com/dogsheep/swarm-to-sqlite", "Issues": "https://github.com/dogsheep/swarm-to-sqlite/issues"}


name: tableau-to-sqlite
summary: Fetch data from Tableau into a SQLite database
classifiers: []
author: Simon Willison
description_content_type: text/markdown
home_page: https://github.com/simonw/tableau-to-sqlite
license: Apache License, Version 2.0
package_url: https://pypi.org/project/tableau-to-sqlite/
project_url: https://pypi.org/project/tableau-to-sqlite/
release_url: https://pypi.org/project/tableau-to-sqlite/0.2.1/
requires_dist: ["click", "TableauScraper (==0.1.2)", "pytest ; extra == 'test'", "vcrpy ; extra == 'test'"]
requires_python: >=3.6
version: 0.2.1
description:
  # tableau-to-sqlite

  [![PyPI](https://img.shields.io/pypi/v/tableau-to-sqlite.svg)](https://pypi.org/project/tableau-to-sqlite/)
  [![Changelog](https://img.shields.io/github/v/release/simonw/tableau-to-sqlite?include_prereleases&label=changelog)](https://github.com/simonw/tableau-to-sqlite/releases)
  [![Tests](https://github.com/simonw/tableau-to-sqlite/workflows/Test/badge.svg)](https://github.com/simonw/tableau-to-sqlite/actions?query=workflow%3ATest)
  [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/tableau-to-sqlite/blob/master/LICENSE)

  Fetch data from Tableau into a SQLite database. A wrapper around [TableauScraper](https://github.com/bertrandmartel/tableau-scraping/).

  ## Installation

  Install this tool using `pip`:

      $ pip install tableau-to-sqlite

  ## Usage

  If you have the URL to a Tableau dashboard like this:

  https://results.mo.gov/t/COVID19/views/VaccinationsDashboard/Vaccinations

  You can pass that directly to the tool:

      tableau-to-sqlite tableau.db \
        https://results.mo.gov/t/COVID19/views/VaccinationsDashboard/Vaccinations

  This will create a SQLite database called `tableau.db` containing one table for each of the worksheets in that dashboard.

  If the dashboard is hosted on https://public.tableau.com/ you can instead provide the view name. This will be two strings separated by a `/` symbol - something like this:

      OregonCOVID-19VaccineProviderEnrollment/COVID-19VaccineProviderEnrollment

  Now run the tool like this:

      tableau-to-sqlite tableau.db \
          OregonCOVID-19VaccineProviderEnrollment/COVID-19VaccineProviderEnrollment

  ## Get the data as JSON or CSV

  If you're building a [git scraper](https://simonwillison.net/2020/Oct/9/git-scraping/) you may want to convert the data gathered by this tool to CSV or JSON to check into your repository.

  You can do that using [sqlite-utils](https://sqlite-utils.datasette.io/). Install it using `pip`:

      pip install sqlite-utils

  You can dump out a table as JSON like so:

      sqlite-utils rows tableau.db \
         'Admin Site and County Map Site No Info' > tableau.json

  Or as CSV like this:

      sqlite-utils rows tableau.db --csv \
         'Admin Site and County Map Site No Info' > tableau.csv

  ## Development

  To contribute to this tool, first checkout the code. Then create a new virtual environment:

      cd tableau-to-sqlite
      python -mvenv venv
      source venv/bin/activate

  Or if you are using `pipenv`:

      pipenv shell

  Now install the dependencies and tests:

      pip install -e '.[test]'

  To run the tests:

      pytest

project_urls:
  {"CI": "https://github.com/simonw/tableau-to-sqlite/actions", "Changelog": "https://github.com/simonw/tableau-to-sqlite/releases", "Homepage": "https://github.com/simonw/tableau-to-sqlite", "Issues": "https://github.com/simonw/tableau-to-sqlite/issues"}


