id,node_id,name,full_name,private,owner,html_url,description,fork,created_at,updated_at,pushed_at,homepage,size,stargazers_count,watchers_count,language,has_issues,has_projects,has_downloads,has_wiki,has_pages,forks_count,archived,disabled,open_issues_count,license,topics,forks,open_issues,watchers,default_branch,permissions,temp_clone_token,organization,network_count,subscribers_count,readme,readme_html,allow_forking,visibility,is_template,template_repository,web_commit_signoff_required,has_discussions
174715153,MDEwOlJlcG9zaXRvcnkxNzQ3MTUxNTM=,datasette-jellyfish,simonw/datasette-jellyfish,0,9599,https://github.com/simonw/datasette-jellyfish,Datasette plugin adding SQL functions for fuzzy text matching powered by Jellyfish,0,2019-03-09T16:02:01Z,2021-02-06T02:33:49Z,2021-02-06T02:34:18Z,https://datasette.io/plugins/datasette-jellyfish,15,9,9,Python,1,1,1,1,0,2,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",2,0,9,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,2,1,"# datasette-jellyfish
[](https://pypi.org/project/datasette-jellyfish/)
[](https://github.com/simonw/datasette-jellyfish/releases)
[](https://github.com/simonw/datasette-jellyfish/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-jellyfish/blob/main/LICENSE)
Datasette plugin that adds custom SQL functions for fuzzy string matching, built on top of the [Jellyfish](https://github.com/jamesturk/jellyfish) Python library by James Turk and Michael Stephens.
Interactive demos:
* [soundex, metaphone, nysiis, match_rating_codex comparison](https://latest-with-plugins.datasette.io/fixtures?sql=SELECT%0D%0A++++soundex%28%3As%29%2C+%0D%0A++++metaphone%28%3As%29%2C+%0D%0A++++nysiis%28%3As%29%2C+%0D%0A++++match_rating_codex%28%3As%29&s=demo).
* [distance functions comparison](https://latest-with-plugins.datasette.io/fixtures?sql=SELECT%0D%0A++++levenshtein_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++damerau_levenshtein_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++hamming_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++jaro_similarity%28%3As1%2C+%3As2%29%2C%0D%0A++++jaro_winkler_similarity%28%3As1%2C+%3As2%29%2C%0D%0A++++match_rating_comparison%28%3As1%2C+%3As2%29%3B&s1=barrack+obama&s2=barrack+h+obama)
Examples:
SELECT soundex(""hello"");
-- Outputs H400
SELECT metaphone(""hello"");
-- Outputs HL
SELECT nysiis(""hello"");
-- Outputs HAL
SELECT match_rating_codex(""hello"");
-- Outputs HLL
SELECT porter_stem(""running"");
-- Outputs run
SELECT levenshtein_distance(""hello"", ""hello world"");
-- Outputs 6
SELECT damerau_levenshtein_distance(""hello"", ""hello world"");
-- Outputs 6
SELECT hamming_distance(""hello"", ""hello world"");
-- Outputs 6
SELECT jaro_similarity(""hello"", ""hello world"");
-- Outputs 0.8181818181818182
SELECT jaro_winkler_similarity(""hello"", ""hello world"");
-- Outputs 0.890909090909091
SELECT match_rating_comparison(""hello"", ""helloo"");
-- Outputs 1
See [the Jellyfish documentation](https://jellyfish.readthedocs.io/en/latest/) for an explanation of each of these functions.","
datasette-jellyfish
Datasette plugin that adds custom SQL functions for fuzzy string matching, built on top of the Jellyfish Python library by James Turk and Michael Stephens.
",,,,,,
175550127,MDEwOlJlcG9zaXRvcnkxNzU1NTAxMjc=,yaml-to-sqlite,simonw/yaml-to-sqlite,0,9599,https://github.com/simonw/yaml-to-sqlite,Utility for converting YAML files to SQLite,0,2019-03-14T04:49:08Z,2021-06-13T09:04:40Z,2021-06-13T04:45:52Z,,19,36,36,Python,1,1,1,1,0,2,0,0,0,apache-2.0,"[""yaml"", ""sqlite"", ""datasette-io"", ""datasette-tool""]",2,0,36,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,2,1,"# yaml-to-sqlite
[](https://pypi.org/project/yaml-to-sqlite/)
[](https://github.com/simonw/yaml-to-sqlite/releases)
[](https://github.com/simonw/yaml-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/yaml-to-sqlite/blob/main/LICENSE)
Load the contents of a YAML file into a SQLite database table.
```
$ yaml-to-sqlite --help
Usage: yaml-to-sqlite [OPTIONS] DB_PATH TABLE YAML_FILE
Convert YAML files to SQLite
Options:
--version Show the version and exit.
--pk TEXT Column to use as a primary key
--single-column TEXT If YAML file is a list of values, populate this column
--help Show this message and exit.
```
## Usage
Given a `news.yml` file containing the following:
```yaml
- date: 2021-06-05
body: |-
[Datasette 0.57](https://docs.datasette.io/en/stable/changelog.html#v0-57) is out with an important security patch.
- date: 2021-05-10
body: |-
[Django SQL Dashboard](https://simonwillison.net/2021/May/10/django-sql-dashboard/) is a new tool that brings a useful authenticated subset of Datasette to Django projects that are built on top of PostgreSQL.
```
Running this command:
```bash
$ yaml-to-sqlite news.db stories news.yml
```
Will create a database file with this schema:
```bash
$ sqlite-utils schema news.db
CREATE TABLE [stories] (
[date] TEXT,
[body] TEXT
);
```
The `--pk` option can be used to set a column as the primary key for the table:
```bash
$ yaml-to-sqlite news.db stories news.yml --pk date
$ sqlite-utils schema news.db
CREATE TABLE [stories] (
[date] TEXT PRIMARY KEY,
[body] TEXT
);
```
## Single column YAML lists
The `--single-column` option can be used when the YAML file is a list of values, for example a file called `dogs.yml` containing the following:
```yaml
- Cleo
- Pancakes
- Nixie
```
Running this command:
```bash
$ yaml-to-sqlite dogs.db dogs.yaml --single-column=name
```
Will create a single `dogs` table with a single `name` column that is the primary key:
```bash
$ sqlite-utils schema dogs.db
CREATE TABLE [dogs] (
[name] TEXT PRIMARY KEY
);
$ sqlite-utils dogs.db 'select * from dogs' -t
name
--------
Cleo
Pancakes
Nixie
```
","
yaml-to-sqlite
Load the contents of a YAML file into a SQLite database table.
$ yaml-to-sqlite --help
Usage: yaml-to-sqlite [OPTIONS] DB_PATH TABLE YAML_FILE
Convert YAML files to SQLite
Options:
--version Show the version and exit.
--pk TEXT Column to use as a primary key
--single-column TEXT If YAML file is a list of values, populate this column
--help Show this message and exit.
Usage
Given a news.yml file containing the following:
- date: 2021-06-05body: |- [Datasette 0.57](https://docs.datasette.io/en/stable/changelog.html#v0-57) is out with an important security patch.
- date: 2021-05-10body: |- [Django SQL Dashboard](https://simonwillison.net/2021/May/10/django-sql-dashboard/) is a new tool that brings a useful authenticated subset of Datasette to Django projects that are built on top of PostgreSQL.
Will create a single dogs table with a single name column that is the primary key:
$ sqlite-utils schema dogs.db
CREATE TABLE [dogs] (
[name] TEXT PRIMARY KEY
);
$ sqlite-utils dogs.db 'select * from dogs' -t
name
--------
Cleo
Pancakes
Nixie
",,,,,,
184168864,MDEwOlJlcG9zaXRvcnkxODQxNjg4NjQ=,datasette-render-html,simonw/datasette-render-html,0,9599,https://github.com/simonw/datasette-render-html,Plugin for selectively rendering the HTML is specific columns,0,2019-04-30T01:21:25Z,2020-09-24T04:44:47Z,2021-03-17T03:57:13Z,,8,2,2,Python,1,1,1,1,0,2,0,0,1,,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",2,1,2,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,2,1,"# datasette-render-html
[](https://pypi.org/project/datasette-render-html/)
[](https://circleci.com/gh/simonw/datasette-render-html)
[](https://github.com/simonw/datasette-render-html/blob/master/LICENSE)
This Datasette plugin lets you configure Datasette to render specific columns as HTML in the table and row interfaces.
This means you can store HTML in those columns and have it rendered as such on those pages.
If you have a database called `docs.db` containing a `glossary` table and you want the `definition` column in that table to be rendered as HTML, you would use a `metadata.json` file that looks like this:
{
""databases"": {
""docs"": {
""tables"": {
""glossary"": {
""plugins"": {
""datasette-render-html"": {
""columns"": [""definition""]
}
}
}
}
}
}
}
## Security
This plugin allows HTML to be rendered exactly as it is stored in the database. As such, you should be sure only to use this against columns with content that you trust - otherwise you could open yourself up to an [XSS attack](https://owasp.org/www-community/attacks/xss/).
It's possible to configure this plugin to apply to columns with specific names across whole databases or the full Datasette instance, but doing so is not safe. It could open you up to XSS vulnerabilities where an attacker composes a SQL query that results in a column containing unsafe HTML.
As such, you should only use this plugin against specific columns in specific tables, as shown in the example above.
","
datasette-render-html
This Datasette plugin lets you configure Datasette to render specific columns as HTML in the table and row interfaces.
This means you can store HTML in those columns and have it rendered as such on those pages.
If you have a database called docs.db containing a glossary table and you want the definition column in that table to be rendered as HTML, you would use a metadata.json file that looks like this:
This plugin allows HTML to be rendered exactly as it is stored in the database. As such, you should be sure only to use this against columns with content that you trust - otherwise you could open yourself up to an XSS attack.
It's possible to configure this plugin to apply to columns with specific names across whole databases or the full Datasette instance, but doing so is not safe. It could open you up to XSS vulnerabilities where an attacker composes a SQL query that results in a column containing unsafe HTML.
As such, you should only use this plugin against specific columns in specific tables, as shown in the example above.
",,,,,,
191022928,MDEwOlJlcG9zaXRvcnkxOTEwMjI5Mjg=,datasette-render-binary,simonw/datasette-render-binary,0,9599,https://github.com/simonw/datasette-render-binary,Datasette plugin for rendering binary data,0,2019-06-09T15:25:52Z,2021-06-02T09:29:20Z,2019-06-13T16:14:31Z,,62,7,7,Python,1,1,1,1,0,0,0,0,1,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,1,7,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-render-binary
[](https://pypi.org/project/datasette-render-binary/)
[](https://circleci.com/gh/simonw/datasette-render-binary)
[](https://github.com/simonw/datasette-render-binary/blob/master/LICENSE)
Datasette plugin for rendering binary data.
Install this plugin in the same environment as Datasette to enable this new functionality:
pip install datasette-render-binary
Binary data in cells will now be rendered as a mixture of characters and octets.
","
datasette-render-binary
Datasette plugin for rendering binary data.
Install this plugin in the same environment as Datasette to enable this new functionality:
pip install datasette-render-binary
Binary data in cells will now be rendered as a mixture of characters and octets.
",,,,,,
195087137,MDEwOlJlcG9zaXRvcnkxOTUwODcxMzc=,datasette-auth-github,simonw/datasette-auth-github,0,9599,https://github.com/simonw/datasette-auth-github,Datasette plugin that authenticates users against GitHub,0,2019-07-03T16:02:53Z,2021-06-03T11:42:54Z,2021-02-25T06:40:17Z,https://datasette-auth-github-demo.datasette.io/,119,34,34,Python,1,1,1,1,0,4,0,0,3,apache-2.0,"[""asgi"", ""datasette"", ""datasette-plugin"", ""datasette-io""]",4,3,34,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,4,1,"# datasette-auth-github
[](https://pypi.org/project/datasette-auth-github/)
[](https://github.com/simonw/datasette-auth-github/releases)
[](https://github.com/simonw/datasette-auth-github/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-auth-github/blob/main/LICENSE)
Datasette plugin that authenticates users against GitHub.
- [Setup instructions](#setup-instructions)
- [The authenticated actor](#the-authenticated-actor)
- [Restricting access to specific users](#restricting-access-to-specific-users)
- [Restricting access to specific GitHub organizations or teams](#restricting-access-to-specific-github-organizations-or-teams)
- [What to do if a user is removed from an organization or team](#what-to-do-if-a-user-is-removed-from-an-organization-or-team)
## Setup instructions
* Install the plugin: `datasette install datasette-auth-github`
* Create a GitHub OAuth app: https://github.com/settings/applications/new
* Set the Authorization callback URL to `http://127.0.0.1:8001/-/github-auth-callback`
* Create a `metadata.json` file with the following structure:
```json
{
""title"": ""datasette-auth-github demo"",
""plugins"": {
""datasette-auth-github"": {
""client_id"": {""$env"": ""GITHUB_CLIENT_ID""},
""client_secret"": {""$env"": ""GITHUB_CLIENT_SECRET""}
}
}
}
```
Now you can start Datasette like this, passing in the secrets as environment variables:
$ GITHUB_CLIENT_ID=XXX GITHUB_CLIENT_SECRET=YYY datasette \
fixtures.db -m metadata.json
Note that hard-coding secrets in `metadata.json` is a bad idea as they will be visible to anyone who can navigate to `/-/metadata`. Instead, we use Datasette's mechanism for [adding secret plugin configuration options](https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values).
By default anonymous users will still be able to interact with Datasette. If you wish all users to have to sign in with a GitHub account first, add this to your ``metadata.json``:
```json
{
""allow"": {
""id"": ""*""
},
""plugins"": {
""datasette-auth-github"": {
""..."": ""...""
}
}
}
```
## The authenticated actor
Visit `/-/actor` when signed in to see the shape of the authenticated actor. It should look something like this:
```json
{
""actor"": {
""display"": ""simonw"",
""gh_id"": ""9599"",
""gh_name"": ""Simon Willison"",
""gh_login"": ""simonw"",
""gh_email"": ""..."",
""gh_orgs"": [
""dogsheep"",
""datasette-project""
],
""gh_teams"": [
""dogsheep/test""
]
}
}
```
The `gh_orgs` and `gh_teams` properties will only be present if you used `load_teams` or `load_orgs`, documented below.
## Restricting access to specific users
You can use Datasette's [permissions mechanism](https://docs.datasette.io/en/stable/authentication.html) to specify which user or users are allowed to access your instance. Here's how to restrict access to just GitHub user `simonw`:
```json
{
""allow"": {
""gh_login"": ""simonw""
},
""plugins"": {
""datasette-auth-github"": {
""..."": ""...""
}
}
}
```
This `""allow""` block can be positioned at the database, table or query level instead: see [Configuring permissions in metadata.json](https://docs.datasette.io/en/stable/authentication.html#configuring-permissions-in-metadata-json) for details.
Note that GitHub allows users to change their username, and it is possible for other people to claim old usernames. If you are concerned that your users may change their usernames you can key the allow blocks against GitHub user IDs instead, which do not change:
```json
{
""allow"": {
""gh_id"": ""9599""
}
}
```
## Restricting access to specific GitHub organizations or teams
You can also restrict access to users who are members of a specific GitHub organization.
You'll need to configure the plugin to check if the user is a member of that organization when they first sign in. You can do that using the `""load_orgs""` plugin configuration option.
Then you can use `""allow"": {""gh_orgs"": [...]}` to specify which organizations are allowed access.
```json
{
""plugins"": {
""datasette-auth-github"": {
""..."": ""..."",
""load_orgs"": [""your-organization""]
}
},
""allow"": {
""gh_orgs"": ""your-organization""
}
}
```
If your organization is [arranged into teams](https://help.github.com/en/articles/organizing-members-into-teams) you can restrict access to a specific team like this:
```json
{
""plugins"": {
""datasette-auth-github"": {
""..."": ""..."",
""load_teams"": [
""your-organization/staff"",
""your-organization/engineering"",
]
}
},
""allows"": {
""gh_team"": ""your-organization/engineering""
}
}
```
## What to do if a user is removed from an organization or team
A user's organization and team memberships are checked once, when they first sign in. Those teams and organizations are then persisted in the user's signed `ds_actor` cookie.
This means that if a user is removed from an organization or team but still has a Datasette cookie, they will still be able to access that Datasette instance.
You can remedy this by rotating the `DATASETTE_SECRET` environment variable any time you make changes to your GitHub organization members.
Changing this value will cause all of your existing users to be signed out, by invalidating their cookies. When they sign back in again their new memberships will be recorded in a new cookie.
See [Configuring the secret](https://docs.datasette.io/en/stable/settings.html?highlight=secret#configuring-the-secret) in the Datasette documentation for more details.
","
datasette-auth-github
Datasette plugin that authenticates users against GitHub.
Note that hard-coding secrets in metadata.json is a bad idea as they will be visible to anyone who can navigate to /-/metadata. Instead, we use Datasette's mechanism for adding secret plugin configuration options.
By default anonymous users will still be able to interact with Datasette. If you wish all users to have to sign in with a GitHub account first, add this to your metadata.json:
The gh_orgs and gh_teams properties will only be present if you used load_teams or load_orgs, documented below.
Restricting access to specific users
You can use Datasette's permissions mechanism to specify which user or users are allowed to access your instance. Here's how to restrict access to just GitHub user simonw:
Note that GitHub allows users to change their username, and it is possible for other people to claim old usernames. If you are concerned that your users may change their usernames you can key the allow blocks against GitHub user IDs instead, which do not change:
{
""allow"": {
""gh_id"": ""9599""
}
}
Restricting access to specific GitHub organizations or teams
You can also restrict access to users who are members of a specific GitHub organization.
You'll need to configure the plugin to check if the user is a member of that organization when they first sign in. You can do that using the ""load_orgs"" plugin configuration option.
Then you can use ""allow"": {""gh_orgs"": [...]} to specify which organizations are allowed access.
What to do if a user is removed from an organization or team
A user's organization and team memberships are checked once, when they first sign in. Those teams and organizations are then persisted in the user's signed ds_actor cookie.
This means that if a user is removed from an organization or team but still has a Datasette cookie, they will still be able to access that Datasette instance.
You can remedy this by rotating the DATASETTE_SECRET environment variable any time you make changes to your GitHub organization members.
Changing this value will cause all of your existing users to be signed out, by invalidating their cookies. When they sign back in again their new memberships will be recorded in a new cookie.
",,,,,,
206202864,MDEwOlJlcG9zaXRvcnkyMDYyMDI4NjQ=,inaturalist-to-sqlite,dogsheep/inaturalist-to-sqlite,0,53015001,https://github.com/dogsheep/inaturalist-to-sqlite,Create a SQLite database containing your observation history from iNaturalist,0,2019-09-04T01:21:21Z,2020-12-19T05:18:38Z,2020-10-22T00:08:58Z,,17,2,2,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""sqlite"", ""inaturalist"", ""datasette"", ""dogsheep"", ""datasette-io"", ""datasette-tool""]",0,0,2,master,"{""admin"": false, ""push"": false, ""pull"": false}",,53015001,0,1,"# inaturalist-to-sqlite
[](https://pypi.org/project/inaturalist-to-sqlite/)
[](https://circleci.com/gh/dogsheep/inaturalist-to-sqlite)
[](https://github.com/dogsheep/inaturalist-to-sqlite/blob/master/LICENSE)
Create a SQLite database containing your observation history from [iNaturalist](https://www.inaturalist.org/).
## How to install
$ pip install inaturalist-to-sqlite
## Usage
$ inaturalist-to-sqlite inaturalist.db yourusername
(Or try `simonw` if you don't yet have an iNaturalist account)
This will import all of your iNaturalist observations into a SQLite database called `inaturalist.db`.","
inaturalist-to-sqlite
Create a SQLite database containing your observation history from iNaturalist.
(Or try simonw if you don't yet have an iNaturalist account)
This will import all of your iNaturalist observations into a SQLite database called inaturalist.db.
",,,,,,
207630174,MDEwOlJlcG9zaXRvcnkyMDc2MzAxNzQ=,datasette-rure,simonw/datasette-rure,0,9599,https://github.com/simonw/datasette-rure,Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine,0,2019-09-10T18:09:33Z,2020-12-04T04:26:53Z,2019-09-11T22:59:38Z,,19,4,4,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""sqlite"", ""regular-expressions"", ""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,4,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-rure
[](https://pypi.org/project/datasette-rure/)
[](https://circleci.com/gh/simonw/datasette-rure)
[](https://github.com/simonw/datasette-rure/blob/master/LICENSE)
Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine
Install this plugin in the same environment as Datasette to enable the `regexp()` SQL function.
$ pip install datasette-rure
The plugin is built on top of the [rure-python](https://github.com/davidblewett/rure-python) library by David Blewett.
## regexp() to test regular expressions
You can test if a value matches a regular expression like this:
select regexp('hi.*there', 'hi there')
-- returns 1
select regexp('not.*there', 'hi there')
-- returns 0
You can also use SQLite's custom syntax to run matches:
select 'hi there' REGEXP 'hi.*there'
-- returns 1
This means you can select rows based on regular expression matches - for example, to select every article where the title begins with an E or an F:
select * from articles where title REGEXP '^[EF]'
Try this out: [REGEXP interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+*+from+articles+where+title+REGEXP+%27%5E%5BEF%5D%27)
## regexp_match() to extract groups
You can extract captured subsets of a pattern using `regexp_match()`.
select regexp_match('.*( and .*)', title) as n from articles where n is not null
-- Returns the ' and X' component of any matching titles, e.g.
-- and Recognition
-- and Transitions Their Place
-- etc
This will return the first parenthesis match when called with two arguments. You can call it with three arguments to indicate which match you would like to extract:
select regexp_match('.*(and)(.*)', title, 2) as n from articles where n is not null
The function will return `null` for invalid inputs e.g. a pattern without capture groups.
Try this out: [regexp_match() interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+%27WHY+%27+%7C%7C+regexp_match%28%27Why+%28.*%29%27%2C+title%29+as+t+from+articles+where+t+is+not+null)
## regexp_matches() to extract multiple matches at once
The `regexp_matches()` function can be used to extract multiple patterns from a single string. The result is returned as a JSON array, which can then be further processed using SQLite's [JSON functions](https://www.sqlite.org/json1.html).
The first argument is a regular expression with named capture groups. The second argument is the string to be matched.
select regexp_matches(
'hello (?P\w+) the (?P\w+)',
'hello bob the dog, hello maggie the cat, hello tarquin the otter'
)
This will return a list of JSON objects, each one representing the named captures from the original regular expression:
[
{""name"": ""bob"", ""species"": ""dog""},
{""name"": ""maggie"", ""species"": ""cat""},
{""name"": ""tarquin"", ""species"": ""otter""}
]
Try this out: [regexp_matches() interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+regexp_matches%28%0D%0A++++%27hello+%28%3FP%3Cname%3E%5Cw%2B%29+the+%28%3FP%3Cspecies%3E%5Cw%2B%29%27%2C%0D%0A++++%27hello+bob+the+dog%2C+hello+maggie+the+cat%2C+hello+tarquin+the+otter%27%0D%0A%29)
","
datasette-rure
Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine
Install this plugin in the same environment as Datasette to enable the regexp() SQL function.
$ pip install datasette-rure
The plugin is built on top of the rure-python library by David Blewett.
regexp() to test regular expressions
You can test if a value matches a regular expression like this:
You can extract captured subsets of a pattern using regexp_match().
select regexp_match('.*( and .*)', title) as n from articles where n is not null
-- Returns the ' and X' component of any matching titles, e.g.
-- and Recognition
-- and Transitions Their Place
-- etc
This will return the first parenthesis match when called with two arguments. You can call it with three arguments to indicate which match you would like to extract:
select regexp_match('.*(and)(.*)', title, 2) as n from articles where n is not null
The function will return null for invalid inputs e.g. a pattern without capture groups.
regexp_matches() to extract multiple matches at once
The regexp_matches() function can be used to extract multiple patterns from a single string. The result is returned as a JSON array, which can then be further processed using SQLite's JSON functions.
The first argument is a regular expression with named capture groups. The second argument is the string to be matched.
select regexp_matches(
'hello (?P<name>\w+) the (?P<species>\w+)',
'hello bob the dog, hello maggie the cat, hello tarquin the otter'
)
This will return a list of JSON objects, each one representing the named captures from the original regular expression:
",,,,,,
217216787,MDEwOlJlcG9zaXRvcnkyMTcyMTY3ODc=,datasette-haversine,simonw/datasette-haversine,0,9599,https://github.com/simonw/datasette-haversine,Datasette plugin that adds a custom SQL function for haversine distances,0,2019-10-24T05:16:14Z,2021-07-28T20:13:38Z,2021-07-28T20:14:24Z,,8,1,1,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,1,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-haversine
[](https://pypi.org/project/datasette-haversine/)
[](https://github.com/simonw/datasette-haversine/releases)
[](https://github.com/simonw/datasette-haversine/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-haversine/blob/main/LICENSE)
Datasette plugin that adds a custom SQL function for haversine distances
Install this plugin in the same environment as Datasette to enable the `haversine()` SQL function.
$ pip install datasette-haversine
The plugin is built on top of the [haversine](https://github.com/mapado/haversine) library.
## haversine() to calculate distances
```sql
select haversine(lat1, lon1, lat2, lon2);
```
This will return the distance in kilometers between the point defined by `(lat1, lon1)` and the point defined by `(lat2, lon2)`.
## Custom units
By default `haversine()` returns results in km. You can pass an optional third argument to get results in a different unit:
- `ft` for feet
- `m` for meters
- `in` for inches
- `mi` for miles
- `nmi` for nautical miles
- `km` for kilometers (the default)
```sql
select haversine(lat1, lon1, lat2, lon2, 'mi');
```
","
datasette-haversine
Datasette plugin that adds a custom SQL function for haversine distances
Install this plugin in the same environment as Datasette to enable the haversine() SQL function.
$ pip install datasette-haversine
The plugin is built on top of the haversine library.
haversine() to calculate distances
select haversine(lat1, lon1, lat2, lon2);
This will return the distance in kilometers between the point defined by (lat1, lon1) and the point defined by (lat2, lon2).
Custom units
By default haversine() returns results in km. You can pass an optional third argument to get results in a different unit:
ft for feet
m for meters
in for inches
mi for miles
nmi for nautical miles
km for kilometers (the default)
select haversine(lat1, lon1, lat2, lon2, 'mi');
",,,,,,
219372133,MDEwOlJlcG9zaXRvcnkyMTkzNzIxMzM=,sqlite-transform,simonw/sqlite-transform,0,9599,https://github.com/simonw/sqlite-transform,Tool for running transformations on columns in a SQLite database,0,2019-11-03T22:07:53Z,2021-08-02T22:06:23Z,2021-08-02T22:07:57Z,,64,29,29,Python,1,1,1,1,0,1,0,0,0,apache-2.0,"[""sqlite"", ""datasette-io"", ""datasette-tool""]",1,0,29,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,1,1,"# sqlite-transform
[](https://pypi.org/project/sqlite-transform/)
[](https://github.com/simonw/sqlite-transform/releases)
[](https://github.com/simonw/sqlite-transform/actions?query=workflow%3ATest)
[](https://github.com/dogsheep/sqlite-transform/blob/main/LICENSE)
Tool for running transformations on columns in a SQLite database.
> **:warning: This tool is no longer maintained**
>
> I added a new tool to [sqlite-utils](https://sqlite-utils.datasette.io/) called [sqlite-utils convert](https://sqlite-utils.datasette.io/en/stable/cli.html#converting-data-in-columns) which provides a super-set of the functionality originally provided here. `sqlite-transform` is no longer maintained, and I recommend switching to using `sqlite-utils convert` instead.
## How to install
pip install sqlite-transform
## parsedate and parsedatetime
These subcommands will run all values in the specified column through `dateutils.parser.parse()` and replace them with the result, formatted as an ISO timestamp or ISO date.
For example, if a row in the database has an `opened` column which contains `10/10/2019 08:10:00 PM`, running the following command:
sqlite-transform parsedatetime my.db mytable opened
Will result in that value being replaced by `2019-10-10T20:10:00`.
Using the `parsedate` subcommand here would result in `2019-10-10` instead.
In the case of ambiguous dates such as `03/04/05` these commands both default to assuming American-style `mm/dd/yy` format. You can pass `--dayfirst` to specify that the day should be assumed to be first, or `--yearfirst` for the year.
## jsonsplit
The `jsonsplit` subcommand takes columns that contain a comma-separated list, for example a `tags` column containing records like `""trees,park,dogs""` and converts it into a JSON array `[""trees"", ""park"", ""dogs""]`.
This is useful for taking advantage of Datasette's [Facet by JSON array](https://docs.datasette.io/en/stable/facets.html#facet-by-json-array) feature.
sqlite-transform jsonsplit my.db mytable tags
It defaults to splitting on commas, but you can specify a different delimiter character using the `--delimiter` option, for example:
sqlite-transform jsonsplit \
my.db mytable tags --delimiter ';'
Values within the array will be treated as strings, so a column containing `123,552,775` will be converted into the JSON array `[""123"", ""552"", ""775""]`.
You can specify a different type for these values using `--type int` or `--type float`, for example:
sqlite-transform jsonsplit \
my.db mytable tags --type int
This will result in that column being converted into `[123, 552, 775]`.
## lambda for executing your own code
The `lambda` subcommand lets you specify Python code which will be executed against the column.
Here's how to convert a column to uppercase:
sqlite-transform lambda my.db mytable mycolumn --code='str(value).upper()'
The code you provide will be compiled into a function that takes `value` as a single argument. You can break your function body into multiple lines, provided the last line is a `return` statement:
sqlite-transform lambda my.db mytable mycolumn --code='value = str(value)
return value.upper()'
You can also specify Python modules that should be imported and made available to your code using one or more `--import` options:
sqlite-transform lambda my.db mytable mycolumn \
--code='""\n"".join(textwrap.wrap(value, 10))' \
--import=textwrap
The `--dry-run` option will output a preview of the transformation against the first ten rows, without modifying the database.
## Saving the result to a separate column
Each of these commands accepts optional `--output` and `--output-type` options. These can be used to save the result of the transformation to a separate column, which will be created if the column does not already exist.
To save the result of `jsonsplit` to a new column called `json_tags`, use the following:
sqlite-transform jsonsplit my.db mytable tags \
--output json_tags
The type of the created column defaults to `text`, but a different column type can be specified using `--output-type`. This example will create a new floating point column called `float_id` with a copy of each item's ID increased by 0.5:
sqlite-transform lambda my.db mytable id \
--code 'float(value) + 0.5' \
--output float_id \
--output-type float
You can drop the original column at the end of the operation by adding `--drop`.
## Splitting a column into multiple columns
Sometimes you may wish to convert a single column into multiple derived columns. For example, you may have a `location` column containing `latitude,longitude` values which you wish to split out into separate `latitude` and `longitude` columns.
You can achieve this using the `--multi` option to `sqlite-transform lambda`. This option expects your `--code` function to return a Python dictionary: new columns well be created and populated for each of the keys in that dictionary.
For the `latitude,longitude` example you would use the following:
sqlite-transform lambda demo.db places location \
--code 'return {
""latitude"": float(value.split("","")[0]),
""longitude"": float(value.split("","")[1]),
}' --multi
The type of the returned values will be taken into account when creating the new columns. In this example, the resulting database schema will look like this:
```sql
CREATE TABLE [places] (
[location] TEXT,
[latitude] FLOAT,
[longitude] FLOAT
);
```
The code function can also return `None`, in which case its output will be ignored.
You can drop the original column at the end of the operation by adding `--drop`.
## Disabling the progress bar
By default each command will show a progress bar. Pass `-s` or `--silent` to hide that progress bar.
","
sqlite-transform
Tool for running transformations on columns in a SQLite database.
⚠️ This tool is no longer maintained
I added a new tool to sqlite-utils called sqlite-utils convert which provides a super-set of the functionality originally provided here. sqlite-transform is no longer maintained, and I recommend switching to using sqlite-utils convert instead.
How to install
pip install sqlite-transform
parsedate and parsedatetime
These subcommands will run all values in the specified column through dateutils.parser.parse() and replace them with the result, formatted as an ISO timestamp or ISO date.
For example, if a row in the database has an opened column which contains 10/10/2019 08:10:00 PM, running the following command:
sqlite-transform parsedatetime my.db mytable opened
Will result in that value being replaced by 2019-10-10T20:10:00.
Using the parsedate subcommand here would result in 2019-10-10 instead.
In the case of ambiguous dates such as 03/04/05 these commands both default to assuming American-style mm/dd/yy format. You can pass --dayfirst to specify that the day should be assumed to be first, or --yearfirst for the year.
jsonsplit
The jsonsplit subcommand takes columns that contain a comma-separated list, for example a tags column containing records like ""trees,park,dogs"" and converts it into a JSON array [""trees"", ""park"", ""dogs""].
This is useful for taking advantage of Datasette's Facet by JSON array feature.
sqlite-transform jsonsplit my.db mytable tags
It defaults to splitting on commas, but you can specify a different delimiter character using the --delimiter option, for example:
Values within the array will be treated as strings, so a column containing 123,552,775 will be converted into the JSON array [""123"", ""552"", ""775""].
You can specify a different type for these values using --type int or --type float, for example:
sqlite-transform jsonsplit \
my.db mytable tags --type int
This will result in that column being converted into [123, 552, 775].
lambda for executing your own code
The lambda subcommand lets you specify Python code which will be executed against the column.
The code you provide will be compiled into a function that takes value as a single argument. You can break your function body into multiple lines, provided the last line is a return statement:
The --dry-run option will output a preview of the transformation against the first ten rows, without modifying the database.
Saving the result to a separate column
Each of these commands accepts optional --output and --output-type options. These can be used to save the result of the transformation to a separate column, which will be created if the column does not already exist.
To save the result of jsonsplit to a new column called json_tags, use the following:
The type of the created column defaults to text, but a different column type can be specified using --output-type. This example will create a new floating point column called float_id with a copy of each item's ID increased by 0.5:
You can drop the original column at the end of the operation by adding --drop.
Splitting a column into multiple columns
Sometimes you may wish to convert a single column into multiple derived columns. For example, you may have a location column containing latitude,longitude values which you wish to split out into separate latitude and longitude columns.
You can achieve this using the --multi option to sqlite-transform lambda. This option expects your --code function to return a Python dictionary: new columns well be created and populated for each of the keys in that dictionary.
For the latitude,longitude example you would use the following:
The type of the returned values will be taken into account when creating the new columns. In this example, the resulting database schema will look like this:
The code function can also return None, in which case its output will be ignored.
You can drop the original column at the end of the operation by adding --drop.
Disabling the progress bar
By default each command will show a progress bar. Pass -s or --silent to hide that progress bar.
",,,,,,
221802296,MDEwOlJlcG9zaXRvcnkyMjE4MDIyOTY=,datasette-template-sql,simonw/datasette-template-sql,0,9599,https://github.com/simonw/datasette-template-sql,Datasette plugin for executing SQL queries from templates,0,2019-11-14T23:05:34Z,2021-05-18T17:58:47Z,2021-05-18T17:58:44Z,https://datasette.io/plugins/datasette-template-sql,23,6,6,Python,1,1,1,1,0,0,0,0,1,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,1,6,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-template-sql
[](https://pypi.org/project/datasette-template-sql/)
[](https://github.com/simonw/datasette-template-sql/releases)
[](https://github.com/simonw/datasette-template-sql/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-template-sql/blob/main/LICENSE)
Datasette plugin for executing SQL queries from templates.
## Examples
[datasette.io](https://datasette.io/) uses this plugin extensively with [custom page templates](https://docs.datasette.io/en/stable/custom_templates.html#custom-pages), check out [simonw/datasette.io](https://github.com/simonw/datasette.io) to see how it works.
[www.niche-museums.com](https://www.niche-museums.com/) uses this plugin to run a custom themed website on top of Datasette. The full source code for the site [is here](https://github.com/simonw/museums) - see also [niche-museums.com, powered by Datasette](https://simonwillison.net/2019/Nov/25/niche-museums/).
[simonw/til](https://github.com/simonw/til) is another simple example, described in [Using a self-rewriting README powered by GitHub Actions to track TILs](https://simonwillison.net/2020/Apr/20/self-rewriting-readme/).
## Installation
Run this command to install the plugin in the same environment as Datasette:
$ pip install datasette-template-sql
## Usage
This plugin makes a new function, `sql(sql_query)`, available to your Datasette templates.
You can use it like this:
```html+jinja
{% for row in sql(""select 1 + 1 as two, 2 * 4 as eight"") %}
{% for key in row.keys() %}
{{ key }}: {{ row[key] }}
{% endfor %}
{% endfor %}
```
The plugin will execute SQL against the current database for the page in `database.html`, `table.html` and `row.html` templates. If a template does not have a current database (`index.html` for example) the query will execute against the first attached database.
### Queries with arguments
You can construct a SQL query using `?` or `:name` parameter syntax by passing a list or dictionary as a second argument:
```html+jinja
{% for row in sql(""select distinct topic from til order by topic"") %}
{{ row.topic }}
{% for til in sql(""select * from til where topic = ?"", [row.topic]) %}
{% endfor %}
```
Here's the same example using the `:topic` style of parameters:
```html+jinja
{% for row in sql(""select distinct topic from til order by topic"") %}
{{ row.topic }}
{% for til in sql(""select * from til where topic = :topic"", {""topic"": row.topic}) %}
{% endfor %}
```
### Querying a different database
You can pass an optional `database=` argument to specify a named database to use for the query. For example, if you have attached a `news.db` database you could use this:
```html+jinja
{% for article in sql(
""select headline, date, summary from articles order by date desc limit 5"",
database=""news""
) %}
{{ article.headline }}
{{ article.date }}
{{ article.summary }}
{% endfor %}
```
","
datasette-template-sql
Datasette plugin for executing SQL queries from templates.
Run this command to install the plugin in the same environment as Datasette:
$ pip install datasette-template-sql
Usage
This plugin makes a new function, sql(sql_query), available to your Datasette templates.
You can use it like this:
{%forrowinsql(""select 1 + 1 as two, 2 * 4 as eight"") %}{%forkeyinrow.keys() %}
{{ key }}: {{ row[key] }}<br>
{%endfor%}{%endfor%}
The plugin will execute SQL against the current database for the page in database.html, table.html and row.html templates. If a template does not have a current database (index.html for example) the query will execute against the first attached database.
Queries with arguments
You can construct a SQL query using ? or :name parameter syntax by passing a list or dictionary as a second argument:
{%forrowinsql(""select distinct topic from til order by topic"") %}
<h2>{{ row.topic }}</h2>
<ul>
{%fortilinsql(""select * from til where topic = ?"", [row.topic]) %}
<li><ahref=""{{ til.url }}"">{{ til.title }}</a> - {{ til.created[:10] }}</li>
{%endfor%}
</ul>
{%endfor%}
Here's the same example using the :topic style of parameters:
{%forrowinsql(""select distinct topic from til order by topic"") %}
<h2>{{ row.topic }}</h2>
<ul>
{%fortilinsql(""select * from til where topic = :topic"", {""topic"": row.topic}) %}
<li><ahref=""{{ til.url }}"">{{ til.title }}</a> - {{ til.created[:10] }}</li>
{%endfor%}
</ul>
{%endfor%}
Querying a different database
You can pass an optional database= argument to specify a named database to use for the query. For example, if you have attached a news.db database you could use this:
{%forarticleinsql(
""select headline, date, summary from articles order by date desc limit 5"",
database=""news""
) %}
<h3>{{ article.headline }}</h2>
<pclass=""date"">{{ article.date }}</p>
<p>{{ article.summary }}</p>
{%endfor%}
",,,,,,
228485806,MDEwOlJlcG9zaXRvcnkyMjg0ODU4MDY=,datasette-configure-asgi,simonw/datasette-configure-asgi,0,9599,https://github.com/simonw/datasette-configure-asgi,Datasette plugin for configuring arbitrary ASGI middleware,0,2019-12-16T22:17:10Z,2020-08-25T15:54:32Z,2019-12-16T22:19:49Z,,6,1,1,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""asgi"", ""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,1,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-configure-asgi
[](https://pypi.org/project/datasette-configure-asgi/)
[](https://circleci.com/gh/simonw/datasette-configure-asgi)
[](https://github.com/simonw/datasette-configure-asgi/blob/master/LICENSE)
Datasette plugin for configuring arbitrary ASGI middleware
## Installation
pip install datasette-configure-asgi
## Usage
This plugin only takes effect if your `metadata.json` file contains relevant top-level plugin configuration in a `""datasette-configure-asgi""` configuration key.
For example, to wrap your Datasette instance in the `asgi-log-to-sqlite` middleware configured to write logs to `/tmp/log.db` you would use the following:
```json
{
""plugins"": {
""datasette-configure-asgi"": [
{
""class"": ""asgi_log_to_sqlite.AsgiLogToSqlite"",
""args"": {
""file"": ""/tmp/log.db""
}
}
]
}
}
```
The `""datasette-configure-asgi""` key should be a list of JSON objects. Each object should have a `""class""` key indicating the class to be used, and an optional `""args""` key providing any necessary arguments to be passed to that class constructor.
## Plugin structure
This plugin can be used to wrap your Datasette instance in any ASGI middleware that conforms to the following structure:
```python
class SomeAsgiMiddleware:
def __init__(self, app, arg1, arg2):
self.app = app
self.arg1 = arg1
self.arg2 = arg2
async def __call__(self, scope, receive, send):
start = time.time()
await self.app(scope, receive, send)
end = time.time()
print(""Time taken: {}"".format(end - start))
```
So the middleware is a class with a constructor which takes the wrapped application as a first argument, `app`, followed by further named arguments to configure the middleware. It provides an `async def __call__(self, scope, receive, send)` method to implement the middleware's behavior.
","
datasette-configure-asgi
Datasette plugin for configuring arbitrary ASGI middleware
Installation
pip install datasette-configure-asgi
Usage
This plugin only takes effect if your metadata.json file contains relevant top-level plugin configuration in a ""datasette-configure-asgi"" configuration key.
For example, to wrap your Datasette instance in the asgi-log-to-sqlite middleware configured to write logs to /tmp/log.db you would use the following:
The ""datasette-configure-asgi"" key should be a list of JSON objects. Each object should have a ""class"" key indicating the class to be used, and an optional ""args"" key providing any necessary arguments to be passed to that class constructor.
Plugin structure
This plugin can be used to wrap your Datasette instance in any ASGI middleware that conforms to the following structure:
So the middleware is a class with a constructor which takes the wrapped application as a first argument, app, followed by further named arguments to configure the middleware. It provides an async def __call__(self, scope, receive, send) method to implement the middleware's behavior.
",,,,,,
240815938,MDEwOlJlcG9zaXRvcnkyNDA4MTU5Mzg=,shapefile-to-sqlite,simonw/shapefile-to-sqlite,0,9599,https://github.com/simonw/shapefile-to-sqlite,Load shapefiles into a SQLite (optionally SpatiaLite) database,0,2020-02-16T01:55:29Z,2021-03-26T08:39:43Z,2020-08-23T06:00:41Z,,54,15,15,Python,1,1,1,1,0,0,0,0,3,apache-2.0,"[""sqlite"", ""gis"", ""spatialite"", ""shapefiles"", ""datasette"", ""datasette-io"", ""datasette-tool""]",0,3,15,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# shapefile-to-sqlite
[](https://pypi.org/project/shapefile-to-sqlite/)
[](https://circleci.com/gh/simonw/shapefile-to-sqlite)
[](https://github.com/simonw/shapefile-to-sqlite/blob/main/LICENSE)
Load shapefiles into a SQLite (optionally SpatiaLite) database.
Project background: [Things I learned about shapefiles building shapefile-to-sqlite](https://simonwillison.net/2020/Feb/19/shapefile-to-sqlite/)
## How to install
$ pip install shapefile-to-sqlite
## How to use
You can run this tool against a shapefile file like so:
$ shapefile-to-sqlite my.db features.shp
This will load the geometries as GeoJSON in a text column.
## Using with SpatiaLite
If you have [SpatiaLite](https://www.gaia-gis.it/fossil/libspatialite/index) available you can load them as SpatiaLite geometries like this:
$ shapefile-to-sqlite my.db features.shp --spatialite
The data will be loaded into a table called `features` - based on the name of the shapefile. You can specify an alternative table name using `--table`:
$ shapefile-to-sqlite my.db features.shp --table=places --spatialite
The tool will search for the SpatiaLite module in the following locations:
- `/usr/lib/x86_64-linux-gnu/mod_spatialite.so`
- `/usr/local/lib/mod_spatialite.dylib`
If you have installed the module in another location, you can use the `--spatialite_mod=xxx` option to specify where:
$ shapefile-to-sqlite my.db features.shp \
--spatialite_mod=/usr/lib/mod_spatialite.dylib
You can use the `--spatial-index` option to create a spatial index on the `geometry` column:
$ shapefile-to-sqlite my.db features.shp --spatial-index
You can omit `--spatialite` if you use either `--spatialite-mod` or `--spatial-index`.
## Projections
By default, this tool will attempt to convert geometries in the shapefile to the WGS 84 projection, for best conformance with the [GeoJSON specification](https://tools.ietf.org/html/rfc7946).
If you want it to leave the data in whatever projection was used by the shapefile, use the `--crs=keep` option.
You can convert the data to another output projection by passing it to the `--crs` option. For example, to convert to [EPSG:2227](https://epsg.io/2227) (California zone 3) use `--crs=espg:2227`.
The full list of formats accepted by the `--crs` option is [documented here](https://pyproj4.github.io/pyproj/stable/api/crs.html#pyproj.crs.CRS.__init__).
## Extracting columns
If your data contains columns with a small number of heavily duplicated values - the names of specific agencies responsible for parcels of land for example - you can extract those columns into separate lookup tables referenced by foreign keys using the `-c` option:
$ shapefile-to-sqlite my.db features.shp -c agency
This will create a `agency` table with `id` and `name` columns, and will create the `agency` column in your main table as an integer foreign key reference to that table.
The `-c` option can be used multiple times.
[CPAD_2020a_Units](https://calands.datasettes.com/calands/CPAD_2020a_Units) is an example of a table created using the `-c` option.
","
shapefile-to-sqlite
Load shapefiles into a SQLite (optionally SpatiaLite) database.
You can omit --spatialite if you use either --spatialite-mod or --spatial-index.
Projections
By default, this tool will attempt to convert geometries in the shapefile to the WGS 84 projection, for best conformance with the GeoJSON specification.
If you want it to leave the data in whatever projection was used by the shapefile, use the --crs=keep option.
You can convert the data to another output projection by passing it to the --crs option. For example, to convert to EPSG:2227 (California zone 3) use --crs=espg:2227.
The full list of formats accepted by the --crs option is documented here.
Extracting columns
If your data contains columns with a small number of heavily duplicated values - the names of specific agencies responsible for parcels of land for example - you can extract those columns into separate lookup tables referenced by foreign keys using the -c option:
This will create a agency table with id and name columns, and will create the agency column in your main table as an integer foreign key reference to that table.
The -c option can be used multiple times.
CPAD_2020a_Units is an example of a table created using the -c option.
",,,,,,
242260583,MDEwOlJlcG9zaXRvcnkyNDIyNjA1ODM=,datasette-mask-columns,simonw/datasette-mask-columns,0,9599,https://github.com/simonw/datasette-mask-columns,Datasette plugin that masks specified database columns,0,2020-02-22T01:29:16Z,2021-06-10T19:50:37Z,2021-06-10T19:51:02Z,https://datasette.io/plugins/datasette-mask-columns,15,2,2,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,2,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-mask-columns
[](https://pypi.org/project/datasette-mask-columns/)
[](https://github.com/simonw/datasette-mask-columns/releases)
[](https://github.com/simonw/datasette-mask-columns/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-mask-columns/blob/main/LICENSE)
Datasette plugin that masks specified database columns
## Installation
pip install datasette-mask-columns
This depends on plugin hook changes in a not-yet released branch of Datasette. See [issue #678](https://github.com/simonw/datasette/issues/678) for details.
## Usage
In your `metadata.json` file add a section like this describing the database and table in which you wish to mask columns:
```json
{
""databases"": {
""my-database"": {
""plugins"": {
""datasette-mask-columns"": {
""users"": [""password""]
}
}
}
}
}
```
All SQL queries against the `users` table in `my-database.db` will now return `null` for the `password` column, no matter what value that column actually holds.
The table page for `users` will display the text `REDACTED` in the masked column. This visual hint will only be available on the table page; it will not display his text for arbitrary queries against the table.
","
datasette-mask-columns
Datasette plugin that masks specified database columns
Installation
pip install datasette-mask-columns
This depends on plugin hook changes in a not-yet released branch of Datasette. See issue #678 for details.
Usage
In your metadata.json file add a section like this describing the database and table in which you wish to mask columns:
All SQL queries against the users table in my-database.db will now return null for the password column, no matter what value that column actually holds.
The table page for users will display the text REDACTED in the masked column. This visual hint will only be available on the table page; it will not display his text for arbitrary queries against the table.
",,,,,,
243887036,MDEwOlJlcG9zaXRvcnkyNDM4ODcwMzY=,datasette-configure-fts,simonw/datasette-configure-fts,0,9599,https://github.com/simonw/datasette-configure-fts,Datasette plugin for enabling full-text search against selected table columns,0,2020-02-29T01:50:57Z,2020-11-01T02:59:12Z,2020-11-01T02:59:10Z,,42,2,2,Python,1,1,1,1,0,0,0,0,2,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,2,2,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-configure-fts
[](https://pypi.org/project/datasette-configure-fts/)
[](https://github.com/simonw/datasette-configure-fts/releases)
[](https://github.com/simonw/datasette-configure-fts/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-configure-fts/blob/main/LICENSE)
Datasette plugin for enabling full-text search against selected table columns
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-configure-fts
## Usage
Having installed the plugin, visit `/-/configure-fts` on your Datasette instance to configure FTS for tables on attached writable databases.
Any time you have permission to configure FTS for a table a menu item will appear in the table actions menu on the table page.
By default only [the root actor](https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor) can access the page - so you'll need to run Datasette with the `--root` option and click on the link shown in the terminal to sign in and access the page.
The `configure-fts` permission governs access. You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to the write interface.
","
datasette-configure-fts
Datasette plugin for enabling full-text search against selected table columns
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-configure-fts
Usage
Having installed the plugin, visit /-/configure-fts on your Datasette instance to configure FTS for tables on attached writable databases.
Any time you have permission to configure FTS for a table a menu item will appear in the table actions menu on the table page.
By default only the root actor can access the page - so you'll need to run Datasette with the --root option and click on the link shown in the terminal to sign in and access the page.
The configure-fts permission governs access. You can use permission plugins such as datasette-permissions-sql to grant additional access to the write interface.
",,,,,,
246108561,MDEwOlJlcG9zaXRvcnkyNDYxMDg1NjE=,datasette-column-inspect,simonw/datasette-column-inspect,0,9599,https://github.com/simonw/datasette-column-inspect,Experimental plugin that adds a column inspector,0,2020-03-09T18:11:00Z,2020-12-09T21:46:10Z,2020-12-09T21:47:38Z,,15,1,1,HTML,1,1,1,1,0,0,0,0,3,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,3,1,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-column-inspect
[](https://pypi.org/project/datasette-column-inspect/)
[](https://github.com/simonw/datasette-column-inspect/releases)
[](https://github.com/simonw/datasette-column-inspect/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-column-inspect/blob/main/LICENSE)
Highly experimental Datasette plugin for inspecting columns.
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-column-inspect
## Usage
This plugin adds an icon to each column on the table page which opens an inspection side panel.
","
datasette-column-inspect
Highly experimental Datasette plugin for inspecting columns.
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-column-inspect
Usage
This plugin adds an icon to each column on the table page which opens an inspection side panel.
",,,,,,
248903544,MDEwOlJlcG9zaXRvcnkyNDg5MDM1NDQ=,hacker-news-to-sqlite,dogsheep/hacker-news-to-sqlite,0,53015001,https://github.com/dogsheep/hacker-news-to-sqlite,Create a SQLite database containing data pulled from Hacker News,0,2020-03-21T04:02:05Z,2021-06-06T22:42:00Z,2021-03-13T19:15:06Z,,19,25,25,Python,1,1,1,1,0,2,0,0,0,apache-2.0,"[""hacker-news"", ""datasette"", ""dogsheep"", ""datasette-io"", ""datasette-tool""]",2,0,25,main,"{""admin"": false, ""push"": false, ""pull"": false}",,53015001,2,1,"# hacker-news-to-sqlite
[](https://pypi.org/project/hacker-news-to-sqlite/)
[](https://github.com/dogsheep/hacker-news-to-sqlite/releases)
[](https://github.com/dogsheep/hacker-news-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/hacker-news-to-sqlite/blob/main/LICENSE)
Create a SQLite database containing data fetched from [Hacker News](https://news.ycombinator.com/).
## How to install
$ pip install hacker-news-to-sqlite
## Usage
$ hacker-news-to-sqlite user hacker-news.db your-username
Importing items: 37%|███████████ | 845/2297 [05:09<11:02, 2.19it/s]
Imports all of your Hacker News submissions and comments into a SQLite database called `hacker-news.db`.
$ hacker-news-to-sqlite trees hacker-news.db 22640038 22643218
Fetches the entire comments tree in which any of those content IDs appears.
## Browsing your data with Datasette
You can use [Datasette](https://datasette.readthedocs.org/) to browse your data. Install Datasette like this:
$ pip install datasette
Now run it against your `hacker-news.db` file like so:
$ datasette hacker-news.db
Visit `https://localhost:8001/` to search and explore your data.
You can improve the display of your data usinng the [datasette-render-timestamps](https://github.com/simonw/datasette-render-timestamps) and [datasette-render-html](https://github.com/simonw/datasette-render-html) plugins. Install them like this:
$ pip install datasette-render-timestamps datasette-render-html
Now save the following configuration in a file called `metadata.json`:
```json
{
""databases"": {
""hacker-news"": {
""tables"": {
""items"": {
""plugins"": {
""datasette-render-html"": {
""columns"": [
""text""
]
},
""datasette-render-timestamps"": {
""columns"": [
""time""
]
}
}
},
""users"": {
""plugins"": {
""datasette-render-timestamps"": {
""columns"": [
""created""
]
}
}
}
}
}
}
}
```
Run Datasette like this:
$ datasette -m metadata.json hacker-news.db
The timestamp columns will now be rendered as human-readable dates, and any HTML in your posts will be displayed as rendered HTML.
","
hacker-news-to-sqlite
Create a SQLite database containing data fetched from Hacker News.
The timestamp columns will now be rendered as human-readable dates, and any HTML in your posts will be displayed as rendered HTML.
",,,,,,
255460347,MDEwOlJlcG9zaXRvcnkyNTU0NjAzNDc=,datasette-clone,simonw/datasette-clone,0,9599,https://github.com/simonw/datasette-clone,Create a local copy of database files from a Datasette instance,0,2020-04-13T23:05:41Z,2021-06-08T15:33:21Z,2021-02-22T19:32:36Z,,20,2,2,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-tool""]",0,0,2,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-clone
[](https://pypi.org/project/datasette-clone/)
[](https://github.com/simonw/datasette-clone/releases)
[](https://github.com/simonw/datasette-clone/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-clone/blob/main/LICENSE)
Create a local copy of database files from a Datasette instance.
See [datasette-clone](https://simonwillison.net/2020/Apr/14/datasette-clone/) on my blog for background on this project.
## How to install
$ pip install datasette-clone
## Usage
This only works against Datasette instances running immutable databases (with the `-i` option). Databases published using the `datasette publish` command should be compatible with this tool.
To download copies of all `.db` files from an instance, run:
datasette-clone https://latest.datasette.io
You can provide an optional second argument to specify a directory:
datasette-clone https://latest.datasette.io /tmp/here-please
The command stores its own copy of a `databases.json` manifest and uses it to only download databases that have changed the next time you run the command.
It also stores a copy of the instance's `metadata.json` to ensure you have a copy of any source and licensing information for the downloaded databases.
If your instance is protected by an API token, you can use `--token` to provide it:
datasette-clone https://latest.datasette.io --token=xyz
For verbose output showing what the tool is doing, use `-v`.
","
datasette-clone
Create a local copy of database files from a Datasette instance.
See datasette-clone on my blog for background on this project.
How to install
$ pip install datasette-clone
Usage
This only works against Datasette instances running immutable databases (with the -i option). Databases published using the datasette publish command should be compatible with this tool.
To download copies of all .db files from an instance, run:
datasette-clone https://latest.datasette.io
You can provide an optional second argument to specify a directory:
The command stores its own copy of a databases.json manifest and uses it to only download databases that have changed the next time you run the command.
It also stores a copy of the instance's metadata.json to ensure you have a copy of any source and licensing information for the downloaded databases.
If your instance is protected by an API token, you can use --token to provide it:
For verbose output showing what the tool is doing, use -v.
",,,,,,
261634807,MDEwOlJlcG9zaXRvcnkyNjE2MzQ4MDc=,datasette-media,simonw/datasette-media,0,9599,https://github.com/simonw/datasette-media,Datasette plugin for serving media based on a SQL query,0,2020-05-06T02:42:57Z,2021-05-03T05:04:39Z,2020-07-30T23:39:29Z,,70,11,11,Python,1,1,1,1,0,0,0,0,8,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,8,11,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-media
[](https://pypi.org/project/datasette-media/)
[](https://github.com/simonw/datasette-media/releases)
[](https://circleci.com/gh/simonw/datasette-media)
[](https://github.com/simonw/datasette-media/blob/master/LICENSE)
Datasette plugin for serving media based on a SQL query.
Use this when you have a database table containing references to files on disk - or binary content stored in BLOB columns - that you would like to be able to serve to your users.
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-media
### HEIC image support
Modern iPhones save their photos using the [HEIC image format](https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format). Processing these images requires an additional dependency, [pyheif](https://pypi.org/project/pyheif/). You can include this dependency by running:
$ pip install datasette-media[heif]
## Usage
You can use this plugin to configure Datasette to serve static media based on SQL queries to an underlying database table.
Media will be served from URLs that start with `/-/media/`. The full URL to each media asset will look like this:
/-/media/type-of-media/media-key
`type-of-media` will correspond to a configured SQL query, and might be something like `photo`. `media-key` will be an identifier that is used as part of the underlying SQL query to find which file should be served.
### Serving static files from disk
The following ``metadata.json`` configuration will cause this plugin to serve files from disk, based on queries to a database table called `apple_photos`.
```json
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath from apple_photos where uuid=:key""
}
}
}
}
```
A request to `/-/media/photo/CF972D33-5324-44F2-8DAE-22CB3182CD31` will execute the following SQL query:
```sql
select filepath from apple_photos where uuid=:key
```
The value from the URL - in this case `CF972D33-5324-44F2-8DAE-22CB3182CD31` - will be passed as the `:key` parameter to the query.
The query returns a `filepath` value that has been read from the table. The plugin will then read that file from disk and serve it in response to the request.
SQL queries default to running against the first connected database. You can specify a different database to execute the query against using `""database"": ""name_of_db""`. To execute against `photos.db`, use this:
```json
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath from apple_photos where uuid=:key"",
""database"": ""photos""
}
}
}
}
```
See [dogsheep-photos](https://github.com/dogsheep/dogsheep-photos) for an example of an application that can benefit from this plugin.
### Serving binary content from BLOB columns
If your SQL query returns a `content` column, this will be served directly to the user:
```json
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select thumbnail as content from photos where uuid=:key"",
""database"": ""thumbs""
}
}
}
}
```
You can also return a `content_type` column which will be used as the `Content-Type` header served to the user:
```json
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select body as content, 'text/html;charset=utf-8' as content_type from documents where id=:key"",
""database"": ""documents""
}
}
}
}
```
If you do not specify a `content_type` the default of `application/octet-stream` will be used.
### Serving content proxied from a URL
To serve content that is itself fetched from elsewhere, return a `content_url` column. This can be particularly useful when combined with the ability to resize images (described in the next section).
```json
{
""plugins"": {
""datasette-media"": {
""photos"": {
""sql"": ""select photo_url as content_url from photos where id=:key"",
""database"": ""photos"",
""enable_transform"": true
}
}
}
}
```
Now you can access resized versions of images from that URL like so:
/-/media/photos/13?w=200
### Setting a download file name
The `content_filename` column can be returned to force browsers to download the content using a specific file name.
```json
{
""plugins"": {
""datasette-media"": {
""hello"": {
""sql"": ""select 'Hello ' || :key as content, 'hello.txt' as content_filename""
}
}
}
}
```
Visiting `/-/media/hello/Groot` will cause your browser to download a file called `hello.txt` containing the text `Hello Groot`.
### Resizing or transforming images
Your SQL query can specify that an image should be resized and/or converted to another format by returning additional columns. All three are optional.
* `resize_width` - the width to resize the image to
* `resize_width` - the height to resize the image to
* `output_format` - the output format to use (e.g. `jpeg` or `png`) - any output format [supported by Pillow](https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html) is allowed here.
If you specify one but not the other of `resize_width` or `resize_height` the unspecified one will be calculated automatically to maintain the aspect ratio of the image.
Here's an example configuration that will resize all images to be JPEGs that are 200 pixels in height:
```json
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath, 200 as resize_height, 'jpeg' as output_format from apple_photos where uuid=:key"",
""database"": ""photos""
}
}
}
}
```
If you enable the `enable_transform` configuration option you can instead specify transform parameters at runtime using querystring parameters. For example:
- `/-/media/photo/CF972D33?w=200` to resize to a fixed width
- `/-/media/photo/CF972D33?h=200` to resize to a fixed height
- `/-/media/photo/CF972D33?format=jpeg` to convert to JPEG
That option is added like so:
```json
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath from apple_photos where uuid=:key"",
""database"": ""photos"",
""enable_transform"": true
}
}
}
}
```
The maximum allowed height or width is 4000 pixels. You can change this limit using the `""max_width_height""` option:
```json
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath from apple_photos where uuid=:key"",
""database"": ""photos"",
""enable_transform"": true,
""max_width_height"": 1000
}
}
}
}
```
## Configuration
In addition to the different named content types, the following special plugin configuration setting is available:
- `transform_threads` - number of threads to use for running transformations (e.g. resizing). Defaults to 4.
This can be used like this:
```json
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath from apple_photos where uuid=:key"",
""database"": ""photos""
},
""transform_threads"": 8
}
}
}
```
","
datasette-media
Datasette plugin for serving media based on a SQL query.
Use this when you have a database table containing references to files on disk - or binary content stored in BLOB columns - that you would like to be able to serve to your users.
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-media
HEIC image support
Modern iPhones save their photos using the HEIC image format. Processing these images requires an additional dependency, pyheif. You can include this dependency by running:
$ pip install datasette-media[heif]
Usage
You can use this plugin to configure Datasette to serve static media based on SQL queries to an underlying database table.
Media will be served from URLs that start with /-/media/. The full URL to each media asset will look like this:
/-/media/type-of-media/media-key
type-of-media will correspond to a configured SQL query, and might be something like photo. media-key will be an identifier that is used as part of the underlying SQL query to find which file should be served.
Serving static files from disk
The following metadata.json configuration will cause this plugin to serve files from disk, based on queries to a database table called apple_photos.
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath from apple_photos where uuid=:key""
}
}
}
}
A request to /-/media/photo/CF972D33-5324-44F2-8DAE-22CB3182CD31 will execute the following SQL query:
select filepath from apple_photos where uuid=:key
The value from the URL - in this case CF972D33-5324-44F2-8DAE-22CB3182CD31 - will be passed as the :key parameter to the query.
The query returns a filepath value that has been read from the table. The plugin will then read that file from disk and serve it in response to the request.
SQL queries default to running against the first connected database. You can specify a different database to execute the query against using ""database"": ""name_of_db"". To execute against photos.db, use this:
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath from apple_photos where uuid=:key"",
""database"": ""photos""
}
}
}
}
See dogsheep-photos for an example of an application that can benefit from this plugin.
Serving binary content from BLOB columns
If your SQL query returns a content column, this will be served directly to the user:
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select thumbnail as content from photos where uuid=:key"",
""database"": ""thumbs""
}
}
}
}
You can also return a content_type column which will be used as the Content-Type header served to the user:
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select body as content, 'text/html;charset=utf-8' as content_type from documents where id=:key"",
""database"": ""documents""
}
}
}
}
If you do not specify a content_type the default of application/octet-stream will be used.
Serving content proxied from a URL
To serve content that is itself fetched from elsewhere, return a content_url column. This can be particularly useful when combined with the ability to resize images (described in the next section).
{
""plugins"": {
""datasette-media"": {
""photos"": {
""sql"": ""select photo_url as content_url from photos where id=:key"",
""database"": ""photos"",
""enable_transform"": true
}
}
}
}
Now you can access resized versions of images from that URL like so:
/-/media/photos/13?w=200
Setting a download file name
The content_filename column can be returned to force browsers to download the content using a specific file name.
Visiting /-/media/hello/Groot will cause your browser to download a file called hello.txt containing the text Hello Groot.
Resizing or transforming images
Your SQL query can specify that an image should be resized and/or converted to another format by returning additional columns. All three are optional.
resize_width - the width to resize the image to
resize_width - the height to resize the image to
output_format - the output format to use (e.g. jpeg or png) - any output format supported by Pillow is allowed here.
If you specify one but not the other of resize_width or resize_height the unspecified one will be calculated automatically to maintain the aspect ratio of the image.
Here's an example configuration that will resize all images to be JPEGs that are 200 pixels in height:
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath, 200 as resize_height, 'jpeg' as output_format from apple_photos where uuid=:key"",
""database"": ""photos""
}
}
}
}
If you enable the enable_transform configuration option you can instead specify transform parameters at runtime using querystring parameters. For example:
/-/media/photo/CF972D33?w=200 to resize to a fixed width
/-/media/photo/CF972D33?h=200 to resize to a fixed height
/-/media/photo/CF972D33?format=jpeg to convert to JPEG
",,,,,,
271408895,MDEwOlJlcG9zaXRvcnkyNzE0MDg4OTU=,datasette-permissions-sql,simonw/datasette-permissions-sql,0,9599,https://github.com/simonw/datasette-permissions-sql,Datasette plugin for configuring permission checks using SQL queries,0,2020-06-10T23:48:13Z,2020-06-12T07:06:12Z,2020-06-12T07:06:15Z,,25,0,0,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,0,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-permissions-sql
[](https://pypi.org/project/datasette-permissions-sql/)
[](https://circleci.com/gh/simonw/datasette-permissions-sql)
[](https://github.com/simonw/datasette-permissions-sql/blob/master/LICENSE)
Datasette plugin for configuring permission checks using SQL queries
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-permissions-sql
## Usage
First, read up on how Datasette's [authentication and permissions system](https://datasette.readthedocs.io/en/latest/authentication.html) works.
This plugin lets you define rules containing SQL queries that are executed to see if the currently authenticated actor has permission to perform certain actions.
Consider a canned query which authenticated users should only be able to execute if a row in the `users` table says that they are a member of staff.
That `users` table in the `mydatabase.db` database could look like this:
| id | username | is_staff |
|--|--------|--------|
| 1 | cleopaws | 0 |
| 2 | simon | 1 |
Authenticated users have an `actor` that looks like this:
```json
{
""id"": 2,
""username"": ""simon""
}
```
To configure the canned query to only be executable by staff users, add the following to your `metadata.json`:
```json
{
""plugins"": {
""datasette-permissions-sql"": [
{
""action"": ""view-query"",
""resource"": [""mydatabase"", ""promote_to_staff""],
""sql"": ""SELECT * FROM users WHERE is_staff = 1 AND id = :actor_id""
}
]
},
""databases"": {
""mydatabase"": {
""queries"": {
""promote_to_staff"": {
""sql"": ""UPDATE users SET is is_staff=1 WHERE id=:id"",
""write"": true
}
}
}
}
}
```
The `""datasette-permissions-sql""` key is a list of rules. Each of those rules has the following shape:
```json
{
""action"": ""name-of-action"",
""resource"": [""resource identifier to run this on""],
""sql"": ""SQL query to execute"",
""database"": ""mydatabase""
}
```
Both `""action""` and `""resource""` are optional. If present, the SQL query will only be executed on permission checks that match the action and, if present, the resource indicators.
`""database""` is also optional: it specifies the named database that the query should be executed against. If it is not present the first connected database will be used.
The Datasette documentation includes a [list of built-in permissions](https://datasette.readthedocs.io/en/stable/authentication.html#built-in-permissions) that you might want to use here.
### The SQL query
If the SQL query returns any rows the action will be allowed. If it returns no rows, the plugin hook will return `False` and deny access to that action.
The SQL query is called with a number of named parameters. You can use any of these as part of the query.
The list of parameters is as follows:
* `action` - the action, e.g. `""view-database""`
* `resource_1` - the first component of the resource, if one was passed
* `resource_2` - the second component of the resource, if available
* `actor_*` - a parameter for every key on the actor. Usually `actor_id` is present.
If any rows are returned, the permission check passes. If no rows are returned the check fails.
Another example table, this time granting explicit access to individual tables. Consider a table called `table_access` that looks like this:
| user_id | database | table |
| - | - | - |
| 1 | mydb | dogs |
| 2 | mydb | dogs |
| 1 | mydb | cats |
The following SQL query would grant access to the `dogs` ttable in the `mydb.db` database to users 1 and 2 - but would forbid access for user 2 to the `cats` table:
```sql
SELECT
*
FROM
table_access
WHERE
user_id = :actor_id
AND ""database"" = :resource_1
AND ""table"" = :resource_2
```
In a `metadata.yaml` configuration file that would look like this:
```yaml
databases:
mydb:
allow_sql: {}
plugins:
datasette-permissions-sql:
- action: view-table
sql: |-
SELECT
*
FROM
table_access
WHERE
user_id = :actor_id
AND ""database"" = :resource_1
AND ""table"" = :resource_2
```
We're using `allow_sql: {}` here to disable arbitrary SQL queries. This prevents users from running `select * from cats` directly to work around the permissions limits.
### Fallback mode
The default behaviour of this plugin is to take full control of specified permissions. The SQL query will directly control if the user is allowed or denied access to the permission.
This means that the default policy for each permission (which in Datasette core is ""allow"" for `view-database` and friends) will be ignored. It also means that any other `permission_allowed` plugins will not get their turn once this plugin has executed.
You can change this on a per-rule basis using ``""fallback"": true``:
```json
{
""action"": ""view-table"",
""resource"": [""mydatabase"", ""mytable""],
""sql"": ""select * from admins where user_id = :actor_id"",
""fallback"": true
}
```
When running in fallback mode, a query result returning no rows will cause the plugin hook to return ``None`` - which means ""I have no opinion on this permission, fall back to other plugins or the default"".
In this mode you can still return `False` (for ""deny access"") by returning a single row with a single value of `-1`. For example:
```json
{
""action"": ""view-table"",
""resource"": [""mydatabase"", ""mytable""],
""sql"": ""select -1 from banned where user_id = :actor_id"",
""fallback"": true
}
```
","
datasette-permissions-sql
Datasette plugin for configuring permission checks using SQL queries
Installation
Install this plugin in the same environment as Datasette.
This plugin lets you define rules containing SQL queries that are executed to see if the currently authenticated actor has permission to perform certain actions.
Consider a canned query which authenticated users should only be able to execute if a row in the users table says that they are a member of staff.
That users table in the mydatabase.db database could look like this:
id
username
is_staff
1
cleopaws
0
2
simon
1
Authenticated users have an actor that looks like this:
{
""id"": 2,
""username"": ""simon""
}
To configure the canned query to only be executable by staff users, add the following to your metadata.json:
{
""plugins"": {
""datasette-permissions-sql"": [
{
""action"": ""view-query"",
""resource"": [""mydatabase"", ""promote_to_staff""],
""sql"": ""SELECT * FROM users WHERE is_staff = 1 AND id = :actor_id""
}
]
},
""databases"": {
""mydatabase"": {
""queries"": {
""promote_to_staff"": {
""sql"": ""UPDATE users SET is is_staff=1 WHERE id=:id"",
""write"": true
}
}
}
}
}
The ""datasette-permissions-sql"" key is a list of rules. Each of those rules has the following shape:
{
""action"": ""name-of-action"",
""resource"": [""resource identifier to run this on""],
""sql"": ""SQL query to execute"",
""database"": ""mydatabase""
}
Both ""action"" and ""resource"" are optional. If present, the SQL query will only be executed on permission checks that match the action and, if present, the resource indicators.
""database"" is also optional: it specifies the named database that the query should be executed against. If it is not present the first connected database will be used.
If the SQL query returns any rows the action will be allowed. If it returns no rows, the plugin hook will return False and deny access to that action.
The SQL query is called with a number of named parameters. You can use any of these as part of the query.
The list of parameters is as follows:
action - the action, e.g. ""view-database""
resource_1 - the first component of the resource, if one was passed
resource_2 - the second component of the resource, if available
actor_* - a parameter for every key on the actor. Usually actor_id is present.
If any rows are returned, the permission check passes. If no rows are returned the check fails.
Another example table, this time granting explicit access to individual tables. Consider a table called table_access that looks like this:
user_id
database
table
1
mydb
dogs
2
mydb
dogs
1
mydb
cats
The following SQL query would grant access to the dogs ttable in the mydb.db database to users 1 and 2 - but would forbid access for user 2 to the cats table:
SELECT*FROM
table_access
WHERE
user_id = :actor_id
AND""database""= :resource_1
AND""table""= :resource_2
In a metadata.yaml configuration file that would look like this:
databases:
mydb:
allow_sql: {}plugins:
datasette-permissions-sql:
- action: view-tablesql: |- SELECT * FROM table_access WHERE user_id = :actor_id AND ""database"" = :resource_1 AND ""table"" = :resource_2
We're using allow_sql: {} here to disable arbitrary SQL queries. This prevents users from running select * from cats directly to work around the permissions limits.
Fallback mode
The default behaviour of this plugin is to take full control of specified permissions. The SQL query will directly control if the user is allowed or denied access to the permission.
This means that the default policy for each permission (which in Datasette core is ""allow"" for view-database and friends) will be ignored. It also means that any other permission_allowed plugins will not get their turn once this plugin has executed.
You can change this on a per-rule basis using ""fallback"": true:
{
""action"": ""view-table"",
""resource"": [""mydatabase"", ""mytable""],
""sql"": ""select * from admins where user_id = :actor_id"",
""fallback"": true
}
When running in fallback mode, a query result returning no rows will cause the plugin hook to return None - which means ""I have no opinion on this permission, fall back to other plugins or the default"".
In this mode you can still return False (for ""deny access"") by returning a single row with a single value of -1. For example:
{
""action"": ""view-table"",
""resource"": [""mydatabase"", ""mytable""],
""sql"": ""select -1 from banned where user_id = :actor_id"",
""fallback"": true
}
",,,,,,
273609879,MDEwOlJlcG9zaXRvcnkyNzM2MDk4Nzk=,datasette-saved-queries,simonw/datasette-saved-queries,0,9599,https://github.com/simonw/datasette-saved-queries,Datasette plugin that lets users save and execute queries,0,2020-06-20T00:20:42Z,2020-09-24T05:08:37Z,2020-08-15T23:38:46Z,,12,2,2,Python,1,1,1,1,0,0,0,0,1,,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,1,2,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-saved-queries
[](https://pypi.org/project/datasette-saved-queries/)
[](https://github.com/simonw/datasette-saved-queries/releases)
[](https://github.com/simonw/datasette-saved-queries/blob/master/LICENSE)
Datasette plugin that lets users save and execute queries
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-saved-queries
## Usage
When the plugin is installed Datasette will automatically create a `saved_queries` table in the first connected database when it starts up.
It also creates a `save_query` writable canned query which you can use to save new queries.
Queries that you save will be added to the query list on the database page.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-saved-queries
python -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-saved-queries
Datasette plugin that lets users save and execute queries
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-saved-queries
Usage
When the plugin is installed Datasette will automatically create a saved_queries table in the first connected database when it starts up.
It also creates a save_query writable canned query which you can use to save new queries.
Queries that you save will be added to the query list on the database page.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-saved-queries
python -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
275615947,MDEwOlJlcG9zaXRvcnkyNzU2MTU5NDc=,datasette-glitch,simonw/datasette-glitch,0,9599,https://github.com/simonw/datasette-glitch,Utilities to help run Datasette on Glitch,0,2020-06-28T15:41:25Z,2020-07-01T22:48:35Z,2020-07-01T22:49:22Z,,3,1,1,Python,1,1,1,1,0,0,0,0,0,,"[""glitch"", ""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,1,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-glitch
[](https://pypi.org/project/datasette-glitch/)
[](https://github.com/simonw/datasette-glitch/releases)
[](https://github.com/simonw/datasette-glitch/blob/master/LICENSE)
Utilities to help run Datasette on Glitch
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-glitch
## Usage
This plugin outputs a special link which will sign you into Datasette as the root user.
Click Tools -> Logs in the Glitch editor interface after your app starts to see the link.
","
datasette-glitch
Utilities to help run Datasette on Glitch
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-glitch
Usage
This plugin outputs a special link which will sign you into Datasette as the root user.
Click Tools -> Logs in the Glitch editor interface after your app starts to see the link.
",,,,,,
291359358,MDEwOlJlcG9zaXRvcnkyOTEzNTkzNTg=,datasette-yaml,simonw/datasette-yaml,0,9599,https://github.com/simonw/datasette-yaml,Export Datasette records as YAML,0,2020-08-29T22:32:15Z,2020-12-28T03:20:36Z,2021-05-13T08:59:53Z,,7,2,2,Python,1,1,1,1,0,1,0,0,1,,"[""yaml"", ""datasette"", ""datasette-plugin"", ""datasette-io""]",1,1,2,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,1,1,"# datasette-yaml
[](https://pypi.org/project/datasette-yaml/)
[](https://github.com/simonw/datasette-yaml/releases)
[](https://github.com/simonw/datasette-yaml/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-yaml/blob/main/LICENSE)
Export Datasette records as YAML
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-yaml
## Usage
Having installed this plugin, every table and query will gain a new `.yaml` export link.
You can also construct these URLs directly: `/dbname/tablename.yaml`
## Demo
The plugin is running on [covid-19.datasettes.com](https://covid-19.datasettes.co/) - for example [/covid/latest_ny_times_counties_with_populations.yaml](https://covid-19.datasettes.com/covid/latest_ny_times_counties_with_populations.yaml)
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-yaml
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-yaml
Export Datasette records as YAML
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-yaml
Usage
Having installed this plugin, every table and query will gain a new .yaml export link.
You can also construct these URLs directly: /dbname/tablename.yaml
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-yaml
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
293164447,MDEwOlJlcG9zaXRvcnkyOTMxNjQ0NDc=,datasette-backup,simonw/datasette-backup,0,9599,https://github.com/simonw/datasette-backup,Plugin adding backup options to Datasette,0,2020-09-05T22:33:29Z,2020-09-24T00:16:59Z,2020-09-07T02:27:30Z,,6,1,1,Python,1,1,1,1,0,0,0,0,3,,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,3,1,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-backup
[](https://pypi.org/project/datasette-backup/)
[](https://github.com/simonw/datasette-backup/releases)
[](https://github.com/simonw/datasette-backup/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-backup/blob/main/LICENSE)
Plugin adding backup options to Datasette
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-backup
## Usage
Once installed, you can download a SQL backup of any of your databases from:
/-/backup/dbname.sql
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-backup
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-backup
Plugin adding backup options to Datasette
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-backup
Usage
Once installed, you can download a SQL backup of any of your databases from:
/-/backup/dbname.sql
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-backup
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
331720824,MDEwOlJlcG9zaXRvcnkzMzE3MjA4MjQ=,datasette-leaflet,simonw/datasette-leaflet,0,9599,https://github.com/simonw/datasette-leaflet,Datasette plugin adding the Leaflet JavaScript library,0,2021-01-21T18:41:19Z,2021-04-20T16:27:35Z,2021-02-01T22:20:28Z,,124,3,3,JavaScript,1,1,1,1,0,0,0,0,2,,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,2,3,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-leaflet
[](https://pypi.org/project/datasette-leaflet/)
[](https://github.com/simonw/datasette-leaflet/releases)
[](https://github.com/simonw/datasette-leaflet/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-leaflet/blob/main/LICENSE)
Datasette plugin adding the [Leaflet](https://leafletjs.com/) JavaScript library.
A growing number of Datasette plugins depend on the Leaflet JavaScript mapping library. They each have their own way of loading Leaflet, which could result in loading it multiple times (with multiple versions) if more than one plugin is installed.
This library is intended to solve this problem, by providing a single plugin they can all depend on that loads Leaflet in a reusable way.
Plugins that use this:
- [datasette-leaflet-freedraw](https://datasette.io/plugins/datasette-leaflet-freedraw)
- [datasette-leaflet-geojson](https://datasette.io/plugins/datasette-leaflet-geojson)
- [datasette-cluster-map](https://datasette.io/plugins/datasette-cluster-map)
## Installation
You can install this plugin like so:
datasette install datasette-leaflet
Usually this plugin will be a dependency of other plugins, so it should be installed automatically when you install them.
## Usage
The plugin makes `leaflet.js` and `leaflet.css` available as static files. It provides two custom template variables with the URLs of those two files.
- `{{ datasette_leaflet_url }}` is the URL to the JavaScript
- `{{ datasette_leaflet_css_url }}` is the URL to the CSS
These URLs are also made available as global JavaScript constants:
- `datasette.leaflet.JAVASCRIPT_URL`
- `datasette.leaflet.CSS_URL`
The JavaScript is packaed as a [JavaScript module](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules). You can dynamically import the JavaScript from a custom template like this:
```html+jinja
```
You can load the CSS like this:
```html+jinja
```
Or dynamically like this:
```html+jinja
```
Here's a full example that loads the JavaScript and CSS and renders a map:
```html+jinja
```
","
datasette-leaflet
Datasette plugin adding the Leaflet JavaScript library.
A growing number of Datasette plugins depend on the Leaflet JavaScript mapping library. They each have their own way of loading Leaflet, which could result in loading it multiple times (with multiple versions) if more than one plugin is installed.
This library is intended to solve this problem, by providing a single plugin they can all depend on that loads Leaflet in a reusable way.
<script>
let link =document.createElement('link');link.rel='stylesheet';link.href='{{ datasette_leaflet_css_url }}';document.head.appendChild(link);
</script>
Here's a full example that loads the JavaScript and CSS and renders a map:
Install this plugin in the same environment as Datasette.
$ datasette install datasette-basemap
Usage
This plugin will make a basemap database available containing OpenStreetMap tiles for zoom levels 0-6 in the mbtiles format. It is designed for use with the datasette-tiles tile server plugin.
Download map tiles and store them in an MBTiles database
Installation
Install this tool using pip:
$ pip install download-tiles
Usage
This tool downloads tiles from a specified TMS (Tile Map Server) server for a specified bounding box and range of zoom levels and stores those tiles in a MBTiles SQLite database. It is a command-line wrapper around the Landez Python libary.
Please use this tool responsibly. Consult the usage policies of the tile servers you are interacting with, for example the OpenStreetMap Tile Usage Policy.
Running the following will download zoom levels 0-3 of OpenStreetMap, 85 tiles total, and store them in a SQLite database called world.mbtiles:
download-tiles world.mbtiles
You can customize which tile and zoom levels are downloaded using command options:
--zoom-levels=0-3 or -z=0-3
The different zoom levels to download. Specify a single number, e.g. 15, or a range of numbers e.g. 0-4. Be careful with this setting as you can easily go over the limits requested by the underlying tile server.
--bbox=3.9,-6.3,14.5,10.2 or -b=3.9,-6.3,14.5,10.2
The bounding box to fetch. Should be specified as min-lon,min-lat,max-lon,max-lat. You can use bboxfinder.com to find these for different areas.
--city=london or --country=madagascar
These options can be used instead of --bbox. The city or country specified will be looked up using the Nominatum API and used to derive a bounding box.
--show-bbox
Use this option to output the bounding box that was retrieved for the --city or --country without downloading any tiles.
--name=Name
A name for this tile collection, used for the name field in the metadata table. If not specified a UUID will be used, or if you used --city or --country the name will be set to the full name of that place.
The tile server URL to use. This should include {z} and {x} and {y} specifiers, and can optionally include {s} for subdomains.
The default URL used here is for OpenStreetMap, http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png
--tiles-subdomains=a,b,c
A comma-separated list of subdomains to use for the {s} parameter.
--verbose
Use this option to turn on verbose logging.
--cache-dir=/tmp/tiles
Provide a directory to cache downloaded tiles between runs. This can be useful if you are worried you might not have used the correct options for the bounding box or zoom levels.
Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd download-tiles
python -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
342126610,MDEwOlJlcG9zaXRvcnkzNDIxMjY2MTA=,datasette-block,simonw/datasette-block,0,9599,https://github.com/simonw/datasette-block,Block all access to specific path prefixes,0,2021-02-25T04:51:08Z,2021-02-25T08:18:28Z,2021-02-25T05:03:45Z,https://datasette.io/plugins/datasette-block,4,1,1,Python,1,1,1,1,0,0,0,0,0,,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,1,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-block
[](https://pypi.org/project/datasette-block/)
[](https://github.com/simonw/datasette-block/releases)
[](https://github.com/simonw/datasette-block/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-block/blob/main/LICENSE)
Block all access to specific path prefixes
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-block
## Configuration
Add the following to `metadata.json` to block specific path prefixes:
```json
{
""plugins"": {
""datasette-block"": {
""prefixes"": [""/all/""]
}
}
}
```
This will cause a 403 error to be returned for any path beginning with `/all/`.
This blocking happens as an ASGI wrapper around Datasette.
## Why would you need this?
You almost always would not. I use it with `datasette-ripgrep` to block access to static assets for unauthenticated users.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-block
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-block
Block all access to specific path prefixes
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-block
Configuration
Add the following to metadata.json to block specific path prefixes:
This will cause a 403 error to be returned for any path beginning with /all/.
This blocking happens as an ASGI wrapper around Datasette.
Why would you need this?
You almost always would not. I use it with datasette-ripgrep to block access to static assets for unauthenticated users.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-block
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
346597557,MDEwOlJlcG9zaXRvcnkzNDY1OTc1NTc=,tableau-to-sqlite,simonw/tableau-to-sqlite,0,9599,https://github.com/simonw/tableau-to-sqlite,Fetch data from Tableau into a SQLite database,0,2021-03-11T06:12:02Z,2021-06-10T04:40:44Z,2021-04-29T16:11:03Z,,212,8,8,Python,1,1,1,1,0,2,0,0,2,apache-2.0,"[""datasette-io"", ""datasette-tool""]",2,2,8,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,2,1,"# tableau-to-sqlite
[](https://pypi.org/project/tableau-to-sqlite/)
[](https://github.com/simonw/tableau-to-sqlite/releases)
[](https://github.com/simonw/tableau-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/tableau-to-sqlite/blob/master/LICENSE)
Fetch data from Tableau into a SQLite database. A wrapper around [TableauScraper](https://github.com/bertrandmartel/tableau-scraping/).
## Installation
Install this tool using `pip`:
$ pip install tableau-to-sqlite
## Usage
If you have the URL to a Tableau dashboard like this:
https://results.mo.gov/t/COVID19/views/VaccinationsDashboard/Vaccinations
You can pass that directly to the tool:
tableau-to-sqlite tableau.db \
https://results.mo.gov/t/COVID19/views/VaccinationsDashboard/Vaccinations
This will create a SQLite database called `tableau.db` containing one table for each of the worksheets in that dashboard.
If the dashboard is hosted on https://public.tableau.com/ you can instead provide the view name. This will be two strings separated by a `/` symbol - something like this:
OregonCOVID-19VaccineProviderEnrollment/COVID-19VaccineProviderEnrollment
Now run the tool like this:
tableau-to-sqlite tableau.db \
OregonCOVID-19VaccineProviderEnrollment/COVID-19VaccineProviderEnrollment
## Get the data as JSON or CSV
If you're building a [git scraper](https://simonwillison.net/2020/Oct/9/git-scraping/) you may want to convert the data gathered by this tool to CSV or JSON to check into your repository.
You can do that using [sqlite-utils](https://sqlite-utils.datasette.io/). Install it using `pip`:
pip install sqlite-utils
You can dump out a table as JSON like so:
sqlite-utils rows tableau.db \
'Admin Site and County Map Site No Info' > tableau.json
Or as CSV like this:
sqlite-utils rows tableau.db --csv \
'Admin Site and County Map Site No Info' > tableau.csv
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd tableau-to-sqlite
python -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
tableau-to-sqlite
Fetch data from Tableau into a SQLite database. A wrapper around TableauScraper.
Installation
Install this tool using pip:
$ pip install tableau-to-sqlite
Usage
If you have the URL to a Tableau dashboard like this:
This will create a SQLite database called tableau.db containing one table for each of the worksheets in that dashboard.
If the dashboard is hosted on https://public.tableau.com/ you can instead provide the view name. This will be two strings separated by a / symbol - something like this:
If you're building a git scraper you may want to convert the data gathered by this tool to CSV or JSON to check into your repository.
You can do that using sqlite-utils. Install it using pip:
pip install sqlite-utils
You can dump out a table as JSON like so:
sqlite-utils rows tableau.db \
'Admin Site and County Map Site No Info' > tableau.json
Or as CSV like this:
sqlite-utils rows tableau.db --csv \
'Admin Site and County Map Site No Info' > tableau.csv
Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd tableau-to-sqlite
python -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
375546675,MDEwOlJlcG9zaXRvcnkzNzU1NDY2NzU=,datasette-placekey,simonw/datasette-placekey,0,9599,https://github.com/simonw/datasette-placekey,SQL functions for working with placekeys,0,2021-06-10T02:31:27Z,2021-06-10T02:33:22Z,2021-06-10T02:32:42Z,https://datasette.io/plugins/datasette-placekey,3,0,0,Python,1,1,1,1,0,0,0,0,1,,"[""datasette"", ""datasette-plugin"", ""datasette-io"", ""placekey""]",0,1,0,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-placekey
[](https://pypi.org/project/datasette-placekey/)
[](https://github.com/simonw/datasette-placekey/releases)
[](https://github.com/simonw/datasette-placekey/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-placekey/blob/main/LICENSE)
SQL functions for working with [placekeys](https://www.placekey.io/).
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-placekey
## Usage
The following SQL functions are exposed - [documentation here](https://placekey.github.io/placekey-py/placekey.html#module-placekey.placekey).
```sql
select
geo_to_placekey(33.0896104,129.7900839),
placekey_to_geo('@6nh-nhh-kvf'),
placekey_to_geo_latitude('@6nh-nhh-kvf'),
placekey_to_geo_longitude('@6nh-nhh-kvf'),
placekey_to_h3('@6nh-nhh-kvf'),
h3_to_placekey('8a30d94e4c87fff'),
placekey_to_geojson('@6nh-nhh-kvf'),
placekey_to_wkt('@6nh-nhh-kvf'),
placekey_format_is_valid('@6nh-nhh-kvf');
```
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-placekey
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","