id,node_id,name,full_name,private,owner,html_url,description,fork,created_at,updated_at,pushed_at,homepage,size,stargazers_count,watchers_count,language,has_issues,has_projects,has_downloads,has_wiki,has_pages,forks_count,archived,disabled,open_issues_count,license,topics,forks,open_issues,watchers,default_branch,permissions,temp_clone_token,organization,network_count,subscribers_count,readme,readme_html,allow_forking,visibility,is_template,template_repository,web_commit_signoff_required,has_discussions
174715153,MDEwOlJlcG9zaXRvcnkxNzQ3MTUxNTM=,datasette-jellyfish,simonw/datasette-jellyfish,0,9599,https://github.com/simonw/datasette-jellyfish,Datasette plugin adding SQL functions for fuzzy text matching powered by Jellyfish,0,2019-03-09T16:02:01Z,2021-02-06T02:33:49Z,2021-02-06T02:34:18Z,https://datasette.io/plugins/datasette-jellyfish,15,9,9,Python,1,1,1,1,0,2,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",2,0,9,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,2,1,"# datasette-jellyfish
[](https://pypi.org/project/datasette-jellyfish/)
[](https://github.com/simonw/datasette-jellyfish/releases)
[](https://github.com/simonw/datasette-jellyfish/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-jellyfish/blob/main/LICENSE)
Datasette plugin that adds custom SQL functions for fuzzy string matching, built on top of the [Jellyfish](https://github.com/jamesturk/jellyfish) Python library by James Turk and Michael Stephens.
Interactive demos:
* [soundex, metaphone, nysiis, match_rating_codex comparison](https://latest-with-plugins.datasette.io/fixtures?sql=SELECT%0D%0A++++soundex%28%3As%29%2C+%0D%0A++++metaphone%28%3As%29%2C+%0D%0A++++nysiis%28%3As%29%2C+%0D%0A++++match_rating_codex%28%3As%29&s=demo).
* [distance functions comparison](https://latest-with-plugins.datasette.io/fixtures?sql=SELECT%0D%0A++++levenshtein_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++damerau_levenshtein_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++hamming_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++jaro_similarity%28%3As1%2C+%3As2%29%2C%0D%0A++++jaro_winkler_similarity%28%3As1%2C+%3As2%29%2C%0D%0A++++match_rating_comparison%28%3As1%2C+%3As2%29%3B&s1=barrack+obama&s2=barrack+h+obama)
Examples:
SELECT soundex(""hello"");
-- Outputs H400
SELECT metaphone(""hello"");
-- Outputs HL
SELECT nysiis(""hello"");
-- Outputs HAL
SELECT match_rating_codex(""hello"");
-- Outputs HLL
SELECT porter_stem(""running"");
-- Outputs run
SELECT levenshtein_distance(""hello"", ""hello world"");
-- Outputs 6
SELECT damerau_levenshtein_distance(""hello"", ""hello world"");
-- Outputs 6
SELECT hamming_distance(""hello"", ""hello world"");
-- Outputs 6
SELECT jaro_similarity(""hello"", ""hello world"");
-- Outputs 0.8181818181818182
SELECT jaro_winkler_similarity(""hello"", ""hello world"");
-- Outputs 0.890909090909091
SELECT match_rating_comparison(""hello"", ""helloo"");
-- Outputs 1
See [the Jellyfish documentation](https://jellyfish.readthedocs.io/en/latest/) for an explanation of each of these functions.","
datasette-jellyfish
Datasette plugin that adds custom SQL functions for fuzzy string matching, built on top of the Jellyfish Python library by James Turk and Michael Stephens.
",,,,,,
175550127,MDEwOlJlcG9zaXRvcnkxNzU1NTAxMjc=,yaml-to-sqlite,simonw/yaml-to-sqlite,0,9599,https://github.com/simonw/yaml-to-sqlite,Utility for converting YAML files to SQLite,0,2019-03-14T04:49:08Z,2021-06-13T09:04:40Z,2021-06-13T04:45:52Z,,19,36,36,Python,1,1,1,1,0,2,0,0,0,apache-2.0,"[""yaml"", ""sqlite"", ""datasette-io"", ""datasette-tool""]",2,0,36,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,2,1,"# yaml-to-sqlite
[](https://pypi.org/project/yaml-to-sqlite/)
[](https://github.com/simonw/yaml-to-sqlite/releases)
[](https://github.com/simonw/yaml-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/yaml-to-sqlite/blob/main/LICENSE)
Load the contents of a YAML file into a SQLite database table.
```
$ yaml-to-sqlite --help
Usage: yaml-to-sqlite [OPTIONS] DB_PATH TABLE YAML_FILE
Convert YAML files to SQLite
Options:
--version Show the version and exit.
--pk TEXT Column to use as a primary key
--single-column TEXT If YAML file is a list of values, populate this column
--help Show this message and exit.
```
## Usage
Given a `news.yml` file containing the following:
```yaml
- date: 2021-06-05
body: |-
[Datasette 0.57](https://docs.datasette.io/en/stable/changelog.html#v0-57) is out with an important security patch.
- date: 2021-05-10
body: |-
[Django SQL Dashboard](https://simonwillison.net/2021/May/10/django-sql-dashboard/) is a new tool that brings a useful authenticated subset of Datasette to Django projects that are built on top of PostgreSQL.
```
Running this command:
```bash
$ yaml-to-sqlite news.db stories news.yml
```
Will create a database file with this schema:
```bash
$ sqlite-utils schema news.db
CREATE TABLE [stories] (
[date] TEXT,
[body] TEXT
);
```
The `--pk` option can be used to set a column as the primary key for the table:
```bash
$ yaml-to-sqlite news.db stories news.yml --pk date
$ sqlite-utils schema news.db
CREATE TABLE [stories] (
[date] TEXT PRIMARY KEY,
[body] TEXT
);
```
## Single column YAML lists
The `--single-column` option can be used when the YAML file is a list of values, for example a file called `dogs.yml` containing the following:
```yaml
- Cleo
- Pancakes
- Nixie
```
Running this command:
```bash
$ yaml-to-sqlite dogs.db dogs.yaml --single-column=name
```
Will create a single `dogs` table with a single `name` column that is the primary key:
```bash
$ sqlite-utils schema dogs.db
CREATE TABLE [dogs] (
[name] TEXT PRIMARY KEY
);
$ sqlite-utils dogs.db 'select * from dogs' -t
name
--------
Cleo
Pancakes
Nixie
```
","
yaml-to-sqlite
Load the contents of a YAML file into a SQLite database table.
$ yaml-to-sqlite --help
Usage: yaml-to-sqlite [OPTIONS] DB_PATH TABLE YAML_FILE
Convert YAML files to SQLite
Options:
--version Show the version and exit.
--pk TEXT Column to use as a primary key
--single-column TEXT If YAML file is a list of values, populate this column
--help Show this message and exit.
Usage
Given a news.yml file containing the following:
- date: 2021-06-05body: |- [Datasette 0.57](https://docs.datasette.io/en/stable/changelog.html#v0-57) is out with an important security patch.
- date: 2021-05-10body: |- [Django SQL Dashboard](https://simonwillison.net/2021/May/10/django-sql-dashboard/) is a new tool that brings a useful authenticated subset of Datasette to Django projects that are built on top of PostgreSQL.
Will create a single dogs table with a single name column that is the primary key:
$ sqlite-utils schema dogs.db
CREATE TABLE [dogs] (
[name] TEXT PRIMARY KEY
);
$ sqlite-utils dogs.db 'select * from dogs' -t
name
--------
Cleo
Pancakes
Nixie
",,,,,,
189321671,MDEwOlJlcG9zaXRvcnkxODkzMjE2NzE=,datasette-jq,simonw/datasette-jq,0,9599,https://github.com/simonw/datasette-jq,Datasette plugin that adds a custom SQL function for executing jq expressions against JSON values,0,2019-05-30T01:06:31Z,2020-12-24T17:35:27Z,2020-04-09T05:43:43Z,,11,10,10,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""jq"", ""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,10,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,2,"# datasette-jq
[](https://pypi.org/project/datasette-jq/)
[](https://circleci.com/gh/simonw/datasette-jq)
[](https://github.com/simonw/datasette-jq/blob/master/LICENSE)
Datasette plugin that adds custom SQL functions for executing [jq](https://stedolan.github.io/jq/) expressions against JSON values.
Install this plugin in the same environment as Datasette to enable the `jq()` SQL function.
Usage:
select jq(
column_with_json,
""{top_3: .classifiers[:3], v: .version}""
)
See [the jq manual](https://stedolan.github.io/jq/manual/#Basicfilters) for full details of supported expression syntax.
## Interactive demo
You can try this plugin out at [datasette-jq-demo.datasette.io](https://datasette-jq-demo.datasette.io/)
Sample query:
select package, ""https://pypi.org/project/"" || package || ""/"" as url,
jq(info, ""{summary: .info.summary, author: .info.author, versions: .releases|keys|reverse}"")
from packages
[Try this query out](https://datasette-jq-demo.datasette.io/demo?sql=select+package%2C+%22https%3A%2F%2Fpypi.org%2Fproject%2F%22+%7C%7C+package+%7C%7C+%22%2F%22+as+url%2C%0D%0Ajq%28info%2C+%22%7Bsummary%3A+.info.summary%2C+author%3A+.info.author%2C+versions%3A+.releases%7Ckeys%7Creverse%7D%22%29%0D%0Afrom+packages) in the interactive demo.
","
datasette-jq
Datasette plugin that adds custom SQL functions for executing jq expressions against JSON values.
Install this plugin in the same environment as Datasette to enable the jq() SQL function.
",,,,,,
207630174,MDEwOlJlcG9zaXRvcnkyMDc2MzAxNzQ=,datasette-rure,simonw/datasette-rure,0,9599,https://github.com/simonw/datasette-rure,Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine,0,2019-09-10T18:09:33Z,2020-12-04T04:26:53Z,2019-09-11T22:59:38Z,,19,4,4,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""sqlite"", ""regular-expressions"", ""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,4,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-rure
[](https://pypi.org/project/datasette-rure/)
[](https://circleci.com/gh/simonw/datasette-rure)
[](https://github.com/simonw/datasette-rure/blob/master/LICENSE)
Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine
Install this plugin in the same environment as Datasette to enable the `regexp()` SQL function.
$ pip install datasette-rure
The plugin is built on top of the [rure-python](https://github.com/davidblewett/rure-python) library by David Blewett.
## regexp() to test regular expressions
You can test if a value matches a regular expression like this:
select regexp('hi.*there', 'hi there')
-- returns 1
select regexp('not.*there', 'hi there')
-- returns 0
You can also use SQLite's custom syntax to run matches:
select 'hi there' REGEXP 'hi.*there'
-- returns 1
This means you can select rows based on regular expression matches - for example, to select every article where the title begins with an E or an F:
select * from articles where title REGEXP '^[EF]'
Try this out: [REGEXP interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+*+from+articles+where+title+REGEXP+%27%5E%5BEF%5D%27)
## regexp_match() to extract groups
You can extract captured subsets of a pattern using `regexp_match()`.
select regexp_match('.*( and .*)', title) as n from articles where n is not null
-- Returns the ' and X' component of any matching titles, e.g.
-- and Recognition
-- and Transitions Their Place
-- etc
This will return the first parenthesis match when called with two arguments. You can call it with three arguments to indicate which match you would like to extract:
select regexp_match('.*(and)(.*)', title, 2) as n from articles where n is not null
The function will return `null` for invalid inputs e.g. a pattern without capture groups.
Try this out: [regexp_match() interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+%27WHY+%27+%7C%7C+regexp_match%28%27Why+%28.*%29%27%2C+title%29+as+t+from+articles+where+t+is+not+null)
## regexp_matches() to extract multiple matches at once
The `regexp_matches()` function can be used to extract multiple patterns from a single string. The result is returned as a JSON array, which can then be further processed using SQLite's [JSON functions](https://www.sqlite.org/json1.html).
The first argument is a regular expression with named capture groups. The second argument is the string to be matched.
select regexp_matches(
'hello (?P\w+) the (?P\w+)',
'hello bob the dog, hello maggie the cat, hello tarquin the otter'
)
This will return a list of JSON objects, each one representing the named captures from the original regular expression:
[
{""name"": ""bob"", ""species"": ""dog""},
{""name"": ""maggie"", ""species"": ""cat""},
{""name"": ""tarquin"", ""species"": ""otter""}
]
Try this out: [regexp_matches() interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+regexp_matches%28%0D%0A++++%27hello+%28%3FP%3Cname%3E%5Cw%2B%29+the+%28%3FP%3Cspecies%3E%5Cw%2B%29%27%2C%0D%0A++++%27hello+bob+the+dog%2C+hello+maggie+the+cat%2C+hello+tarquin+the+otter%27%0D%0A%29)
","
datasette-rure
Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine
Install this plugin in the same environment as Datasette to enable the regexp() SQL function.
$ pip install datasette-rure
The plugin is built on top of the rure-python library by David Blewett.
regexp() to test regular expressions
You can test if a value matches a regular expression like this:
You can extract captured subsets of a pattern using regexp_match().
select regexp_match('.*( and .*)', title) as n from articles where n is not null
-- Returns the ' and X' component of any matching titles, e.g.
-- and Recognition
-- and Transitions Their Place
-- etc
This will return the first parenthesis match when called with two arguments. You can call it with three arguments to indicate which match you would like to extract:
select regexp_match('.*(and)(.*)', title, 2) as n from articles where n is not null
The function will return null for invalid inputs e.g. a pattern without capture groups.
regexp_matches() to extract multiple matches at once
The regexp_matches() function can be used to extract multiple patterns from a single string. The result is returned as a JSON array, which can then be further processed using SQLite's JSON functions.
The first argument is a regular expression with named capture groups. The second argument is the string to be matched.
select regexp_matches(
'hello (?P<name>\w+) the (?P<species>\w+)',
'hello bob the dog, hello maggie the cat, hello tarquin the otter'
)
This will return a list of JSON objects, each one representing the named captures from the original regular expression:
",,,,,,
209091256,MDEwOlJlcG9zaXRvcnkyMDkwOTEyNTY=,datasette-atom,simonw/datasette-atom,0,9599,https://github.com/simonw/datasette-atom,Datasette plugin that adds a .atom output format,0,2019-09-17T15:31:01Z,2021-03-26T02:06:51Z,2021-01-24T23:59:36Z,,47,10,10,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,10,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,2,"# datasette-atom
[](https://pypi.org/project/datasette-atom/)
[](https://github.com/simonw/datasette-atom/releases)
[](https://github.com/simonw/datasette-atom/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-atom/blob/main/LICENSE)
Datasette plugin that adds support for generating [Atom feeds](https://validator.w3.org/feed/docs/atom.html) with the results of a SQL query.
## Installation
Install this plugin in the same environment as Datasette to enable the `.atom` output extension.
$ pip install datasette-atom
## Usage
To create an Atom feed you need to define a custom SQL query that returns a required set of columns:
* `atom_id` - a unique ID for each row. [This article](https://web.archive.org/web/20080211143232/http://diveintomark.org/archives/2004/05/28/howto-atom-id) has suggestions about ways to create these IDs.
* `atom_title` - a title for that row.
* `atom_updated` - an [RFC 3339](http://www.faqs.org/rfcs/rfc3339.html) timestamp representing the last time the entry was modified in a significant way. This can usually be the time that the row was created.
The following columns are optional:
* `atom_content` - content that should be shown in the feed. This will be treated as a regular string, so any embedded HTML tags will be escaped when they are displayed.
* `atom_content_html` - content that should be shown in the feed. This will be treated as an HTML string, and will be sanitized using [Bleach](https://github.com/mozilla/bleach) to ensure it does not have any malicious code in it before being returned as part of a `` Atom element. If both are provided, this will be used in place of `atom_content`.
* `atom_link` - a URL that should be used as the link that the feed entry points to.
* `atom_author_name` - the name of the author of the entry. If you provide this you can also provide `atom_author_uri` and `atom_author_email` with a URL and e-mail address for that author.
A query that returns these columns can then be returned as an Atom feed by adding the `.atom` extension.
## Example
Here is an example SQL query which generates an Atom feed for new entries on [www.niche-museums.com](https://www.niche-museums.com/):
```sql
select
'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id,
name as atom_title,
created as atom_updated,
'https://www.niche-museums.com/browse/museums/' || id as atom_link,
coalesce(
'',
''
) || '
' || description || '
' as atom_content_html
from
museums
order by
created desc
limit
15
```
You can try this query by [pasting it in here](https://www.niche-museums.com/browse) - then click the `.atom` link to see it as an Atom feed.
## Using a canned query
Datasette's [canned query mechanism](https://docs.datasette.io/en/stable/sql_queries.html#canned-queries) is a useful way to configure feeds. If a canned query definition has a `title` that will be used as the title of the Atom feed.
Here's an example, defined using a `metadata.yaml` file:
```yaml
databases:
browse:
queries:
feed:
title: Niche Museums
sql: |-
select
'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id,
name as atom_title,
created as atom_updated,
'https://www.niche-museums.com/browse/museums/' || id as atom_link,
coalesce(
'',
''
) || '
' || description || '
' as atom_content_html
from
museums
order by
created desc
limit
15
```
## Disabling HTML filtering
The HTML allow-list used by Bleach for the `atom_content_html` column can be found in the `clean(html)` function at the bottom of [datasette_atom/__init__.py](https://github.com/simonw/datasette-atom/blob/main/datasette_atom/__init__.py).
You can disable Bleach entirely for Atom feeds generated using a canned query. You should only do this if you are certain that no user-provided HTML could be included in that value.
Here's how to do that in `metadata.json`:
```json
{
""plugins"": {
""datasette-atom"": {
""allow_unsafe_html_in_canned_queries"": true
}
}
}
```
Setting this to `true` will disable Bleach filtering for all canned queries across all databases.
You can disable Bleach filtering just for a specific list of canned queries like so:
```json
{
""plugins"": {
""datasette-atom"": {
""allow_unsafe_html_in_canned_queries"": {
""museums"": [""latest"", ""moderation""]
}
}
}
}
```
This will disable Bleach just for the canned queries called `latest` and `moderation` in the `museums.db` database.
","
datasette-atom
Datasette plugin that adds support for generating Atom feeds with the results of a SQL query.
Installation
Install this plugin in the same environment as Datasette to enable the .atom output extension.
$ pip install datasette-atom
Usage
To create an Atom feed you need to define a custom SQL query that returns a required set of columns:
atom_id - a unique ID for each row. This article has suggestions about ways to create these IDs.
atom_title - a title for that row.
atom_updated - an RFC 3339 timestamp representing the last time the entry was modified in a significant way. This can usually be the time that the row was created.
The following columns are optional:
atom_content - content that should be shown in the feed. This will be treated as a regular string, so any embedded HTML tags will be escaped when they are displayed.
atom_content_html - content that should be shown in the feed. This will be treated as an HTML string, and will be sanitized using Bleach to ensure it does not have any malicious code in it before being returned as part of a <content type=""html""> Atom element. If both are provided, this will be used in place of atom_content.
atom_link - a URL that should be used as the link that the feed entry points to.
atom_author_name - the name of the author of the entry. If you provide this you can also provide atom_author_uri and atom_author_email with a URL and e-mail address for that author.
A query that returns these columns can then be returned as an Atom feed by adding the .atom extension.
Example
Here is an example SQL query which generates an Atom feed for new entries on www.niche-museums.com:
select'tag:niche-museums.com,'|| substr(created, 0, 11) ||':'|| id as atom_id,
name as atom_title,
created as atom_updated,
'https://www.niche-museums.com/browse/museums/'|| id as atom_link,
coalesce(
'<img src=""'|| photo_url ||'?w=800&h=400&fit=crop&auto=compress"">',
''
) ||'<p>'|| description ||'</p>'as atom_content_html
from
museums
order by
created desclimit15
You can try this query by pasting it in here - then click the .atom link to see it as an Atom feed.
Using a canned query
Datasette's canned query mechanism is a useful way to configure feeds. If a canned query definition has a title that will be used as the title of the Atom feed.
Here's an example, defined using a metadata.yaml file:
databases:
browse:
queries:
feed:
title: Niche Museumssql: |- select 'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id, name as atom_title, created as atom_updated, 'https://www.niche-museums.com/browse/museums/' || id as atom_link, coalesce( '<img src=""' || photo_url || '?w=800&h=400&fit=crop&auto=compress"">', '' ) || '<p>' || description || '</p>' as atom_content_html from museums order by created desc limit 15
Disabling HTML filtering
The HTML allow-list used by Bleach for the atom_content_html column can be found in the clean(html) function at the bottom of datasette_atom/init.py.
You can disable Bleach entirely for Atom feeds generated using a canned query. You should only do this if you are certain that no user-provided HTML could be included in that value.
This will disable Bleach just for the canned queries called latest and moderation in the museums.db database.
",,,,,,
214299267,MDEwOlJlcG9zaXRvcnkyMTQyOTkyNjc=,datasette-render-timestamps,simonw/datasette-render-timestamps,0,9599,https://github.com/simonw/datasette-render-timestamps,Datasette plugin for rendering timestamps,0,2019-10-10T22:50:50Z,2020-10-17T11:09:42Z,2020-03-22T17:57:17Z,,17,4,4,Python,1,1,1,1,0,1,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",1,0,4,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,1,2,"# datasette-render-timestamps
[](https://pypi.org/project/datasette-render-timestamps/)
[](https://circleci.com/gh/simonw/datasette-render-timestamps)
[](https://github.com/simonw/datasette-render-timestamps/blob/master/LICENSE)
Datasette plugin for rendering timestamps.
## Installation
Install this plugin in the same environment as Datasette to enable this new functionality:
pip install datasette-render-timestamps
The plugin will then look out for integer numbers that are likely to be timestamps - anything that would be a number of seconds from 5 years ago to 5 years in the future.
These will then be rendered in a more readable format.
## Configuration
You can disable automatic column detection in favour of explicitly listing the columns that you would like to render using [plugin configuration](https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration) in a `metadata.json` file.
Add a `""datasette-render-timestamps""` configuration block and use a `""columns""` key to list the columns you would like to treat as timestamp values:
```json
{
""plugins"": {
""datasette-render-timestamps"": {
""columns"": [""created"", ""updated""]
}
}
}
```
This will cause any `created` or `updated` columns in any table to be treated as timestamps and rendered.
Save this to `metadata.json` and run datasette with the `--metadata` flag to load this configuration:
datasette serve mydata.db --metadata metadata.json
To disable automatic timestamp detection entirely, you can use `""columnns"": []`.
This configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the `entries` table in the `news.db` database:
```json
{
""databases"": {
""news"": {
""tables"": {
""entries"": {
""plugins"": {
""datasette-render-timestamps"": {
""columns"": [""created"", ""updated""]
}
}
}
}
}
}
}
```
And here's how to apply it to every `created` column in every table in the `news.db` database:
```json
{
""databases"": {
""news"": {
""plugins"": {
""datasette-render-timestamps"": {
""columns"": [""created"", ""updated""]
}
}
}
}
}
```
### Customizing the date format
The default format is `%B %d, %Y - %H:%M:%S UTC` which renders for example: `October 10, 2019 - 07:18:29 UTC`. If you want another format, the date format can be customized using plugin configuration. Any format string supported by [strftime](http://strftime.org/) may be used. For example:
```json
{
""plugins"": {
""datasette-render-timestamps"": {
""format"": ""%Y-%m-%d-%H:%M:%S""
}
}
}
```
","
datasette-render-timestamps
Datasette plugin for rendering timestamps.
Installation
Install this plugin in the same environment as Datasette to enable this new functionality:
pip install datasette-render-timestamps
The plugin will then look out for integer numbers that are likely to be timestamps - anything that would be a number of seconds from 5 years ago to 5 years in the future.
These will then be rendered in a more readable format.
Configuration
You can disable automatic column detection in favour of explicitly listing the columns that you would like to render using plugin configuration in a metadata.json file.
Add a ""datasette-render-timestamps"" configuration block and use a ""columns"" key to list the columns you would like to treat as timestamp values:
To disable automatic timestamp detection entirely, you can use ""columnns"": [].
This configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the entries table in the news.db database:
The default format is %B %d, %Y - %H:%M:%S UTC which renders for example: October 10, 2019 - 07:18:29 UTC. If you want another format, the date format can be customized using plugin configuration. Any format string supported by strftime may be used. For example:
",,,,,,
217216787,MDEwOlJlcG9zaXRvcnkyMTcyMTY3ODc=,datasette-haversine,simonw/datasette-haversine,0,9599,https://github.com/simonw/datasette-haversine,Datasette plugin that adds a custom SQL function for haversine distances,0,2019-10-24T05:16:14Z,2021-07-28T20:13:38Z,2021-07-28T20:14:24Z,,8,1,1,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,1,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-haversine
[](https://pypi.org/project/datasette-haversine/)
[](https://github.com/simonw/datasette-haversine/releases)
[](https://github.com/simonw/datasette-haversine/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-haversine/blob/main/LICENSE)
Datasette plugin that adds a custom SQL function for haversine distances
Install this plugin in the same environment as Datasette to enable the `haversine()` SQL function.
$ pip install datasette-haversine
The plugin is built on top of the [haversine](https://github.com/mapado/haversine) library.
## haversine() to calculate distances
```sql
select haversine(lat1, lon1, lat2, lon2);
```
This will return the distance in kilometers between the point defined by `(lat1, lon1)` and the point defined by `(lat2, lon2)`.
## Custom units
By default `haversine()` returns results in km. You can pass an optional third argument to get results in a different unit:
- `ft` for feet
- `m` for meters
- `in` for inches
- `mi` for miles
- `nmi` for nautical miles
- `km` for kilometers (the default)
```sql
select haversine(lat1, lon1, lat2, lon2, 'mi');
```
","
datasette-haversine
Datasette plugin that adds a custom SQL function for haversine distances
Install this plugin in the same environment as Datasette to enable the haversine() SQL function.
$ pip install datasette-haversine
The plugin is built on top of the haversine library.
haversine() to calculate distances
select haversine(lat1, lon1, lat2, lon2);
This will return the distance in kilometers between the point defined by (lat1, lon1) and the point defined by (lat2, lon2).
Custom units
By default haversine() returns results in km. You can pass an optional third argument to get results in a different unit:
ft for feet
m for meters
in for inches
mi for miles
nmi for nautical miles
km for kilometers (the default)
select haversine(lat1, lon1, lat2, lon2, 'mi');
",,,,,,
219372133,MDEwOlJlcG9zaXRvcnkyMTkzNzIxMzM=,sqlite-transform,simonw/sqlite-transform,0,9599,https://github.com/simonw/sqlite-transform,Tool for running transformations on columns in a SQLite database,0,2019-11-03T22:07:53Z,2021-08-02T22:06:23Z,2021-08-02T22:07:57Z,,64,29,29,Python,1,1,1,1,0,1,0,0,0,apache-2.0,"[""sqlite"", ""datasette-io"", ""datasette-tool""]",1,0,29,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,1,1,"# sqlite-transform
[](https://pypi.org/project/sqlite-transform/)
[](https://github.com/simonw/sqlite-transform/releases)
[](https://github.com/simonw/sqlite-transform/actions?query=workflow%3ATest)
[](https://github.com/dogsheep/sqlite-transform/blob/main/LICENSE)
Tool for running transformations on columns in a SQLite database.
> **:warning: This tool is no longer maintained**
>
> I added a new tool to [sqlite-utils](https://sqlite-utils.datasette.io/) called [sqlite-utils convert](https://sqlite-utils.datasette.io/en/stable/cli.html#converting-data-in-columns) which provides a super-set of the functionality originally provided here. `sqlite-transform` is no longer maintained, and I recommend switching to using `sqlite-utils convert` instead.
## How to install
pip install sqlite-transform
## parsedate and parsedatetime
These subcommands will run all values in the specified column through `dateutils.parser.parse()` and replace them with the result, formatted as an ISO timestamp or ISO date.
For example, if a row in the database has an `opened` column which contains `10/10/2019 08:10:00 PM`, running the following command:
sqlite-transform parsedatetime my.db mytable opened
Will result in that value being replaced by `2019-10-10T20:10:00`.
Using the `parsedate` subcommand here would result in `2019-10-10` instead.
In the case of ambiguous dates such as `03/04/05` these commands both default to assuming American-style `mm/dd/yy` format. You can pass `--dayfirst` to specify that the day should be assumed to be first, or `--yearfirst` for the year.
## jsonsplit
The `jsonsplit` subcommand takes columns that contain a comma-separated list, for example a `tags` column containing records like `""trees,park,dogs""` and converts it into a JSON array `[""trees"", ""park"", ""dogs""]`.
This is useful for taking advantage of Datasette's [Facet by JSON array](https://docs.datasette.io/en/stable/facets.html#facet-by-json-array) feature.
sqlite-transform jsonsplit my.db mytable tags
It defaults to splitting on commas, but you can specify a different delimiter character using the `--delimiter` option, for example:
sqlite-transform jsonsplit \
my.db mytable tags --delimiter ';'
Values within the array will be treated as strings, so a column containing `123,552,775` will be converted into the JSON array `[""123"", ""552"", ""775""]`.
You can specify a different type for these values using `--type int` or `--type float`, for example:
sqlite-transform jsonsplit \
my.db mytable tags --type int
This will result in that column being converted into `[123, 552, 775]`.
## lambda for executing your own code
The `lambda` subcommand lets you specify Python code which will be executed against the column.
Here's how to convert a column to uppercase:
sqlite-transform lambda my.db mytable mycolumn --code='str(value).upper()'
The code you provide will be compiled into a function that takes `value` as a single argument. You can break your function body into multiple lines, provided the last line is a `return` statement:
sqlite-transform lambda my.db mytable mycolumn --code='value = str(value)
return value.upper()'
You can also specify Python modules that should be imported and made available to your code using one or more `--import` options:
sqlite-transform lambda my.db mytable mycolumn \
--code='""\n"".join(textwrap.wrap(value, 10))' \
--import=textwrap
The `--dry-run` option will output a preview of the transformation against the first ten rows, without modifying the database.
## Saving the result to a separate column
Each of these commands accepts optional `--output` and `--output-type` options. These can be used to save the result of the transformation to a separate column, which will be created if the column does not already exist.
To save the result of `jsonsplit` to a new column called `json_tags`, use the following:
sqlite-transform jsonsplit my.db mytable tags \
--output json_tags
The type of the created column defaults to `text`, but a different column type can be specified using `--output-type`. This example will create a new floating point column called `float_id` with a copy of each item's ID increased by 0.5:
sqlite-transform lambda my.db mytable id \
--code 'float(value) + 0.5' \
--output float_id \
--output-type float
You can drop the original column at the end of the operation by adding `--drop`.
## Splitting a column into multiple columns
Sometimes you may wish to convert a single column into multiple derived columns. For example, you may have a `location` column containing `latitude,longitude` values which you wish to split out into separate `latitude` and `longitude` columns.
You can achieve this using the `--multi` option to `sqlite-transform lambda`. This option expects your `--code` function to return a Python dictionary: new columns well be created and populated for each of the keys in that dictionary.
For the `latitude,longitude` example you would use the following:
sqlite-transform lambda demo.db places location \
--code 'return {
""latitude"": float(value.split("","")[0]),
""longitude"": float(value.split("","")[1]),
}' --multi
The type of the returned values will be taken into account when creating the new columns. In this example, the resulting database schema will look like this:
```sql
CREATE TABLE [places] (
[location] TEXT,
[latitude] FLOAT,
[longitude] FLOAT
);
```
The code function can also return `None`, in which case its output will be ignored.
You can drop the original column at the end of the operation by adding `--drop`.
## Disabling the progress bar
By default each command will show a progress bar. Pass `-s` or `--silent` to hide that progress bar.
","
sqlite-transform
Tool for running transformations on columns in a SQLite database.
⚠️ This tool is no longer maintained
I added a new tool to sqlite-utils called sqlite-utils convert which provides a super-set of the functionality originally provided here. sqlite-transform is no longer maintained, and I recommend switching to using sqlite-utils convert instead.
How to install
pip install sqlite-transform
parsedate and parsedatetime
These subcommands will run all values in the specified column through dateutils.parser.parse() and replace them with the result, formatted as an ISO timestamp or ISO date.
For example, if a row in the database has an opened column which contains 10/10/2019 08:10:00 PM, running the following command:
sqlite-transform parsedatetime my.db mytable opened
Will result in that value being replaced by 2019-10-10T20:10:00.
Using the parsedate subcommand here would result in 2019-10-10 instead.
In the case of ambiguous dates such as 03/04/05 these commands both default to assuming American-style mm/dd/yy format. You can pass --dayfirst to specify that the day should be assumed to be first, or --yearfirst for the year.
jsonsplit
The jsonsplit subcommand takes columns that contain a comma-separated list, for example a tags column containing records like ""trees,park,dogs"" and converts it into a JSON array [""trees"", ""park"", ""dogs""].
This is useful for taking advantage of Datasette's Facet by JSON array feature.
sqlite-transform jsonsplit my.db mytable tags
It defaults to splitting on commas, but you can specify a different delimiter character using the --delimiter option, for example:
Values within the array will be treated as strings, so a column containing 123,552,775 will be converted into the JSON array [""123"", ""552"", ""775""].
You can specify a different type for these values using --type int or --type float, for example:
sqlite-transform jsonsplit \
my.db mytable tags --type int
This will result in that column being converted into [123, 552, 775].
lambda for executing your own code
The lambda subcommand lets you specify Python code which will be executed against the column.
The code you provide will be compiled into a function that takes value as a single argument. You can break your function body into multiple lines, provided the last line is a return statement:
The --dry-run option will output a preview of the transformation against the first ten rows, without modifying the database.
Saving the result to a separate column
Each of these commands accepts optional --output and --output-type options. These can be used to save the result of the transformation to a separate column, which will be created if the column does not already exist.
To save the result of jsonsplit to a new column called json_tags, use the following:
The type of the created column defaults to text, but a different column type can be specified using --output-type. This example will create a new floating point column called float_id with a copy of each item's ID increased by 0.5:
You can drop the original column at the end of the operation by adding --drop.
Splitting a column into multiple columns
Sometimes you may wish to convert a single column into multiple derived columns. For example, you may have a location column containing latitude,longitude values which you wish to split out into separate latitude and longitude columns.
You can achieve this using the --multi option to sqlite-transform lambda. This option expects your --code function to return a Python dictionary: new columns well be created and populated for each of the keys in that dictionary.
For the latitude,longitude example you would use the following:
The type of the returned values will be taken into account when creating the new columns. In this example, the resulting database schema will look like this:
The code function can also return None, in which case its output will be ignored.
You can drop the original column at the end of the operation by adding --drop.
Disabling the progress bar
By default each command will show a progress bar. Pass -s or --silent to hide that progress bar.
",,,,,,
228485806,MDEwOlJlcG9zaXRvcnkyMjg0ODU4MDY=,datasette-configure-asgi,simonw/datasette-configure-asgi,0,9599,https://github.com/simonw/datasette-configure-asgi,Datasette plugin for configuring arbitrary ASGI middleware,0,2019-12-16T22:17:10Z,2020-08-25T15:54:32Z,2019-12-16T22:19:49Z,,6,1,1,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""asgi"", ""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,1,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-configure-asgi
[](https://pypi.org/project/datasette-configure-asgi/)
[](https://circleci.com/gh/simonw/datasette-configure-asgi)
[](https://github.com/simonw/datasette-configure-asgi/blob/master/LICENSE)
Datasette plugin for configuring arbitrary ASGI middleware
## Installation
pip install datasette-configure-asgi
## Usage
This plugin only takes effect if your `metadata.json` file contains relevant top-level plugin configuration in a `""datasette-configure-asgi""` configuration key.
For example, to wrap your Datasette instance in the `asgi-log-to-sqlite` middleware configured to write logs to `/tmp/log.db` you would use the following:
```json
{
""plugins"": {
""datasette-configure-asgi"": [
{
""class"": ""asgi_log_to_sqlite.AsgiLogToSqlite"",
""args"": {
""file"": ""/tmp/log.db""
}
}
]
}
}
```
The `""datasette-configure-asgi""` key should be a list of JSON objects. Each object should have a `""class""` key indicating the class to be used, and an optional `""args""` key providing any necessary arguments to be passed to that class constructor.
## Plugin structure
This plugin can be used to wrap your Datasette instance in any ASGI middleware that conforms to the following structure:
```python
class SomeAsgiMiddleware:
def __init__(self, app, arg1, arg2):
self.app = app
self.arg1 = arg1
self.arg2 = arg2
async def __call__(self, scope, receive, send):
start = time.time()
await self.app(scope, receive, send)
end = time.time()
print(""Time taken: {}"".format(end - start))
```
So the middleware is a class with a constructor which takes the wrapped application as a first argument, `app`, followed by further named arguments to configure the middleware. It provides an `async def __call__(self, scope, receive, send)` method to implement the middleware's behavior.
","
datasette-configure-asgi
Datasette plugin for configuring arbitrary ASGI middleware
Installation
pip install datasette-configure-asgi
Usage
This plugin only takes effect if your metadata.json file contains relevant top-level plugin configuration in a ""datasette-configure-asgi"" configuration key.
For example, to wrap your Datasette instance in the asgi-log-to-sqlite middleware configured to write logs to /tmp/log.db you would use the following:
The ""datasette-configure-asgi"" key should be a list of JSON objects. Each object should have a ""class"" key indicating the class to be used, and an optional ""args"" key providing any necessary arguments to be passed to that class constructor.
Plugin structure
This plugin can be used to wrap your Datasette instance in any ASGI middleware that conforms to the following structure:
So the middleware is a class with a constructor which takes the wrapped application as a first argument, app, followed by further named arguments to configure the middleware. It provides an async def __call__(self, scope, receive, send) method to implement the middleware's behavior.
",,,,,,
242260583,MDEwOlJlcG9zaXRvcnkyNDIyNjA1ODM=,datasette-mask-columns,simonw/datasette-mask-columns,0,9599,https://github.com/simonw/datasette-mask-columns,Datasette plugin that masks specified database columns,0,2020-02-22T01:29:16Z,2021-06-10T19:50:37Z,2021-06-10T19:51:02Z,https://datasette.io/plugins/datasette-mask-columns,15,2,2,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,2,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-mask-columns
[](https://pypi.org/project/datasette-mask-columns/)
[](https://github.com/simonw/datasette-mask-columns/releases)
[](https://github.com/simonw/datasette-mask-columns/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-mask-columns/blob/main/LICENSE)
Datasette plugin that masks specified database columns
## Installation
pip install datasette-mask-columns
This depends on plugin hook changes in a not-yet released branch of Datasette. See [issue #678](https://github.com/simonw/datasette/issues/678) for details.
## Usage
In your `metadata.json` file add a section like this describing the database and table in which you wish to mask columns:
```json
{
""databases"": {
""my-database"": {
""plugins"": {
""datasette-mask-columns"": {
""users"": [""password""]
}
}
}
}
}
```
All SQL queries against the `users` table in `my-database.db` will now return `null` for the `password` column, no matter what value that column actually holds.
The table page for `users` will display the text `REDACTED` in the masked column. This visual hint will only be available on the table page; it will not display his text for arbitrary queries against the table.
","
datasette-mask-columns
Datasette plugin that masks specified database columns
Installation
pip install datasette-mask-columns
This depends on plugin hook changes in a not-yet released branch of Datasette. See issue #678 for details.
Usage
In your metadata.json file add a section like this describing the database and table in which you wish to mask columns:
All SQL queries against the users table in my-database.db will now return null for the password column, no matter what value that column actually holds.
The table page for users will display the text REDACTED in the masked column. This visual hint will only be available on the table page; it will not display his text for arbitrary queries against the table.
",,,,,,
255460347,MDEwOlJlcG9zaXRvcnkyNTU0NjAzNDc=,datasette-clone,simonw/datasette-clone,0,9599,https://github.com/simonw/datasette-clone,Create a local copy of database files from a Datasette instance,0,2020-04-13T23:05:41Z,2021-06-08T15:33:21Z,2021-02-22T19:32:36Z,,20,2,2,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-tool""]",0,0,2,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-clone
[](https://pypi.org/project/datasette-clone/)
[](https://github.com/simonw/datasette-clone/releases)
[](https://github.com/simonw/datasette-clone/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-clone/blob/main/LICENSE)
Create a local copy of database files from a Datasette instance.
See [datasette-clone](https://simonwillison.net/2020/Apr/14/datasette-clone/) on my blog for background on this project.
## How to install
$ pip install datasette-clone
## Usage
This only works against Datasette instances running immutable databases (with the `-i` option). Databases published using the `datasette publish` command should be compatible with this tool.
To download copies of all `.db` files from an instance, run:
datasette-clone https://latest.datasette.io
You can provide an optional second argument to specify a directory:
datasette-clone https://latest.datasette.io /tmp/here-please
The command stores its own copy of a `databases.json` manifest and uses it to only download databases that have changed the next time you run the command.
It also stores a copy of the instance's `metadata.json` to ensure you have a copy of any source and licensing information for the downloaded databases.
If your instance is protected by an API token, you can use `--token` to provide it:
datasette-clone https://latest.datasette.io --token=xyz
For verbose output showing what the tool is doing, use `-v`.
","
datasette-clone
Create a local copy of database files from a Datasette instance.
See datasette-clone on my blog for background on this project.
How to install
$ pip install datasette-clone
Usage
This only works against Datasette instances running immutable databases (with the -i option). Databases published using the datasette publish command should be compatible with this tool.
To download copies of all .db files from an instance, run:
datasette-clone https://latest.datasette.io
You can provide an optional second argument to specify a directory:
The command stores its own copy of a databases.json manifest and uses it to only download databases that have changed the next time you run the command.
It also stores a copy of the instance's metadata.json to ensure you have a copy of any source and licensing information for the downloaded databases.
If your instance is protected by an API token, you can use --token to provide it:
For verbose output showing what the tool is doing, use -v.
",,,,,,
271408895,MDEwOlJlcG9zaXRvcnkyNzE0MDg4OTU=,datasette-permissions-sql,simonw/datasette-permissions-sql,0,9599,https://github.com/simonw/datasette-permissions-sql,Datasette plugin for configuring permission checks using SQL queries,0,2020-06-10T23:48:13Z,2020-06-12T07:06:12Z,2020-06-12T07:06:15Z,,25,0,0,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,0,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-permissions-sql
[](https://pypi.org/project/datasette-permissions-sql/)
[](https://circleci.com/gh/simonw/datasette-permissions-sql)
[](https://github.com/simonw/datasette-permissions-sql/blob/master/LICENSE)
Datasette plugin for configuring permission checks using SQL queries
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-permissions-sql
## Usage
First, read up on how Datasette's [authentication and permissions system](https://datasette.readthedocs.io/en/latest/authentication.html) works.
This plugin lets you define rules containing SQL queries that are executed to see if the currently authenticated actor has permission to perform certain actions.
Consider a canned query which authenticated users should only be able to execute if a row in the `users` table says that they are a member of staff.
That `users` table in the `mydatabase.db` database could look like this:
| id | username | is_staff |
|--|--------|--------|
| 1 | cleopaws | 0 |
| 2 | simon | 1 |
Authenticated users have an `actor` that looks like this:
```json
{
""id"": 2,
""username"": ""simon""
}
```
To configure the canned query to only be executable by staff users, add the following to your `metadata.json`:
```json
{
""plugins"": {
""datasette-permissions-sql"": [
{
""action"": ""view-query"",
""resource"": [""mydatabase"", ""promote_to_staff""],
""sql"": ""SELECT * FROM users WHERE is_staff = 1 AND id = :actor_id""
}
]
},
""databases"": {
""mydatabase"": {
""queries"": {
""promote_to_staff"": {
""sql"": ""UPDATE users SET is is_staff=1 WHERE id=:id"",
""write"": true
}
}
}
}
}
```
The `""datasette-permissions-sql""` key is a list of rules. Each of those rules has the following shape:
```json
{
""action"": ""name-of-action"",
""resource"": [""resource identifier to run this on""],
""sql"": ""SQL query to execute"",
""database"": ""mydatabase""
}
```
Both `""action""` and `""resource""` are optional. If present, the SQL query will only be executed on permission checks that match the action and, if present, the resource indicators.
`""database""` is also optional: it specifies the named database that the query should be executed against. If it is not present the first connected database will be used.
The Datasette documentation includes a [list of built-in permissions](https://datasette.readthedocs.io/en/stable/authentication.html#built-in-permissions) that you might want to use here.
### The SQL query
If the SQL query returns any rows the action will be allowed. If it returns no rows, the plugin hook will return `False` and deny access to that action.
The SQL query is called with a number of named parameters. You can use any of these as part of the query.
The list of parameters is as follows:
* `action` - the action, e.g. `""view-database""`
* `resource_1` - the first component of the resource, if one was passed
* `resource_2` - the second component of the resource, if available
* `actor_*` - a parameter for every key on the actor. Usually `actor_id` is present.
If any rows are returned, the permission check passes. If no rows are returned the check fails.
Another example table, this time granting explicit access to individual tables. Consider a table called `table_access` that looks like this:
| user_id | database | table |
| - | - | - |
| 1 | mydb | dogs |
| 2 | mydb | dogs |
| 1 | mydb | cats |
The following SQL query would grant access to the `dogs` ttable in the `mydb.db` database to users 1 and 2 - but would forbid access for user 2 to the `cats` table:
```sql
SELECT
*
FROM
table_access
WHERE
user_id = :actor_id
AND ""database"" = :resource_1
AND ""table"" = :resource_2
```
In a `metadata.yaml` configuration file that would look like this:
```yaml
databases:
mydb:
allow_sql: {}
plugins:
datasette-permissions-sql:
- action: view-table
sql: |-
SELECT
*
FROM
table_access
WHERE
user_id = :actor_id
AND ""database"" = :resource_1
AND ""table"" = :resource_2
```
We're using `allow_sql: {}` here to disable arbitrary SQL queries. This prevents users from running `select * from cats` directly to work around the permissions limits.
### Fallback mode
The default behaviour of this plugin is to take full control of specified permissions. The SQL query will directly control if the user is allowed or denied access to the permission.
This means that the default policy for each permission (which in Datasette core is ""allow"" for `view-database` and friends) will be ignored. It also means that any other `permission_allowed` plugins will not get their turn once this plugin has executed.
You can change this on a per-rule basis using ``""fallback"": true``:
```json
{
""action"": ""view-table"",
""resource"": [""mydatabase"", ""mytable""],
""sql"": ""select * from admins where user_id = :actor_id"",
""fallback"": true
}
```
When running in fallback mode, a query result returning no rows will cause the plugin hook to return ``None`` - which means ""I have no opinion on this permission, fall back to other plugins or the default"".
In this mode you can still return `False` (for ""deny access"") by returning a single row with a single value of `-1`. For example:
```json
{
""action"": ""view-table"",
""resource"": [""mydatabase"", ""mytable""],
""sql"": ""select -1 from banned where user_id = :actor_id"",
""fallback"": true
}
```
","
datasette-permissions-sql
Datasette plugin for configuring permission checks using SQL queries
Installation
Install this plugin in the same environment as Datasette.
This plugin lets you define rules containing SQL queries that are executed to see if the currently authenticated actor has permission to perform certain actions.
Consider a canned query which authenticated users should only be able to execute if a row in the users table says that they are a member of staff.
That users table in the mydatabase.db database could look like this:
id
username
is_staff
1
cleopaws
0
2
simon
1
Authenticated users have an actor that looks like this:
{
""id"": 2,
""username"": ""simon""
}
To configure the canned query to only be executable by staff users, add the following to your metadata.json:
{
""plugins"": {
""datasette-permissions-sql"": [
{
""action"": ""view-query"",
""resource"": [""mydatabase"", ""promote_to_staff""],
""sql"": ""SELECT * FROM users WHERE is_staff = 1 AND id = :actor_id""
}
]
},
""databases"": {
""mydatabase"": {
""queries"": {
""promote_to_staff"": {
""sql"": ""UPDATE users SET is is_staff=1 WHERE id=:id"",
""write"": true
}
}
}
}
}
The ""datasette-permissions-sql"" key is a list of rules. Each of those rules has the following shape:
{
""action"": ""name-of-action"",
""resource"": [""resource identifier to run this on""],
""sql"": ""SQL query to execute"",
""database"": ""mydatabase""
}
Both ""action"" and ""resource"" are optional. If present, the SQL query will only be executed on permission checks that match the action and, if present, the resource indicators.
""database"" is also optional: it specifies the named database that the query should be executed against. If it is not present the first connected database will be used.
If the SQL query returns any rows the action will be allowed. If it returns no rows, the plugin hook will return False and deny access to that action.
The SQL query is called with a number of named parameters. You can use any of these as part of the query.
The list of parameters is as follows:
action - the action, e.g. ""view-database""
resource_1 - the first component of the resource, if one was passed
resource_2 - the second component of the resource, if available
actor_* - a parameter for every key on the actor. Usually actor_id is present.
If any rows are returned, the permission check passes. If no rows are returned the check fails.
Another example table, this time granting explicit access to individual tables. Consider a table called table_access that looks like this:
user_id
database
table
1
mydb
dogs
2
mydb
dogs
1
mydb
cats
The following SQL query would grant access to the dogs ttable in the mydb.db database to users 1 and 2 - but would forbid access for user 2 to the cats table:
SELECT*FROM
table_access
WHERE
user_id = :actor_id
AND""database""= :resource_1
AND""table""= :resource_2
In a metadata.yaml configuration file that would look like this:
databases:
mydb:
allow_sql: {}plugins:
datasette-permissions-sql:
- action: view-tablesql: |- SELECT * FROM table_access WHERE user_id = :actor_id AND ""database"" = :resource_1 AND ""table"" = :resource_2
We're using allow_sql: {} here to disable arbitrary SQL queries. This prevents users from running select * from cats directly to work around the permissions limits.
Fallback mode
The default behaviour of this plugin is to take full control of specified permissions. The SQL query will directly control if the user is allowed or denied access to the permission.
This means that the default policy for each permission (which in Datasette core is ""allow"" for view-database and friends) will be ignored. It also means that any other permission_allowed plugins will not get their turn once this plugin has executed.
You can change this on a per-rule basis using ""fallback"": true:
{
""action"": ""view-table"",
""resource"": [""mydatabase"", ""mytable""],
""sql"": ""select * from admins where user_id = :actor_id"",
""fallback"": true
}
When running in fallback mode, a query result returning no rows will cause the plugin hook to return None - which means ""I have no opinion on this permission, fall back to other plugins or the default"".
In this mode you can still return False (for ""deny access"") by returning a single row with a single value of -1. For example:
{
""action"": ""view-table"",
""resource"": [""mydatabase"", ""mytable""],
""sql"": ""select -1 from banned where user_id = :actor_id"",
""fallback"": true
}
",,,,,,
274264484,MDEwOlJlcG9zaXRvcnkyNzQyNjQ0ODQ=,sqlite-generate,simonw/sqlite-generate,0,9599,https://github.com/simonw/sqlite-generate,Tool for generating demo SQLite databases,0,2020-06-22T23:36:44Z,2021-02-27T15:25:26Z,2021-02-27T15:25:24Z,https://sqlite-generate-demo.datasette.io/,56,17,17,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""sqlite"", ""datasette-io"", ""datasette-tool""]",0,0,17,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,2,"# sqlite-generate
[](https://pypi.org/project/sqlite-generate/)
[](https://github.com/simonw/sqlite-generate/releases)
[](https://github.com/simonw/sqlite-generate/blob/master/LICENSE)
Tool for generating demo SQLite databases
## Installation
Install this plugin using `pip`:
$ pip install sqlite-generate
## Demo
You can see a demo of the database generated using this command running in [Datasette](https://github.com/simonw/datasette) at https://sqlite-generate-demo.datasette.io/
The demo is generated using the following command:
sqlite-generate demo.db --seed seed --fts --columns=10 --fks=0,3 --pks=0,2
## Usage
To generate a SQLite database file called `data.db` with 10 randomly named tables in it, run the following:
sqlite-generate data.db
You can use the `--tables` option to generate a different number of tables:
sqlite-generate data.db --tables 20
You can run the command against the same database file multiple times to keep adding new tables, using different settings for each batch of generated tables.
By default each table will contain a random number of rows between 0 and 200. You can customize this with the `--rows` option:
sqlite-generate data.db --rows 20
This will insert 20 rows into each table.
sqlite-generate data.db --rows 500,2000
This inserts a random number of rows between 500 and 2000 into each table.
Each table will have 5 columns. You can change this using `--columns`:
sqlite-generate data.db --columns 10
`--columns` can also accept a range:
sqlite-generate data.db --columns 5,15
You can control the random number seed used with the `--seed` option. This will result in the exact same database file being created by multiple runs of the tool:
sqlite-generate data.db --seed=myseed
By default each table will contain between 0 and 2 foreign key columns to other tables. You can control this using the `--fks` option, with either a single number or a range:
sqlite-generate data.db --columns=20 --fks=5,15
Each table will have a single primary key column called `id`. You can use the `--pks=` option to change the number of primary key columns on each table. Drop it to 0 to generate [rowid tables](https://www.sqlite.org/rowidtable.html). Increase it above 1 to generate tables with compound primary keys. Or use a range to get a random selection of different primary key layouts:
sqlite-generate data.db --pks=0,2
To configure [SQLite full-text search](https://www.sqlite.org/fts5.html) for all columns of type text, use `--fts`:
sqlite-generate data.db --fts
This will use FTS5 by default. To use [FTS4](https://www.sqlite.org/fts3.html) instead, use `--fts4`.
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd sqlite-generate
python -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
To generate a SQLite database file called data.db with 10 randomly named tables in it, run the following:
sqlite-generate data.db
You can use the --tables option to generate a different number of tables:
sqlite-generate data.db --tables 20
You can run the command against the same database file multiple times to keep adding new tables, using different settings for each batch of generated tables.
By default each table will contain a random number of rows between 0 and 200. You can customize this with the --rows option:
sqlite-generate data.db --rows 20
This will insert 20 rows into each table.
sqlite-generate data.db --rows 500,2000
This inserts a random number of rows between 500 and 2000 into each table.
Each table will have 5 columns. You can change this using --columns:
sqlite-generate data.db --columns 10
--columns can also accept a range:
sqlite-generate data.db --columns 5,15
You can control the random number seed used with the --seed option. This will result in the exact same database file being created by multiple runs of the tool:
sqlite-generate data.db --seed=myseed
By default each table will contain between 0 and 2 foreign key columns to other tables. You can control this using the --fks option, with either a single number or a range:
sqlite-generate data.db --columns=20 --fks=5,15
Each table will have a single primary key column called id. You can use the --pks= option to change the number of primary key columns on each table. Drop it to 0 to generate rowid tables. Increase it above 1 to generate tables with compound primary keys. Or use a range to get a random selection of different primary key layouts:
This will use FTS5 by default. To use FTS4 instead, use --fts4.
Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd sqlite-generate
python -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
275615947,MDEwOlJlcG9zaXRvcnkyNzU2MTU5NDc=,datasette-glitch,simonw/datasette-glitch,0,9599,https://github.com/simonw/datasette-glitch,Utilities to help run Datasette on Glitch,0,2020-06-28T15:41:25Z,2020-07-01T22:48:35Z,2020-07-01T22:49:22Z,,3,1,1,Python,1,1,1,1,0,0,0,0,0,,"[""glitch"", ""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,1,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-glitch
[](https://pypi.org/project/datasette-glitch/)
[](https://github.com/simonw/datasette-glitch/releases)
[](https://github.com/simonw/datasette-glitch/blob/master/LICENSE)
Utilities to help run Datasette on Glitch
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-glitch
## Usage
This plugin outputs a special link which will sign you into Datasette as the root user.
Click Tools -> Logs in the Glitch editor interface after your app starts to see the link.
","
datasette-glitch
Utilities to help run Datasette on Glitch
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-glitch
Usage
This plugin outputs a special link which will sign you into Datasette as the root user.
Click Tools -> Logs in the Glitch editor interface after your app starts to see the link.
",,,,,,
275624346,MDEwOlJlcG9zaXRvcnkyNzU2MjQzNDY=,datasette-init,simonw/datasette-init,0,9599,https://github.com/simonw/datasette-init,Ensure specific tables and views exist on startup,0,2020-06-28T16:26:29Z,2021-06-14T19:43:55Z,2020-07-01T22:47:09Z,,9,1,1,Python,1,1,1,1,0,0,0,0,0,,[],0,0,1,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-init
[](https://pypi.org/project/datasette-init/)
[](https://github.com/simonw/datasette-init/releases)
[](https://github.com/simonw/datasette-init/blob/master/LICENSE)
Ensure specific tables and views exist on startup
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-init
## Usage
This plugin is configured using `metadata.json` (or `metadata.yaml`).
### Creating tables
Add a block like this that specifies the tables you would like to ensure exist:
```json
{
""plugins"": {
""datasette-init"": {
""my_database"": {
""tables"": {
""dogs"": {
""columns"": {
""id"": ""integer"",
""name"": ""text"",
""age"": ""integer"",
""weight"": ""float""
},
""pk"": ""id""
}
}
}
}
}
}
```
Any tables that do not yet exist will be created when Datasette first starts.
Valid column types are `""integer""`, `""text""`, `""float""` and `""blob""`.
The `""pk""` is optional, and is used to define the primary key. To define a compound primary key (across more than one column) use a list of column names here:
```json
""pk"": [""id1"", ""id2""]
```
### Creating views
The plugin can also be used to create views:
```json
{
""plugins"": {
""datasette-init"": {
""my_database"": {
""views"": {
""my_view"": ""select 1 + 1""
}
}
}
}
}
```
Each view in the ``""views""`` block will be created when the Database first starts. If a view with the same name already exists it will be replaced with the new definition.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-init
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-init
Ensure specific tables and views exist on startup
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-init
Usage
This plugin is configured using metadata.json (or metadata.yaml).
Creating tables
Add a block like this that specifies the tables you would like to ensure exist:
Any tables that do not yet exist will be created when Datasette first starts.
Valid column types are ""integer"", ""text"", ""float"" and ""blob"".
The ""pk"" is optional, and is used to define the primary key. To define a compound primary key (across more than one column) use a list of column names here:
Each view in the ""views"" block will be created when the Database first starts. If a view with the same name already exists it will be replaced with the new definition.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-init
python3 -mvenv venv
source venv/bin/activate
Install this plugin in the same environment as Datasette.
$ datasette install datasette-basemap
Usage
This plugin will make a basemap database available containing OpenStreetMap tiles for zoom levels 0-6 in the mbtiles format. It is designed for use with the datasette-tiles tile server plugin.
Download map tiles and store them in an MBTiles database
Installation
Install this tool using pip:
$ pip install download-tiles
Usage
This tool downloads tiles from a specified TMS (Tile Map Server) server for a specified bounding box and range of zoom levels and stores those tiles in a MBTiles SQLite database. It is a command-line wrapper around the Landez Python libary.
Please use this tool responsibly. Consult the usage policies of the tile servers you are interacting with, for example the OpenStreetMap Tile Usage Policy.
Running the following will download zoom levels 0-3 of OpenStreetMap, 85 tiles total, and store them in a SQLite database called world.mbtiles:
download-tiles world.mbtiles
You can customize which tile and zoom levels are downloaded using command options:
--zoom-levels=0-3 or -z=0-3
The different zoom levels to download. Specify a single number, e.g. 15, or a range of numbers e.g. 0-4. Be careful with this setting as you can easily go over the limits requested by the underlying tile server.
--bbox=3.9,-6.3,14.5,10.2 or -b=3.9,-6.3,14.5,10.2
The bounding box to fetch. Should be specified as min-lon,min-lat,max-lon,max-lat. You can use bboxfinder.com to find these for different areas.
--city=london or --country=madagascar
These options can be used instead of --bbox. The city or country specified will be looked up using the Nominatum API and used to derive a bounding box.
--show-bbox
Use this option to output the bounding box that was retrieved for the --city or --country without downloading any tiles.
--name=Name
A name for this tile collection, used for the name field in the metadata table. If not specified a UUID will be used, or if you used --city or --country the name will be set to the full name of that place.
The tile server URL to use. This should include {z} and {x} and {y} specifiers, and can optionally include {s} for subdomains.
The default URL used here is for OpenStreetMap, http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png
--tiles-subdomains=a,b,c
A comma-separated list of subdomains to use for the {s} parameter.
--verbose
Use this option to turn on verbose logging.
--cache-dir=/tmp/tiles
Provide a directory to cache downloaded tiles between runs. This can be useful if you are worried you might not have used the correct options for the bounding box or zoom levels.
Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd download-tiles
python -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
342126610,MDEwOlJlcG9zaXRvcnkzNDIxMjY2MTA=,datasette-block,simonw/datasette-block,0,9599,https://github.com/simonw/datasette-block,Block all access to specific path prefixes,0,2021-02-25T04:51:08Z,2021-02-25T08:18:28Z,2021-02-25T05:03:45Z,https://datasette.io/plugins/datasette-block,4,1,1,Python,1,1,1,1,0,0,0,0,0,,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,1,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-block
[](https://pypi.org/project/datasette-block/)
[](https://github.com/simonw/datasette-block/releases)
[](https://github.com/simonw/datasette-block/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-block/blob/main/LICENSE)
Block all access to specific path prefixes
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-block
## Configuration
Add the following to `metadata.json` to block specific path prefixes:
```json
{
""plugins"": {
""datasette-block"": {
""prefixes"": [""/all/""]
}
}
}
```
This will cause a 403 error to be returned for any path beginning with `/all/`.
This blocking happens as an ASGI wrapper around Datasette.
## Why would you need this?
You almost always would not. I use it with `datasette-ripgrep` to block access to static assets for unauthenticated users.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-block
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-block
Block all access to specific path prefixes
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-block
Configuration
Add the following to metadata.json to block specific path prefixes:
This will cause a 403 error to be returned for any path beginning with /all/.
This blocking happens as an ASGI wrapper around Datasette.
Why would you need this?
You almost always would not. I use it with datasette-ripgrep to block access to static assets for unauthenticated users.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-block
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
393999598,MDEwOlJlcG9zaXRvcnkzOTM5OTk1OTg=,datasette-pyinstrument,simonw/datasette-pyinstrument,0,9599,https://github.com/simonw/datasette-pyinstrument,Use pyinstrument to analyze Datasette page performance,0,2021-08-08T15:33:29Z,2021-08-08T15:50:54Z,2021-08-08T15:50:52Z,,0,0,0,Python,1,1,1,1,0,0,0,0,0,,[],0,0,0,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-pyinstrument
[](https://pypi.org/project/datasette-pyinstrument/)
[](https://github.com/simonw/datasette-pyinstrument/releases)
[](https://github.com/simonw/datasette-pyinstrument/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-pyinstrument/blob/main/LICENSE)
Use pyinstrument to analyze Datasette page performance
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-pyinstrument
## Usage
Once installed, adding `?_pyinstrument=1` to any URL within Datasette will replace the output of that page with the pyinstrument profiler results for it.
## Demo
You can see the output of this plugin at https://latest-with-plugins.datasette.io/fixtures/sortable?_pyinstrument=1
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-pyinstrument
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-pyinstrument
Use pyinstrument to analyze Datasette page performance
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-pyinstrument
Usage
Once installed, adding ?_pyinstrument=1 to any URL within Datasette will replace the output of that page with the pyinstrument profiler results for it.
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-pyinstrument
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
394107614,MDEwOlJlcG9zaXRvcnkzOTQxMDc2MTQ=,datasette-query-links,simonw/datasette-query-links,0,9599,https://github.com/simonw/datasette-query-links,Turn SELECT queries returned by a query into links to execute them,0,2021-08-09T01:16:59Z,2021-08-09T04:31:40Z,2021-08-09T02:56:40Z,,7,3,3,Python,1,1,1,1,0,0,0,0,0,,[],0,0,3,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-query-links
[](https://pypi.org/project/datasette-query-links/)
[](https://github.com/simonw/datasette-query-links/releases)
[](https://github.com/simonw/datasette-query-links/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-query-links/blob/main/LICENSE)
Turn SELECT queries returned by a query into links to execute them
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-query-links
## Usage
This is an experimental plugin, requiring Datasette 0.59a1 or higher.
Any SQL query that returns a value that itself looks like a valid SQL query will be converted into a link to execute that SQL query when it is displayed in the Datasette interface.
These links will only show for valid SQL query - if a SQL query would return an error it will not be turned into a link.
## Demo
* [Here's an example query](https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++%27select+*+from+%5Bfacetable%5D%27+as+query%0D%0Aunion%0D%0Aselect%0D%0A++%27select+sqlite_version()%27%0D%0Aunion%0D%0Aselect%0D%0A++%27select+this+is+invalid+SQL+so+will+not+be+linked%27) showing the plugin in action.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-query-links
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-query-links
Turn SELECT queries returned by a query into links to execute them
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-query-links
Usage
This is an experimental plugin, requiring Datasette 0.59a1 or higher.
Any SQL query that returns a value that itself looks like a valid SQL query will be converted into a link to execute that SQL query when it is displayed in the Datasette interface.
These links will only show for valid SQL query - if a SQL query would return an error it will not be turned into a link.