id,node_id,name,full_name,private,owner,html_url,description,fork,created_at,updated_at,pushed_at,homepage,size,stargazers_count,watchers_count,language,has_issues,has_projects,has_downloads,has_wiki,has_pages,forks_count,archived,disabled,open_issues_count,license,topics,forks,open_issues,watchers,default_branch,permissions,temp_clone_token,organization,network_count,subscribers_count,readme,readme_html,allow_forking,visibility,is_template,template_repository,web_commit_signoff_required,has_discussions
107914493,MDEwOlJlcG9zaXRvcnkxMDc5MTQ0OTM=,datasette,simonw/datasette,0,9599,https://github.com/simonw/datasette,An open source multi-tool for exploring and publishing data,0,2017-10-23T00:39:03Z,2022-11-15T23:16:27Z,2022-11-16T03:47:14Z,https://datasette.io,5770,6628,6628,Python,1,0,1,1,0,463,0,0,435,apache-2.0,"[""asgi"", ""automatic-api"", ""csv"", ""datasets"", ""datasette"", ""datasette-io"", ""docker"", ""json"", ""python"", ""sql"", ""sqlite""]",463,435,6628,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,463,97,,,1,public,0,,0,1
110509816,MDEwOlJlcG9zaXRvcnkxMTA1MDk4MTY=,csvs-to-sqlite,simonw/csvs-to-sqlite,0,9599,https://github.com/simonw/csvs-to-sqlite,Convert CSV files into a SQLite database,0,2017-11-13T06:38:21Z,2021-11-18T16:33:39Z,2021-11-18T16:35:33Z,,138,655,655,Python,1,1,1,1,0,50,0,0,34,apache-2.0,"[""click"", ""csv"", ""datasette"", ""datasette-io"", ""datasette-tool"", ""pandas"", ""python"", ""sqlite""]",50,34,655,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,50,17,"# csvs-to-sqlite
[](https://pypi.org/project/csvs-to-sqlite/)
[](https://github.com/simonw/csvs-to-sqlite/releases)
[](https://github.com/simonw/csvs-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/csvs-to-sqlite/blob/main/LICENSE)
Convert CSV files into a SQLite database. Browse and publish that SQLite database with [Datasette](https://github.com/simonw/datasette).
Basic usage:
csvs-to-sqlite myfile.csv mydatabase.db
This will create a new SQLite database called `mydatabase.db` containing a
single table, `myfile`, containing the CSV content.
You can provide multiple CSV files:
csvs-to-sqlite one.csv two.csv bundle.db
The `bundle.db` database will contain two tables, `one` and `two`.
This means you can use wildcards:
csvs-to-sqlite ~/Downloads/*.csv my-downloads.db
If you pass a path to one or more directories, the script will recursively
search those directories for CSV files and create tables for each one.
csvs-to-sqlite ~/path/to/directory all-my-csvs.db
## Handling TSV (tab-separated values)
You can use the `-s` option to specify a different delimiter. If you want
to use a tab character you'll need to apply shell escaping like so:
csvs-to-sqlite my-file.tsv my-file.db -s $'\t'
## Refactoring columns into separate lookup tables
Let's say you have a CSV file that looks like this:
county,precinct,office,district,party,candidate,votes
Clark,1,President,,REP,John R. Kasich,5
Clark,2,President,,REP,John R. Kasich,0
Clark,3,President,,REP,John R. Kasich,7
([Real example taken from the Open Elections project](https://github.com/openelections/openelections-data-sd/blob/master/2016/20160607__sd__primary__clark__precinct.csv))
You can now convert selected columns into separate lookup tables using the new
`--extract-column` option (shortname: `-c`) - for example:
csvs-to-sqlite openelections-data-*/*.csv \
-c county:County:name \
-c precinct:Precinct:name \
-c office -c district -c party -c candidate \
openelections.db
The format is as follows:
column_name:optional_table_name:optional_table_value_column_name
If you just specify the column name e.g. `-c office`, the following table will
be created:
CREATE TABLE ""office"" (
""id"" INTEGER PRIMARY KEY,
""value"" TEXT
);
If you specify all three options, e.g. `-c precinct:Precinct:name` the table
will look like this:
CREATE TABLE ""Precinct"" (
""id"" INTEGER PRIMARY KEY,
""name"" TEXT
);
The original tables will be created like this:
CREATE TABLE ""ca__primary__san_francisco__precinct"" (
""county"" INTEGER,
""precinct"" INTEGER,
""office"" INTEGER,
""district"" INTEGER,
""party"" INTEGER,
""candidate"" INTEGER,
""votes"" INTEGER,
FOREIGN KEY (county) REFERENCES County(id),
FOREIGN KEY (party) REFERENCES party(id),
FOREIGN KEY (precinct) REFERENCES Precinct(id),
FOREIGN KEY (office) REFERENCES office(id),
FOREIGN KEY (candidate) REFERENCES candidate(id)
);
They will be populated with IDs that reference the new derived tables.
## Installation
$ pip install csvs-to-sqlite
`csvs-to-sqlite` now requires Python 3. If you are running Python 2 you can install the last version to support Python 2:
$ pip install csvs-to-sqlite==0.9.2
## csvs-to-sqlite --help
```
Usage: csvs-to-sqlite [OPTIONS] PATHS... DBNAME
PATHS: paths to individual .csv files or to directories containing .csvs
DBNAME: name of the SQLite database file to create
Options:
-s, --separator TEXT Field separator in input .csv
-q, --quoting INTEGER Control field quoting behavior per csv.QUOTE_*
constants. Use one of QUOTE_MINIMAL (0),
QUOTE_ALL (1), QUOTE_NONNUMERIC (2) or
QUOTE_NONE (3).
--skip-errors Skip lines with too many fields instead of
stopping the import
--replace-tables Replace tables if they already exist
-t, --table TEXT Table to use (instead of using CSV filename)
-c, --extract-column TEXT One or more columns to 'extract' into a
separate lookup table. If you pass a simple
column name that column will be replaced with
integer foreign key references to a new table
of that name. You can customize the name of
the table like so: state:States:state_name
This will pull unique values from the 'state'
column and use them to populate a new 'States'
table, with an id column primary key and a
state_name column containing the strings from
the original column.
-d, --date TEXT One or more columns to parse into ISO
formatted dates
-dt, --datetime TEXT One or more columns to parse into ISO
formatted datetimes
-df, --datetime-format TEXT One or more custom date format strings to try
when parsing dates/datetimes
-pk, --primary-key TEXT One or more columns to use as the primary key
-f, --fts TEXT One or more columns to use to populate a full-
text index
-i, --index TEXT Add index on this column (or a compound index
with -i col1,col2)
--shape TEXT Custom shape for the DB table - format is
csvcol:dbcol(TYPE),...
--filename-column TEXT Add a column with this name and populate with
CSV file name
--fixed-column ... Populate column with a fixed string
--fixed-column-int ...
Populate column with a fixed integer
--fixed-column-float ...
Populate column with a fixed float
--no-index-fks Skip adding index to foreign key columns
created using --extract-column (default is to
add them)
--no-fulltext-fks Skip adding full-text index on values
extracted using --extract-column (default is
to add them)
--just-strings Import all columns as text strings by default
(and, if specified, still obey --shape,
--date/datetime, and --datetime-format)
--version Show the version and exit.
--help Show this message and exit.
```
","
csvs-to-sqlite
Convert CSV files into a SQLite database. Browse and publish that SQLite database with Datasette.
Basic usage:
csvs-to-sqlite myfile.csv mydatabase.db
This will create a new SQLite database called mydatabase.db containing a
single table, myfile, containing the CSV content.
You can provide multiple CSV files:
csvs-to-sqlite one.csv two.csv bundle.db
The bundle.db database will contain two tables, one and two.
This means you can use wildcards:
csvs-to-sqlite ~/Downloads/*.csv my-downloads.db
If you pass a path to one or more directories, the script will recursively
search those directories for CSV files and create tables for each one.
csvs-to-sqlite ~/path/to/directory all-my-csvs.db
Handling TSV (tab-separated values)
You can use the -s option to specify a different delimiter. If you want
to use a tab character you'll need to apply shell escaping like so:
csvs-to-sqlite my-file.tsv my-file.db -s $'\t'
Refactoring columns into separate lookup tables
Let's say you have a CSV file that looks like this:
county,precinct,office,district,party,candidate,votes
Clark,1,President,,REP,John R. Kasich,5
Clark,2,President,,REP,John R. Kasich,0
Clark,3,President,,REP,John R. Kasich,7
They will be populated with IDs that reference the new derived tables.
Installation
$ pip install csvs-to-sqlite
csvs-to-sqlite now requires Python 3. If you are running Python 2 you can install the last version to support Python 2:
$ pip install csvs-to-sqlite==0.9.2
csvs-to-sqlite --help
Usage: csvs-to-sqlite [OPTIONS] PATHS... DBNAME
PATHS: paths to individual .csv files or to directories containing .csvs
DBNAME: name of the SQLite database file to create
Options:
-s, --separator TEXT Field separator in input .csv
-q, --quoting INTEGER Control field quoting behavior per csv.QUOTE_*
constants. Use one of QUOTE_MINIMAL (0),
QUOTE_ALL (1), QUOTE_NONNUMERIC (2) or
QUOTE_NONE (3).
--skip-errors Skip lines with too many fields instead of
stopping the import
--replace-tables Replace tables if they already exist
-t, --table TEXT Table to use (instead of using CSV filename)
-c, --extract-column TEXT One or more columns to 'extract' into a
separate lookup table. If you pass a simple
column name that column will be replaced with
integer foreign key references to a new table
of that name. You can customize the name of
the table like so: state:States:state_name
This will pull unique values from the 'state'
column and use them to populate a new 'States'
table, with an id column primary key and a
state_name column containing the strings from
the original column.
-d, --date TEXT One or more columns to parse into ISO
formatted dates
-dt, --datetime TEXT One or more columns to parse into ISO
formatted datetimes
-df, --datetime-format TEXT One or more custom date format strings to try
when parsing dates/datetimes
-pk, --primary-key TEXT One or more columns to use as the primary key
-f, --fts TEXT One or more columns to use to populate a full-
text index
-i, --index TEXT Add index on this column (or a compound index
with -i col1,col2)
--shape TEXT Custom shape for the DB table - format is
csvcol:dbcol(TYPE),...
--filename-column TEXT Add a column with this name and populate with
CSV file name
--fixed-column <TEXT TEXT>... Populate column with a fixed string
--fixed-column-int <TEXT INTEGER>...
Populate column with a fixed integer
--fixed-column-float <TEXT FLOAT>...
Populate column with a fixed float
--no-index-fks Skip adding index to foreign key columns
created using --extract-column (default is to
add them)
--no-fulltext-fks Skip adding full-text index on values
extracted using --extract-column (default is
to add them)
--just-strings Import all columns as text strings by default
(and, if specified, still obey --shape,
--date/datetime, and --datetime-format)
--version Show the version and exit.
--help Show this message and exit.
Run datasette install datasette-cluster-map to add this plugin to your Datasette virtual environment. Datasette will automatically load the plugin if it is installed in this way.
If you are deploying using the datasette publish command you can use the --install option:
If any of your tables have a latitude and longitude column, a map will be automatically displayed.
Configuration
If your columns are called something else you can configure the column names using plugin configuration in a metadata.json file. For example, if all of your columns are called xlat and xlng you can create a metadata.json file like this:
{
""title"": ""Regular metadata keys can go here too"",
""plugins"": {
""datasette-cluster-map"": {
""latitude_column"": ""xlat"",
""longitude_column"": ""xlng""
}
}
}
Then run Datasette like this:
datasette mydata.db -m metadata.json
This will configure the required column names for every database loaded by that Datasette instance.
If you want to customize the column names for just one table in one database, you can do something like this:
You can also use a custom SQL query to rename those columns to latitude and longitude, for example:
select*,
""Capture Latitude""as latitude,
""Capture Longitude""as longitude
from [USGS_WC_eartag_deployments_2009-2011]
The map defaults to being displayed above the main results table on the page. You can use the ""container"" plugin setting to provide a CSS selector indicating an element that the map should be appended to instead.
Custom tile layers
You can customize the tile layer used by the maps using the tile_layer and tile_layer_options configuration settings. For example, to use the Stamen Watercolor tiles you can use these settings:
The marker popup defaults to displaying the data for the underlying database row.
You can customize this by including a popup column in your results containing JSON that defines a more useful popup.
The JSON in the popup column should look something like this:
{
""image"": ""https://niche-museums.imgix.net/dodgems.heic?w=800&h=400&fit=crop"",
""alt"": ""Dingles Fairground Heritage Centre"",
""title"": ""Dingles Fairground Heritage Centre"",
""description"": ""Home of the National Fairground Collection, Dingles has over 45,000 indoor square feet of vintage fairground rides... and you can go on them! Highlights include the last complete surviving and opera"",
""link"": ""/browse/museums/26""
}
Each of these columns is optional.
title is the title to show at the top of the popup
image is the URL to an image to display in the popup
alt is the alt attribute to use for that image
description is a longer string of text to use as a description
link is a URL that the marker content should link to
You can use the SQLite json_object() function to construct this data dynamically as part of your SQL query. Here's an example:
select json_object(
'image', photo_url ||'?w=800&h=400&fit=crop',
'title', name,
'description', substr(description, 0, 200),
'link', '/browse/museums/'|| id
) as popup,
latitude, longitude from museums
where id in (26, 27) order by id
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-cluster-map
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
135007287,MDEwOlJlcG9zaXRvcnkxMzUwMDcyODc=,datasette-leaflet-geojson,simonw/datasette-leaflet-geojson,0,9599,https://github.com/simonw/datasette-leaflet-geojson,Datasette plugin that replaces any GeoJSON column values with a Leaflet map.,0,2018-05-27T01:42:30Z,2022-08-26T23:27:11Z,2022-08-26T23:27:08Z,,91,9,9,Python,1,1,1,1,0,4,0,0,3,,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""gis"", ""leaflet""]",4,3,9,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,4,2,"# datasette-leaflet-geojson
[](https://pypi.org/project/datasette-leaflet-geojson/)
[](https://github.com/simonw/datasette-leaflet-geojson/releases)
[](https://github.com/simonw/datasette-leaflet-geojson/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-leaflet-geojson/blob/main/LICENSE)
Datasette plugin that replaces any GeoJSON column values with a Leaflet map
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-leaflet-geojson
## Usage
Any columns containing valid GeoJSON strings will have their contents replaced with a Leaflet map when they are displayed in the Datasette interface.
## Demo
You can try this plugin out at https://calands.datasettes.com/calands/superunits_with_maps
## Configuration
By default this plugin displays maps for the first ten rows, and shows a ""Click to load map"" prompt for rows past the first ten.
You can change this limit using the `default_maps_to_load` plugin configuration setting. Add this to your `metadata.json`:
```json
{
""plugins"": {
""datasette-leaflet-geojson"": {
""default_maps_to_load"": 20
}
}
}
```
Then run Datasette with `datasette mydb.db -m metadata.json`.
","
datasette-leaflet-geojson
Datasette plugin that replaces any GeoJSON column values with a Leaflet map
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-leaflet-geojson
Usage
Any columns containing valid GeoJSON strings will have their contents replaced with a Leaflet map when they are displayed in the Datasette interface.
Then run Datasette with datasette mydb.db -m metadata.json.
",1,public,0,,0,
138669673,MDEwOlJlcG9zaXRvcnkxMzg2Njk2NzM=,datasette-vega,simonw/datasette-vega,0,9599,https://github.com/simonw/datasette-vega,Datasette plugin for visualizing data using Vega,0,2018-06-26T01:40:54Z,2021-12-10T22:20:46Z,2021-12-10T22:20:43Z,,59,42,42,JavaScript,1,1,1,1,0,2,0,0,31,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""plugin"", ""react"", ""vega""]",2,31,42,master,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,2,2,"# datasette-vega
[](https://pypi.org/project/datasette-vega/)
[](https://github.com/simonw/datasette-vega/blob/master/LICENSE)
A [Datasette](https://github.com/simonw/datasette) plugin that provides tools
for generating charts using [Vega](https://vega.github.io/).
Try out the latest master build as a live demo at https://datasette-vega-latest.datasette.io/ or try the latest release installed as a plugin at https://fivethirtyeight.datasettes.com/
To add this to your Datasette installation, install the plugin like so:
pip install datasette-vega
The plugin will then add itself to every Datasette table view.
If you are publishing data using the `datasette publish` command, you can
include this plugin like so:
datasette publish now mydatabase.db --install=datasette-vega
","
datasette-vega
A Datasette plugin that provides tools
for generating charts using Vega.
To add this to your Datasette installation, install the plugin like so:
pip install datasette-vega
The plugin will then add itself to every Datasette table view.
If you are publishing data using the datasette publish command, you can
include this plugin like so:
datasette publish now mydatabase.db --install=datasette-vega
",1,public,0,,,
140912432,MDEwOlJlcG9zaXRvcnkxNDA5MTI0MzI=,sqlite-utils,simonw/sqlite-utils,0,9599,https://github.com/simonw/sqlite-utils,Python CLI utility and library for manipulating SQLite databases,0,2018-07-14T03:21:46Z,2022-11-15T18:12:16Z,2022-11-15T15:53:38Z,https://sqlite-utils.datasette.io,1437,1029,1029,Python,1,1,1,1,0,79,0,0,72,apache-2.0,"[""cli"", ""click"", ""datasette"", ""datasette-io"", ""datasette-tool"", ""python"", ""sqlite"", ""sqlite-database""]",79,72,1029,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,79,16,"# sqlite-utils
[](https://pypi.org/project/sqlite-utils/)
[](https://sqlite-utils.datasette.io/en/stable/changelog.html)
[](https://pypi.org/project/sqlite-utils/)
[](https://github.com/simonw/sqlite-utils/actions?query=workflow%3ATest)
[](http://sqlite-utils.datasette.io/en/stable/?badge=stable)
[](https://codecov.io/gh/simonw/sqlite-utils)
[](https://github.com/simonw/sqlite-utils/blob/main/LICENSE)
[](https://discord.gg/Ass7bCAMDw)
Python CLI utility and library for manipulating SQLite databases.
## Some feature highlights
- [Pipe JSON](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data) (or [CSV or TSV](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-csv-or-tsv-data)) directly into a new SQLite database file, automatically creating a table with the appropriate schema
- [Run in-memory SQL queries](https://sqlite-utils.datasette.io/en/stable/cli.html#querying-data-directly-using-an-in-memory-database), including joins, directly against data in CSV, TSV or JSON files and view the results
- [Configure SQLite full-text search](https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search) against your database tables and run search queries against them, ordered by relevance
- Run [transformations against your tables](https://sqlite-utils.datasette.io/en/stable/cli.html#transforming-tables) to make schema changes that SQLite `ALTER TABLE` does not directly support, such as changing the type of a column
- [Extract columns](https://sqlite-utils.datasette.io/en/stable/cli.html#extracting-columns-into-a-separate-table) into separate tables to better normalize your existing data
Read more on my blog, in this series of posts on [New features in sqlite-utils](https://simonwillison.net/series/sqlite-utils-features/) and other [entries tagged sqliteutils](https://simonwillison.net/tags/sqliteutils/).
## Installation
pip install sqlite-utils
Or if you use [Homebrew](https://brew.sh/) for macOS:
brew install sqlite-utils
## Using as a CLI tool
Now you can do things with the CLI utility like this:
$ sqlite-utils memory dogs.csv ""select * from t""
[{""id"": 1, ""age"": 4, ""name"": ""Cleo""},
{""id"": 2, ""age"": 2, ""name"": ""Pancakes""}]
$ sqlite-utils insert dogs.db dogs dogs.csv --csv
[####################################] 100%
$ sqlite-utils tables dogs.db --counts
[{""table"": ""dogs"", ""count"": 2}]
$ sqlite-utils dogs.db ""select id, name from dogs""
[{""id"": 1, ""name"": ""Cleo""},
{""id"": 2, ""name"": ""Pancakes""}]
$ sqlite-utils dogs.db ""select * from dogs"" --csv
id,age,name
1,4,Cleo
2,2,Pancakes
$ sqlite-utils dogs.db ""select * from dogs"" --table
id age name
---- ----- --------
1 4 Cleo
2 2 Pancakes
You can import JSON data into a new database table like this:
$ curl https://api.github.com/repos/simonw/sqlite-utils/releases \
| sqlite-utils insert releases.db releases - --pk id
Or for data in a CSV file:
$ sqlite-utils insert dogs.db dogs dogs.csv --csv
`sqlite-utils memory` lets you import CSV or JSON data into an in-memory database and run SQL queries against it in a single command:
$ cat dogs.csv | sqlite-utils memory - ""select name, age from stdin""
See the [full CLI documentation](https://sqlite-utils.datasette.io/en/stable/cli.html) for comprehensive coverage of many more commands.
## Using as a library
You can also `import sqlite_utils` and use it as a Python library like this:
```python
import sqlite_utils
db = sqlite_utils.Database(""demo_database.db"")
# This line creates a ""dogs"" table if one does not already exist:
db[""dogs""].insert_all([
{""id"": 1, ""age"": 4, ""name"": ""Cleo""},
{""id"": 2, ""age"": 2, ""name"": ""Pancakes""}
], pk=""id"")
```
Check out the [full library documentation](https://sqlite-utils.datasette.io/en/stable/python-api.html) for everything else you can do with the Python library.
## Related projects
* [Datasette](https://datasette.io/): A tool for exploring and publishing data
* [csvs-to-sqlite](https://github.com/simonw/csvs-to-sqlite): Convert CSV files into a SQLite database
* [db-to-sqlite](https://github.com/simonw/db-to-sqlite): CLI tool for exporting a MySQL or PostgreSQL database as a SQLite file
* [dogsheep](https://dogsheep.github.io/): A family of tools for personal analytics, built on top of `sqlite-utils`
","
sqlite-utils
Python CLI utility and library for manipulating SQLite databases.
Some feature highlights
Pipe JSON (or CSV or TSV) directly into a new SQLite database file, automatically creating a table with the appropriate schema
Run in-memory SQL queries, including joins, directly against data in CSV, TSV or JSON files and view the results
Run transformations against your tables to make schema changes that SQLite ALTER TABLE does not directly support, such as changing the type of a column
Extract columns into separate tables to better normalize your existing data
You can also import sqlite_utils and use it as a Python library like this:
importsqlite_utilsdb=sqlite_utils.Database(""demo_database.db"")
# This line creates a ""dogs"" table if one does not already exist:db[""dogs""].insert_all([
{""id"": 1, ""age"": 4, ""name"": ""Cleo""},
{""id"": 2, ""age"": 2, ""name"": ""Pancakes""}
], pk=""id"")
Datasette: A tool for exploring and publishing data
csvs-to-sqlite: Convert CSV files into a SQLite database
db-to-sqlite: CLI tool for exporting a MySQL or PostgreSQL database as a SQLite file
dogsheep: A family of tools for personal analytics, built on top of sqlite-utils
",1,public,0,,0,0
142967347,MDEwOlJlcG9zaXRvcnkxNDI5NjczNDc=,datasette-json-html,simonw/datasette-json-html,0,9599,https://github.com/simonw/datasette-json-html,Datasette plugin for rendering HTML based on JSON values,0,2018-07-31T05:41:39Z,2022-03-15T04:54:15Z,2022-03-22T01:43:59Z,,46,19,19,Python,1,1,1,1,0,1,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""plugin""]",1,0,19,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,1,4,"# datasette-json-html
[](https://pypi.org/project/datasette-json-html/)
[](https://github.com/simonw/datasette-json-html/releases)
[](https://github.com/simonw/datasette-remote-metadata/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-json-html/blob/main/LICENSE)
Datasette plugin for rendering HTML based on JSON values, using the [render_cell plugin hook](https://docs.datasette.io/en/stable/plugin_hooks.html#render-cell-value-column-table-database-datasette).
This plugin looks for cell values that match a very specific JSON format and converts them into HTML when they are rendered by the Datasette interface.
## Links
{
""href"": ""https://simonwillison.net/"",
""label"": ""Simon Willison""
}
Will be rendered as an `` link:
Simon Willison
You can set a tooltip on the link using a `""title""` key:
{
""href"": ""https://simonwillison.net/"",
""label"": ""Simon Willison"",
""title"": ""My blog""
}
Produces:
Simon Willison
You can also include a description, which will be displayed below the link. If descriptions include newlines they will be converted to ` ` elements:
select json_object(
""href"", ""https://simonwillison.net/"",
""label"", ""Simon Willison"",
""description"", ""This can contain"" || x'0a' || ""newlines""
)
Produces:
Simon Willison This can contain newlines
* [Literal JSON link demo](https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22href%22%3A+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++%22label%22%3A+%22Simon+Willison%22%2C%0D%0A++++%22title%22%3A+%22My+blog%22%0D%0A%7D%27)
## List of links
[
{
""href"": ""https://simonwillison.net/"",
""label"": ""Simon Willison""
},
{
""href"": ""https://github.com/simonw/datasette"",
""label"": ""Datasette""
}
]
Will be rendered as a comma-separated list of `` links:
Simon Willison,
Datasette
The `href` property must begin with `https://` or `http://` or `/`, to avoid potential XSS injection attacks (for example URLs that begin with `javascript:`).
Lists of links cannot include `""description""` keys.
* [Literal list of links demo](https://datasette-json-html.datasette.io/demo?sql=select+%27%5B%0D%0A++++%7B%0D%0A++++++++%22href%22%3A+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++++++%22label%22%3A+%22Simon+Willison%22%0D%0A++++%7D%2C%0D%0A++++%7B%0D%0A++++++++%22href%22%3A+%22https%3A%2F%2Fgithub.com%2Fsimonw%2Fdatasette%22%2C%0D%0A++++++++%22label%22%3A+%22Datasette%22%0D%0A++++%7D%0D%0A%5D%27)
## Images
The image tag is more complex. The most basic version looks like this:
{
""img_src"": ""https://placekitten.com/200/300""
}
This will render as:
But you can also include one or more of `alt`, `caption`, `width` and `href`.
If you include width or alt, they will be added as attributes:
{
""img_src"": ""https://placekitten.com/200/300"",
""alt"": ""Kitten"",
""width"": 200
}
Produces:
* [Literal image demo](https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22img_src%22%3A+%22https%3A%2F%2Fplacekitten.com%2F200%2F300%22%2C%0D%0A++++%22alt%22%3A+%22Kitten%22%2C%0D%0A++++%22width%22%3A+200%0D%0A%7D%27)
The `href` key will cause the image to be wrapped in a link:
{
""img_src"": ""https://placekitten.com/200/300"",
""href"": ""http://www.example.com""
}
Produces:
The `caption` key wraps everything in a fancy figure/figcaption block:
{
""img_src"": ""https://placekitten.com/200/300"",
""caption"": ""Kitten caption""
}
Produces:
Kitten caption
## Preformatted text
You can use `{""pre"": ""text""}` to render text in a `
` HTML tag:
{
""pre"": ""This\nhas\nnewlines""
}
Produces:
This
has
newlines
If the value attached to the `""pre""` key is itself a JSON object, that JSON will be pretty-printed:
{
""pre"": {
""this"": {
""object"": [""is"", ""nested""]
}
}
}
Produces:
{
"this": {
"object": [
"is",
"nested"
]
}
}
* [Preformatted text with JSON demo](https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22pre%22%3A+%7B%0D%0A++++++++%22this%22%3A+%7B%0D%0A++++++++++++%22object%22%3A+%5B%22is%22%2C+%22nested%22%5D%0D%0A++++++++%7D%0D%0A++++%7D%0D%0A%7D%27)
* [Preformatted text demo showing the Mandelbrot Set](https://datasette-json-html.datasette.io/demo?sql=WITH+RECURSIVE%0D%0A++xaxis%28x%29+AS+%28VALUES%28-2.0%29+UNION+ALL+SELECT+x%2B0.05+FROM+xaxis+WHERE+x%3C1.2%29%2C%0D%0A++yaxis%28y%29+AS+%28VALUES%28-1.0%29+UNION+ALL+SELECT+y%2B0.1+FROM+yaxis+WHERE+y%3C1.0%29%2C%0D%0A++m%28iter%2C+cx%2C+cy%2C+x%2C+y%29+AS+%28%0D%0A++++SELECT+0%2C+x%2C+y%2C+0.0%2C+0.0+FROM+xaxis%2C+yaxis%0D%0A++++UNION+ALL%0D%0A++++SELECT+iter%2B1%2C+cx%2C+cy%2C+x*x-y*y+%2B+cx%2C+2.0*x*y+%2B+cy+FROM+m+%0D%0A+++++WHERE+%28x*x+%2B+y*y%29+%3C+4.0+AND+iter%3C28%0D%0A++%29%2C%0D%0A++m2%28iter%2C+cx%2C+cy%29+AS+%28%0D%0A++++SELECT+max%28iter%29%2C+cx%2C+cy+FROM+m+GROUP+BY+cx%2C+cy%0D%0A++%29%2C%0D%0A++a%28t%29+AS+%28%0D%0A++++SELECT+group_concat%28+substr%28%27+.%2B*%23%27%2C+1%2Bmin%28iter%2F7%2C4%29%2C+1%29%2C+%27%27%29+%0D%0A++++FROM+m2+GROUP+BY+cy%0D%0A++%29%0D%0ASELECT+json_object%28%27pre%27%2C+group_concat%28rtrim%28t%29%2Cx%270a%27%29%29+FROM+a%3B) using [this example](https://www.sqlite.org/lang_with.html#outlandish_recursive_query_examples) from the SQLite documentation
## Using these with SQLite JSON functions
The most powerful way to make use of this plugin is in conjunction with SQLite's [JSON functions](https://www.sqlite.org/json1.html). For example:
select json_object(
""href"", ""https://simonwillison.net/"",
""label"", ""Simon Willison""
);
* [json_object() link demo](https://datasette-json-html.datasette.io/demo?sql=select+json_object%28%0D%0A++++%22href%22%2C+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++%22label%22%2C+%22Simon+Willison%22%0D%0A%29%3B)
You can use these functions to construct JSON objects that work with the plugin from data in a table:
select id, json_object(
""href"", url, ""label"", text
) from mytable;
* [Demo that builds links against a table](https://datasette-json-html.datasette.io/demo?sql=select+json_object%28%22href%22%2C+url%2C+%22label%22%2C+package%2C+%22title%22%2C+package+%7C%7C+%22+%22+%7C%7C+url%29+as+package+from+packages)
The `json_group_array()` function is an aggregate function similar to `group_concat()` - it allows you to construct lists of JSON objects in conjunction with a `GROUP BY` clause.
This means you can use it to construct dynamic lists of links, for example:
select
substr(package, 0, 12) as prefix,
json_group_array(
json_object(
""href"", url,
""label"", package
)
) as package_links
from packages
group by prefix
* [Demo of json_group_array()](https://datasette-json-html.datasette.io/demo?sql=select%0D%0A++++substr%28package%2C+0%2C+12%29+as+prefix%2C%0D%0A++++json_group_array%28%0D%0A++++++++json_object%28%0D%0A++++++++++++%22href%22%2C+url%2C%0D%0A++++++++++++%22label%22%2C+package%0D%0A++++++++%29%0D%0A++++%29+as+package_links%0D%0Afrom+packages%0D%0Agroup+by+prefix)
## The `urllib_quote_plus()` SQL function
Since this plugin is designed to be used with SQL that constructs the underlying JSON structure, it is likely you will need to construct dynamic URLs from results returned by a SQL query.
This plugin registers a custom SQLite function called `urllib_quote_plus()` to help you do that. It lets you use Python's [urllib.parse.quote\_plus() function](https://docs.python.org/3/library/urllib.parse.html#urllib.parse.quote_plus) from within a SQL query.
Here's an example of how you might use it:
select id, json_object(
""href"",
""/mydatabase/other_table?_search="" || urllib_quote_plus(text),
""label"", text
) from mytable;
","
datasette-json-html
Datasette plugin for rendering HTML based on JSON values, using the render_cell plugin hook.
This plugin looks for cell values that match a very specific JSON format and converts them into HTML when they are rendered by the Datasette interface.
The json_group_array() function is an aggregate function similar to group_concat() - it allows you to construct lists of JSON objects in conjunction with a GROUP BY clause.
This means you can use it to construct dynamic lists of links, for example:
select
substr(package, 0, 12) as prefix,
json_group_array(
json_object(
""href"", url,
""label"", package
)
) as package_links
from packages
group by prefix
Since this plugin is designed to be used with SQL that constructs the underlying JSON structure, it is likely you will need to construct dynamic URLs from results returned by a SQL query.
This plugin registers a custom SQLite function called urllib_quote_plus() to help you do that. It lets you use Python's urllib.parse.quote_plus() function from within a SQL query.
Here's an example of how you might use it:
select id, json_object(
""href"",
""/mydatabase/other_table?_search="" || urllib_quote_plus(text),
""label"", text
) from mytable;
",1,public,0,,,
145483077,MDEwOlJlcG9zaXRvcnkxNDU0ODMwNzc=,datasette-render-images,simonw/datasette-render-images,0,9599,https://github.com/simonw/datasette-render-images,Datasette plugin that renders binary blob images using data-uris,0,2018-08-21T00:05:47Z,2022-08-11T16:06:11Z,2022-08-11T16:06:08Z,https://datasette-render-images-demo.datasette.io/favicons/favicons,35,14,14,Python,1,1,1,1,0,2,0,0,3,,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""plugin""]",2,3,14,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,2,2,"# datasette-render-images
[](https://pypi.org/project/datasette-render-images/)
[](https://github.com/simonw/datasette-render-images/releases)
[](https://github.com/simonw/datasette-render-images/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-render-images/blob/main/LICENSE)
A Datasette plugin that renders binary blob images with data-uris, using the [render_cell() plugin hook](https://docs.datasette.io/en/stable/plugins.html#render-cell-value-column-table-database-datasette).
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-render-images
## Usage
If a database row contains binary image data (PNG, GIF or JPEG), this plugin will detect that it is an image (using the [imghdr module](https://docs.python.org/3/library/imghdr.html) and render that cell using an `` element.
Here's a [demo of the plugin in action](https://datasette-render-images-demo.datasette.io/favicons/favicons).
## Creating a compatible database table
You can use the [sqlite-utils insert-files](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-data-from-files) command to insert image files into a database table:
$ pip install sqlite-utils
$ sqlite-utils insert-files gifs.db images *.gif
See [Fun with binary data and SQLite](https://simonwillison.net/2020/Jul/30/fun-binary-data-and-sqlite/) for more on this tool.
## Configuration
By default the plugin will only render images that are smaller than 100KB. You can adjust this limit using the `size_limit` plugin configuration option - for example, to increase the limit to 1MB (1000000 bytes) use the following in `metadata.json`:
```json
{
""plugins"": {
""datasette-render-images"": {
""size_limit"": 1000000
}
}
}
```
","
datasette-render-images
A Datasette plugin that renders binary blob images with data-uris, using the render_cell() plugin hook.
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-render-images
Usage
If a database row contains binary image data (PNG, GIF or JPEG), this plugin will detect that it is an image (using the imghdr module and render that cell using an <img src=""data:image/png;base64,...""> element.
By default the plugin will only render images that are smaller than 100KB. You can adjust this limit using the size_limit plugin configuration option - for example, to increase the limit to 1MB (1000000 bytes) use the following in metadata.json:
",1,public,0,,0,
163790822,MDEwOlJlcG9zaXRvcnkxNjM3OTA4MjI=,datasette-sqlite-fts4,simonw/datasette-sqlite-fts4,0,9599,https://github.com/simonw/datasette-sqlite-fts4,Datasette plugin that adds custom SQL functions for working with SQLite FTS4,0,2019-01-02T03:40:41Z,2022-07-31T16:33:25Z,2022-07-31T14:46:26Z,https://datasette.io/plugins/datasette-sqlite-fts4,14,3,3,Python,1,1,1,1,0,1,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""plugin""]",1,0,3,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,1,2,"# datasette-sqlite-fts4
[](https://pypi.org/project/datasette-sqlite-fts4/)
[](https://github.com/simonw/datasette-sqlite-fts4/releases)
[](https://github.com/simonw/datasette-sqlite-fts4/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-sqlite-fts4/blob/main/LICENSE)
Datasette plugin that exposes the custom SQL functions from [sqlite-fts4](https://github.com/simonw/sqlite-fts4).
[Interactive demo](https://datasette-sqlite-fts4.datasette.io/24ways-fts4?sql=select%0D%0A++++json_object%28%0D%0A++++++++""label""%2C+articles.title%2C+""href""%2C+articles.url%0D%0A++++%29+as+article%2C%0D%0A++++articles.author%2C%0D%0A++++rank_score%28matchinfo%28articles_fts%2C+""pcx""%29%29+as+score%2C%0D%0A++++rank_bm25%28matchinfo%28articles_fts%2C+""pcnalx""%29%29+as+bm25%2C%0D%0A++++json_object%28%0D%0A++++++++""pre""%2C+annotate_matchinfo%28matchinfo%28articles_fts%2C+""pcxnalyb""%29%2C+""pcxnalyb""%29%0D%0A++++%29+as+annotated_matchinfo%2C%0D%0A++++matchinfo%28articles_fts%2C+""pcxnalyb""%29+as+matchinfo%2C%0D%0A++++decode_matchinfo%28matchinfo%28articles_fts%2C+""pcxnalyb""%29%29+as+decoded_matchinfo%0D%0Afrom%0D%0A++++articles_fts+join+articles+on+articles.rowid+%3D+articles_fts.rowid%0D%0Awhere%0D%0A++++articles_fts+match+%3Asearch%0D%0Aorder+by+bm25&search=jquery+maps). Read [Exploring search relevance algorithms with SQLite](https://simonwillison.net/2019/Jan/7/exploring-search-relevance-algorithms-sqlite/) for further details on this project.
## Installation
pip install datasette-sqlite-fts4
If you are deploying a database using `datasette publish` you can include this plugin using the `--install` option:
datasette publish now mydb.db --install=datasette-sqlite-fts4
","
datasette-sqlite-fts4
Datasette plugin that exposes the custom SQL functions from sqlite-fts4.
If you are deploying a database using datasette publish you can include this plugin using the --install option:
datasette publish now mydb.db --install=datasette-sqlite-fts4
",1,public,0,,0,
167730071,MDEwOlJlcG9zaXRvcnkxNjc3MzAwNzE=,datasette-pretty-json,simonw/datasette-pretty-json,0,9599,https://github.com/simonw/datasette-pretty-json,Datasette plugin that pretty-prints any column values that are valid JSON objects or arrays,0,2019-01-26T19:30:43Z,2022-09-24T06:13:11Z,2022-09-28T21:06:31Z,,14,8,8,Python,1,1,1,1,0,0,0,0,1,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""json""]",0,1,8,master,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-pretty-json
[](https://pypi.org/project/datasette-pretty-json/)
[](https://github.com/simonw/datasette-pretty-json/releases)
[](https://github.com/simonw/datasette-pretty-json/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-pretty-json/blob/main/LICENSE)
[Datasette](https://github.com/simonw/datasette) plugin that pretty-prints any column values that are valid JSON objects or arrays.
You may also be interested in [datasette-json-html](https://github.com/simonw/datasette-json-html).
","
datasette-pretty-json
Datasette plugin that pretty-prints any column values that are valid JSON objects or arrays.
",1,public,0,,0,
197882382,MDEwOlJlcG9zaXRvcnkxOTc4ODIzODI=,healthkit-to-sqlite,dogsheep/healthkit-to-sqlite,0,53015001,https://github.com/dogsheep/healthkit-to-sqlite,Convert an Apple Healthkit export zip to a SQLite database,0,2019-07-20T05:03:12Z,2021-08-20T00:55:34Z,2021-08-20T00:56:17Z,https://datasette.io/tools/healthkit-to-sqlite,29,91,91,Python,1,1,1,1,0,4,0,0,8,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-tool"", ""dogsheep"", ""healthkit"", ""sqlite""]",4,8,91,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,53015001,4,3,"# healthkit-to-sqlite
[](https://pypi.org/project/healthkit-to-sqlite/)
[](https://github.com/dogsheep/healthkit-to-sqlite/releases)
[](https://github.com/dogsheep/healthkit-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/dogsheep/healthkit-to-sqlite/blob/main/LICENSE)
Convert an Apple Healthkit export zip to a SQLite database
## How to install
$ pip install healthkit-to-sqlite
## How to use
First you need to export your Apple HealthKit data.
1. On your iPhone, open the ""Health"" app
2. Click the profile icon in the top right
3. Click ""Export Health Data"" at the bottom of that page
4. Save the resulting file somewhere you can access it, or AirDrop it directly to your laptop.
Now you can convert the resulting `export.zip` file to SQLite like so:
$ healthkit-to-sqlite export.zip healthkit.db
A progress bar will be displayed. You can disable this using `--silent`.
```
Importing from HealthKit [#-------------] 5% 00:01:33
```
You can explore the resulting data using [Datasette](https://datasette.readthedocs.io/) like this:
$ datasette healthkit.db
","
healthkit-to-sqlite
Convert an Apple Healthkit export zip to a SQLite database
How to install
$ pip install healthkit-to-sqlite
How to use
First you need to export your Apple HealthKit data.
On your iPhone, open the ""Health"" app
Click the profile icon in the top right
Click ""Export Health Data"" at the bottom of that page
Save the resulting file somewhere you can access it, or AirDrop it directly to your laptop.
Now you can convert the resulting export.zip file to SQLite like so:
$ healthkit-to-sqlite export.zip healthkit.db
A progress bar will be displayed. You can disable this using --silent.
Importing from HealthKit [#-------------] 5% 00:01:33
You can explore the resulting data using Datasette like this:
$ datasette healthkit.db
",,,,,,
206156866,MDEwOlJlcG9zaXRvcnkyMDYxNTY4NjY=,twitter-to-sqlite,dogsheep/twitter-to-sqlite,0,53015001,https://github.com/dogsheep/twitter-to-sqlite,Save data from Twitter to a SQLite database,0,2019-09-03T19:30:08Z,2021-12-26T18:08:43Z,2021-12-26T18:08:40Z,,298,269,269,Python,1,1,1,1,0,13,0,0,10,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-tool"", ""dogsheep"", ""sqlite"", ""twitter"", ""twitter-api""]",13,10,269,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,53015001,13,5,"# twitter-to-sqlite
[](https://pypi.org/project/twitter-to-sqlite/)
[](https://github.com/dogsheep/twitter-to-sqlite/releases)
[](https://github.com/dogsheep/twitter-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/dogsheep/twitter-to-sqlite/blob/main/LICENSE)
Save data from Twitter to a SQLite database.
**This tool currently uses Twitter API v1**. You may be unable to use it if you do not have an API key for that version of the API.
- [How to install](#how-to-install)
- [Authentication](#authentication)
- [Retrieving tweets by specific accounts](#retrieving-tweets-by-specific-accounts)
- [Retrieve user profiles in bulk](#retrieve-user-profiles-in-bulk)
- [Retrieve tweets in bulk](#retrieve-tweets-in-bulk)
- [Retrieving Twitter followers](#retrieving-twitter-followers)
- [Retrieving friends](#retrieving-friends)
- [Retrieving favorited tweets](#retrieving-favorited-tweets)
- [Retrieving Twitter lists](#retrieving-twitter-lists)
- [Retrieving Twitter list memberships](#retrieving-twitter-list-memberships)
- [Retrieving just follower and friend IDs](#retrieving-just-follower-and-friend-ids)
- [Retrieving tweets from your home timeline](#retrieving-tweets-from-your-home-timeline)
- [Retrieving your mentions](#retrieving-your-mentions)
- [Providing input from a SQL query with --sql and --attach](#providing-input-from-a-sql-query-with---sql-and---attach)
- [Running searches](#running-searches)
- [Capturing tweets in real-time with track and follow](#capturing-tweets-in-real-time-with-track-and-follow)
* [track](#track)
* [follow](#follow)
- [Importing data from your Twitter archive](#importing-data-from-your-twitter-archive)
- [Design notes](#design-notes)
## How to install
$ pip install twitter-to-sqlite
## Authentication
First, you will need to create a Twitter application at https://developer.twitter.com/en/apps. You may need to apply for a Twitter developer account - if so, you may find this [example of an email application](https://raw.githubusercontent.com/dogsheep/twitter-to-sqlite/main/email.png) useful that has been approved in the past.
Once you have created your application, navigate to the ""Keys and tokens"" page and make note of the following:
* Your API key
* Your API secret key
* Your access token
* Your access token secret
You will need to save all four of these values to a JSON file in order to use this tool.
You can create that JSON file by running the following command and pasting in the values at the prompts:
$ twitter-to-sqlite auth
Create an app here: https://developer.twitter.com/en/apps
Then navigate to 'Keys and tokens' and paste in the following:
API key: xxx
API secret key: xxx
Access token: xxx
Access token secret: xxx
This will create a file called `auth.json` in your current directory containing the required values. To save the file at a different path or filename, use the `--auth=myauth.json` option.
## Retrieving tweets by specific accounts
The `user-timeline` command retrieves all of the tweets posted by the specified user accounts. It defaults to the account belonging to the authenticated user:
$ twitter-to-sqlite user-timeline twitter.db
Importing tweets [#####-------------------------------] 2799/17780 00:01:39
All of these commands assume that there is an `auth.json` file in the current directory. You can provide the path to your `auth.json` file using `-a`:
$ twitter-to-sqlite user-timeline twitter.db -a /path/to/auth.json
To load tweets for other users, pass their screen names as arguments:
$ twitter-to-sqlite user-timeline twitter.db cleopaws nichemuseums
Twitter's API only returns up to around 3,200 tweets for most user accounts, but you may find that it returns all available tweets for your own user account.
You can pass numeric Twitter user IDs instead of screen names using the `--ids` parameter.
You can use `--since` to retrieve every tweet since the last time you imported for that user, or `--since_id=xxx` to retrieve every tweet since a specific tweet ID.
This command also accepts `--sql` and `--attach` options, documented below.
## Retrieve user profiles in bulk
If you have a list of Twitter screen names (or user IDs) you can bulk fetch their fully inflated Twitter profiles using the `users-lookup` command:
$ twitter-to-sqlite users-lookup users.db simonw cleopaws
You can pass user IDs instead using the `--ids` option:
$ twitter-to-sqlite users-lookup users.db 12497 3166449535 --ids
This command also accepts `--sql` and `--attach` options, documented below.
## Retrieve tweets in bulk
If you have a list of tweet IDS you can bulk fetch them using the `statuses-lookup` command:
$ twitter-to-sqlite statuses-lookup tweets.db 1122154819815239680 1122154178493575169
The `--sql` and `--attach` options are supported.
Here's a recipe to retrieve any tweets that existing tweets are in-reply-to which have not yet been stored in your database:
$ twitter-to-sqlite statuses-lookup tweets.db \
--sql='
select in_reply_to_status_id
from tweets
where in_reply_to_status_id is not null' \
--skip-existing
The `--skip-existing` option means that tweets that have already been stored in the database will not be fetched again.
## Retrieving Twitter followers
The `followers` command retrieves details of every follower of the specified accounts. You can use it to retrieve your own followers, or you can pass one or more screen names to pull the followers for other accounts.
The following command pulls your followers and saves them in a SQLite database file called `twitter.db`:
$ twitter-to-sqlite followers twitter.db
This command is **extremely slow**, because Twitter impose a rate limit of no more than one request per minute to this endpoint! If you are running it against an account with thousands of followers you should expect this to take several hours.
To retrieve followers for another account, use:
$ twitter-to-sqlite followers twitter.db cleopaws
This command also accepts the `--ids`, `--sql` and `--attach` options.
See [Analyzing my Twitter followers with Datasette](https://simonwillison.net/2018/Jan/28/analyzing-my-twitter-followers/) for the original inspiration for this command.
## Retrieving friends
The `friends` command works like the `followers` command, but retrieves the specified (or currently authenticated) user's friends - defined as accounts that the user is following.
$ twitter-to-sqlite friends twitter.db
It takes the same options as the `followers` command.
## Retrieving favorited tweets
The `favorites` command retrieves tweets that have been favorited by a specified user. Called without any extra arguments it retrieves tweets favorited by the currently authenticated user:
$ twitter-to-sqlite favorites faves.db
You can also use the `--screen_name` or `--user_id` arguments to retrieve favorite tweets for another user:
$ twitter-to-sqlite favorites faves-obama.db --screen_name=BarackObama
Use the `--stop_after=xxx` argument to retrieve only the most recent number of favorites, e.g. to get the authenticated user's 50 most recent favorites:
$ twitter-to-sqlite favorites faves.db --stop_after=50
## Retrieving Twitter lists
The `lists` command retrieves all of the lists belonging to one or more users.
$ twitter-to-sqlite lists lists.db simonw dogsheep
This command also accepts the `--sql` and `--attach` and `--ids` options.
To additionally fetch the list of members for each list, use `--members`.
## Retrieving Twitter list memberships
The `list-members` command can be used to retrieve details of one or more Twitter lists, including all of their members.
$ twitter-to-sqlite list-members members.db simonw/the-good-place
You can pass multiple `screen_name/list_slug` identifiers.
If you know the numeric IDs of the lists instead, you can use `--ids`:
$ twitter-to-sqlite list-members members.db 927913322841653248 --ids
## Retrieving just follower and friend IDs
It's also possible to retrieve just the numeric Twitter IDs of the accounts that specific users are following (""friends"" in Twitter's API terminology) or followed-by:
$ twitter-to-sqlite followers-ids members.db simonw cleopaws
This will populate the `following` table with `followed_id`/`follower_id` pairs for the two specified accounts, listing every account ID that is following either of those two accounts.
$ twitter-to-sqlite friends-ids members.db simonw cleopaws
This will do the same thing but pull the IDs that those accounts are following.
Both of these commands also support `--sql` and `--attach` as an alternative to passing screen names as direct command-line arguments. You can use `--ids` to process the inputs as user IDs rather than screen names.
The underlying Twitter APIs have a rate limit of 15 requests every 15 minutes - though they do return up to 5,000 IDs in each call. By default both of these subcommands will wait for 61 seconds between API calls in order to stay within the rate limit - you can adjust this behaviour down to just one second delay if you know you will not be making many calls using `--sleep=1`.
## Retrieving tweets from your home timeline
The `home-timeline` command retrieves up to 800 tweets from the home timeline of the authenticated user - generally this means tweets from people you follow.
$ twitter-to-sqlite home-timeline twitter.db
Importing timeline [#################--------] 591/800 00:01:14
The tweets are stored in the `tweets` table, and a record is added to the `timeline_tweets` table noting that this tweet came in due to being spotted in the timeline of your user.
You can use `--since` to retrieve just tweets that have been posted since the last time this command was run, or `--since_id=xxx` to explicitly pass in a tweet ID to use as the last position.
You can then view your timeline in Datasette using the following URL:
`/tweets/tweets?_where=id+in+(select+tweet+from+[timeline_tweets])&_sort_desc=id&_facet=user`
This will filter your tweets table to just tweets that appear in your timeline, ordered by most recent first and use faceting to show you which users are responsible for the most tweets.
## Retrieving your mentions
The `mentions-timeline` command works like `home-timeline` except it retrieves tweets that mention the authenticated user's account. It records the user account that was mentioned in a `mentions_tweets` table.
It supports `--since` and `--since_id` in the same was as `home-timeline` does.
## Providing input from a SQL query with --sql and --attach
This option is available for some subcommands - run `twitter-to-sqlite command-name --help` to check.
You can provide Twitter screen names (or user IDs or tweet IDs) directly as command-line arguments, or you can provide those screen names or IDs by executing a SQL query.
For example: consider a SQLite database with an `attendees` table listing names and Twitter accounts - something like this:
| First | Last | Twitter |
|---------|------------|--------------|
| Simon | Willison | simonw |
| Avril | Lavigne | AvrilLavigne |
You can run the `users-lookup` command to pull the Twitter profile of every user listed in that database by loading the screen names using a `--sql` query:
$ twitter-to-sqlite users-lookup my.db --sql=""select Twitter from attendees""
If your database table contains Twitter IDs, you can select those IDs and pass the `--ids` argument. For example, to fetch the profiles of users who have had their user IDs inserted into the `following` table using the `twitter-to-sqlite friends-ids` command:
$ twitter-to-sqlite users-lookup my.db --sql=""select follower_id from following"" --ids
Or to avoid re-fetching users that have already been fetched:
$ twitter-to-sqlite users-lookup my.db \
--sql=""select followed_id from following where followed_id not in (
select id from users)"" --ids
If your data lives in a separate database file you can attach it using `--attach`. For example, consider the attendees example above but the data lives in an `attendees.db` file, and you want to fetch the user profiles into a `tweets.db` file. You could do that like this:
$ twitter-to-sqlite users-lookup tweets.db \
--attach=attendees.db \
--sql=""select Twitter from attendees.attendees""
The filename (without the extension) will be used as the database alias within SQLite. If you want a different alias for some reason you can specify that with a colon like this:
$ twitter-to-sqlite users-lookup tweets.db \
--attach=foo:attendees.db \
--sql=""select Twitter from foo.attendees""
## Running searches
The `search` command runs a search against the Twitter [standard search API](https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets).
$ twitter-to-sqlite search tweets.db ""dogsheep""
This will import up to around 320 tweets that match that search term into the `tweets` table. It will also create a record in the `search_runs` table recording that the search took place, and many-to-many records in the `search_runs_tweets` table recording which tweets were seen for that search at that time.
You can use the `--since` parameter to check for previous search runs with the same arguments and only retrieve tweets that were posted since the last retrieved matching tweet.
The following additional options for `search` are supported:
* `--geocode`: `latitude,longitude,radius` where radius is a number followed by mi or km
* `--lang`: ISO 639-1 language code e.g. `en` or `es`
* `--locale`: Locale: only `ja` is currently effective
* `--result_type`: `mixed`, `recent` or `popular`. Defaults to `mixed`
* `--count`: Number of results per page, defaults to the maximum of 100
* `--stop_after`: Stop after this many results
* `--since_id`: Pull tweets since this Tweet ID. You probably want to use `--since` instead of this.
## Capturing tweets in real-time with track and follow
This functionality is **experimental**. Please [file bug reports](https://github.com/dogsheep/twitter-to-sqlite/issues) if you find any!
Twitter provides a real-time API which can be used to subscribe to tweets as they happen. `twitter-to-sqlite` can use this API to continually update a SQLite database with tweets matching certain keywords, or referencing specific users.
### track
To track keywords, use the `track` command:
$ twitter-to-sqlite track tweets.db kakapo
This command will continue to run until you hit Ctrl+C. It will capture any tweets mentioning the keyword [kakapo](https://en.wikipedia.org/wiki/Kakapo) and store them in the `tweets.db` database file.
You can pass multiple keywords as a space separated list. This will capture tweets matching either of those keywords:
$ twitter-to-sqlite track tweets.db kakapo raccoon
You can enclose phrases in quotes to search for tweets matching both of those keywords:
$ twitter-to-sqlite track tweets.db 'trash panda'
See [the Twitter track documentation](https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/basic-stream-parameters#track) for advanced tips on using this command.
Add the `--verbose` option to see matching tweets (in their verbose JSON form) displayed to the terminal as they are captured:
$ twitter-to-sqlite track tweets.db raccoon --verbose
### follow
The `follow` command will capture all tweets that are relevant to one or more specific Twitter users.
$ twitter-to-sqlite follow tweets.db nytimes
This includes tweets by those users, tweets that reply to or quote those users and retweets by that user. See [the Twitter follow documentation](https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/basic-stream-parameters#follow) for full details.
The command accepts one or more screen names.
You can feed it numeric Twitter user IDs instead of screen names by using the `--ids` flag.
The command also supports the `--sql` and `--attach` options, and the `--verbose` option for displaying tweets as they are captured.
Here's how to start following tweets from every user ID currently represented as being followed in the `following` table (populated using the `friends-ids` command):
$ twitter-to-sqlite follow tweets.db \
--sql=""select distinct followed_id from following"" \
--ids
## Importing data from your Twitter archive
You can request an archive of your Twitter data by [following these instructions](https://help.twitter.com/en/managing-your-account/how-to-download-your-twitter-archive).
Twitter will send you a link to download a `.zip` file. You can import the contents of that file into a set of tables in a new database file called `archive.db` (each table beginning with the `archive_` prefix) using the `import` command:
$ twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2.zip
This command does not populate any of the regular tables, since Twitter's export data does not exactly match the schema returned by the Twitter API.
It will delete and recreate the corresponding `archive_*` tables every time you run it. If this is not what you want, run the command against a new SQLite database file name rather than running it against one that already exists.
If you have already decompressed your archive, you can run this against the directory that you decompressed it to:
$ twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2/
You can also run it against one or more specific files within that folder. For example, to import just the follower.js and following.js files:
$ twitter-to-sqlite import archive.db \
~/Downloads/twitter-2019-06-25-b31f2/follower.js \
~/Downloads/twitter-2019-06-25-b31f2/following.js
You may want to use other commands to populate tables based on data from the archive. For example, to retrieve full API versions of each of the tweets you have favourited in your archive, you could run the following:
$ twitter-to-sqlite statuses-lookup archive.db \
--sql='select tweetId from archive_like' \
--skip-existing
If you want these imported tweets to then be reflected in the `favorited_by` table, you can do so by applying the following SQL query:
$ sqlite3 archive.db
SQLite version 3.22.0 2018-01-22 18:45:57
Enter "".help"" for usage hints.
sqlite> INSERT OR IGNORE INTO favorited_by (tweet, user)
...> SELECT tweetId, 'YOUR_TWITTER_ID' FROM archive_like;
Replace YOUR_TWITTER_ID with your numeric Twitter ID. If you don't know that ID you can find it out by running the following:
$ twitter-to-sqlite fetch \
""https://api.twitter.com/1.1/account/verify_credentials.json"" \
| grep '""id""' | head -n 1
## Design notes
* Tweet IDs are stored as integers, to afford sorting by ID in a sensible way
* While we configure foreign key relationships between tables, we do not ask SQLite to enforce them. This is used by the `following` table to allow the `followers-ids` and `friends-ids` commands to populate it with user IDs even if the user accounts themselves are not yet present in the `users` table.
","
twitter-to-sqlite
Save data from Twitter to a SQLite database.
This tool currently uses Twitter API v1. You may be unable to use it if you do not have an API key for that version of the API.
Once you have created your application, navigate to the ""Keys and tokens"" page and make note of the following:
Your API key
Your API secret key
Your access token
Your access token secret
You will need to save all four of these values to a JSON file in order to use this tool.
You can create that JSON file by running the following command and pasting in the values at the prompts:
$ twitter-to-sqlite auth
Create an app here: https://developer.twitter.com/en/apps
Then navigate to 'Keys and tokens' and paste in the following:
API key: xxx
API secret key: xxx
Access token: xxx
Access token secret: xxx
This will create a file called auth.json in your current directory containing the required values. To save the file at a different path or filename, use the --auth=myauth.json option.
Retrieving tweets by specific accounts
The user-timeline command retrieves all of the tweets posted by the specified user accounts. It defaults to the account belonging to the authenticated user:
Twitter's API only returns up to around 3,200 tweets for most user accounts, but you may find that it returns all available tweets for your own user account.
You can pass numeric Twitter user IDs instead of screen names using the --ids parameter.
You can use --since to retrieve every tweet since the last time you imported for that user, or --since_id=xxx to retrieve every tweet since a specific tweet ID.
This command also accepts --sql and --attach options, documented below.
Retrieve user profiles in bulk
If you have a list of Twitter screen names (or user IDs) you can bulk fetch their fully inflated Twitter profiles using the users-lookup command:
Here's a recipe to retrieve any tweets that existing tweets are in-reply-to which have not yet been stored in your database:
$ twitter-to-sqlite statuses-lookup tweets.db \
--sql='
select in_reply_to_status_id
from tweets
where in_reply_to_status_id is not null' \
--skip-existing
The --skip-existing option means that tweets that have already been stored in the database will not be fetched again.
Retrieving Twitter followers
The followers command retrieves details of every follower of the specified accounts. You can use it to retrieve your own followers, or you can pass one or more screen names to pull the followers for other accounts.
The following command pulls your followers and saves them in a SQLite database file called twitter.db:
$ twitter-to-sqlite followers twitter.db
This command is extremely slow, because Twitter impose a rate limit of no more than one request per minute to this endpoint! If you are running it against an account with thousands of followers you should expect this to take several hours.
To retrieve followers for another account, use:
$ twitter-to-sqlite followers twitter.db cleopaws
This command also accepts the --ids, --sql and --attach options.
The friends command works like the followers command, but retrieves the specified (or currently authenticated) user's friends - defined as accounts that the user is following.
$ twitter-to-sqlite friends twitter.db
It takes the same options as the followers command.
Retrieving favorited tweets
The favorites command retrieves tweets that have been favorited by a specified user. Called without any extra arguments it retrieves tweets favorited by the currently authenticated user:
$ twitter-to-sqlite favorites faves.db
You can also use the --screen_name or --user_id arguments to retrieve favorite tweets for another user:
It's also possible to retrieve just the numeric Twitter IDs of the accounts that specific users are following (""friends"" in Twitter's API terminology) or followed-by:
This will populate the following table with followed_id/follower_id pairs for the two specified accounts, listing every account ID that is following either of those two accounts.
This will do the same thing but pull the IDs that those accounts are following.
Both of these commands also support --sql and --attach as an alternative to passing screen names as direct command-line arguments. You can use --ids to process the inputs as user IDs rather than screen names.
The underlying Twitter APIs have a rate limit of 15 requests every 15 minutes - though they do return up to 5,000 IDs in each call. By default both of these subcommands will wait for 61 seconds between API calls in order to stay within the rate limit - you can adjust this behaviour down to just one second delay if you know you will not be making many calls using --sleep=1.
Retrieving tweets from your home timeline
The home-timeline command retrieves up to 800 tweets from the home timeline of the authenticated user - generally this means tweets from people you follow.
The tweets are stored in the tweets table, and a record is added to the timeline_tweets table noting that this tweet came in due to being spotted in the timeline of your user.
You can use --since to retrieve just tweets that have been posted since the last time this command was run, or --since_id=xxx to explicitly pass in a tweet ID to use as the last position.
You can then view your timeline in Datasette using the following URL:
This will filter your tweets table to just tweets that appear in your timeline, ordered by most recent first and use faceting to show you which users are responsible for the most tweets.
Retrieving your mentions
The mentions-timeline command works like home-timeline except it retrieves tweets that mention the authenticated user's account. It records the user account that was mentioned in a mentions_tweets table.
It supports --since and --since_id in the same was as home-timeline does.
Providing input from a SQL query with --sql and --attach
This option is available for some subcommands - run twitter-to-sqlite command-name --help to check.
You can provide Twitter screen names (or user IDs or tweet IDs) directly as command-line arguments, or you can provide those screen names or IDs by executing a SQL query.
For example: consider a SQLite database with an attendees table listing names and Twitter accounts - something like this:
First
Last
Twitter
Simon
Willison
simonw
Avril
Lavigne
AvrilLavigne
You can run the users-lookup command to pull the Twitter profile of every user listed in that database by loading the screen names using a --sql query:
$ twitter-to-sqlite users-lookup my.db --sql=""select Twitter from attendees""
If your database table contains Twitter IDs, you can select those IDs and pass the --ids argument. For example, to fetch the profiles of users who have had their user IDs inserted into the following table using the twitter-to-sqlite friends-ids command:
$ twitter-to-sqlite users-lookup my.db --sql=""select follower_id from following"" --ids
Or to avoid re-fetching users that have already been fetched:
$ twitter-to-sqlite users-lookup my.db \
--sql=""select followed_id from following where followed_id not in (
select id from users)"" --ids
If your data lives in a separate database file you can attach it using --attach. For example, consider the attendees example above but the data lives in an attendees.db file, and you want to fetch the user profiles into a tweets.db file. You could do that like this:
The filename (without the extension) will be used as the database alias within SQLite. If you want a different alias for some reason you can specify that with a colon like this:
This will import up to around 320 tweets that match that search term into the tweets table. It will also create a record in the search_runs table recording that the search took place, and many-to-many records in the search_runs_tweets table recording which tweets were seen for that search at that time.
You can use the --since parameter to check for previous search runs with the same arguments and only retrieve tweets that were posted since the last retrieved matching tweet.
The following additional options for search are supported:
--geocode: latitude,longitude,radius where radius is a number followed by mi or km
--lang: ISO 639-1 language code e.g. en or es
--locale: Locale: only ja is currently effective
--result_type: mixed, recent or popular. Defaults to mixed
--count: Number of results per page, defaults to the maximum of 100
--stop_after: Stop after this many results
--since_id: Pull tweets since this Tweet ID. You probably want to use --since instead of this.
Capturing tweets in real-time with track and follow
This functionality is experimental. Please file bug reports if you find any!
Twitter provides a real-time API which can be used to subscribe to tweets as they happen. twitter-to-sqlite can use this API to continually update a SQLite database with tweets matching certain keywords, or referencing specific users.
track
To track keywords, use the track command:
$ twitter-to-sqlite track tweets.db kakapo
This command will continue to run until you hit Ctrl+C. It will capture any tweets mentioning the keyword kakapo and store them in the tweets.db database file.
You can pass multiple keywords as a space separated list. This will capture tweets matching either of those keywords:
The follow command will capture all tweets that are relevant to one or more specific Twitter users.
$ twitter-to-sqlite follow tweets.db nytimes
This includes tweets by those users, tweets that reply to or quote those users and retweets by that user. See the Twitter follow documentation for full details.
The command accepts one or more screen names.
You can feed it numeric Twitter user IDs instead of screen names by using the --ids flag.
The command also supports the --sql and --attach options, and the --verbose option for displaying tweets as they are captured.
Here's how to start following tweets from every user ID currently represented as being followed in the following table (populated using the friends-ids command):
Twitter will send you a link to download a .zip file. You can import the contents of that file into a set of tables in a new database file called archive.db (each table beginning with the archive_ prefix) using the import command:
This command does not populate any of the regular tables, since Twitter's export data does not exactly match the schema returned by the Twitter API.
It will delete and recreate the corresponding archive_* tables every time you run it. If this is not what you want, run the command against a new SQLite database file name rather than running it against one that already exists.
If you have already decompressed your archive, you can run this against the directory that you decompressed it to:
You may want to use other commands to populate tables based on data from the archive. For example, to retrieve full API versions of each of the tweets you have favourited in your archive, you could run the following:
If you want these imported tweets to then be reflected in the favorited_by table, you can do so by applying the following SQL query:
$ sqlite3 archive.db
SQLite version 3.22.0 2018-01-22 18:45:57
Enter "".help"" for usage hints.
sqlite> INSERT OR IGNORE INTO favorited_by (tweet, user)
...> SELECT tweetId, 'YOUR_TWITTER_ID' FROM archive_like;
<Ctrl+D>
Replace YOUR_TWITTER_ID with your numeric Twitter ID. If you don't know that ID you can find it out by running the following:
Tweet IDs are stored as integers, to afford sorting by ID in a sensible way
While we configure foreign key relationships between tables, we do not ask SQLite to enforce them. This is used by the following table to allow the followers-ids and friends-ids commands to populate it with user IDs even if the user accounts themselves are not yet present in the users table.
",1,public,0,,,
207052882,MDEwOlJlcG9zaXRvcnkyMDcwNTI4ODI=,github-to-sqlite,dogsheep/github-to-sqlite,0,53015001,https://github.com/dogsheep/github-to-sqlite,Save data from GitHub to a SQLite database,0,2019-09-08T02:50:28Z,2022-09-20T04:36:37Z,2022-09-28T21:07:54Z,https://github-to-sqlite.dogsheep.net/,143,235,235,Python,1,1,1,1,0,32,0,0,20,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-tool"", ""dogsheep"", ""github-api"", ""sqlite""]",32,20,235,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,53015001,32,6,"# github-to-sqlite
[](https://pypi.org/project/github-to-sqlite/)
[](https://github.com/dogsheep/github-to-sqlite/releases)
[](https://github.com/dogsheep/github-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/dogsheep/github-to-sqlite/blob/main/LICENSE)
Save data from GitHub to a SQLite database.
- [Demo](#demo)
- [How to install](#how-to-install)
- [Authentication](#authentication)
- [Fetching issues for a repository](#fetching-issues-for-a-repository)
- [Fetching pull requests for a repository](#fetching-pull-requests-for-a-repository)
- [Fetching issue comments for a repository](#fetching-issue-comments-for-a-repository)
- [Fetching commits for a repository](#fetching-commits-for-a-repository)
- [Fetching releases for a repository](#fetching-releases-for-a-repository)
- [Fetching tags for a repository](#fetching-tags-for-a-repository)
- [Fetching contributors to a repository](#fetching-contributors-to-a-repository)
- [Fetching repos belonging to a user or organization](#fetching-repos-belonging-to-a-user-or-organization)
- [Fetching specific repositories](#fetching-specific-repositories)
- [Fetching repos that have been starred by a user](#fetching-repos-that-have-been-starred-by-a-user)
- [Fetching users that have starred specific repos](#fetching-users-that-have-starred-specific-repos)
- [Fetching GitHub Actions workflows](#fetching-github-actions-workflows)
- [Scraping dependents for a repository](#scraping-dependents-for-a-repository)
- [Fetching emojis](#fetching-emojis)
- [Making authenticated API calls](#making-authenticated-api-calls)
## Demo
https://github-to-sqlite.dogsheep.net/ hosts a [Datasette](https://datasette.io/) demo of a database created by [running this tool](https://github.com/dogsheep/github-to-sqlite/blob/main/.github/workflows/deploy-demo.yml#L40-L60) against all of the repositories in the [Dogsheep GitHub organization](https://github.com/dogsheep), plus the [datasette](https://github.com/simonw/datasette) and [sqlite-utils](https://github.com/simonw/sqlite-utils) repositories.
## How to install
$ pip install github-to-sqlite
## Authentication
Create a GitHub personal access token: https://github.com/settings/tokens
Run this command and paste in your new token:
$ github-to-sqlite auth
This will create a file called `auth.json` in your current directory containing the required value. To save the file at a different path or filename, use the `--auth=myauth.json` option.
As an alternative to using an `auth.json` file you can add your access token to an environment variable called `GITHUB_TOKEN`.
## Fetching issues for a repository
The `issues` command retrieves all of the issues belonging to a specified repository.
$ github-to-sqlite issues github.db simonw/datasette
If an `auth.json` file is present it will use the token from that file. It works without authentication for public repositories but you should be aware that GitHub have strict IP-based rate limits for unauthenticated requests.
You can point to a different location of `auth.json` using `-a`:
$ github-to-sqlite issues github.db simonw/datasette -a /path/to/auth.json
You can use the `--issue` option one or more times to load specific issues:
$ github-to-sqlite issues github.db simonw/datasette --issue=1
Example: [issues table](https://github-to-sqlite.dogsheep.net/github/issues)
## Fetching pull requests for a repository
While pull requests are a type of issue, you will get more information on pull requests by pulling them separately. For example, whether a pull request has been merged and when.
Following the API of issues, the `pull-requests` command retrieves all of the pull requests belonging to a specified repository.
$ github-to-sqlite pull-requests github.db simonw/datasette
You can use the `--pull-request` option one or more times to load specific pull request:
$ github-to-sqlite pull-requests github.db simonw/datasette --pull-request=81
Note that the `merged_by` column on the `pull_requests` table will only be populated for pull requests that are loaded using the `--pull-request` option - the GitHub API does not return this field for pull requests that are loaded in bulk.
Example: [pull_requests table](https://github-to-sqlite.dogsheep.net/github/pull_requests)
## Fetching issue comments for a repository
The `issue-comments` command retrieves all of the comments on all of the issues in a repository.
It is recommended you run `issues` first, so that each imported comment can have a foreign key poining to its issue.
$ github-to-sqlite issues github.db simonw/datasette
$ github-to-sqlite issue-comments github.db simonw/datasette
You can use the `--issue` option to only load comments for a specific issue within that repository, for example:
$ github-to-sqlite issue-comments github.db simonw/datasette --issue=1
Example: [issue_comments table](https://github-to-sqlite.dogsheep.net/github/issue_comments)
## Fetching commits for a repository
The `commits` command retrieves details of all of the commits for one or more repositories. It currently fetches the sha, commit message and author and committer details - it does no retrieve the full commit body.
$ github-to-sqlite commits github.db simonw/datasette simonw/sqlite-utils
The command accepts one or more repositories.
By default it will stop as soon as it sees a commit that has previously been retrieved. You can force it to retrieve all commits (including those that have been previously inserted) using `--all`.
Example: [commits table](https://github-to-sqlite.dogsheep.net/github/commits)
## Fetching releases for a repository
The `releases` command retrieves the releases for one or more repositories.
$ github-to-sqlite releases github.db simonw/datasette simonw/sqlite-utils
The command accepts one or more repositories.
Example: [releases table](https://github-to-sqlite.dogsheep.net/github/releases)
## Fetching tags for a repository
The `tags` command retrieves all of the tags for one or more repositories.
$ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils
Example: [tags table](https://github-to-sqlite.dogsheep.net/github/tags)
## Fetching contributors to a repository
The `contributors` command retrieves details of all of the contributors for one or more repositories.
$ github-to-sqlite contributors github.db simonw/datasette simonw/sqlite-utils
The command accepts one or more repositories. It populates a `contributors` table, with foreign keys to `repos` and `users` and a `contributions` table listing the number of commits to that repository for each contributor.
Example: [contributors table](https://github-to-sqlite.dogsheep.net/github/contributors)
## Fetching repos belonging to a user or organization
The `repos` command fetches repos belonging to a user or organization.
Without any other arguments, this command will fetch all repos that the currently authenticated user owns, collaborates on or can access via one of their organizations:
$ github-to-sqlite repos github.db
To fetch repos belonging to a specific user or organization, provide their username as an argument:
$ github-to-sqlite repos github.db dogsheep # organization
$ github-to-sqlite repos github.db simonw # user
You can pass more than one username to fetch for multiple users or organizations at once:
$ github-to-sqlite repos github.db simonw dogsheep
Add the `--readme` option to save the README for the repo in a column called `readme`. Add `--readme-html` to save the HTML rendered version of the README into a collumn called `readme_html`.
Example: [repos table](https://github-to-sqlite.dogsheep.net/github/repos)
## Fetching specific repositories
You can use `-r` with the `repos` command one or more times to fetch just specific repositories.
$ github-to-sqlite repos github.db -r simonw/datasette -r dogsheep/github-to-sqlite
## Fetching repos that have been starred by a user
The `starred` command fetches the repos that have been starred by a user.
$ github-to-sqlite starred github.db simonw
If you are using an `auth.json` file you can omit the username to retrieve the starred repos for the authenticated user.
Example: [stars table](https://github-to-sqlite.dogsheep.net/github/stars)
## Fetching users that have starred specific repos
The `stargazers` command fetches the users that have starred the specified repos.
$ github-to-sqlite stargazers github.db simonw/datasette dogsheep/github-to-sqlite
You can specify one or more repository using `owner/repo` syntax.
Users fetched using this command will be inserted into the `users` table. Many-to-many records showing which repository they starred will be added to the `stars` table.
## Fetching GitHub Actions workflows
The `workflows` command fetches the YAML workflow configurations from each repository's `.github/workflows` directory and parses them to populate `workflows`, `jobs` and `steps` tables.
$ github-to-sqlite workflows github.db simonw/datasette dogsheep/github-to-sqlite
You can specify one or more repository using `owner/repo` syntax.
Example: [workflows table](https://github-to-sqlite.dogsheep.net/github/workflows), [jobs table](https://github-to-sqlite.dogsheep.net/github/jobs), [steps table](https://github-to-sqlite.dogsheep.net/github/steps)
## Scraping dependents for a repository
The GitHub dependency graph can show other GitHub projects that depend on a specific repo, for example [simonw/datasette/network/dependents](https://github.com/simonw/datasette/network/dependents).
This data is not yet available through the GitHub API. The `scrape-dependents` command scrapes those pages and uses the GitHub API to load full versions of the dependent repositories.
$ github-to-sqlite scrape-dependents github.db simonw/datasette
The command accepts one or more repositories.
Add `-v` for verbose output.
Example: [dependents table](https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc)
## Fetching emojis
You can fetch a list of every emoji supported by GitHub using the `emojis` command:
$ github-to-sqlite emojis github.db
This will create a table callad `emojis` with a primary key `name` and a `url` column.
If you add the `--fetch` option the command will also fetch the binary content of the images and place them in an `image` column:
$ github-to-sqlite emojis emojis.db -f
[########----------------------------] 397/1799 22% 00:03:43
You can then use the [datasette-render-images](https://github.com/simonw/datasette-render-images) plugin to browse them visually.
Example: [emojis table](https://github-to-sqlite.dogsheep.net/github/emojis)
## Making authenticated API calls
The `github-to-sqlite get` command provides a convenient shortcut for making authenticated calls to the API. Once you have created your `auth.json` file (or set a `GITHUB_TOKEN` environment variable) you can use it like this:
$ github-to-sqlite get https://api.github.com/gists
This will make an authenticated call to the URL you provide and pretty-print the resulting JSON to the console.
You can ommit the `https://api.github.com/` prefix, for example:
$ github-to-sqlite get /gists
Many GitHub APIs are [paginated using the HTTP Link header](https://docs.github.com/en/rest/guides/traversing-with-pagination). You can follow this pagination and output a list of all of the resulting items using `--paginate`:
$ github-to-sqlite get /users/simonw/repos --paginate
You can outline newline-delimited JSON for each item using `--nl`. This can be useful for streaming items into another tool.
$ github-to-sqlite get /users/simonw/repos --nl
","
This will create a file called auth.json in your current directory containing the required value. To save the file at a different path or filename, use the --auth=myauth.json option.
As an alternative to using an auth.json file you can add your access token to an environment variable called GITHUB_TOKEN.
Fetching issues for a repository
The issues command retrieves all of the issues belonging to a specified repository.
If an auth.json file is present it will use the token from that file. It works without authentication for public repositories but you should be aware that GitHub have strict IP-based rate limits for unauthenticated requests.
You can point to a different location of auth.json using -a:
$ github-to-sqlite issues github.db simonw/datasette -a /path/to/auth.json
You can use the --issue option one or more times to load specific issues:
While pull requests are a type of issue, you will get more information on pull requests by pulling them separately. For example, whether a pull request has been merged and when.
Following the API of issues, the pull-requests command retrieves all of the pull requests belonging to a specified repository.
Note that the merged_by column on the pull_requests table will only be populated for pull requests that are loaded using the --pull-request option - the GitHub API does not return this field for pull requests that are loaded in bulk.
The commits command retrieves details of all of the commits for one or more repositories. It currently fetches the sha, commit message and author and committer details - it does no retrieve the full commit body.
By default it will stop as soon as it sees a commit that has previously been retrieved. You can force it to retrieve all commits (including those that have been previously inserted) using --all.
The command accepts one or more repositories. It populates a contributors table, with foreign keys to repos and users and a contributions table listing the number of commits to that repository for each contributor.
Fetching repos belonging to a user or organization
The repos command fetches repos belonging to a user or organization.
Without any other arguments, this command will fetch all repos that the currently authenticated user owns, collaborates on or can access via one of their organizations:
$ github-to-sqlite repos github.db
To fetch repos belonging to a specific user or organization, provide their username as an argument:
Add the --readme option to save the README for the repo in a column called readme. Add --readme-html to save the HTML rendered version of the README into a collumn called readme_html.
You can specify one or more repository using owner/repo syntax.
Users fetched using this command will be inserted into the users table. Many-to-many records showing which repository they starred will be added to the stars table.
Fetching GitHub Actions workflows
The workflows command fetches the YAML workflow configurations from each repository's .github/workflows directory and parses them to populate workflows, jobs and steps tables.
This data is not yet available through the GitHub API. The scrape-dependents command scrapes those pages and uses the GitHub API to load full versions of the dependent repositories.
The github-to-sqlite get command provides a convenient shortcut for making authenticated calls to the API. Once you have created your auth.json file (or set a GITHUB_TOKEN environment variable) you can use it like this:
$ github-to-sqlite get https://api.github.com/gists
This will make an authenticated call to the URL you provide and pretty-print the resulting JSON to the console.
You can ommit the https://api.github.com/ prefix, for example:
$ github-to-sqlite get /gists
Many GitHub APIs are paginated using the HTTP Link header. You can follow this pagination and output a list of all of the resulting items using --paginate:
$ github-to-sqlite get /users/simonw/repos --paginate
You can outline newline-delimited JSON for each item using --nl. This can be useful for streaming items into another tool.
$ github-to-sqlite get /users/simonw/repos --nl
",1,public,0,,0,
213286752,MDEwOlJlcG9zaXRvcnkyMTMyODY3NTI=,pocket-to-sqlite,dogsheep/pocket-to-sqlite,0,53015001,https://github.com/dogsheep/pocket-to-sqlite,Create a SQLite database containing data from your Pocket account,0,2019-10-07T03:24:14Z,2022-08-21T21:11:59Z,2022-08-22T16:21:34Z,,20,63,63,Python,1,1,1,1,0,3,0,0,5,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-tool"", ""dogsheep"", ""pocket"", ""pocket-api"", ""sqlite""]",3,5,63,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,53015001,3,4,"# pocket-to-sqlite
[](https://pypi.org/project/pocket-to-sqlite/)
[](https://github.com/dogsheep/pocket-to-sqlite/releases)
[](https://github.com/dogsheep/pocket-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/dogsheep/pocket-to-sqlite/blob/main/LICENSE)
Create a SQLite database containing data from your [Pocket](https://getpocket.com/) account.
## How to install
$ pip install pocket-to-sqlite
## Usage
You will need to first obtain a valid OAuth token for your Pocket account. You can do this by running the `auth` command and following the prompts:
$ pocket-to-sqlite auth
Visit this page and sign in with your Pocket account:
https://getpocket.com/auth/author...
Once you have signed in there, hit to continue
Authentication tokens written to auth.json
Now you can fetch all of your items from Pocket like this:
$ pocket-to-sqlite fetch pocket.db
The first time you run this command it will fetch all of your items, and display a progress bar while it does it.
On subsequent runs it will only fetch new items.
You can force it to fetch everything from the beginning again using `--all`. Use `--silent` to disable the progress bar.
## Using with Datasette
The SQLite database produced by this tool is designed to be browsed using [Datasette](https://datasette.readthedocs.io/). Use the [datasette-render-timestamps](https://github.com/simonw/datasette-render-timestamps) plugin to improve the display of the timestamp values.
","
pocket-to-sqlite
Create a SQLite database containing data from your Pocket account.
How to install
$ pip install pocket-to-sqlite
Usage
You will need to first obtain a valid OAuth token for your Pocket account. You can do this by running the auth command and following the prompts:
$ pocket-to-sqlite auth
Visit this page and sign in with your Pocket account:
https://getpocket.com/auth/author...
Once you have signed in there, hit <enter> to continue
Authentication tokens written to auth.json
Now you can fetch all of your items from Pocket like this:
$ pocket-to-sqlite fetch pocket.db
The first time you run this command it will fetch all of your items, and display a progress bar while it does it.
On subsequent runs it will only fetch new items.
You can force it to fetch everything from the beginning again using --all. Use --silent to disable the progress bar.
Using with Datasette
The SQLite database produced by this tool is designed to be browsed using Datasette. Use the datasette-render-timestamps plugin to improve the display of the timestamp values.
",1,public,0,,0,
220716822,MDEwOlJlcG9zaXRvcnkyMjA3MTY4MjI=,datasette-render-markdown,simonw/datasette-render-markdown,0,9599,https://github.com/simonw/datasette-render-markdown,Datasette plugin for rendering Markdown,0,2019-11-09T23:28:31Z,2022-05-26T04:58:56Z,2022-07-18T19:35:10Z,,57,11,11,Python,1,1,1,1,0,0,0,0,1,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""markdown""]",0,1,11,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-render-markdown
[](https://pypi.org/project/datasette-render-markdown/)
[](https://github.com/simonw/datasette-render-markdown/releases)
[](https://github.com/simonw/datasette-render-markdown/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-render-markdown/blob/main/LICENSE)
Datasette plugin for rendering Markdown.
## Installation
Install this plugin in the same environment as Datasette to enable this new functionality:
$ pip install datasette-render-markdown
## Usage
You can explicitly list the columns you would like to treat as Markdown using [plugin configuration](https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration) in a `metadata.json` file.
Add a `""datasette-render-markdown""` configuration block and use a `""columns""` key to list the columns you would like to treat as Markdown values:
```json
{
""plugins"": {
""datasette-render-markdown"": {
""columns"": [""body""]
}
}
}
```
This will cause any `body` column in any table to be treated as markdown and safely rendered using [Python-Markdown](https://python-markdown.github.io/). The resulting HTML is then run through [Bleach](https://bleach.readthedocs.io/) to avoid the risk of XSS security problems.
Save this to `metadata.json` and run Datasette with the `--metadata` flag to load this configuration:
$ datasette serve mydata.db --metadata metadata.json
The configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the `entries` table in the `news.db` database:
```json
{
""databases"": {
""news"": {
""tables"": {
""entries"": {
""plugins"": {
""datasette-render-markdown"": {
""columns"": [""body""]
}
}
}
}
}
}
}
```
And here's how to apply it to every `body` column in every table in the `news.db` database:
```json
{
""databases"": {
""news"": {
""plugins"": {
""datasette-render-markdown"": {
""columns"": [""body""]
}
}
}
}
}
```
## Columns that match a naming convention
This plugin can also render markdown in any columns that match a specific naming convention.
By default, columns that have a name ending in `_markdown` will be rendered.
You can try this out using the following query:
```sql
select '# Hello there
* This is a list
* of items
[And a link](https://github.com/simonw/datasette-render-markdown).'
as demo_markdown
```
You can configure a different list of wildcard patterns using the `""patterns""` configuration key. Here's how to render columns that end in either `_markdown` or `_md`:
```json
{
""plugins"": {
""datasette-render-markdown"": {
""patterns"": [""*_markdown"", ""*_md""]
}
}
}
```
To disable wildcard column matching entirely, set `""patterns"": []` in your plugin metadata configuration.
## Markdown extensions
The [Python-Markdown library](https://python-markdown.github.io/) that powers this plugin supports extensions, both [bundled](https://python-markdown.github.io/extensions/) and [third-party](https://github.com/Python-Markdown/markdown/wiki/Third-Party-Extensions). These can be used to enable additional Markdown features such as [table support](https://python-markdown.github.io/extensions/tables/).
You can configure support for extensions using the `""extensions""` key in your plugin metadata configuration.
Since extensions may introduce new HTML tags, you will also need to add those tags to the list of tags that are allowed by the [Bleach](https://bleach.readthedocs.io/) sanitizer. You can do that using the `""extra_tags""` key, and you can whitelist additional HTML attributes using `""extra_attrs""`. See [the Bleach documentation](https://bleach.readthedocs.io/en/latest/clean.html#allowed-tags-tags) for more information on this.
Here's how to enable support for [Markdown tables](https://python-markdown.github.io/extensions/tables/):
```json
{
""plugins"": {
""datasette-render-markdown"": {
""extensions"": [""tables""],
""extra_tags"": [""table"", ""thead"", ""tr"", ""th"", ""td"", ""tbody""]
}
}
}
```
### GitHub-Flavored Markdown
Enabling [GitHub-Flavored Markdown](https://help.github.com/en/github/writing-on-github) (useful for if you are working with data imported from GitHub using [github-to-sqlite](https://github.com/dogsheep/github-to-sqlite)) is a little more complicated.
First, you will need to install the [py-gfm](https://py-gfm.readthedocs.io) package:
$ pip install py-gfm
Note that `py-gfm` has [a bug](https://github.com/Zopieux/py-gfm/issues/13) that causes it to pin to `Markdown<3.0` - so if you are using it you should install it _before_ installing `datasette-render-markdown` to ensure you get a compatibly version of that dependency.
Now you can configure it like this. Note that the extension name is `mdx_gfm:GithubFlavoredMarkdownExtension` and you need to whitelist several extra HTML tags and attributes:
```json
{
""plugins"": {
""datasette-render-markdown"": {
""extra_tags"": [
""hr"",
""br"",
""details"",
""summary"",
""input""
],
""extra_attrs"": {
""input"": [
""type"",
""disabled"",
""checked""
],
},
""extensions"": [
""mdx_gfm:GithubFlavoredMarkdownExtension""
]
}
}
}
```
The `` attributes are needed to support rendering checkboxes in issue descriptions.
## Markdown in templates
The plugin also adds a new template function: `render_markdown(value)`. You can use this in your templates like so:
```html+jinja
{{ render_markdown(""""""
# This is markdown
* One
* Two
* Three
"""""") }}
```
You can load additional extensions and whitelist tags by passing extra arguments to the function like this:
```html+jinja
{{ render_markdown(""""""
## Markdown table
First Header | Second Header
------------- | -------------
Content Cell | Content Cell
Content Cell | Content Cell
"""""", extensions=[""tables""],
extra_tags=[""table"", ""thead"", ""tr"", ""th"", ""td"", ""tbody""])) }}
```
","
datasette-render-markdown
Datasette plugin for rendering Markdown.
Installation
Install this plugin in the same environment as Datasette to enable this new functionality:
$ pip install datasette-render-markdown
Usage
You can explicitly list the columns you would like to treat as Markdown using plugin configuration in a metadata.json file.
Add a ""datasette-render-markdown"" configuration block and use a ""columns"" key to list the columns you would like to treat as Markdown values:
This will cause any body column in any table to be treated as markdown and safely rendered using Python-Markdown. The resulting HTML is then run through Bleach to avoid the risk of XSS security problems.
Save this to metadata.json and run Datasette with the --metadata flag to load this configuration:
The configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the entries table in the news.db database:
This plugin can also render markdown in any columns that match a specific naming convention.
By default, columns that have a name ending in _markdown will be rendered.
You can try this out using the following query:
select'# Hello there* This is a list* of items[And a link](https://github.com/simonw/datasette-render-markdown).'as demo_markdown
You can configure a different list of wildcard patterns using the ""patterns"" configuration key. Here's how to render columns that end in either _markdown or _md:
You can configure support for extensions using the ""extensions"" key in your plugin metadata configuration.
Since extensions may introduce new HTML tags, you will also need to add those tags to the list of tags that are allowed by the Bleach sanitizer. You can do that using the ""extra_tags"" key, and you can whitelist additional HTML attributes using ""extra_attrs"". See the Bleach documentation for more information on this.
First, you will need to install the py-gfm package:
$ pip install py-gfm
Note that py-gfm has a bug that causes it to pin to Markdown<3.0 - so if you are using it you should install it before installing datasette-render-markdown to ensure you get a compatibly version of that dependency.
Now you can configure it like this. Note that the extension name is mdx_gfm:GithubFlavoredMarkdownExtension and you need to whitelist several extra HTML tags and attributes:
",1,public,0,,0,
234825790,MDEwOlJlcG9zaXRvcnkyMzQ4MjU3OTA=,datasette-upload-csvs,simonw/datasette-upload-csvs,0,9599,https://github.com/simonw/datasette-upload-csvs,Datasette plugin for uploading CSV files and converting them to database tables,0,2020-01-19T02:07:05Z,2022-07-03T20:58:20Z,2022-09-09T16:23:59Z,https://datasette.io/plugins/datasette-upload-csvs,58,9,9,Python,1,1,1,1,0,1,0,0,4,apache-2.0,"[""csvs"", ""datasette"", ""datasette-io"", ""datasette-plugin""]",1,4,9,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,1,2,"# datasette-upload-csvs
[](https://pypi.org/project/datasette-upload-csvs/)
[](https://github.com/simonw/datasette-upload-csvs/releases)
[](https://github.com/simonw/datasette-upload-csvs/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-upload-csvs/blob/main/LICENSE)
Datasette plugin for uploading CSV files and converting them to database tables
## Installation
datasette install datasette-upload-csvs
## Usage
The plugin adds an interface at `/-/upload-csvs` for uploading a CSV file and using it to create a new database table.
By default only [the root actor](https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor) can access the page - so you'll need to run Datasette with the `--root` option and click on the link shown in the terminal to sign in and access the page.
The `upload-csvs` permission governs access. You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to the write interface.
","
datasette-upload-csvs
Datasette plugin for uploading CSV files and converting them to database tables
Installation
datasette install datasette-upload-csvs
Usage
The plugin adds an interface at /-/upload-csvs for uploading a CSV file and using it to create a new database table.
By default only the root actor can access the page - so you'll need to run Datasette with the --root option and click on the link shown in the terminal to sign in and access the page.
The upload-csvs permission governs access. You can use permission plugins such as datasette-permissions-sql to grant additional access to the write interface.
",1,public,0,,0,
236110759,MDEwOlJlcG9zaXRvcnkyMzYxMTA3NTk=,datasette-auth-existing-cookies,simonw/datasette-auth-existing-cookies,0,9599,https://github.com/simonw/datasette-auth-existing-cookies,Datasette plugin that authenticates users based on existing domain cookies,0,2020-01-25T01:20:31Z,2022-05-28T01:50:15Z,2022-05-30T17:10:11Z,,54,3,3,Python,1,1,1,1,0,1,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin""]",1,0,3,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,1,3,"# datasette-auth-existing-cookies
[](https://pypi.org/project/datasette-auth-existing-cookies/)
[](https://github.com/simonw/datasette-auth-existing-cookies/releases)
[](https://github.com/simonw/datasette-auth-existing-cookies/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-auth-existing-cookies/blob/master/LICENSE)
Datasette plugin that authenticates users based on existing domain cookies.
## When to use this
This plugin allows you to build custom authentication for Datasette when you are hosting a Datasette instance on the same domain as another, authenticated website.
Consider a website on `www.example.com` which supports user authentication.
You could run Datasette on `data.example.com` in a way that lets it see cookies that were set for the `.example.com` domain.
Using this plugin, you could build an API endpoint at `www.example.com/user-for-cookies` which returns a JSON object representing the currently signed-in user, based on their cookies.
The plugin running on `data.example.com` will then make the `actor` available to the rest of Datasette based on the response from that API.
Read about [Datasette's authentication and permissions system](https://docs.datasette.io/en/stable/authentication.html) for more on how actors and permissions work.
## Configuration
This plugin requires some configuration in the Datasette [metadata.json file](https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration).
The following configuration options are supported:
- `api_url`: this is the API endpoint that Datasette should call with the user's cookies in order to identify the logged in user.
- `cookies`: optional. A list of cookie names that should be passed through to the API endpoint - if left blank, the default is to send all cookies.
- `ttl`: optional. By default Datasette will make a request to the API endpoint for every HTTP request recieved by Datasette itself. A `ttl` value of 5 will cause Datasette to cache the actor associated with the user's cookies for 5 seconds, reducing that API traffic.
- `headers`: an optional list of other headers to forward to the API endpoint as query string parameters.
Here is an example that uses all four of these settings:
```json
{
""plugins"": {
""datasette-auth-existing-cookies"": {
""api_url"": ""http://www.example.com/user-from-cookies"",
""cookies"": [""sessionid""],
""headers"": [""host""],
""ttl"": 10
}
}
}
```
With this configuration any hit to a Datasette hosted at `data.example.com` will result in the following request being made to the `http://www.example.com/user-from-cookies` API endpoint:
```
GET http://www.example.com/user-from-cookies?host=data.example.com
Cookie: sessionid=abc123
```
That API is expected to return a JSON object representing the current user:
```json
{
""id"": 1,
""name"": ""Barry""
}
```
Since `ttl` is set to 10 that actor will be cached for ten seconds against that exact combination of cookies and headers. When that cache expires another hit will be made to the API.
When deciding on a TTL value, take into account that users who lose access to the core site - maybe because their session expires, or their account is disabled - will still be able to access the Datasette instance until that cache expires.
","
datasette-auth-existing-cookies
Datasette plugin that authenticates users based on existing domain cookies.
When to use this
This plugin allows you to build custom authentication for Datasette when you are hosting a Datasette instance on the same domain as another, authenticated website.
Consider a website on www.example.com which supports user authentication.
You could run Datasette on data.example.com in a way that lets it see cookies that were set for the .example.com domain.
Using this plugin, you could build an API endpoint at www.example.com/user-for-cookies which returns a JSON object representing the currently signed-in user, based on their cookies.
The plugin running on data.example.com will then make the actor available to the rest of Datasette based on the response from that API.
This plugin requires some configuration in the Datasette metadata.json file.
The following configuration options are supported:
api_url: this is the API endpoint that Datasette should call with the user's cookies in order to identify the logged in user.
cookies: optional. A list of cookie names that should be passed through to the API endpoint - if left blank, the default is to send all cookies.
ttl: optional. By default Datasette will make a request to the API endpoint for every HTTP request recieved by Datasette itself. A ttl value of 5 will cause Datasette to cache the actor associated with the user's cookies for 5 seconds, reducing that API traffic.
headers: an optional list of other headers to forward to the API endpoint as query string parameters.
Here is an example that uses all four of these settings:
With this configuration any hit to a Datasette hosted at data.example.com will result in the following request being made to the http://www.example.com/user-from-cookies API endpoint:
GET http://www.example.com/user-from-cookies?host=data.example.com
Cookie: sessionid=abc123
That API is expected to return a JSON object representing the current user:
{
""id"": 1,
""name"": ""Barry""
}
Since ttl is set to 10 that actor will be cached for ten seconds against that exact combination of cookies and headers. When that cache expires another hit will be made to the API.
When deciding on a TTL value, take into account that users who lose access to the core site - maybe because their session expires, or their account is disabled - will still be able to access the Datasette instance until that cache expires.
",1,public,0,,,
236867027,MDEwOlJlcG9zaXRvcnkyMzY4NjcwMjc=,datasette-sentry,simonw/datasette-sentry,0,9599,https://github.com/simonw/datasette-sentry,Datasette plugin for configuring Sentry,0,2020-01-28T23:41:27Z,2022-07-18T20:28:25Z,2022-10-06T22:31:29Z,,26,6,6,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""sentry""]",0,0,6,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-sentry
[](https://pypi.org/project/datasette-sentry/)
[](https://github.com/simonw/datasette-sentry/releases)
[](https://github.com/simonw/datasette-sentry/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-sentry/blob/main/LICENSE)
Datasette plugin for configuring Sentry for error reporting
## Installation
pip install datasette-sentry
## Usage
This plugin only takes effect if your `metadata.json` file contains relevant top-level plugin configuration in a `""datasette-sentry""` configuration key.
You will need a Sentry DSN - see their [Getting Started instructions](https://docs.sentry.io/error-reporting/quickstart/?platform=python).
Add it to `metadata.json` like this:
```json
{
""plugins"": {
""datasette-sentry"": {
""dsn"": ""https://KEY@sentry.io/PROJECTID""
}
}
}
```
Settings in `metadata.json` are visible to anyone who visits the `/-/metadata` URL so this is a good place to take advantage of Datasette's [secret configuration values](https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values), in which case your configuration will look more like this:
```json
{
""plugins"": {
""datasette-sentry"": {
""dsn"": {
""$env"": ""SENTRY_DSN""
}
}
}
}
```
Then make a `SENTRY_DSN` environment variable available to Datasette.
## Configuration
In addition to the `dsn` setting, you can also configure the Sentry [sample rate](https://docs.sentry.io/platforms/python/configuration/sampling/) by setting `sample_rate` to a floating point number between 0 and 1.
For example, to capture 25% of errors you would do this:
```json
{
""plugins"": {
""datasette-sentry"": {
""dsn"": {
""$env"": ""SENTRY_DSN""
},
""sample_rate"": 0.25
}
}
}
```
","
datasette-sentry
Datasette plugin for configuring Sentry for error reporting
Installation
pip install datasette-sentry
Usage
This plugin only takes effect if your metadata.json file contains relevant top-level plugin configuration in a ""datasette-sentry"" configuration key.
Settings in metadata.json are visible to anyone who visits the /-/metadata URL so this is a good place to take advantage of Datasette's secret configuration values, in which case your configuration will look more like this:
",1,public,0,,0,
243710733,MDEwOlJlcG9zaXRvcnkyNDM3MTA3MzM=,datasette-ics,simonw/datasette-ics,0,9599,https://github.com/simonw/datasette-ics,Datasette plugin for outputting iCalendar files,0,2020-02-28T08:11:01Z,2022-07-07T14:11:49Z,2022-07-12T02:08:10Z,https://datasette.io/plugins/datasette-ics,34,13,13,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""icalendar"", ""ics""]",0,0,13,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-ics
[](https://pypi.org/project/datasette-ics/)
[](https://github.com/simonw/datasette-ics/releases)
[](https://github.com/simonw/datasette-ics/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-ics/blob/main/LICENSE)
Datasette plugin that adds support for generating [iCalendar .ics files](https://tools.ietf.org/html/rfc5545) with the results of a SQL query.
## Installation
Install this plugin in the same environment as Datasette to enable the `.ics` output extension.
$ pip install datasette-ics
## Usage
To create an iCalendar file you need to define a custom SQL query that returns a required set of columns:
* `event_name` - the short name for the event
* `event_dtstart` - when the event starts
The following columns are optional:
* `event_dtend` - when the event ends
* `event_duration` - the duration of the event (use instead of `dtend`)
* `event_description` - a longer description of the event
* `event_uid` - a globally unique identifier for this event
* `event_tzid` - the timezone for the event, e.g. `America/Chicago`
A query that returns these columns can then be returned as an ics feed by adding the `.ics` extension.
## Demo
[This SQL query]([https://www.rockybeaches.com/data?sql=with+inner+as+(%0D%0A++select%0D%0A++++datetime%2C%0D%0A++++substr(datetime%2C+0%2C+11)+as+date%2C%0D%0A++++mllw_feet%2C%0D%0A++++lag(mllw_feet)+over+win+as+previous_mllw_feet%2C%0D%0A++++lead(mllw_feet)+over+win+as+next_mllw_feet%0D%0A++from%0D%0A++++tide_predictions%0D%0A++where%0D%0A++++station_id+%3D+%3Astation_id%0D%0A++++and+datetime+%3E%3D+date()%0D%0A++++window+win+as+(%0D%0A++++++order+by%0D%0A++++++++datetime%0D%0A++++)%0D%0A++order+by%0D%0A++++datetime%0D%0A)%2C%0D%0Alowest_tide_per_day+as+(%0D%0A++select%0D%0A++++date%2C%0D%0A++++datetime%2C%0D%0A++++mllw_feet%0D%0A++from%0D%0A++++inner%0D%0A++where%0D%0A++++mllw_feet+%3C%3D+previous_mllw_feet%0D%0A++++and+mllw_feet+%3C%3D+next_mllw_feet%0D%0A)%0D%0Aselect%0D%0A++min(datetime)+as+event_dtstart%2C%0D%0A++%27Low+tide%3A+%27+||+mllw_feet+||+%27+feet%27+as+event_name%2C%0D%0A++%27America%2FLos_Angeles%27+as+event_tzid%0D%0Afrom%0D%0A++lowest_tide_per_day%0D%0Agroup+by%0D%0A++date%0D%0Aorder+by%0D%0A++date&station_id=9414131) calculates the lowest tide per day at Pillar Point in Half Moon Bay, California.
Since the query returns `event_name`, `event_dtstart` and `event_tzid` columns it produces [this ICS feed](https://www.rockybeaches.com/data.ics?sql=with+inner+as+(%0D%0A++select%0D%0A++++datetime%2C%0D%0A++++substr(datetime%2C+0%2C+11)+as+date%2C%0D%0A++++mllw_feet%2C%0D%0A++++lag(mllw_feet)+over+win+as+previous_mllw_feet%2C%0D%0A++++lead(mllw_feet)+over+win+as+next_mllw_feet%0D%0A++from%0D%0A++++tide_predictions%0D%0A++where%0D%0A++++station_id+%3D+%3Astation_id%0D%0A++++and+datetime+%3E%3D+date()%0D%0A++++window+win+as+(%0D%0A++++++order+by%0D%0A++++++++datetime%0D%0A++++)%0D%0A++order+by%0D%0A++++datetime%0D%0A)%2C%0D%0Alowest_tide_per_day+as+(%0D%0A++select%0D%0A++++date%2C%0D%0A++++datetime%2C%0D%0A++++mllw_feet%0D%0A++from%0D%0A++++inner%0D%0A++where%0D%0A++++mllw_feet+%3C%3D+previous_mllw_feet%0D%0A++++and+mllw_feet+%3C%3D+next_mllw_feet%0D%0A)%0D%0Aselect%0D%0A++min(datetime)+as+event_dtstart%2C%0D%0A++%27Low+tide%3A+%27+||+mllw_feet+||+%27+feet%27+as+event_name%2C%0D%0A++%27America%2FLos_Angeles%27+as+event_tzid%0D%0Afrom%0D%0A++lowest_tide_per_day%0D%0Agroup+by%0D%0A++date%0D%0Aorder+by%0D%0A++date&station_id=9414131). If you subscribe to that in a calendar application such as Apple Calendar you get something that looks like this:
## Using a canned query
Datasette's [canned query mechanism](https://datasette.readthedocs.io/en/stable/sql_queries.html#canned-queries) can be used to configure calendars. If a canned query definition has a `title` that will be used as the title of the calendar.
Here's an example, defined using a `metadata.yaml` file:
```yaml
databases:
mydatabase:
queries:
calendar:
title: My Calendar
sql: |-
select
title as event_name,
start as event_dtstart,
description as event_description
from
events
order by
start
limit
100
```
This will result in a calendar feed at `http://localhost:8001/mydatabase/calendar.ics`
","
datasette-ics
Datasette plugin that adds support for generating iCalendar .ics files with the results of a SQL query.
Installation
Install this plugin in the same environment as Datasette to enable the .ics output extension.
$ pip install datasette-ics
Usage
To create an iCalendar file you need to define a custom SQL query that returns a required set of columns:
event_name - the short name for the event
event_dtstart - when the event starts
The following columns are optional:
event_dtend - when the event ends
event_duration - the duration of the event (use instead of dtend)
event_description - a longer description of the event
event_uid - a globally unique identifier for this event
event_tzid - the timezone for the event, e.g. America/Chicago
A query that returns these columns can then be returned as an ics feed by adding the .ics extension.
Demo
This SQL query calculates the lowest tide per day at Pillar Point in Half Moon Bay, California.
Since the query returns event_name, event_dtstart and event_tzid columns it produces this ICS feed. If you subscribe to that in a calendar application such as Apple Calendar you get something that looks like this:
Using a canned query
Datasette's canned query mechanism can be used to configure calendars. If a canned query definition has a title that will be used as the title of the calendar.
Here's an example, defined using a metadata.yaml file:
databases:
mydatabase:
queries:
calendar:
title: My Calendarsql: |- select title as event_name, start as event_dtstart, description as event_description from events order by start limit 100
This will result in a calendar feed at http://localhost:8001/mydatabase/calendar.ics
",1,public,0,,0,
245856731,MDEwOlJlcG9zaXRvcnkyNDU4NTY3MzE=,datasette-search-all,simonw/datasette-search-all,0,9599,https://github.com/simonw/datasette-search-all,Datasette plugin for searching all searchable tables at once,0,2020-03-08T17:21:54Z,2021-12-19T04:06:49Z,2022-10-05T01:53:33Z,,186,6,6,Python,1,1,1,1,0,2,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""search""]",2,0,6,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,2,2,"# datasette-search-all
[](https://pypi.org/project/datasette-search-all/)
[](https://github.com/simonw/datasette-search-all/releases)
[](https://github.com/simonw/datasette-search-all/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-search-all/blob/main/LICENSE)
Datasette plugin for searching all searchable tables at once.
## Installation
Install the plugin in the same Python environment as Datasette:
pip install datasette-search-all
## Background
See [datasette-search-all: a new plugin for searching multiple Datasette tables at once](https://simonwillison.net/2020/Mar/9/datasette-search-all/) for background on this project. You can try the plugin out at https://fara.datasettes.com/
## Usage
This plugin only works if at least one of the tables connected to your Datasette instance has been configured for SQLite's full-text search.
The [Datasette search documentation](https://docs.datasette.io/en/stable/full_text_search.html) includes details on how to enable full-text search for a table.
You can also use the following tools:
* [sqlite-utils](https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search) includes a command-line tool for enabling full-text search.
* [datasette-enable-fts](https://github.com/simonw/datasette-enable-fts) is a Datasette plugin that adds a web interface for enabling search for specific columns.
If the plugin detects at least one searchable table it will add a search form to the homepage.
You can also navigate to `/-/search` on your Datasette instance to use the search interface directly.
## Screenshot
","
datasette-search-all
Datasette plugin for searching all searchable tables at once.
Installation
Install the plugin in the same Python environment as Datasette:
sqlite-utils includes a command-line tool for enabling full-text search.
datasette-enable-fts is a Datasette plugin that adds a web interface for enabling search for specific columns.
If the plugin detects at least one searchable table it will add a search form to the homepage.
You can also navigate to /-/search on your Datasette instance to use the search interface directly.
Screenshot
",1,public,0,,0,
247527438,MDEwOlJlcG9zaXRvcnkyNDc1Mjc0Mzg=,datasette-edit-schema,simonw/datasette-edit-schema,0,9599,https://github.com/simonw/datasette-edit-schema,Datasette plugin for modifying table schemas,0,2020-03-15T18:34:06Z,2022-07-01T22:20:25Z,2022-08-22T22:45:58Z,,133,6,6,JavaScript,1,1,1,1,0,0,0,0,10,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin""]",0,10,6,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-edit-schema
[](https://pypi.org/project/datasette-edit-schema/)
[](https://github.com/simonw/datasette-edit-schema/releases)
[](https://github.com/simonw/datasette-edit-schema/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-edit-schema/blob/master/LICENSE)
Datasette plugin for modifying table schemas
## Features
* Add new columns to a table
* Rename columns in a table
* Modify the type of columns in a table
* Re-order the columns in a table
* Rename a table
* Delete a table
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-edit-schema
## Usage
Navigate to `/-/edit-schema/dbname/tablename` on your Datasette instance to edit a specific table.
Use `/-/edit-schema/dbname` to create a new table in a specific database.
By default only [the root actor](https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor) can access the page - so you'll need to run Datasette with the `--root` option and click on the link shown in the terminal to sign in and access the page.
## Permissions
The `edit-schema` permission governs access. You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to the write interface.
These permission checks will call the `permission_allowed()` plugin hook with three arguments:
- `action` will be the string `""edit-schema""`
- `actor` will be the currently authenticated actor - usually a dictionary
- `resource` will be the string name of the database
## Screenshot
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-edit-schema
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-edit-schema
Datasette plugin for modifying table schemas
Features
Add new columns to a table
Rename columns in a table
Modify the type of columns in a table
Re-order the columns in a table
Rename a table
Delete a table
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-edit-schema
Usage
Navigate to /-/edit-schema/dbname/tablename on your Datasette instance to edit a specific table.
Use /-/edit-schema/dbname to create a new table in a specific database.
By default only the root actor can access the page - so you'll need to run Datasette with the --root option and click on the link shown in the terminal to sign in and access the page.
Permissions
The edit-schema permission governs access. You can use permission plugins such as datasette-permissions-sql to grant additional access to the write interface.
These permission checks will call the permission_allowed() plugin hook with three arguments:
action will be the string ""edit-schema""
actor will be the currently authenticated actor - usually a dictionary
resource will be the string name of the database
Screenshot
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-edit-schema
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,0,
248385299,MDEwOlJlcG9zaXRvcnkyNDgzODUyOTk=,datasette-publish-fly,simonw/datasette-publish-fly,0,9599,https://github.com/simonw/datasette-publish-fly,Datasette plugin for publishing data using Fly,0,2020-03-19T01:47:01Z,2022-09-29T22:28:45Z,2022-09-29T17:25:15Z,,50,10,10,Python,1,1,1,1,0,3,0,0,4,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""fly""]",3,4,10,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,3,3,"# datasette-publish-fly
[](https://pypi.org/project/datasette-publish-fly/)
[](https://github.com/simonw/datasette-publish-fly/releases)
[](https://github.com/simonw/datasette-publish-fly/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-publish-fly/blob/main/LICENSE)
[Datasette](https://datasette.io/) plugin for deploying Datasette instances to [Fly.io](https://fly.io/).
Project background: [Using SQLite and Datasette with Fly Volumes](https://simonwillison.net/2022/Feb/15/fly-volumes/)
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-publish-fly
## Deploying read-only data
First, install the `flyctl` command-line tool by [following their instructions](https://fly.io/docs/getting-started/installing-flyctl/).
Run `flyctl auth signup` to create an account there, or `flyctl auth login` if you already have one.
You can now use `datasette publish fly` to publish one or more SQLite database files:
datasette publish fly my-database.db --app=""my-data-app""
The argument you pass to `--app` will be used for the URL of your application: `my-data-app.fly.dev`.
To update an application, run the publish command passing the same application name to the `--app` option.
Fly have [a free tier](https://fly.io/docs/about/pricing/#free-allowances), beyond which they will charge you monthly for each application you have live. Details of their pricing can be [found on their site](https://fly.io/docs/pricing/).
Your application will be deployed at `https://your-app-name.fly.io/` - be aware that it may take several minutes to start working the first time you deploy it.
## Using Fly volumes for writable databases
Fly [Volumes](https://fly.io/docs/reference/volumes/) provide persistant disk storage for Fly applications. Volumes can be 1GB or more in size and the Fly free tier includes 3GB of volume space.
Datasette plugins such as [datasette-uploads-csvs](https://datasette.io/plugins/datasette-upload-csvs) and [datasette-tiddlywiki](https://datasette.io/plugins/datasette-tiddlywiki) can be deployed to Fly and store their mutable data in a volume.
> :warning: **You should only run a single instance of your application** if your database accepts writes. Fly has excellent support for running multiple instances in different geographical regions, but `datasette-publish-fly` with volumes is not yet compatible with that model. You should probably [use Fly PostgreSQL instead](https://fly.io/blog/globally-distributed-postgres/).
Here's how to deploy `datasette-tiddlywiki` with authentication provided by `datasette-auth-passwords`.
First, you'll need to create a root password hash to use to sign into the instance.
You can do that by installing the plugin and running the `datasette hash-password` command, or by using [this hosted tool](https://datasette-auth-passwords-demo.datasette.io/-/password-tool).
The hash should look like `pbkdf2_sha256$...` - you'll need this for the next step.
In this example we're also deploying a read-only database called `content.db`.
Pick a name for your new application, then run the following:
datasette publish fly \
content.db \
--app your-application-name \
--create-volume 1 \
--create-db tiddlywiki \
--install datasette-auth-passwords \
--install datasette-tiddlywiki \
--plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$...'
This will create the new application, deploy the `content.db` read-only database, create a 1GB volume for that application, create a new database in that volume called `tiddlywiki.db`, then install the two plugins and configure the password you specified.
### Updating applications that use a volume
Once you have deployed an application using a volume, you can update that application without needing the `--create-volume` or `--create-db` options. To add the [datasette-graphq](https://datasette.io/plugins/datasette-graphql) plugin to your deployed application you would run the following:
datasette publish fly \
content.db \
--app your-application-name \
--install datasette-auth-passwords \
--install datasette-tiddlywiki \
--install datasette-graphql \
--plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$...' \
Since the application name is the same you don't need the `--create-volume` or `--create-db` options - these are persisted automatically between deploys.
You do need to specify the full list of plugins that you want to have installed, and any plugin secrets.
You also need to include any read-only database files that are part of the instance - `content.db` in this example - otherwise the new deployment will not include them.
### Advanced volume usage
`datasette publish fly` will add a volume called `datasette` to your Fly application. You can customize the name using the `--volume name custom_name` option.
Fly can be used to scale applications to run multiple instances in multiple regions around the world. This works well with read-only Datasette but is not currently recommended using Datasette with volumes, since each Fly replica would need its own volume and data stored in one instance would not be visible in others.
If you want to use multiple instances with volumes you will need to switch to using the `flyctl` command directly. The `--generate-dir` option, described below, can help with this.
## Generating without deploying
Use the `--generate-dir` option to generate a directory that can be deployed to Fly rather than deploying directly:
datasette publish fly my-database.db \
--app=""my-generated-app"" \
--generate-dir /tmp/deploy-this
You can then manually deploy your generated application using the following:
cd /tmp/deploy-this
flyctl apps create my-generated-app
flyctl deploy
## datasette publish fly --help
```
Usage: datasette publish fly [OPTIONS] [FILES]...
Deploy an application to Fly that runs Datasette against the provided database
files.
Usage example:
datasette publish fly my-database.db --app=""my-data-app""
Full documentation: https://datasette.io/plugins/datasette-publish-fly
Options:
-m, --metadata FILENAME Path to JSON/YAML file containing metadata to
publish
--extra-options TEXT Extra options to pass to datasette serve
--branch TEXT Install datasette from a GitHub branch e.g.
main
--template-dir DIRECTORY Path to directory containing custom templates
--plugins-dir DIRECTORY Path to directory containing custom plugins
--static MOUNT:DIRECTORY Serve static files from this directory at
/MOUNT/...
--install TEXT Additional packages (e.g. plugins) to install
--plugin-secret ...
Secrets to pass to plugins, e.g. --plugin-
secret datasette-auth-github client_id xxx
--version-note TEXT Additional note to show on /-/versions
--secret TEXT Secret used for signing secure values, such as
signed cookies
--title TEXT Title for metadata
--license TEXT License label for metadata
--license_url TEXT License URL for metadata
--source TEXT Source label for metadata
--source_url TEXT Source URL for metadata
--about TEXT About label for metadata
--about_url TEXT About URL for metadata
--spatialite Enable SpatialLite extension
--region TEXT Fly region to deploy to, e.g sjc - see
https://fly.io/docs/reference/regions/
--create-volume INTEGER RANGE Create and attach volume of this size in GB
[x>=1]
--create-db TEXT Names of read-write database files to create
--volume-name TEXT Volume name to use
-a, --app TEXT Name of Fly app to deploy [required]
-o, --org TEXT Name of Fly org to deploy to
--generate-dir DIRECTORY Output generated application files and stop
without deploying
--show-files Output the generated Dockerfile, metadata.json
and fly.toml
--help Show this message and exit.
```
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd datasette-publish-fly
python -m venv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
### Integration tests
The tests in `tests/test_integration.py` make actual calls to Fly to deploy a test application.
These tests are skipped by default. If you have `flyctl` installed and configured, you can run the integration tests like this:
pytest --integration -s
The `-s` option here ensures that output from the deploys will be visible to you - otherwise it can look like the tests have hung.
The tests will create applications on Fly that start with the prefix `publish-fly-temp-` and then delete them at the end of the run.
","
datasette-publish-fly
Datasette plugin for deploying Datasette instances to Fly.io.
The argument you pass to --app will be used for the URL of your application: my-data-app.fly.dev.
To update an application, run the publish command passing the same application name to the --app option.
Fly have a free tier, beyond which they will charge you monthly for each application you have live. Details of their pricing can be found on their site.
Your application will be deployed at https://your-app-name.fly.io/ - be aware that it may take several minutes to start working the first time you deploy it.
Using Fly volumes for writable databases
Fly Volumes provide persistant disk storage for Fly applications. Volumes can be 1GB or more in size and the Fly free tier includes 3GB of volume space.
⚠️You should only run a single instance of your application if your database accepts writes. Fly has excellent support for running multiple instances in different geographical regions, but datasette-publish-fly with volumes is not yet compatible with that model. You should probably use Fly PostgreSQL instead.
Here's how to deploy datasette-tiddlywiki with authentication provided by datasette-auth-passwords.
First, you'll need to create a root password hash to use to sign into the instance.
You can do that by installing the plugin and running the datasette hash-password command, or by using this hosted tool.
The hash should look like pbkdf2_sha256$... - you'll need this for the next step.
In this example we're also deploying a read-only database called content.db.
Pick a name for your new application, then run the following:
This will create the new application, deploy the content.db read-only database, create a 1GB volume for that application, create a new database in that volume called tiddlywiki.db, then install the two plugins and configure the password you specified.
Updating applications that use a volume
Once you have deployed an application using a volume, you can update that application without needing the --create-volume or --create-db options. To add the datasette-graphq plugin to your deployed application you would run the following:
Since the application name is the same you don't need the --create-volume or --create-db options - these are persisted automatically between deploys.
You do need to specify the full list of plugins that you want to have installed, and any plugin secrets.
You also need to include any read-only database files that are part of the instance - content.db in this example - otherwise the new deployment will not include them.
Advanced volume usage
datasette publish fly will add a volume called datasette to your Fly application. You can customize the name using the --volume name custom_name option.
Fly can be used to scale applications to run multiple instances in multiple regions around the world. This works well with read-only Datasette but is not currently recommended using Datasette with volumes, since each Fly replica would need its own volume and data stored in one instance would not be visible in others.
If you want to use multiple instances with volumes you will need to switch to using the flyctl command directly. The --generate-dir option, described below, can help with this.
Generating without deploying
Use the --generate-dir option to generate a directory that can be deployed to Fly rather than deploying directly:
You can then manually deploy your generated application using the following:
cd /tmp/deploy-this
flyctl apps create my-generated-app
flyctl deploy
datasette publish fly --help
Usage: datasette publish fly [OPTIONS] [FILES]...
Deploy an application to Fly that runs Datasette against the provided database
files.
Usage example:
datasette publish fly my-database.db --app=""my-data-app""
Full documentation: https://datasette.io/plugins/datasette-publish-fly
Options:
-m, --metadata FILENAME Path to JSON/YAML file containing metadata to
publish
--extra-options TEXT Extra options to pass to datasette serve
--branch TEXT Install datasette from a GitHub branch e.g.
main
--template-dir DIRECTORY Path to directory containing custom templates
--plugins-dir DIRECTORY Path to directory containing custom plugins
--static MOUNT:DIRECTORY Serve static files from this directory at
/MOUNT/...
--install TEXT Additional packages (e.g. plugins) to install
--plugin-secret <TEXT TEXT TEXT>...
Secrets to pass to plugins, e.g. --plugin-
secret datasette-auth-github client_id xxx
--version-note TEXT Additional note to show on /-/versions
--secret TEXT Secret used for signing secure values, such as
signed cookies
--title TEXT Title for metadata
--license TEXT License label for metadata
--license_url TEXT License URL for metadata
--source TEXT Source label for metadata
--source_url TEXT Source URL for metadata
--about TEXT About label for metadata
--about_url TEXT About URL for metadata
--spatialite Enable SpatialLite extension
--region TEXT Fly region to deploy to, e.g sjc - see
https://fly.io/docs/reference/regions/
--create-volume INTEGER RANGE Create and attach volume of this size in GB
[x>=1]
--create-db TEXT Names of read-write database files to create
--volume-name TEXT Volume name to use
-a, --app TEXT Name of Fly app to deploy [required]
-o, --org TEXT Name of Fly org to deploy to
--generate-dir DIRECTORY Output generated application files and stop
without deploying
--show-files Output the generated Dockerfile, metadata.json
and fly.toml
--help Show this message and exit.
Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd datasette-publish-fly
python -m venv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
Integration tests
The tests in tests/test_integration.py make actual calls to Fly to deploy a test application.
These tests are skipped by default. If you have flyctl installed and configured, you can run the integration tests like this:
pytest --integration -s
The -s option here ensures that output from the deploys will be visible to you - otherwise it can look like the tests have hung.
The tests will create applications on Fly that start with the prefix publish-fly-temp- and then delete them at the end of the run.
",1,public,0,,0,
253632948,MDEwOlJlcG9zaXRvcnkyNTM2MzI5NDg=,datasette-publish-vercel,simonw/datasette-publish-vercel,0,9599,https://github.com/simonw/datasette-publish-vercel,Datasette plugin for publishing data using Vercel,0,2020-04-06T22:47:13Z,2022-07-29T17:09:47Z,2022-08-24T17:43:41Z,,55,27,27,Python,1,1,1,1,0,5,0,0,17,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""vercel"", ""zeit-now""]",5,17,27,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,5,4,"# datasette-publish-vercel
[](https://pypi.org/project/datasette-publish-vercel/)
[](https://github.com/simonw/datasette-publish-vercel/releases)
[](https://github.com/simonw/datasette-publish-vercel/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-publish-vercel/blob/main/LICENSE)
Datasette plugin for publishing data using [Vercel](https://vercel.com/).
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-publish-vercel
## Usage
First, install the Vercel CLI tool by [following their instructions](https://vercel.com/download).
Run `vercel login` to login to (or create) an account.
Now you can use `datasette publish vercel` to publish your data:
datasette publish vercel my-database.db --project=my-database
The `--project` argument is required - it specifies the project name that should be used for your deployment. This will be used as part of the deployment's URL.
### Other options
* `--no-prod` deploys to the project without updating the ""production"" URL alias to point to that new deployment. Without that option all deploys go directly to production.
* `--debug` enables the Vercel CLI debug output.
* `--token` allows you to pass a Now authentication token, rather than needing to first run `now login` to configure the tool. Tokens can be created in the Vercel web dashboard under Account Settings -> Tokens.
* `--public` runs `vercel --public` to publish the application source code at `/_src` e.g. https://datasette-public.now.sh/_src and make recent logs visible at `/_logs` e.g. https://datasette-public.now.sh/_logs
* `--generate-dir` - by default this tool generates a new Vercel app in a temporary directory, deploys it and then deletes the directory. Use `--generate-dir=my-app` to output the generated application files to a new directory of your choice instead. You can then deploy it by running `vercel` in that directory.
* `--setting default_page_size 10` - use this to set Datasette settings, as described in [the documentation](https://docs.datasette.io/en/stable/settings.html). This is a replacement for the unsupported `--extra-options` option.
### Full help
**Warning:** Some of these options are not yet implemented by this plugin. In particular, the following do not yet work:
* `--extra-options` - use `--setting` described above instead.
* `--plugin-secret`
* `--version-note`
```
$ datasette publish vercel --help
Usage: datasette publish vercel [OPTIONS] [FILES]...
Publish to https://vercel.com/
Options:
-m, --metadata FILENAME Path to JSON/YAML file containing metadata to publish
--extra-options TEXT Extra options to pass to datasette serve
--branch TEXT Install datasette from a GitHub branch e.g. main
--template-dir DIRECTORY Path to directory containing custom templates
--plugins-dir DIRECTORY Path to directory containing custom plugins
--static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/...
--install TEXT Additional packages (e.g. plugins) to install
--plugin-secret ...
Secrets to pass to plugins, e.g. --plugin-secret
datasette-auth-github client_id xxx
--version-note TEXT Additional note to show on /-/versions
--secret TEXT Secret used for signing secure values, such as signed
cookies
--title TEXT Title for metadata
--license TEXT License label for metadata
--license_url TEXT License URL for metadata
--source TEXT Source label for metadata
--source_url TEXT Source URL for metadata
--about TEXT About label for metadata
--about_url TEXT About URL for metadata
--token TEXT Auth token to use for deploy
--project PROJECT Vercel project name to use [required]
--scope TEXT Optional Vercel scope (e.g. a team name)
--no-prod Don't deploy directly to production
--debug Enable Vercel CLI debug output
--public Publish source with Vercel CLI --public
--generate-dir DIRECTORY Output generated application files and stop without
deploying
--generate-vercel-json Output generated vercel.json file and stop without
deploying
--vercel-json FILENAME Custom vercel.json file to use instead of generating
one
--setting SETTING... Setting, see docs.datasette.io/en/stable/settings.html
--crossdb Enable cross-database SQL queries
--help Show this message and exit.
```
## Using a custom `vercel.json` file
If you want to add additional redirects or similar to your Vercel configuration you may want to provide a custom `vercel.json` file.
To do this, first generate a configuration file (without running a deploy) using the `--generate-vercel-json` option:
datasette publish vercel my-database.db \
--project=my-database \
--generate-vercel-json > vercel.json
You can now edit the `vercel.json` file that this creates to add your custom options.
Then run the deploy using:
datasette publish vercel my-database.db \
--project=my-database \
--vercel-json=vercel.json
## Setting a `DATASETTE_SECRET`
Datasette uses [a secret string](https://docs.datasette.io/en/stable/settings.html#configuring-the-secret) for purposes such as signing authentication cookies. This secret is reset when the server restarts, which will sign out any users who are authenticated using a signed cookie.
You can avoid this by generating a `DATASETTE_SECRET` secret string and setting that as a [Vercel environment variable](https://vercel.com/docs/concepts/projects/environment-variables). If you do this the secret will stay consistent and your users will not be signed out.
## Using this with GitHub Actions
This plugin can be used together with [GitHub Actions](https://github.com/features/actions) to deploy Datasette instances automatically on new pushes to a repo, or on a schedule.
The GitHub Actions runners already have the Vercel deployment tool installed. You'll need to create an API token for your account at [vercel.com/account/tokens](https://vercel.com/account/tokens), and store that as a secret in your GitHub repository called `VERCEL_TOKEN`.
Make sure your workflow has installed `datasette` and `datasette-publish-vercel` using `pip`, then add the following step to your GitHub Actions workflow:
```
- name: Deploy Datasette using Vercel
env:
VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
run: |-
datasette publish vercel mydb.db \
--token $VERCEL_TOKEN \
--project my-vercel-project
```
You can see a full example of a workflow that uses Vercel in this way [in the simonw/til repository](https://github.com/simonw/til/blob/12b3f0d3679320cbeafa5df164bbc08ba703625d/.github/workflows/build.yml).
","
datasette-publish-vercel
Datasette plugin for publishing data using Vercel.
Installation
Install this plugin in the same environment as Datasette.
The --project argument is required - it specifies the project name that should be used for your deployment. This will be used as part of the deployment's URL.
Other options
--no-prod deploys to the project without updating the ""production"" URL alias to point to that new deployment. Without that option all deploys go directly to production.
--debug enables the Vercel CLI debug output.
--token allows you to pass a Now authentication token, rather than needing to first run now login to configure the tool. Tokens can be created in the Vercel web dashboard under Account Settings -> Tokens.
--generate-dir - by default this tool generates a new Vercel app in a temporary directory, deploys it and then deletes the directory. Use --generate-dir=my-app to output the generated application files to a new directory of your choice instead. You can then deploy it by running vercel in that directory.
--setting default_page_size 10 - use this to set Datasette settings, as described in the documentation. This is a replacement for the unsupported --extra-options option.
Full help
Warning: Some of these options are not yet implemented by this plugin. In particular, the following do not yet work:
--extra-options - use --setting described above instead.
--plugin-secret
--version-note
$ datasette publish vercel --help
Usage: datasette publish vercel [OPTIONS] [FILES]...
Publish to https://vercel.com/
Options:
-m, --metadata FILENAME Path to JSON/YAML file containing metadata to publish
--extra-options TEXT Extra options to pass to datasette serve
--branch TEXT Install datasette from a GitHub branch e.g. main
--template-dir DIRECTORY Path to directory containing custom templates
--plugins-dir DIRECTORY Path to directory containing custom plugins
--static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/...
--install TEXT Additional packages (e.g. plugins) to install
--plugin-secret <TEXT TEXT TEXT>...
Secrets to pass to plugins, e.g. --plugin-secret
datasette-auth-github client_id xxx
--version-note TEXT Additional note to show on /-/versions
--secret TEXT Secret used for signing secure values, such as signed
cookies
--title TEXT Title for metadata
--license TEXT License label for metadata
--license_url TEXT License URL for metadata
--source TEXT Source label for metadata
--source_url TEXT Source URL for metadata
--about TEXT About label for metadata
--about_url TEXT About URL for metadata
--token TEXT Auth token to use for deploy
--project PROJECT Vercel project name to use [required]
--scope TEXT Optional Vercel scope (e.g. a team name)
--no-prod Don't deploy directly to production
--debug Enable Vercel CLI debug output
--public Publish source with Vercel CLI --public
--generate-dir DIRECTORY Output generated application files and stop without
deploying
--generate-vercel-json Output generated vercel.json file and stop without
deploying
--vercel-json FILENAME Custom vercel.json file to use instead of generating
one
--setting SETTING... Setting, see docs.datasette.io/en/stable/settings.html
--crossdb Enable cross-database SQL queries
--help Show this message and exit.
Using a custom vercel.json file
If you want to add additional redirects or similar to your Vercel configuration you may want to provide a custom vercel.json file.
To do this, first generate a configuration file (without running a deploy) using the --generate-vercel-json option:
Datasette uses a secret string for purposes such as signing authentication cookies. This secret is reset when the server restarts, which will sign out any users who are authenticated using a signed cookie.
You can avoid this by generating a DATASETTE_SECRET secret string and setting that as a Vercel environment variable. If you do this the secret will stay consistent and your users will not be signed out.
Using this with GitHub Actions
This plugin can be used together with GitHub Actions to deploy Datasette instances automatically on new pushes to a repo, or on a schedule.
The GitHub Actions runners already have the Vercel deployment tool installed. You'll need to create an API token for your account at vercel.com/account/tokens, and store that as a secret in your GitHub repository called VERCEL_TOKEN.
Make sure your workflow has installed datasette and datasette-publish-vercel using pip, then add the following step to your GitHub Actions workflow:
",1,public,0,,0,
256834907,MDEwOlJlcG9zaXRvcnkyNTY4MzQ5MDc=,dogsheep-photos,dogsheep/dogsheep-photos,0,53015001,https://github.com/dogsheep/dogsheep-photos,Upload your photos to S3 and import metadata about them into a SQLite database,0,2020-04-18T19:22:13Z,2021-11-04T20:45:03Z,2021-11-04T20:45:00Z,,68,124,124,Python,1,1,1,1,0,7,0,0,19,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-tool"", ""dogsheep"", ""sqlite""]",7,19,124,master,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,53015001,7,10,"# dogsheep-photos
[](https://pypi.org/project/dogsheep-photos/)
[](https://github.com/dogsheep/dogsheep-photos/releases)
[](https://circleci.com/gh/dogsheep/dogsheep-photos)
[](https://github.com/dogsheep/dogsheep-photos/blob/master/LICENSE)
Save details of your photos to a SQLite database and upload them to S3.
See [Using SQL to find my best photo of a pelican according to Apple Photos](https://simonwillison.net/2020/May/21/apple-photos-sqlite/) for background information on this project.
## What these tools do
These tools are a work-in-progress mechanism for taking full ownership of your photos. The core idea is to help implement the following:
* Every photo you have taken lives in a single, private Amazon S3 bucket
* You have a single SQLite database file which stores metadata about those photos - potentially pulled from multiple different places. This may include EXIF data, Apple Photos, the results of running machine learning APIs against photos and much more besides.
* You can then use [Datasette](https://github.com/simonw/datasette) to explore your own photos.
I'm a heavy user of Apple Photos so the initial releases of this tool will have a bias towards that, but ideally I would like a subset of these tools to be useful to people no matter which core photo solution they are using.
## Installation
$ pip install dogsheep-photos
## Authentication (if using S3)
If you want to use S3 to store your photos, you will need to first create S3 credentials for a new, dedicated bucket.
You may find the [s3-credentials tool](https://github.com/simonw/s3-credentials) useful for this.
Run this command and paste in your credentials. You will need three values: the name of your S3 bucket, your Access key ID and your Secret access key.
$ dogsheep-photos s3-auth
This will create a file called `auth.json` in your current directory containing the required values. To save the file at a different path or filename, use the `--auth=myauth.json` option.
## Uploading photos
Run this command to upload every photo in a specific directory to your S3 bucket:
$ dogsheep-photos upload photos.db \
~/Pictures/Photos\ Library.photoslibrary/original
The command will only upload photos that have not yet been uploaded, based on their sha256 hash.
`photos.db` will be created with an `uploads` table containing details of which files were uploaded.
To see what the command would do without uploading any files, use the `--dry-run` option.
The sha256 hash of the photo contents will be used as the name of the file in the bucket, with an extension matching the type of file. This is an implementation of the [Content addressable storage](https://en.wikipedia.org/wiki/Content-addressable_storage) pattern.
## Importing Apple Photos metadata
The `apple-photos` command imports metadata from your Apple Photos library.
$ photo-to-sqlite apple-photos photos.db
Imported metadata includes places, people, albums, quality scores and machine learning labels for the photo contents.
## Creating a subset database
You can create a new, subset database of photos using the `create-subset` command.
This is useful for creating a shareable SQLite database that only contains metadata for a selected set of photos.
Since photo metadata contains latitude and longitude you may not want to share a database that includes photos taken at your home address.
`create-subset` takes three arguments: an existing database file created using the `apple-photos` command, the name of the new, shareable database file you would like to create and a SQL query that returns the `sha256` hash values of the photos you would like to include in that database.
For example, here's how to create a shareable database of just the photos that have been added to albums containing the word ""Public"":
$ dogsheep-photos create-subset \
photos.db \
public.db \
""select sha256 from apple_photos where albums like '%Public%'""
## Serving photos locally with datasette-media
If you don't want to upload your photos to S3 but you still want to browse them using Datasette you can do so using the [datasette-media](https://github.com/simonw/datasette-media) plugin. This plugin adds the ability to serve images and other static files directly from disk, configured using a SQL query.
To use it, first install Datasette and the plugin:
$ pip install datasette datasette-media
If any of your photos are `.HEIC` images taken by an iPhone you should also install the optional `pyheif` dependency:
$ pip install pyheif
Now create a `metadata.yaml` file configuring the plugin:
```yaml
plugins:
datasette-media:
thumbnail:
sql: |-
select path as filepath, 200 as resize_height from apple_photos where uuid = :key
large:
sql: |-
select path as filepath, 1024 as resize_height from apple_photos where uuid = :key
```
This will configure two URL endpoints - one for 200 pixel high thumbnails and one for 1024 pixel high larger images.
Create your `photos.db` database using the `apple-photos` command, then run Datasette like this:
$ datasette -m metadata.yaml
Your photos will be served on URLs that look like this:
http://127.0.0.1:8001/-/media/thumbnail/F4469918-13F3-43D8-9EC1-734C0E6B60AD
http://127.0.0.1:8001/-/media/large/F4469918-13F3-43D8-9EC1-734C0E6B60AD
You can find the UUIDs for use in these URLs by running `select uuid from photos_with_apple_metadata`.
### Displaying images using datasette-json-html
If you are using `datasette-media` to serve photos you can include images directly in Datasette query results using the [datasette-json-html](https://github.com/simonw/datasette-json-html) plugin.
Run `pip install datasette-json-html` to install the plugin, then use queries like this to view your images:
```sql
select
json_object(
'img_src',
'/-/media/thumbnail/' || uuid
) as photo,
uuid,
date
from
apple_photos
order by
date desc
limit 10;
```
The `photo` column returned by this query should render as image tags that display the correct images.
### Displaying images using custom template pages
Datasette's [custom pages](https://datasette.readthedocs.io/en/stable/custom_templates.html#custom-pages) feature lets you create custom pages for a Datasette instance by dropping HTML templates into a `templates/pages` directory and then running Datasette using `datasette --template-dir=templates/`.
You can combine that ability with the [datasette-template-sql](https://github.com/simonw/datasette-template-sql) plugin to create custom template pages that directly display photos served by `datasette-media`.
Install the plugin using `pip install datasette-template-sql`.
Create a `templates/pages` folder and add the following files:
`recent-photos.html`
```html+jinja
Recent photos
{% for photo in sql(""select * from apple_photos order by date desc limit 20"") %}
{% endfor %}
```
`random-photos.html`
```html+jinja
Random photos
{% for photo in sql(""with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 20"") %}
{% endfor %}
```
Now run Datasette like this:
$ datasette photos.db -m metadata.yaml --template-dir=templates/
Visiting `http://localhost:8001/recent-photos` will display 20 recent photos. Visiting `http://localhost:8001/random-photos` will display 20 photos randomly selected from your 5,000 most recent.
","
dogsheep-photos
Save details of your photos to a SQLite database and upload them to S3.
These tools are a work-in-progress mechanism for taking full ownership of your photos. The core idea is to help implement the following:
Every photo you have taken lives in a single, private Amazon S3 bucket
You have a single SQLite database file which stores metadata about those photos - potentially pulled from multiple different places. This may include EXIF data, Apple Photos, the results of running machine learning APIs against photos and much more besides.
You can then use Datasette to explore your own photos.
I'm a heavy user of Apple Photos so the initial releases of this tool will have a bias towards that, but ideally I would like a subset of these tools to be useful to people no matter which core photo solution they are using.
Installation
$ pip install dogsheep-photos
Authentication (if using S3)
If you want to use S3 to store your photos, you will need to first create S3 credentials for a new, dedicated bucket.
Run this command and paste in your credentials. You will need three values: the name of your S3 bucket, your Access key ID and your Secret access key.
$ dogsheep-photos s3-auth
This will create a file called auth.json in your current directory containing the required values. To save the file at a different path or filename, use the --auth=myauth.json option.
Uploading photos
Run this command to upload every photo in a specific directory to your S3 bucket:
The command will only upload photos that have not yet been uploaded, based on their sha256 hash.
photos.db will be created with an uploads table containing details of which files were uploaded.
To see what the command would do without uploading any files, use the --dry-run option.
The sha256 hash of the photo contents will be used as the name of the file in the bucket, with an extension matching the type of file. This is an implementation of the Content addressable storage pattern.
Importing Apple Photos metadata
The apple-photos command imports metadata from your Apple Photos library.
$ photo-to-sqlite apple-photos photos.db
Imported metadata includes places, people, albums, quality scores and machine learning labels for the photo contents.
Creating a subset database
You can create a new, subset database of photos using the create-subset command.
This is useful for creating a shareable SQLite database that only contains metadata for a selected set of photos.
Since photo metadata contains latitude and longitude you may not want to share a database that includes photos taken at your home address.
create-subset takes three arguments: an existing database file created using the apple-photos command, the name of the new, shareable database file you would like to create and a SQL query that returns the sha256 hash values of the photos you would like to include in that database.
For example, here's how to create a shareable database of just the photos that have been added to albums containing the word ""Public"":
$ dogsheep-photos create-subset \
photos.db \
public.db \
""select sha256 from apple_photos where albums like '%Public%'""
Serving photos locally with datasette-media
If you don't want to upload your photos to S3 but you still want to browse them using Datasette you can do so using the datasette-media plugin. This plugin adds the ability to serve images and other static files directly from disk, configured using a SQL query.
To use it, first install Datasette and the plugin:
$ pip install datasette datasette-media
If any of your photos are .HEIC images taken by an iPhone you should also install the optional pyheif dependency:
$ pip install pyheif
Now create a metadata.yaml file configuring the plugin:
plugins:
datasette-media:
thumbnail:
sql: |- select path as filepath, 200 as resize_height from apple_photos where uuid = :keylarge:
sql: |- select path as filepath, 1024 as resize_height from apple_photos where uuid = :key
This will configure two URL endpoints - one for 200 pixel high thumbnails and one for 1024 pixel high larger images.
Create your photos.db database using the apple-photos command, then run Datasette like this:
$ datasette -m metadata.yaml
Your photos will be served on URLs that look like this:
You can find the UUIDs for use in these URLs by running select uuid from photos_with_apple_metadata.
Displaying images using datasette-json-html
If you are using datasette-media to serve photos you can include images directly in Datasette query results using the datasette-json-html plugin.
Run pip install datasette-json-html to install the plugin, then use queries like this to view your images:
select
json_object(
'img_src',
'/-/media/thumbnail/'|| uuid
) as photo,
uuid,
datefrom
apple_photos
order bydatedesclimit10;
The photo column returned by this query should render as image tags that display the correct images.
Displaying images using custom template pages
Datasette's custom pages feature lets you create custom pages for a Datasette instance by dropping HTML templates into a templates/pages directory and then running Datasette using datasette --template-dir=templates/.
You can combine that ability with the datasette-template-sql plugin to create custom template pages that directly display photos served by datasette-media.
Install the plugin using pip install datasette-template-sql.
Create a templates/pages folder and add the following files:
recent-photos.html
<h1>Recent photos</h1>
<div>
{%forphotoinsql(""select * from apple_photos order by date desc limit 20"") %}
<imgsrc=""/-/media/photo/{{ photo['uuid'] }}"">
{%endfor%}
</div>
random-photos.html
<h1>Random photos</h1>
<div>
{%forphotoinsql(""with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 20"") %}
<imgsrc=""/-/media/photo/{{ photo['uuid'] }}"">
{%endfor%}
</div>
Visiting http://localhost:8001/recent-photos will display 20 recent photos. Visiting http://localhost:8001/random-photos will display 20 photos randomly selected from your 5,000 most recent.
",1,public,0,,,
271665336,MDEwOlJlcG9zaXRvcnkyNzE2NjUzMzY=,datasette-auth-tokens,simonw/datasette-auth-tokens,0,9599,https://github.com/simonw/datasette-auth-tokens,Datasette plugin for authenticating access using API tokens,0,2020-06-11T23:23:30Z,2021-10-15T00:52:53Z,2021-10-15T00:54:20Z,,34,4,4,Python,1,1,1,1,0,1,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin""]",1,0,4,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,1,3,"# datasette-auth-tokens
[](https://pypi.org/project/datasette-auth-tokens/)
[](https://github.com/simonw/datasette-auth-tokens/releases)
[](https://github.com/simonw/datasette-auth-tokens/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-auth-tokens/blob/main/LICENSE)
Datasette plugin for authenticating access using API tokens
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-auth-tokens
## Hard-coded tokens
Read about Datasette's [authentication and permissions system](https://datasette.readthedocs.io/en/latest/authentication.html).
This plugin lets you configure secret API tokens which can be used to make authenticated requests to Datasette.
First, create a random API token. A useful recipe for doing that is the following:
$ python -c 'import secrets; print(secrets.token_hex(32))'
5f9a486dd807de632200b17508c75002bb66ca6fde1993db1de6cbd446362589
Decide on the actor that this token should represent, for example:
```json
{
""bot_id"": ""my-bot""
}
```
You can then use `""allow""` blocks to provide that token with permission to access specific actions. To enable access to a configured writable SQL query you could use this in your `metadata.json`:
```json
{
""plugins"": {
""datasette-auth-tokens"": {
""tokens"": [
{
""token"": {
""$env"": ""BOT_TOKEN""
},
""actor"": {
""bot_id"": ""my-bot""
}
}
]
}
},
""databases"": {
"":memory:"": {
""queries"": {
""show_version"": {
""sql"": ""select sqlite_version()"",
""allow"": {
""bot_id"": ""my-bot""
}
}
}
}
}
}
```
This uses Datasette's [secret configuration values mechanism](https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values) to allow the secret token to be passed as an environment variable.
Run Datasette like this:
BOT_TOKEN=""this-is-the-secret-token"" \
datasette -m metadata.json
You can now run authenticated API queries like this:
$ curl -H 'Authorization: Bearer this-is-the-secret-token' \
'http://127.0.0.1:8001/:memory:/show_version.json?_shape=array'
[{""sqlite_version()"": ""3.31.1""}]
Additionally you can allow passing the token as a query string parameter, although that's disabled by default given the security implications of URLs with secret tokens included. This may be useful to easily allow embedding data between different services.
Simply enable it using the `param` config value:
```json
{
""plugins"": {
""datasette-auth-tokens"": {
""tokens"": [
{
""token"": {
""$env"": ""BOT_TOKEN""
},
""actor"": {
""bot_id"": ""my-bot""
},
}
],
""param"": ""_auth_token""
}
},
""databases"": {
"":memory:"": {
""queries"": {
""show_version"": {
""sql"": ""select sqlite_version()"",
""allow"": {
""bot_id"": ""my-bot""
}
}
}
}
}
}
```
You can now run authenticated API queries like this:
$ curl http://127.0.0.1:8001/:memory:/show_version.json?_shape=array&_auth_token=this-is-the-secret-token
[{""sqlite_version()"": ""3.31.1""}]
## Tokens from your database
As an alternative (or in addition) to the hard-coded list of tokens you can store tokens in a database table and configure the plugin to access them using a SQL query.
Your query needs to take a `:token_id` parameter and return at least two columns: one called `token_secret` and one called `actor_*` - usually `actor_id`. Further `actor_` prefixed columns can be returned to provide more details for the authenticated actor.
Here's a simple example of a configuration query:
```sql
select actor_id, actor_name, token_secret from tokens where token_id = :token_id
```
This can run against a table like this one:
| token_id | token_secret | actor_id | actor_name |
| -------- | ------------ | -------- | ---------- |
| 1 | bd3c94f51fcd | 78 | Cleopaws |
| 2 | 86681b4d6f66 | 32 | Pancakes |
The tokens are formed as the token ID, then a hyphen, then the token secret. For example:
- `1-bd3c94f51fcd`
- `2-86681b4d6f66`
The SQL query will be executed with the portion before the hyphen as the `:token_id` parameter.
The `token_secret` value returned by the query will be compared to the portion of the token after the hyphen to check if the token is valid.
Columns with a prefix of `actor_` will be used to populate the actor dictionary. In the above example, a token of `2-86681b4d6f66` will become an actor dictionary of `{""id"": 32, ""name"": ""Pancakes""}`.
To configure this, use a `""query""` block in your plugin configuration like this:
```json
{
""plugins"": {
""datasette-auth-tokens"": {
""query"": {
""sql"": ""select actor_id, actor_name, token_secret from tokens where token_id = :token_id"",
""database"": ""tokens""
}
}
},
""databases"": {
""tokens"": {
""allow"": {}
}
}
}
```
The `""sql""` key here contains the SQL query. The `""database""` key has the name of the attached database file that the query should be executed against - in this case it would execute against `tokens.db`.
### Securing your tokens
Anyone with access to your Datasette instance can use it to read the `token_secret` column in your tokens table. This probably isn't what you want!
To avoid this, you should lock down access to that table. The configuration example above shows how to do this using an `""allow"": {}` block. Consult Datasette's [Permissions documentation](https://datasette.readthedocs.io/en/stable/authentication.html#permissions) for more information about how to lock down this kind of access.
","
datasette-auth-tokens
Datasette plugin for authenticating access using API tokens
Installation
Install this plugin in the same environment as Datasette.
Decide on the actor that this token should represent, for example:
{
""bot_id"": ""my-bot""
}
You can then use ""allow"" blocks to provide that token with permission to access specific actions. To enable access to a configured writable SQL query you could use this in your metadata.json:
Additionally you can allow passing the token as a query string parameter, although that's disabled by default given the security implications of URLs with secret tokens included. This may be useful to easily allow embedding data between different services.
As an alternative (or in addition) to the hard-coded list of tokens you can store tokens in a database table and configure the plugin to access them using a SQL query.
Your query needs to take a :token_id parameter and return at least two columns: one called token_secret and one called actor_* - usually actor_id. Further actor_ prefixed columns can be returned to provide more details for the authenticated actor.
Here's a simple example of a configuration query:
select actor_id, actor_name, token_secret from tokens where token_id = :token_id
This can run against a table like this one:
token_id
token_secret
actor_id
actor_name
1
bd3c94f51fcd
78
Cleopaws
2
86681b4d6f66
32
Pancakes
The tokens are formed as the token ID, then a hyphen, then the token secret. For example:
1-bd3c94f51fcd
2-86681b4d6f66
The SQL query will be executed with the portion before the hyphen as the :token_id parameter.
The token_secret value returned by the query will be compared to the portion of the token after the hyphen to check if the token is valid.
Columns with a prefix of actor_ will be used to populate the actor dictionary. In the above example, a token of 2-86681b4d6f66 will become an actor dictionary of {""id"": 32, ""name"": ""Pancakes""}.
To configure this, use a ""query"" block in your plugin configuration like this:
The ""sql"" key here contains the SQL query. The ""database"" key has the name of the attached database file that the query should be executed against - in this case it would execute against tokens.db.
Securing your tokens
Anyone with access to your Datasette instance can use it to read the token_secret column in your tokens table. This probably isn't what you want!
To avoid this, you should lock down access to that table. The configuration example above shows how to do this using an ""allow"": {} block. Consult Datasette's Permissions documentation for more information about how to lock down this kind of access.
",1,public,0,,,
272098486,MDEwOlJlcG9zaXRvcnkyNzIwOTg0ODY=,datasette-psutil,simonw/datasette-psutil,0,9599,https://github.com/simonw/datasette-psutil,Datasette plugin adding a /-/psutil debugging endpoint,0,2020-06-13T22:57:07Z,2022-03-07T15:36:30Z,2022-03-07T15:35:57Z,https://datasette.io/plugins/datasette-psutil,12,2,2,Python,1,1,1,1,0,0,0,0,1,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""psutil""]",0,1,2,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-psutil
[](https://pypi.org/project/datasette-psutil/)
[](https://github.com/simonw/datasette-psutil/releases)
[](https://github.com/simonw/datasette-psutil/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-psutil/blob/main/LICENSE)
Datasette plugin adding a `/-/psutil` debugging endpoint
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-psutil
## Usage
Visit `/-/psutil` on your Datasette instance to see various information provided by [psutil](https://psutil.readthedocs.io/).
## Demo
https://latest-with-plugins.datasette.io/-/psutil is a live demo of this plugin, hosted on Google Cloud Run.
","
datasette-psutil
Datasette plugin adding a /-/psutil debugging endpoint
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-psutil
Usage
Visit /-/psutil on your Datasette instance to see various information provided by psutil.
",1,public,0,,,
274293597,MDEwOlJlcG9zaXRvcnkyNzQyOTM1OTc=,datasette-block-robots,simonw/datasette-block-robots,0,9599,https://github.com/simonw/datasette-block-robots,Datasette plugin that blocks robots and crawlers using robots.txt,0,2020-06-23T02:52:23Z,2022-08-30T16:13:40Z,2022-08-30T16:25:38Z,https://datasette.io/plugins/datasette-block-robots,21,2,2,Python,1,1,1,1,0,0,0,0,0,,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""robots-txt""]",0,0,2,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-block-robots
[](https://pypi.org/project/datasette-block-robots/)
[](https://github.com/simonw/datasette-block-robots/releases)
[](https://github.com/simonw/datasette-block-robots/blob/master/LICENSE)
Datasette plugin that blocks robots and crawlers using robots.txt
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-block-robots
## Usage
Having installed the plugin, `/robots.txt` on your Datasette instance will return the following:
User-agent: *
Disallow: /
This will request all robots and crawlers not to visit any of the pages on your site.
Here's a demo of the plugin in action: https://sqlite-generate-demo.datasette.io/robots.txt
## Configuration
By default the plugin will block all access to the site, using `Disallow: /`.
If you want the index page to be indexed by search engines without crawling the database, table or row pages themselves, you can use the following:
```json
{
""plugins"": {
""datasette-block-robots"": {
""allow_only_index"": true
}
}
}
```
This will return a `/robots.txt` like so:
User-agent: *
Disallow: /db1
Disallow: /db2
With a `Disallow` line for every attached database.
To block access to specific areas of the site using custom paths, add this to your `metadata.json` configuration file:
```json
{
""plugins"": {
""datasette-block-robots"": {
""disallow"": [""/mydatabase/mytable""]
}
}
}
```
This will result in a `/robots.txt` that looks like this:
User-agent: *
Disallow: /mydatabase/mytable
Alternatively you can set the full contents of the `robots.txt` file using the `literal` configuration option. Here's how to do that if you are using YAML rather than JSON and have a `metadata.yml` file:
```yaml
plugins:
datasette-block-robots:
literal: |-
User-agent: *
Disallow: /
User-agent: Bingbot
User-agent: Googlebot
Disallow:
```
This example would block all crawlers with the exception of Googlebot and Bingbot, which are allowed to crawl the entire site.
## Extending this with other plugins
This plugin adds a new [plugin hook](https://docs.datasette.io/en/stable/plugin_hooks.html) to Datasete called `block_robots_extra_lines()` which can be used by other plugins to add their own additional lines to the `robots.txt` file.
The hook can optionally accept these parameters:
- `datasette`: The current [Datasette instance](https://docs.datasette.io/en/stable/internals.html#datasette-class). You can use this to execute SQL queries or read plugin configuration settings.
- `request`: The [Request object](https://docs.datasette.io/en/stable/internals.html#request-object) representing the incoming request to `/robots.txt`.
The hook should return a list of strings, each representing a line to be added to the `robots.txt` file.
It can also return an `async def` function, which will be awaited and used to generate a list of lines. Use this option if you need to make `await` calls inside you hook implementation.
This example uses the hook to add a `Sitemap: http://example.com/sitemap.xml` line to the `robots.txt` file:
```python
from datasette import hookimpl
@hookimpl
def block_robots_extra_lines(datasette, request):
return [
""Sitemap: {}"".format(datasette.absolute_url(request, ""/sitemap.xml"")),
]
```
This example blocks access to paths based on a database query:
```python
@hookimpl
def block_robots_extra_lines(datasette):
async def inner():
db = datasette.get_database()
result = await db.execute(""select path from mytable"")
return [
""Disallow: /{}"".format(row[""path""]) for row in result
]
return inner
```
[datasette-sitemap](https://datasette.io/plugins/datasette-sitemap) is an example of a plugin that uses this hook.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-block-robots
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-block-robots
Datasette plugin that blocks robots and crawlers using robots.txt
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-block-robots
Usage
Having installed the plugin, /robots.txt on your Datasette instance will return the following:
User-agent: *
Disallow: /
This will request all robots and crawlers not to visit any of the pages on your site.
This will result in a /robots.txt that looks like this:
User-agent: *
Disallow: /mydatabase/mytable
Alternatively you can set the full contents of the robots.txt file using the literal configuration option. Here's how to do that if you are using YAML rather than JSON and have a metadata.yml file:
This example would block all crawlers with the exception of Googlebot and Bingbot, which are allowed to crawl the entire site.
Extending this with other plugins
This plugin adds a new plugin hook to Datasete called block_robots_extra_lines() which can be used by other plugins to add their own additional lines to the robots.txt file.
The hook can optionally accept these parameters:
datasette: The current Datasette instance. You can use this to execute SQL queries or read plugin configuration settings.
request: The Request object representing the incoming request to /robots.txt.
The hook should return a list of strings, each representing a line to be added to the robots.txt file.
It can also return an async def function, which will be awaited and used to generate a list of lines. Use this option if you need to make await calls inside you hook implementation.
This example uses the hook to add a Sitemap: http://example.com/sitemap.xml line to the robots.txt file:
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-block-robots
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,0,
279357123,MDEwOlJlcG9zaXRvcnkyNzkzNTcxMjM=,datasette-auth-passwords,simonw/datasette-auth-passwords,0,9599,https://github.com/simonw/datasette-auth-passwords,Datasette plugin for authentication using passwords,0,2020-07-13T16:34:39Z,2022-02-10T22:07:52Z,2022-03-22T01:49:50Z,https://datasette-auth-passwords-demo.datasette.io,52,12,12,Python,1,1,1,1,0,0,0,0,3,,"[""datasette"", ""datasette-io"", ""datasette-plugin""]",0,3,12,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-auth-passwords
[](https://pypi.org/project/datasette-auth-passwords/)
[](https://github.com/simonw/datasette-auth-passwords/releases)
[](https://github.com/simonw/datasette-auth-passwords/blob/master/LICENSE)
Datasette plugin for authenticating access using passwords
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-auth-passwords
## Demo
A demo of this plugin is running at https://datasette-auth-passwords-demo.datasette.io/
The demo is configured to show the `public.db` database to everyone, but the `private.db` database only to logged in users.
You can log in at https://datasette-auth-passwords-demo.datasette.io/-/login with username `root` and password `password!`.
## Usage
This plugin works based on a list of username/password accounts that are hard-coded into the plugin configuration.
First, you'll need to create a password hash. There are three ways to do that:
- Install the plugin, then use the interactive tool located at `/-/password-tool`
- Use the hosted version of that tool at https://datasette-auth-passwords-demo.datasette.io/-/password-tool
- Use the `datasette hash-password` command, described below
Now add the following to your `metadata.json`:
```json
{
""plugins"": {
""datasette-auth-passwords"": {
""someusername_password_hash"": {
""$env"": ""PASSWORD_HASH_1""
}
}
}
}
```
The password hash can now be specified in an environment variable when you run Datasette. You can do that like so:
PASSWORD_HASH_1='pbkdf2_sha256$...' \
datasette -m metadata.json
Be sure to use single quotes here otherwise the `$` symbols in the password hash may be incorrectly interpreted by your shell.
You will now be able to log in to your instance using the form at `/-/login` with `someusername` as the username and the password that you used to create your hash as the password.
You can include as many accounts as you like in the configuration, each with different usernames.
### datasette hash-password
The plugin exposes a new CLI command, `datasette hash-password`. You can run this without arguments to interactively create a new password hash:
```
% datasette hash-password
Password:
Repeat for confirmation:
pbkdf2_sha256$260000$1513...
```
Or if you want to use it as part of a script, you can add the `--no-confirm` option to generate a hash directly from a value passed to standard input:
```
% echo 'my password' | datasette hash-password --no-confirm
pbkdf2_sha256$260000$daa...
```
### Specifying actors
By default, a logged in user will result in an [actor block](https://datasette.readthedocs.io/en/stable/authentication.html#actors) that just contains their username:
```json
{
""id"": ""someusername""
}
```
You can customize the actor that will be used for a username by including an `""actors""` configuration block, like this:
```json
{
""plugins"": {
""datasette-auth-passwords"": {
""someusername_password_hash"": {
""$env"": ""PASSWORD_HASH_1""
},
""actors"": {
""someusername"": {
""id"": ""someusername"",
""name"": ""Some user""
}
}
}
}
}
```
### HTTP Basic authentication option
This plugin defaults to implementing login using an HTML form that sets a signed authentication cookie.
You can alternatively configure it to use [HTTP Basic authentication](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#basic_authentication_scheme) instead.
Do this by adding `""http_basic_auth"": true` to the `datasette-auth-passwords` block in your plugin configuration.
This option introduces the following behaviour:
- Account usernames and passwords are configured in the same way as form-based authentication
- Every page within Datasette - even pages that normally do not use authentication, such as static assets - will display a browser login prompt
- Users will be unable to log out without closing their browser entirely
There is a demo of this mode at https://datasette-auth-passwords-http-basic-demo.datasette.io/ - sign in with username `root` and password `password!`
### Using with datasette publish
If you are publishing data using a [datasette publish](https://datasette.readthedocs.io/en/stable/publish.html#datasette-publish) command you can use the `--plugin-secret` option to securely configure your password hashes (see [secret configuration values](https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values)).
You would run the command something like this:
datasette publish cloudrun mydatabase.db \
--install datasette-auth-passwords \
--plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$...' \
--service datasette-auth-passwords-demo
This will allow you to log in as username `root` using the password that you used to create the hash.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-auth-passwords
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-auth-passwords
Datasette plugin for authenticating access using passwords
Installation
Install this plugin in the same environment as Datasette.
Be sure to use single quotes here otherwise the $ symbols in the password hash may be incorrectly interpreted by your shell.
You will now be able to log in to your instance using the form at /-/login with someusername as the username and the password that you used to create your hash as the password.
You can include as many accounts as you like in the configuration, each with different usernames.
datasette hash-password
The plugin exposes a new CLI command, datasette hash-password. You can run this without arguments to interactively create a new password hash:
% datasette hash-password
Password:
Repeat for confirmation:
pbkdf2_sha256$260000$1513...
Or if you want to use it as part of a script, you can add the --no-confirm option to generate a hash directly from a value passed to standard input:
If you are publishing data using a datasette publish command you can use the --plugin-secret option to securely configure your password hashes (see secret configuration values).
This will allow you to log in as username root using the password that you used to create the hash.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-auth-passwords
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
280500027,MDEwOlJlcG9zaXRvcnkyODA1MDAwMjc=,datasette-insert,simonw/datasette-insert,0,9599,https://github.com/simonw/datasette-insert,Datasette plugin for inserting and updating data,0,2020-07-17T18:40:34Z,2022-06-27T02:54:14Z,2022-07-22T17:52:23Z,,54,9,9,Python,1,1,1,1,0,0,0,0,1,,"[""datasette"", ""datasette-io"", ""datasette-plugin""]",0,1,9,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-insert
[](https://pypi.org/project/datasette-insert/)
[](https://github.com/simonw/datasette-insert/releases)
[](https://github.com/simonw/datasette-insert/blob/master/LICENSE)
Datasette plugin for inserting and updating data
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-insert
This plugin should always be deployed with additional configuration to prevent unauthenticated access, see notes below.
If you are trying it out on your own local machine, you can `pip install` the [datasette-insert-unsafe](https://github.com/simonw/datasette-insert-unsafe) plugin to allow access without needing to set up authentication or permissions separately.
## Inserting data and creating tables
Start datasette and make sure it has a writable SQLite database attached to it. If you have not yet created a database file you can use this:
datasette data.db --create
The `--create` option will create a new empty `data.db` database file if it does not already exist.
The plugin adds an endpoint that allows data to be inserted or updated and tables to be created by POSTing JSON data to the following URL:
/-/insert/name-of-database/name-of-table
The JSON should look like this:
```json
[
{
""id"": 1,
""name"": ""Cleopaws"",
""age"": 5
},
{
""id"": 2,
""name"": ""Pancakes"",
""age"": 5
}
]
```
The first time data is posted to the URL a table of that name will be created if it does not aready exist, with the desired columns.
You can specify which column should be used as the primary key using the `?pk=` URL argument.
Here's how to POST to a database and create a new table using the Python `requests` library:
```python
import requests
requests.post(""http://localhost:8001/-/insert/data/dogs?pk=id"", json=[
{
""id"": 1,
""name"": ""Cleopaws"",
""age"": 5
},
{
""id"": 2,
""name"": ""Pancakes"",
""age"": 4
}
])
```
And here's how to do the same thing using `curl`:
```
curl --request POST \
--data '[
{
""id"": 1,
""name"": ""Cleopaws"",
""age"": 5
},
{
""id"": 2,
""name"": ""Pancakes"",
""age"": 4
}
]' \
'http://localhost:8001/-/insert/data/dogs?pk=id'
```
Or by piping in JSON like so:
cat dogs.json | curl --request POST -d @- \
'http://localhost:8001/-/insert/data/dogs?pk=id'
### Inserting a single row
If you are inserting a single row you can optionally send it as a dictionary rather than a list with a single item:
```
curl --request POST \
--data '{
""id"": 1,
""name"": ""Cleopaws"",
""age"": 5
}' \
'http://localhost:8001/-/insert/data/dogs?pk=id'
```
### Automatically adding new columns
If you send data to an existing table with keys that are not reflected by the existing columns, you will get an HTTP 400 error with a JSON response like this:
```json
{
""status"": 400,
""error"": ""Unknown keys: 'foo'"",
""error_code"": ""unknown_keys""
}
```
If you add `?alter=1` to the URL you are posting to any missing columns will be automatically added:
```
curl --request POST \
--data '[
{
""id"": 3,
""name"": ""Boris"",
""age"": 1,
""breed"": ""Husky""
}
]' \
'http://localhost:8001/-/insert/data/dogs?alter=1'
```
## Upserting data
An ""upsert"" operation can be used to partially update a record. With upserts you can send a subset of the keys and, if the ID matches the specified primary key, they will be used to update an existing record.
Upserts can be sent to the `/-/upsert` API endpoint.
This example will update the dog with ID=1's age from 5 to 7:
```
curl --request POST \
--data '{
""id"": 1,
""age"": 7
}' \
'http://localhost:3322/-/upsert/data/dogs?pk=id'
```
Like the `/-/insert` endpoint, the `/-/upsert` endpoint can accept an array of objects too. It also supports the `?alter=1` option.
## Permissions and authentication
This plugin defaults to denying all access, to help ensure people don't accidentally deploy it on the open internet in an unsafe configuration.
You can read about [Datasette's approach to authentication](https://datasette.readthedocs.io/en/stable/authentication.html) in the Datasette manual.
You can install the `datasette-insert-unsafe` plugin to run in unsafe mode, where all access is allowed by default.
I recommend using this plugin in conjunction with [datasette-auth-tokens](https://github.com/simonw/datasette-auth-tokens), which provides a mechanism for making authenticated calls using API tokens.
You can then use [""allow"" blocks](https://datasette.readthedocs.io/en/stable/authentication.html#defining-permissions-with-allow-blocks) in the `datasette-insert` plugin configuration to specify which authenticated tokens are allowed to make use of the API.
Here's an example `metadata.json` file which restricts access to the `/-/insert` API to an API token defined in an `INSERT_TOKEN` environment variable:
```json
{
""plugins"": {
""datasette-insert"": {
""allow"": {
""bot"": ""insert-bot""
}
},
""datasette-auth-tokens"": {
""tokens"": [
{
""token"": {
""$env"": ""INSERT_TOKEN""
},
""actor"": {
""bot"": ""insert-bot""
}
}
]
}
}
}
```
With this configuration in place you can start Datasette like this:
INSERT_TOKEN=abc123 datasette data.db -m metadata.json
You can now send data to the API using `curl` like this:
```
curl --request POST \
-H ""Authorization: Bearer abc123"" \
--data '[
{
""id"": 3,
""name"": ""Boris"",
""age"": 1,
""breed"": ""Husky""
}
]' \
'http://localhost:8001/-/insert/data/dogs'
```
Or using the Python `requests` library like so:
```python
requests.post(
""http://localhost:8001/-/insert/data/dogs"",
json={""id"": 1, ""name"": ""Cleopaws"", ""age"": 5},
headers={""Authorization"": ""bearer abc123""},
)
```
### Finely grained permissions
Using an `""allow""` block as described above grants full permission to the features enabled by the API.
The API implements several new Datasett permissions, which other plugins can use to make more finely grained decisions.
The full set of permissions are as follows:
- `insert:all` - all permissions - this is used by the `""allow""` block described above. Argument: `database_name`
- `insert:insert-update` - the ability to insert data into an existing table, or to update data by its primary key. Arguments: `(database_name, table_name)`
- `insert:create-table` - the ability to create a new table. Argument: `database_name`
- `insert:alter-table` - the ability to add columns to an existing table (using `?alter=1`). Arguments: `(database_name, table_name)`
You can use plugins like [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to hook into these more detailed permissions for finely grained control over what actions each authenticated actor can take.
Plugins that implement the [permission_allowed()](https://datasette.readthedocs.io/en/stable/plugin_hooks.html#plugin-hook-permission-allowed) plugin hook can take full control over these permission decisions.
## CORS
If you start Datasette with the `datasette --cors` option the following HTTP headers will be added to resources served by this plugin:
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: content-type,authorization
Access-Control-Allow-Methods: POST
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-insert
python3 -m venv venv
source venv/bin/activate
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-insert
Datasette plugin for inserting and updating data
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-insert
This plugin should always be deployed with additional configuration to prevent unauthenticated access, see notes below.
If you are trying it out on your own local machine, you can pip install the datasette-insert-unsafe plugin to allow access without needing to set up authentication or permissions separately.
Inserting data and creating tables
Start datasette and make sure it has a writable SQLite database attached to it. If you have not yet created a database file you can use this:
datasette data.db --create
The --create option will create a new empty data.db database file if it does not already exist.
The plugin adds an endpoint that allows data to be inserted or updated and tables to be created by POSTing JSON data to the following URL:
If you send data to an existing table with keys that are not reflected by the existing columns, you will get an HTTP 400 error with a JSON response like this:
An ""upsert"" operation can be used to partially update a record. With upserts you can send a subset of the keys and, if the ID matches the specified primary key, they will be used to update an existing record.
Upserts can be sent to the /-/upsert API endpoint.
This example will update the dog with ID=1's age from 5 to 7:
You can install the datasette-insert-unsafe plugin to run in unsafe mode, where all access is allowed by default.
I recommend using this plugin in conjunction with datasette-auth-tokens, which provides a mechanism for making authenticated calls using API tokens.
You can then use ""allow"" blocks in the datasette-insert plugin configuration to specify which authenticated tokens are allowed to make use of the API.
Here's an example metadata.json file which restricts access to the /-/insert API to an API token defined in an INSERT_TOKEN environment variable:
Using an ""allow"" block as described above grants full permission to the features enabled by the API.
The API implements several new Datasett permissions, which other plugins can use to make more finely grained decisions.
The full set of permissions are as follows:
insert:all - all permissions - this is used by the ""allow"" block described above. Argument: database_name
insert:insert-update - the ability to insert data into an existing table, or to update data by its primary key. Arguments: (database_name, table_name)
insert:create-table - the ability to create a new table. Argument: database_name
insert:alter-table - the ability to add columns to an existing table (using ?alter=1). Arguments: (database_name, table_name)
You can use plugins like datasette-permissions-sql to hook into these more detailed permissions for finely grained control over what actions each authenticated actor can take.
Plugins that implement the permission_allowed() plugin hook can take full control over these permission decisions.
CORS
If you start Datasette with the datasette --cors option the following HTTP headers will be added to resources served by this plugin:
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: content-type,authorization
Access-Control-Allow-Methods: POST
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-insert
python3 -m venv venv
source venv/bin/activate
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,0,
281481347,MDEwOlJlcG9zaXRvcnkyODE0ODEzNDc=,datasette-copyable,simonw/datasette-copyable,0,9599,https://github.com/simonw/datasette-copyable,Datasette plugin for outputting tables in formats suitable for copy and paste,0,2020-07-21T19:04:08Z,2022-03-26T20:02:45Z,2022-03-26T20:02:42Z,,11,11,11,Python,1,1,1,1,0,0,0,0,0,,"[""datasette"", ""datasette-io"", ""datasette-plugin""]",0,0,11,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-copyable
[](https://pypi.org/project/datasette-copyable/)
[](https://github.com/simonw/datasette-copyable/releases)
[](https://github.com/simonw/datasette-copyable/blob/master/LICENSE)
Datasette plugin for outputting tables in formats suitable for copy and paste
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-copyable
## Demo
You can try this plugin on [fivethirtyeight.datasettes.com](https://fivethirtyeight.datasettes.com/) - browse for tables or queries there and look for the ""copyable"" link. Here's an example for a table of [airline safety data](https://fivethirtyeight.datasettes.com/fivethirtyeight/airline-safety~2Fairline-safety.copyable).
## Usage
This plugin adds a `.copyable` output extension to every table, view and query.
Navigating to this page will show an interface allowing you to select a format for copying and pasting the demo. The default is TSV, which is suitable for copying into Google Sheets or Excel.
You can add `?_raw=1` to get back just the raw data.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-copyable
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-copyable
Datasette plugin for outputting tables in formats suitable for copy and paste
Installation
Install this plugin in the same environment as Datasette.
This plugin adds a .copyable output extension to every table, view and query.
Navigating to this page will show an interface allowing you to select a format for copying and pasting the demo. The default is TSV, which is suitable for copying into Google Sheets or Excel.
You can add ?_raw=1 to get back just the raw data.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-copyable
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
284383265,MDEwOlJlcG9zaXRvcnkyODQzODMyNjU=,datasette-graphql,simonw/datasette-graphql,0,9599,https://github.com/simonw/datasette-graphql,Datasette plugin providing an automatic GraphQL API for your SQLite databases,0,2020-08-02T03:31:58Z,2022-07-17T02:00:26Z,2022-07-18T21:13:34Z,https://datasette-graphql-demo.datasette.io/,715,63,63,Python,1,1,1,1,0,5,0,0,8,,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""graphql"", ""sqlite""]",5,8,63,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,5,3,"# datasette-graphql
[](https://pypi.org/project/datasette-graphql/)
[](https://github.com/simonw/datasette-graphql/releases)
[](https://github.com/simonw/datasette-graphql/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-graphql/blob/main/LICENSE)
**Datasette plugin providing an automatic GraphQL API for your SQLite databases**
Read more about this project: [GraphQL in Datasette with the new datasette-graphql plugin](https://simonwillison.net/2020/Aug/7/datasette-graphql/)
Try out a live demo at [datasette-graphql-demo.datasette.io/graphql](https://datasette-graphql-demo.datasette.io/graphql?query=%7B%0A%20%20repos(first%3A10%2C%20search%3A%20%22sql%22%2C%20sort_desc%3A%20created_at)%20%7B%0A%20%20%20%20totalCount%0A%20%20%20%20pageInfo%20%7B%0A%20%20%20%20%20%20endCursor%0A%20%20%20%20%20%20hasNextPage%0A%20%20%20%20%7D%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20description_%0A%20%20%20%20%09stargazers_count%0A%20%20%20%20%20%20created_at%0A%20%20%20%20%20%20owner%20%7B%0A%20%20%20%20%20%20%20%20name%0A%20%20%20%20%20%20%20%20html_url%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)
- [Installation](#installation)
- [Usage](#usage)
* [Querying for tables and columns](#querying-for-tables-and-columns)
* [Fetching a single record](#fetching-a-single-record)
* [Accessing nested objects](#accessing-nested-objects)
* [Accessing related objects](#accessing-related-objects)
* [Filtering tables](#filtering-tables)
* [Sorting](#sorting)
* [Pagination](#pagination)
* [Search](#search)
* [Columns containing JSON strings](#columns-containing-json-strings)
* [Auto camelCase](#auto-camelcase)
* [CORS](#cors)
* [Execution limits](#execution-limits)
- [The graphql() template function](#the-graphql-template-function)
- [Adding custom fields with plugins](#adding-custom-fields-with-plugins)
- [Development](#development)
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-graphql
## Usage
This plugin sets up `/graphql` as a GraphQL endpoint for the first attached database.
If you have multiple attached databases each will get its own endpoint at `/graphql/name_of_database`.
The automatically generated GraphQL schema is available at `/graphql/name_of_database.graphql` - here's [an example](https://datasette-graphql-demo.datasette.io/graphql/github.graphql).
### Querying for tables and columns
Individual tables (and SQL views) can be queried like this:
```graphql
{
repos {
nodes {
id
full_name
description_
}
}
}
```
[Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%20%7B%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20id%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20description_%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)
In this example query the underlying database table is called `repos` and its columns include `id`, `full_name` and `description`. Since `description` is a reserved word the query needs to ask for `description_` instead.
### Fetching a single record
If you only want to fetch a single record - for example if you want to fetch a row by its primary key - you can use the `tablename_row` field:
```graphql
{
repos_row(id: 107914493) {
id
full_name
description_
}
}
```
[Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos_row%28id%3A%20107914493%29%20%7B%0A%20%20%20%20id%0A%20%20%20%20full_name%0A%20%20%20%20description_%0A%20%20%7D%0A%7D%0A)
The `tablename_row` field accepts the primary key column (or columns) as arguments. It also supports the same `filter:`, `search:`, `sort:` and `sort_desc:` arguments as the `tablename` field, described below.
### Accessing nested objects
If a column is a foreign key to another table, you can request columns from the table pointed to by that foreign key using a nested query like this:
```graphql
{
repos {
nodes {
id
full_name
owner {
id
login
}
}
}
}
```
[Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%20%7B%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20id%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20owner%20%7B%0A%20%20%20%20%20%20%20%20id%0A%20%20%20%20%20%20%20%20login%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)
### Accessing related objects
If another table has a foreign key back to the table you are accessing, you can fetch rows from that related table.
Consider a `users` table which is related to `repos` - a repo has a foreign key back to the user that owns the repository. The `users` object type will have a `repos_by_owner_list` field which can be used to access those related repos:
```graphql
{
users(first: 1, search: ""simonw"") {
nodes {
name
repos_by_owner_list(first: 5) {
totalCount
nodes {
full_name
}
}
}
}
}
```
[Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20users%28first%3A%201%2C%20search%3A%20%22simonw%22%29%20%7B%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20name%0A%20%20%20%20%20%20repos_by_owner_list%28first%3A%205%29%20%7B%0A%20%20%20%20%20%20%20%20totalCount%0A%20%20%20%20%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)
### Filtering tables
You can filter the rows returned for a specific table using the `filter:` argument. This accepts a filter object mapping columns to operations. For example, to return just repositories with the Apache 2 license and more than 10 stars:
```graphql
{
repos(filter: {license: {eq: ""apache-2.0""}, stargazers_count: {gt: 10}}) {
nodes {
full_name
stargazers_count
license {
key
}
}
}
}
```
[Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28filter%3A%20%7Blicense%3A%20%7Beq%3A%20%22apache-2.0%22%7D%2C%20stargazers_count%3A%20%7Bgt%3A%2010%7D%7D%29%20%7B%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20stargazers_count%0A%20%20%20%20%20%20license%20%7B%0A%20%20%20%20%20%20%20%20key%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)
See [table filters examples](https://github.com/simonw/datasette-graphql/blob/main/examples/filters.md) for more operations, and [column filter arguments](https://docs.datasette.io/en/stable/json_api.html#column-filter-arguments) in the Datasette documentation for details of how those operations work.
These same filters can be used on nested relationships, like so:
```graphql
{
users_row(id: 9599) {
name
repos_by_owner_list(filter: {name: {startswith: ""datasette-""}}) {
totalCount
nodes {
full_name
}
}
}
}
```
[Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20users_row%28id%3A%209599%29%20%7B%0A%20%20%20%20name%0A%20%20%20%20repos_by_owner_list%28filter%3A%20%7Bname%3A%20%7Bstartswith%3A%20%22datasette-%22%7D%7D%29%20%7B%0A%20%20%20%20%20%20totalCount%0A%20%20%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)
The `where:` argument can be used as an alternative to `filter:` when the thing you are expressing is too complex to be modeled using a filter expression. It accepts a string fragment of SQL that will be included in the `WHERE` clause of the SQL query.
```graphql
{
repos(where: ""name='sqlite-utils' or name like 'datasette-%'"") {
totalCount
nodes {
full_name
}
}
}
```
[Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28where%3A%20%22name%3D%27sqlite-utils%27%20or%20name%20like%20%27datasette-%25%27%22%29%20%7B%0A%20%20%20%20totalCount%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)
### Sorting
You can set a sort order for results from a table using the `sort:` or `sort_desc:` arguments. The value for this argument should be the name of the column you wish to sort (or sort-descending) by.
```graphql
{
repos(sort_desc: stargazers_count) {
nodes {
full_name
stargazers_count
}
}
}
```
[Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28sort_desc%3A%20stargazers_count%29%20%7B%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20stargazers_count%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)
### Pagination
By default the first 10 rows will be returned. You can control this using the `first:` argument.
```graphql
{
repos(first: 20) {
totalCount
pageInfo {
hasNextPage
endCursor
}
nodes {
full_name
stargazers_count
license {
key
}
}
}
}
```
[Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28first%3A%2020%29%20%7B%0A%20%20%20%20totalCount%0A%20%20%20%20pageInfo%20%7B%0A%20%20%20%20%20%20hasNextPage%0A%20%20%20%20%20%20endCursor%0A%20%20%20%20%7D%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20stargazers_count%0A%20%20%20%20%20%20license%20%7B%0A%20%20%20%20%20%20%20%20key%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)
The `totalCount` field returns the total number of records that match the query.
Requesting the `pageInfo.endCursor` field provides you with the value you need to request the next page. You can pass this to the `after:` argument to request the next page.
```graphql
{
repos(first: 20, after: ""134874019"") {
totalCount
pageInfo {
hasNextPage
endCursor
}
nodes {
full_name
stargazers_count
license {
key
}
}
}
}
```
[Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28first%3A%2020%2C%20after%3A%20%22134874019%22%29%20%7B%0A%20%20%20%20totalCount%0A%20%20%20%20pageInfo%20%7B%0A%20%20%20%20%20%20hasNextPage%0A%20%20%20%20%20%20endCursor%0A%20%20%20%20%7D%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20stargazers_count%0A%20%20%20%20%20%20license%20%7B%0A%20%20%20%20%20%20%20%20key%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)
The `hasNextPage` field tells you if there are any more records.
### Search
If a table has been configured to use SQLite full-text search you can execute searches against it using the `search:` argument:
```graphql
{
repos(search: ""datasette"") {
totalCount
pageInfo {
hasNextPage
endCursor
}
nodes {
full_name
description_
}
}
}
```
[Try this query](https://datasette-graphql-demo.datasette.io/graphql?query=%0A%7B%0A%20%20repos%28search%3A%20%22datasette%22%29%20%7B%0A%20%20%20%20totalCount%0A%20%20%20%20pageInfo%20%7B%0A%20%20%20%20%20%20hasNextPage%0A%20%20%20%20%20%20endCursor%0A%20%20%20%20%7D%0A%20%20%20%20nodes%20%7B%0A%20%20%20%20%20%20full_name%0A%20%20%20%20%20%20description_%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A)
The [sqlite-utils](https://sqlite-utils.datasette.io/) Python library and CLI tool can be used to add full-text search to an existing database table.
### Columns containing JSON strings
If your table has a column that contains data encoded as JSON, `datasette-graphql` will make that column available as an encoded JSON string. Clients calling your API will need to parse the string as JSON in order to access the data.
You can return the data as a nested structure by configuring that column to be treated as a JSON column. The [plugin configuration](https://docs.datasette.io/en/stable/plugins.html#plugin-configuration) for that in `metadata.json` looks like this:
```json
{
""databases"": {
""test"": {
""tables"": {
""repos"": {
""plugins"": {
""datasette-graphql"": {
""json_columns"": [
""tags""
]
}
}
}
}
}
}
}
```
### Auto camelCase
The names of your columns and tables default to being matched by their representations in GraphQL.
If you have tables with `names_like_this` you may want to work with them in GraphQL using `namesLikeThis`, for consistency with GraphQL and JavaScript conventions.
You can turn on automatic camelCase using the `""auto_camelcase""` plugin configuration setting in `metadata.json`, like this:
```json
{
""plugins"": {
""datasette-graphql"": {
""auto_camelcase"": true
}
}
}
```
### CORS
This plugin obeys the `--cors` option passed to the `datasette` command-line tool. If you pass `--cors` it adds the following CORS HTTP headers to allow JavaScript running on other domains to access the GraphQL API:
access-control-allow-headers: content-type
access-control-allow-method: POST
access-control-allow-origin: *
### Execution limits
The plugin implements two limits by default:
- The total time spent executing all of the underlying SQL queries that make up the GraphQL execution must not exceed 1000ms (one second)
- The total number of SQL table queries executed as a result of nested GraphQL fields must not exceed 100
These limits can be customized using the `num_queries_limit` and `time_limit_ms` plugin configuration settings, for example in `metadata.json`:
```json
{
""plugins"": {
""datasette-graphql"": {
""num_queries_limit"": 200,
""time_limit_ms"": 5000
}
}
}
```
Setting these to `0` will disable the limit checks entirely.
## The graphql() template function
The plugin also makes a Jinja template function available called `graphql()`. You can use that function in your Datasette [custom templates](https://docs.datasette.io/en/stable/custom_templates.html#custom-templates) like so:
```html+jinja
{% set users = graphql(""""""
{
users {
nodes {
name
points
score
}
}
}
"""""")[""users""] %}
{% for user in users.nodes %}
{% endfor %}
```
The function executes a GraphQL query against the generated schema and returns the results. You can assign those results to a variable in your template and then loop through and display them.
By default the query will be run against the first attached database. You can use the optional second argument to the function to specify a different database - for example, to run against an attached `github.db` database you would do this:
```html+jinja
{% set user = graphql(""""""
{
users_row(id:9599) {
name
login
avatar_url
}
}
"""""", ""github"")[""users_row""] %}
Hello, {{ user.name }}
```
You can use [GraphQL variables](https://graphql.org/learn/queries/#variables) in these template calls by passing them to the `variables=` argument:
```html+jinja
{% set user = graphql(""""""
query ($id: Int) {
users_row(id: $id) {
name
login
avatar_url
}
}
"""""", database=""github"", variables={""id"": 9599})[""users_row""] %}
Hello, {{ user.name }}
```
## Adding custom fields with plugins
`datasette-graphql` adds a new [plugin hook](https://docs.datasette.io/en/stable/writing_plugins.html) to Datasette which can be used to add custom fields to your GraphQL schema.
The plugin hook looks like this:
```python
@hookimpl
def graphql_extra_fields(datasette, database):
""A list of (name, field_type) tuples to include in the GraphQL schema""
```
You can use this hook to return a list of tuples describing additional fields that should be exposed in your schema. Each tuple should consist of a string naming the new field, plus a [Graphene Field object](https://docs.graphene-python.org/en/latest/types/objecttypes/) that specifies the schema and provides a `resolver` function.
This example implementation uses `pkg_resources` to return a list of currently installed Python packages:
```python
import graphene
from datasette import hookimpl
import pkg_resources
@hookimpl
def graphql_extra_fields():
class Package(graphene.ObjectType):
""An installed package""
name = graphene.String()
version = graphene.String()
def resolve_packages(root, info):
return [
{""name"": d.project_name, ""version"": d.version}
for d in pkg_resources.working_set
]
return [
(
""packages"",
graphene.Field(
graphene.List(Package),
description=""List of installed packages"",
resolver=resolve_packages,
),
),
]
```
With this plugin installed, the following GraphQL query can be used to retrieve a list of installed packages:
```graphql
{
packages {
name
version
}
}
```
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-graphql
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-graphql
Datasette plugin providing an automatic GraphQL API for your SQLite databases
In this example query the underlying database table is called repos and its columns include id, full_name and description. Since description is a reserved word the query needs to ask for description_ instead.
Fetching a single record
If you only want to fetch a single record - for example if you want to fetch a row by its primary key - you can use the tablename_row field:
The tablename_row field accepts the primary key column (or columns) as arguments. It also supports the same filter:, search:, sort: and sort_desc: arguments as the tablename field, described below.
Accessing nested objects
If a column is a foreign key to another table, you can request columns from the table pointed to by that foreign key using a nested query like this:
If another table has a foreign key back to the table you are accessing, you can fetch rows from that related table.
Consider a users table which is related to repos - a repo has a foreign key back to the user that owns the repository. The users object type will have a repos_by_owner_list field which can be used to access those related repos:
You can filter the rows returned for a specific table using the filter: argument. This accepts a filter object mapping columns to operations. For example, to return just repositories with the Apache 2 license and more than 10 stars:
The where: argument can be used as an alternative to filter: when the thing you are expressing is too complex to be modeled using a filter expression. It accepts a string fragment of SQL that will be included in the WHERE clause of the SQL query.
{
repos(where: ""name='sqlite-utils' or name like 'datasette-%'"") {
totalCountnodes {
full_name
}
}
}
You can set a sort order for results from a table using the sort: or sort_desc: arguments. The value for this argument should be the name of the column you wish to sort (or sort-descending) by.
The totalCount field returns the total number of records that match the query.
Requesting the pageInfo.endCursor field provides you with the value you need to request the next page. You can pass this to the after: argument to request the next page.
The sqlite-utils Python library and CLI tool can be used to add full-text search to an existing database table.
Columns containing JSON strings
If your table has a column that contains data encoded as JSON, datasette-graphql will make that column available as an encoded JSON string. Clients calling your API will need to parse the string as JSON in order to access the data.
You can return the data as a nested structure by configuring that column to be treated as a JSON column. The plugin configuration for that in metadata.json looks like this:
The names of your columns and tables default to being matched by their representations in GraphQL.
If you have tables with names_like_this you may want to work with them in GraphQL using namesLikeThis, for consistency with GraphQL and JavaScript conventions.
You can turn on automatic camelCase using the ""auto_camelcase"" plugin configuration setting in metadata.json, like this:
This plugin obeys the --cors option passed to the datasette command-line tool. If you pass --cors it adds the following CORS HTTP headers to allow JavaScript running on other domains to access the GraphQL API:
access-control-allow-headers: content-type
access-control-allow-method: POST
access-control-allow-origin: *
Execution limits
The plugin implements two limits by default:
The total time spent executing all of the underlying SQL queries that make up the GraphQL execution must not exceed 1000ms (one second)
The total number of SQL table queries executed as a result of nested GraphQL fields must not exceed 100
These limits can be customized using the num_queries_limit and time_limit_ms plugin configuration settings, for example in metadata.json:
The function executes a GraphQL query against the generated schema and returns the results. You can assign those results to a variable in your template and then loop through and display them.
By default the query will be run against the first attached database. You can use the optional second argument to the function to specify a different database - for example, to run against an attached github.db database you would do this:
datasette-graphql adds a new plugin hook to Datasette which can be used to add custom fields to your GraphQL schema.
The plugin hook looks like this:
@hookimpldefgraphql_extra_fields(datasette, database):
""A list of (name, field_type) tuples to include in the GraphQL schema""
You can use this hook to return a list of tuples describing additional fields that should be exposed in your schema. Each tuple should consist of a string naming the new field, plus a Graphene Field object that specifies the schema and provides a resolver function.
This example implementation uses pkg_resources to return a list of currently installed Python packages:
With this plugin installed, the following GraphQL query can be used to retrieve a list of installed packages:
{
packages {
nameversion
}
}
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-graphql
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,0,
288629766,MDEwOlJlcG9zaXRvcnkyODg2Mjk3NjY=,datasette-schema-versions,simonw/datasette-schema-versions,0,9599,https://github.com/simonw/datasette-schema-versions,Datasette plugin that shows the schema version of every attached database,0,2020-08-19T04:04:39Z,2021-09-11T02:42:37Z,2021-09-11T02:44:32Z,,5,0,0,Python,1,1,1,1,0,0,0,0,0,,"[""datasette"", ""datasette-io"", ""datasette-plugin""]",0,0,0,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-schema-versions
[](https://pypi.org/project/datasette-schema-versions/)
[](https://github.com/simonw/datasette-schema-versions/releases)
[](https://github.com/simonw/datasette-schema-versions/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-schema-versions/blob/main/LICENSE)
Datasette plugin that shows the schema version of every attached database
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-schema-versions
## Usage
Visit `/-/schema-versions` on your Datasette instance to see a numeric version for the schema for each of your databases.
Any changes you make to the schema will increase this version number.
","
datasette-schema-versions
Datasette plugin that shows the schema version of every attached database
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-schema-versions
Usage
Visit /-/schema-versions on your Datasette instance to see a numeric version for the schema for each of your databases.
Any changes you make to the schema will increase this version number.
",,,,,,
294706267,MDEwOlJlcG9zaXRvcnkyOTQ3MDYyNjc=,datasette-seaborn,simonw/datasette-seaborn,0,9599,https://github.com/simonw/datasette-seaborn,Statistical visualizations for Datasette using Seaborn,0,2020-09-11T13:43:08Z,2022-03-22T01:49:39Z,2022-03-22T01:49:36Z,https://datasette-seaborn-demo.datasette.io/,24,11,11,Python,1,1,1,1,0,0,0,0,5,,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""seaborn"", ""visualization""]",0,5,11,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-seaborn
[](https://pypi.org/project/datasette-seaborn/)
[](https://github.com/simonw/datasette-seaborn/releases)
[](https://github.com/simonw/datasette-seaborn/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-seaborn/blob/main/LICENSE)
Statistical visualizations for Datasette using Seaborn
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-seaborn
## Usage
Navigate to the new `.seaborn` extension for any Datasette table.
The `_seaborn` argument specifies a method on `sns` to execute, e.g. `?_seaborn=relplot`.
Extra arguments to those methods can be specified using e.g. `&_seaborn_x=column_name`.
## Configuration
The plugin implements a default rendering time limit of five seconds. You can customize this limit using the `render_time_limit` setting, which accepts a floating point number of seconds. Add this to your `metadata.json`:
```json
{
""plugins"": {
""datasette-seaborn"": {
""render_time_limit"": 1.0
}
}
}
```
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-seaborn
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-seaborn
Statistical visualizations for Datasette using Seaborn
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-seaborn
Usage
Navigate to the new .seaborn extension for any Datasette table.
The _seaborn argument specifies a method on sns to execute, e.g. ?_seaborn=relplot.
Extra arguments to those methods can be specified using e.g. &_seaborn_x=column_name.
Configuration
The plugin implements a default rendering time limit of five seconds. You can customize this limit using the render_time_limit setting, which accepts a floating point number of seconds. Add this to your metadata.json:
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-seaborn
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
299143849,MDEwOlJlcG9zaXRvcnkyOTkxNDM4NDk=,datasette-dateutil,simonw/datasette-dateutil,0,9599,https://github.com/simonw/datasette-dateutil,dateutil functions for Datasette,0,2020-09-28T00:14:20Z,2022-03-01T00:09:57Z,2022-03-01T01:40:21Z,,18,6,6,Python,1,1,1,1,0,0,0,0,2,,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""dateutil""]",0,2,6,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-dateutil
[](https://pypi.org/project/datasette-dateutil/)
[](https://github.com/simonw/datasette-dateutil/releases)
[](https://github.com/simonw/datasette-dateutil/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-dateutil/blob/main/LICENSE)
dateutil functions for Datasette
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-dateutil
## Usage
This function adds custom SQL functions that expose functionality from the [dateutil](https://dateutil.readthedocs.io/) Python library.
Once installed, the following SQL functions become available:
### Parsing date strings
- `dateutil_parse(text)` - returns an ISO8601 date string parsed from the text, or `null` if the input could not be parsed. `dateutil_parse(""10 october 2020 3pm"")` returns `2020-10-10T15:00:00`.
- `dateutil_parse_fuzzy(text)` - same as `dateutil_parse()` but this also works against strings that contain a date somewhere within them - that date will be returned, or `null` if no dates could be found. `dateutil_parse_fuzzy(""This is due 10 september"")` returns `2020-09-10T00:00:00` (but will start returning the 2021 version of that if the year is 2021).
The `dateutil_parse()` and `dateutil_parse_fuzzy()` functions both follow the American convention of assuming that `1/2/2020` lists the month first, evaluating this example to the 2nd of January.
If you want to assume that the day comes first, use these two functions instead:
- `dateutil_parse_dayfirst(text)`
- `dateutil_parse_fuzzy_dayfirst(text)`
Here's a query demonstrating these functions:
```sql
select
dateutil_parse(""10 october 2020 3pm""),
dateutil_parse_fuzzy(""This is due 10 september""),
dateutil_parse(""1/2/2020""),
dateutil_parse(""2020-03-04""),
dateutil_parse_dayfirst(""2020-03-04"");
```
[Try that query](https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_parse%28%2210+october+2020+3pm%22%29%2C%0D%0A++dateutil_parse_fuzzy%28%22This+is+due+10+september%22%29%2C%0D%0A++dateutil_parse%28%221%2F2%2F2020%22%29%2C%0D%0A++dateutil_parse%28%222020-03-04%22%29%2C%0D%0A++dateutil_parse_dayfirst%28%222020-03-04%22%29%3B)
### Optional default dates
The `dateutil_parse()`, `dateutil_parse_fuzzy()`, `dateutil_parse_dayfirst()` and `dateutil_parse_fuzzy_dayfirst()` functions all accept an optional second argument specifying a ""default"" datetime to consider if some of the details are missing. For example, the following:
```sql
select dateutil_parse('1st october', '1985-01-01')
```
Will return `1985-10-01T00:00:00` - the missing year is replaced with the year from the default date.
[Example query demonstrating the default date argument](https://latest-with-plugins.datasette.io/fixtures?sql=with+times+as+%28%0D%0A++select%0D%0A++++datetime%28%27now%27%29+as+t%0D%0A++union%0D%0A++select%0D%0A++++datetime%28%27now%27%2C+%27-1+year%27%29%0D%0A++union%0D%0A++select%0D%0A++++datetime%28%27now%27%2C+%27-3+years%27%29%0D%0A%29%0D%0Aselect+t%2C+dateutil_parse_fuzzy%28%22This+is+due+10+september%22%2C+t%29+from+times)
### Calculating Easter
- `dateutil_easter(year)` - returns the date for Easter in that year, for example `dateutil_easter(""2020"")` returns `2020-04-12`.
[Example Easter query](https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_easter%282019%29%2C%0D%0A++dateutil_easter%282020%29%2C%0D%0A++dateutil_easter%282021%29)
### JSON arrays of dates
Several functions return JSON arrays of date strings. These can be used with SQLite's `json_each()` function to perform joins against dates from a specific date range or recurrence rule.
These functions can return up to 10,000 results. They will return an error if more than 10,000 dates would be returned - this is to protect against denial of service attacks.
- `dateutil_dates_between('1 january 2020', '5 jan 2020')` - given two dates (in any format that can be handled by `dateutil_parse()`) this function returns a JSON string containing the dates between those two days, inclusive. This example returns `[""2020-01-01"", ""2020-01-02"", ""2020-01-03"", ""2020-01-04"", ""2020-01-05""]`.
- `dateutil_dates_between('1 january 2020', '5 jan 2020', 0)` - set the optional third argument to `0` to specify that you would like this to be exclusive of the last day. This example returns `[""2020-01-01"", ""2020-01-02"", ""2020-01-03"", ""2020-01-04""]`.
[Try these queries](https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_dates_between%28%271+january+2020%27%2C+%275+jan+2020%27%29%2C%0D%0A++dateutil_dates_between%28%271+january+2020%27%2C+%275+jan+2020%27%2C+0%29)
The `dateutil_rrule()` and `dateutil_rrule_date()` functions accept the iCalendar standard ``rrule` format - see [the dateutil documentation](https://dateutil.readthedocs.io/en/stable/rrule.html#rrulestr-examples) for more examples.
This format lets you specify recurrence rules such as ""the next four last mondays of the month"".
- `dateutil_rrule(rrule, optional_dtsart)` - given an rrule returns a JSON array of ISO datetimes. The second argument is optional and will be treated as the start date for the rule.
- `dateutil_rrule_date(rrule, optional_dtsart)` - same as `dateutil_rrule()` but returns ISO dates.
Example query:
```sql
select
dateutil_rrule('FREQ=HOURLY;COUNT=5'),
dateutil_rrule_date(
'FREQ=DAILY;COUNT=3',
'1st jan 2020'
);
```
[Try the rrule example query](https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_rrule('FREQ%3DHOURLY%3BCOUNT%3D5')%2C%0D%0A++dateutil_rrule_date(%0D%0A++++'FREQ%3DDAILY%3BCOUNT%3D3'%2C%0D%0A++++'1st+jan+2020'%0D%0A++)%3B)
### Joining data using json_each()
SQLite's [json_each() function](https://www.sqlite.org/json1.html#jeach) can be used to turn a JSON array of dates into a table that can be joined against other data. Here's a query that returns a table showing every day in January 2019:
```sql
select
value as date
from
json_each(
dateutil_dates_between('1 Jan 2019', '31 Jan 2019')
)
```
[Try that query](https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++value+as+date%0D%0Afrom%0D%0A++json_each%28%0D%0A++++dateutil_dates_between%28%271+Jan+2019%27%2C+%2731+Jan+2019%27%29%0D%0A++%29)
You can run joins against this table by assigning it a name using SQLite's [support for Common Table Expressions (CTEs)](https://sqlite.org/lang_with.html).
This example query uses `substr(created, 0, 11)` to retrieve the date portion of the `created` column in the [facetable demo table](https://latest-with-plugins.datasette.io/fixtures/facetable), then joins that against the table of days in January to calculate the count of rows created on each day. The `LEFT JOIN` against `days_in_january` ensures that days which had no created records are still returned in the results, with a count of 0.
```sql
with created_dates as (
select
substr(created, 0, 11) as date
from
facetable
),
days_in_january as (
select
value as date
from
json_each(
dateutil_dates_between('1 Jan 2019', '31 Jan 2019')
)
)
select
days_in_january.date,
count(created_dates.date) as total
from
days_in_january
left join created_dates on days_in_january.date = created_dates.date
group by
days_in_january.date;
```
[Try that query](https://latest-with-plugins.datasette.io/fixtures?sql=with+created_dates+as+%28%0D%0A++select%0D%0A++++substr%28created%2C+0%2C+11%29+as+date%0D%0A++from%0D%0A++++facetable%0D%0A%29%2C%0D%0Adays_in_january+as+%28%0D%0A++select%0D%0A++++value+as+date%0D%0A++from%0D%0A++++json_each%28%0D%0A++++++dateutil_dates_between%28%271+Jan+2019%27%2C+%2731+Jan+2019%27%29%0D%0A++++%29%0D%0A%29%0D%0Aselect%0D%0A++days_in_january.date%2C%0D%0A++count%28created_dates.date%29+as+total%0D%0Afrom%0D%0A++days_in_january%0D%0A++left+join+created_dates+on+days_in_january.date+%3D+created_dates.date%0D%0Agroup+by%0D%0A++days_in_january.date%3B#g.mark=bar&g.x_column=date&g.x_type=ordinal&g.y_column=total&g.y_type=quantitative) with a bar chart rendered using the [datasette-vega](https://github.com/simonw/datasette-vega) plugin.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-dateutil
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-dateutil
dateutil functions for Datasette
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-dateutil
Usage
This function adds custom SQL functions that expose functionality from the dateutil Python library.
Once installed, the following SQL functions become available:
Parsing date strings
dateutil_parse(text) - returns an ISO8601 date string parsed from the text, or null if the input could not be parsed. dateutil_parse(""10 october 2020 3pm"") returns 2020-10-10T15:00:00.
dateutil_parse_fuzzy(text) - same as dateutil_parse() but this also works against strings that contain a date somewhere within them - that date will be returned, or null if no dates could be found. dateutil_parse_fuzzy(""This is due 10 september"") returns 2020-09-10T00:00:00 (but will start returning the 2021 version of that if the year is 2021).
The dateutil_parse() and dateutil_parse_fuzzy() functions both follow the American convention of assuming that 1/2/2020 lists the month first, evaluating this example to the 2nd of January.
If you want to assume that the day comes first, use these two functions instead:
dateutil_parse_dayfirst(text)
dateutil_parse_fuzzy_dayfirst(text)
Here's a query demonstrating these functions:
select
dateutil_parse(""10 october 2020 3pm""),
dateutil_parse_fuzzy(""This is due 10 september""),
dateutil_parse(""1/2/2020""),
dateutil_parse(""2020-03-04""),
dateutil_parse_dayfirst(""2020-03-04"");
The dateutil_parse(), dateutil_parse_fuzzy(), dateutil_parse_dayfirst() and dateutil_parse_fuzzy_dayfirst() functions all accept an optional second argument specifying a ""default"" datetime to consider if some of the details are missing. For example, the following:
Several functions return JSON arrays of date strings. These can be used with SQLite's json_each() function to perform joins against dates from a specific date range or recurrence rule.
These functions can return up to 10,000 results. They will return an error if more than 10,000 dates would be returned - this is to protect against denial of service attacks.
dateutil_dates_between('1 january 2020', '5 jan 2020') - given two dates (in any format that can be handled by dateutil_parse()) this function returns a JSON string containing the dates between those two days, inclusive. This example returns [""2020-01-01"", ""2020-01-02"", ""2020-01-03"", ""2020-01-04"", ""2020-01-05""].
dateutil_dates_between('1 january 2020', '5 jan 2020', 0) - set the optional third argument to 0 to specify that you would like this to be exclusive of the last day. This example returns [""2020-01-01"", ""2020-01-02"", ""2020-01-03"", ""2020-01-04""].
The dateutil_rrule() and dateutil_rrule_date() functions accept the iCalendar standard ``rrule` format - see the dateutil documentation for more examples.
This format lets you specify recurrence rules such as ""the next four last mondays of the month"".
dateutil_rrule(rrule, optional_dtsart) - given an rrule returns a JSON array of ISO datetimes. The second argument is optional and will be treated as the start date for the rule.
dateutil_rrule_date(rrule, optional_dtsart) - same as dateutil_rrule() but returns ISO dates.
Example query:
select
dateutil_rrule('FREQ=HOURLY;COUNT=5'),
dateutil_rrule_date(
'FREQ=DAILY;COUNT=3',
'1st jan 2020'
);
SQLite's json_each() function can be used to turn a JSON array of dates into a table that can be joined against other data. Here's a query that returns a table showing every day in January 2019:
select
value asdatefrom
json_each(
dateutil_dates_between('1 Jan 2019', '31 Jan 2019')
)
This example query uses substr(created, 0, 11) to retrieve the date portion of the created column in the facetable demo table, then joins that against the table of days in January to calculate the count of rows created on each day. The LEFT JOIN against days_in_january ensures that days which had no created records are still returned in the results, with a count of 0.
with created_dates as (
select
substr(created, 0, 11) asdatefrom
facetable
),
days_in_january as (
select
value asdatefrom
json_each(
dateutil_dates_between('1 Jan 2019', '31 Jan 2019')
)
)
selectdays_in_january.date,
count(created_dates.date) as total
from
days_in_january
left join created_dates ondays_in_january.date=created_dates.dategroup bydays_in_january.date;
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-dateutil
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
299198369,MDEwOlJlcG9zaXRvcnkyOTkxOTgzNjk=,datasette-import-table,simonw/datasette-import-table,0,9599,https://github.com/simonw/datasette-import-table,Datasette plugin for importing tables from other Datasette instances,0,2020-09-28T05:30:07Z,2022-06-09T15:27:33Z,2022-06-09T16:40:22Z,,20,0,0,Python,1,1,1,1,0,0,0,0,2,,"[""datasette"", ""datasette-io"", ""datasette-plugin""]",0,2,0,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-import-table
[](https://pypi.org/project/datasette-import-table/)
[](https://github.com/simonw/datasette-import-table/releases)
[](https://github.com/simonw/datasette-import-table/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-import-table/blob/main/LICENSE)
Datasette plugin for importing tables from other Datasette instances
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-import-table
## Usage
Visit `/-/import-table` for the interface. Paste in the URL to a table page on another Datasette instance and click the button to import that table.
By default only [the root actor](https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor) can access the page - so you'll need to run Datasette with the `--root` option and click on the link shown in the terminal to sign in and access the page.
The `import-table` permission governs access. You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to the write interface.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-import-table
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-import-table
Datasette plugin for importing tables from other Datasette instances
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-import-table
Usage
Visit /-/import-table for the interface. Paste in the URL to a table page on another Datasette instance and click the button to import that table.
By default only the root actor can access the page - so you'll need to run Datasette with the --root option and click on the link shown in the terminal to sign in and access the page.
The import-table permission governs access. You can use permission plugins such as datasette-permissions-sql to grant additional access to the write interface.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-import-table
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
312934001,MDEwOlJlcG9zaXRvcnkzMTI5MzQwMDE=,datasette-indieauth,simonw/datasette-indieauth,0,9599,https://github.com/simonw/datasette-indieauth,Datasette authentication using IndieAuth and RelMeAuth,0,2020-11-15T01:18:21Z,2022-10-25T01:00:43Z,2022-10-25T01:34:47Z,,51,8,8,Python,1,1,1,1,0,0,0,0,1,,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""indieauth""]",0,1,8,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,3,,,1,public,0,,0,
315796015,MDEwOlJlcG9zaXRvcnkzMTU3OTYwMTU=,datasette-ripgrep,simonw/datasette-ripgrep,0,9599,https://github.com/simonw/datasette-ripgrep,"Web interface for searching your code using ripgrep, built as a Datasette plugin",0,2020-11-25T01:26:36Z,2022-04-24T03:48:42Z,2022-06-30T22:45:03Z,https://ripgrep.datasette.io,55,58,58,Python,1,1,1,1,0,1,0,0,6,apache-2.0,"[""codesearch"", ""datasette"", ""datasette-io"", ""datasette-plugin"", ""ripgrep""]",1,6,58,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,1,3,"# datasette-ripgrep
[](https://pypi.org/project/datasette-ripgrep/)
[](https://github.com/simonw/datasette-ripgrep/releases)
[](https://github.com/simonw/datasette-ripgrep/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-ripgrep/blob/main/LICENSE)
Web interface for searching your code using [ripgrep](https://github.com/BurntSushi/ripgrep), built as a [Datasette](https://datasette.io/) plugin
For background on this project see [datasette-ripgrep: deploy a regular expression search engine for your source code](https://simonwillison.net/2020/Nov/28/datasette-ripgrep/).
## Demo
Try this plugin out at https://ripgrep.datasette.io/-/ripgrep - where you can run regular expression searches across the source code of Datasette and all of the `datasette-*` plugins belonging to the [simonw GitHub user](https://github.com/simonw).
Some example searches:
- [with.\*AsyncClient](https://ripgrep.datasette.io/-/ripgrep?pattern=with.*AsyncClient) - regular expression search for `with.*AsyncClient`
- [.plugin_config, literal=on](https://ripgrep.datasette.io/-/ripgrep?pattern=.plugin_config\(&literal=on) - a non-regular expression search for `.plugin_config(`
- [with.\*AsyncClient glob=datasette/\*\*](https://ripgrep.datasette.io/-/ripgrep?pattern=with.*AsyncClient&glob=datasette%2F%2A%2A) - search for that pattern only within the `datasette/` top folder
- [""sqlite-utils\["">\] glob=setup.py](https://ripgrep.datasette.io/-/ripgrep?pattern=%22sqlite-utils%5B%22%3E%5D&glob=setup.py) - a regular expression search for packages that depend on either `sqlite-utils` or `sqlite-utils>=some-version`
- [test glob=!\*.html](https://ripgrep.datasette.io/-/ripgrep?pattern=test&glob=%21*.html) - search for the string `test` but exclude results in HTML files
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-ripgrep
The `rg` executable needs to be [installed](https://github.com/BurntSushi/ripgrep/blob/master/README.md#installation) such that it can be run by this tool.
## Usage
This plugin requires configuration: it needs to a `path` setting so that it knows where to run searches.
Create a `metadata.json` file that looks like this:
```json
{
""plugins"": {
""datasette-ripgrep"": {
""path"": ""/path/to/your/files""
}
}
}
```
Now run Datasette using `datasette -m metadata.json`. The plugin will add an interface at `/-/ripgrep` for running searches.
## Plugin configuration
The `""path""` configuration is required. Optional extra configuration options are:
- `time_limit` - floating point number. The `rg` process will be terminated if it takes longer than this limit. The default is one second, `1.0`.
- `max_lines` - integer. The `rg` process will be terminated if it returns more than this number of lines. The default is `2000`.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-ripgrep
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-ripgrep
Web interface for searching your code using ripgrep, built as a Datasette plugin
Now run Datasette using datasette -m metadata.json. The plugin will add an interface at /-/ripgrep for running searches.
Plugin configuration
The ""path"" configuration is required. Optional extra configuration options are:
time_limit - floating point number. The rg process will be terminated if it takes longer than this limit. The default is one second, 1.0.
max_lines - integer. The rg process will be terminated if it returns more than this number of lines. The default is 2000.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-ripgrep
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,0,
331151708,MDEwOlJlcG9zaXRvcnkzMzExNTE3MDg=,datasette-leaflet-freedraw,simonw/datasette-leaflet-freedraw,0,9599,https://github.com/simonw/datasette-leaflet-freedraw,Draw polygons on maps in Datasette,0,2021-01-20T00:55:03Z,2021-12-17T22:07:50Z,2022-02-03T20:24:37Z,,1177,9,9,Python,1,1,1,1,0,2,0,0,2,,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""leafletjs""]",2,2,9,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,2,2,"# datasette-leaflet-freedraw
[](https://pypi.org/project/datasette-leaflet-freedraw/)
[](https://github.com/simonw/datasette-leaflet-freedraw/releases)
[](https://github.com/simonw/datasette-leaflet-freedraw/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-leaflet-freedraw/blob/main/LICENSE)
Draw polygons on maps in Datasette
Project background: [Drawing shapes on a map to query a SpatiaLite database](https://simonwillison.net/2021/Jan/24/drawing-shapes-spatialite/).
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-leaflet-freedraw
## Usage
If a table has a SpatiaLite `geometry` column, the plugin will add a map interface to the table page allowing users to draw a shape on the map to find rows with a geometry that intersects that shape.
The plugin can also work with arbitrary SQL queries. There it looks for input fields with a name of `freedraw` or that ends in `_freedraw` and replaces them with a map interface.
The map interface uses the [FreeDraw](https://freedraw.herokuapp.com/) Leaflet plugin.
## Demo
You can try out this plugin to run searches against the GreenInfo Network California Protected Areas Database. Here's [an example query](https://calands.datasettes.com/calands?sql=select%0D%0A++AsGeoJSON%28geometry%29%2C+*%0D%0Afrom%0D%0A++CPAD_2020a_SuperUnits%0D%0Awhere%0D%0A++PARK_NAME+like+%27%25mini%25%27+and%0D%0A++Intersects%28GeomFromGeoJSON%28%3Afreedraw%29%2C+geometry%29+%3D+1%0D%0A++and+CPAD_2020a_SuperUnits.rowid+in+%28%0D%0A++++select%0D%0A++++++rowid%0D%0A++++from%0D%0A++++++SpatialIndex%0D%0A++++where%0D%0A++++++f_table_name+%3D+%27CPAD_2020a_SuperUnits%27%0D%0A++++++and+search_frame+%3D+GeomFromGeoJSON%28%3Afreedraw%29%0D%0A++%29&freedraw=%7B%22type%22%3A%22MultiPolygon%22%2C%22coordinates%22%3A%5B%5B%5B%5B-122.42202758789064%2C37.82280243352759%5D%2C%5B-122.39868164062501%2C37.823887203271454%5D%2C%5B-122.38220214843751%2C37.81846319511331%5D%2C%5B-122.35061645507814%2C37.77071473849611%5D%2C%5B-122.34924316406251%2C37.74465712069939%5D%2C%5B-122.37258911132814%2C37.703380457832374%5D%2C%5B-122.39044189453125%2C37.690340943717715%5D%2C%5B-122.41241455078126%2C37.680559803205135%5D%2C%5B-122.44262695312501%2C37.67295135774715%5D%2C%5B-122.47283935546876%2C37.67295135774715%5D%2C%5B-122.52502441406251%2C37.68382032669382%5D%2C%5B-122.53463745117189%2C37.6892542140253%5D%2C%5B-122.54699707031251%2C37.690340943717715%5D%2C%5B-122.55798339843751%2C37.72945260537781%5D%2C%5B-122.54287719726564%2C37.77831314799672%5D%2C%5B-122.49893188476564%2C37.81303878836991%5D%2C%5B-122.46185302734376%2C37.82822612280363%5D%2C%5B-122.42889404296876%2C37.82822612280363%5D%2C%5B-122.42202758789064%2C37.82280243352759%5D%5D%5D%5D%7D) showing mini parks in San Francisco:
```sql
select
AsGeoJSON(geometry), *
from
CPAD_2020a_SuperUnits
where
PARK_NAME like '%mini%' and
Intersects(GeomFromGeoJSON(:freedraw), geometry) = 1
and CPAD_2020a_SuperUnits.rowid in (
select
rowid
from
SpatialIndex
where
f_table_name = 'CPAD_2020a_SuperUnits'
and search_frame = GeomFromGeoJSON(:freedraw)
)
```
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-leaflet-freedraw
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
Install this plugin in the same environment as Datasette.
$ datasette install datasette-leaflet-freedraw
Usage
If a table has a SpatiaLite geometry column, the plugin will add a map interface to the table page allowing users to draw a shape on the map to find rows with a geometry that intersects that shape.
The plugin can also work with arbitrary SQL queries. There it looks for input fields with a name of freedraw or that ends in _freedraw and replaces them with a map interface.
The map interface uses the FreeDraw Leaflet plugin.
Demo
You can try out this plugin to run searches against the GreenInfo Network California Protected Areas Database. Here's an example query showing mini parks in San Francisco:
select
AsGeoJSON(geometry), *from
CPAD_2020a_SuperUnits
where
PARK_NAME like'%mini%'and
Intersects(GeomFromGeoJSON(:freedraw), geometry) =1andCPAD_2020a_SuperUnits.rowidin (
select
rowid
from
SpatialIndex
where
f_table_name ='CPAD_2020a_SuperUnits'and search_frame = GeomFromGeoJSON(:freedraw)
)
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-leaflet-freedraw
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
335175637,MDEwOlJlcG9zaXRvcnkzMzUxNzU2Mzc=,datasette-tiles,simonw/datasette-tiles,0,9599,https://github.com/simonw/datasette-tiles,"Mapping tile server for Datasette, serving tiles from MBTiles packages",0,2021-02-02T05:11:12Z,2022-03-22T01:52:30Z,2022-03-22T01:52:27Z,https://datasette.io/plugins/datasette-tiles,54,4,4,Python,1,1,1,1,0,4,0,0,8,,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""mbtiles""]",4,8,4,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,4,3,"# datasette-tiles
[](https://pypi.org/project/datasette-tiles/)
[](https://github.com/simonw/datasette-tiles/releases)
[](https://github.com/simonw/datasette-tiles/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-tiles/blob/main/LICENSE)
Datasette plugin for serving MBTiles map tiles
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-tiles
## Demo
You can try this plugin out at https://datasette-tiles-demo.datasette.io/-/tiles
## Usage
This plugin scans all database files connected to Datasette to see if any of them are valid MBTiles databases.
It can then serve tiles from those databases at the following URL:
/-/tiles/db-name/zoom/x/y.png
An example map for each database demonstrating the configured minimum and maximum zoom for that database can be found at `/-/tiles/db-name` - this can also be accessed via the table and database action menus for that database.
Visit `/-/tiles` for an index page of attached valid databases.
You can install the [datasette-basemap](https://datasette.io/plugins/datasette-basemap) plugin to get a `basemap` default set of tiles, handling zoom levels 0 to 6 using OpenStreetMap.
### Tile coordinate systems
There are two tile coordinate systems in common use for online maps. The first is used by OpenStreetMap and Google Maps, the second is from a specification called [Tile Map Service](https://en.wikipedia.org/wiki/Tile_Map_Service), or TMS.
Both systems use three components: `z/x/y` - where `z` is the zoom level, `x` is the column and `y` is the row.
The difference is in the way the `y` value is counted. OpenStreetMap has y=0 at the top. TMS has y=0 at the bottom.
An illustrative example: at zoom level 2 the map is divided into 16 total tiles. The OpenStreetMap scheme numbers them like so:
0/0 1/0 2/0 3/0
0/1 1/1 2/1 3/1
0/2 1/2 2/2 3/2
0/3 1/3 2/3 3/3
The TMS scheme looks like this:
0/3 1/3 2/3 3/3
0/2 1/2 2/2 3/2
0/1 1/1 2/1 3/1
0/0 1/0 2/0 3/0
`datasette-tiles` can serve tiles using either of these standards. For the OpenStreetMap / Google Maps 0-at-the-top system, use the following URL:
/-/tiles/database-name/{z}/{x}/{y}.png
For the TMS 0-at-the-bottom system, use this:
/-/tiles-tms/database-name/{z}/{x}/{y}.png
### Configuring a Leaflet tile layer
The following JavaScript will configure a [Leaflet TileLayer](https://leafletjs.com/reference-1.7.1.html#tilelayer) for use with this plugin:
```javascript
var tiles = leaflet.tileLayer(""/-/tiles/basemap/{z}/{x}/{y}.png"", {
minZoom: 0,
maxZoom: 6,
attribution: ""\u00a9 OpenStreetMap contributors""
});
```
### Tile stacks
`datasette-tiles` can be configured to serve tiles from multiple attached MBTiles files, searching each database in order for a tile and falling back to the next in line if that tile is not found.
For a demo of this in action, visit https://datasette-tiles-demo.datasette.io/-/tiles-stack and zoom in on Japan. It should start showing [Stamen's Toner map](maps.stamen.com) of Japan once you get to zoom level 6 and 7.
The `/-/tiles-stack/{z}/{x}/{y}.png` endpoint provides this feature.
If you start Datasette like this:
datasette world.mbtiles country.mbtiles city1.mbtiles city2.mbtiles
Any requests for a tile from the `/-/tiles-stack` path will first check the `city2` database, than `city1`, then `country`, then `world`.
If you have the [datasette-basemap](https://datasette.io/plugins/datasette-basemap) plugin installed it will be given special treatment: the `basemap` database will always be the last database checked for a tile.
Rather than rely on the order in which databases were attached, you can instead configure an explicit order using the `tiles-stack-order` plugin setting. Add the following to your `metadata.json` file:
```json
{
""plugins"": {
""datasette-tiles"": {
""tiles-stack-order"": [""world"", ""country""]
}
}
}
```
You can then run Datasette like this:
datasette -m metadata.json country.mbtiles world.mbtiles
This endpoint serves tiles using the OpenStreetMap / Google Maps coordinate system. To load tiles using the TMS coordinate system use this endpoint instead:
/-/tiles-stack-tms/{z}/{x}/{y}.png
### Retina tiles
Retina (double resolution) tiles are supported by `datasette-tiles` if the MBTiles database file contains 512x512 tile images as opposed to the default of 256x256. JavaScript libraries such as Leaflet will serve these tiles with a fixed 256x256 size, which will cause them to be displayed correctly by capable operating systems.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-tiles
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-tiles
Datasette plugin for serving MBTiles map tiles
Installation
Install this plugin in the same environment as Datasette.
This plugin scans all database files connected to Datasette to see if any of them are valid MBTiles databases.
It can then serve tiles from those databases at the following URL:
/-/tiles/db-name/zoom/x/y.png
An example map for each database demonstrating the configured minimum and maximum zoom for that database can be found at /-/tiles/db-name - this can also be accessed via the table and database action menus for that database.
Visit /-/tiles for an index page of attached valid databases.
You can install the datasette-basemap plugin to get a basemap default set of tiles, handling zoom levels 0 to 6 using OpenStreetMap.
Tile coordinate systems
There are two tile coordinate systems in common use for online maps. The first is used by OpenStreetMap and Google Maps, the second is from a specification called Tile Map Service, or TMS.
Both systems use three components: z/x/y - where z is the zoom level, x is the column and y is the row.
The difference is in the way the y value is counted. OpenStreetMap has y=0 at the top. TMS has y=0 at the bottom.
An illustrative example: at zoom level 2 the map is divided into 16 total tiles. The OpenStreetMap scheme numbers them like so:
datasette-tiles can be configured to serve tiles from multiple attached MBTiles files, searching each database in order for a tile and falling back to the next in line if that tile is not found.
Any requests for a tile from the /-/tiles-stack path will first check the city2 database, than city1, then country, then world.
If you have the datasette-basemap plugin installed it will be given special treatment: the basemap database will always be the last database checked for a tile.
Rather than rely on the order in which databases were attached, you can instead configure an explicit order using the tiles-stack-order plugin setting. Add the following to your metadata.json file:
This endpoint serves tiles using the OpenStreetMap / Google Maps coordinate system. To load tiles using the TMS coordinate system use this endpoint instead:
/-/tiles-stack-tms/{z}/{x}/{y}.png
Retina tiles
Retina (double resolution) tiles are supported by datasette-tiles if the MBTiles database file contains 512x512 tile images as opposed to the default of 256x256. JavaScript libraries such as Leaflet will serve these tiles with a fixed 256x256 size, which will cause them to be displayed correctly by capable operating systems.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-tiles
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
361014273,MDEwOlJlcG9zaXRvcnkzNjEwMTQyNzM=,datasette-dashboards,rclement/datasette-dashboards,0,1238873,https://github.com/rclement/datasette-dashboards,Datasette plugin providing data dashboards from metadata,0,2021-04-23T21:56:48Z,2022-09-21T13:03:39Z,2022-10-07T07:18:03Z,https://datasette-dashboards-demo.vercel.app,1746,74,74,Python,1,1,1,1,0,3,0,0,3,apache-2.0,"[""dashboards"", ""data-visualization"", ""datasette"", ""datasette-io"", ""datasette-plugin"", ""sql"", ""vega-lite""]",3,3,74,master,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,3,1,"# datasette-dashboards
> Datasette plugin providing data dashboards from metadata
[](https://pypi.org/project/datasette-dashboards/)
[](https://github.com/rclement/datasette-dashboards/actions/workflows/ci-cd.yml)
[](https://codecov.io/gh/rclement/datasette-dashboards)
[](https://github.com/rclement/datasette-dashboards/blob/master/LICENSE)
Try out a live demo at [https://datasette-dashboards-demo.vercel.app](https://datasette-dashboards-demo.vercel.app/-/dashboards)
**WARNING**: this plugin is still experimental and not ready for production.
Some breaking changes might happen between releases before reaching a stable version.
Use it at your own risks!
## Installation
Install this plugin in the same environment as Datasette:
```bash
$ datasette install datasette-dashboards
```
## Usage
Define dashboards within `metadata.yml` / `metadata.json`:
```yaml
plugins:
datasette-dashboards:
my-dashboard:
title: My Dashboard
description: Showing some nice metrics
layout:
- [analysis-note, events-count]
- [analysis-note, events-source]
filters:
date_start:
name: Date Start
type: date
default: ""2021-01-01""
date_end:
name: Date End
type: date
charts:
analysis-note:
library: markdown
display: |-
# Analysis notes
> A quick rundown of events statistics and KPIs
events-count:
title: Total number of events
db: jobs
query: SELECT count(*) as count FROM events
library: metric
display:
field: count
prefix:
suffix:
events-source:
title: Number of events by source
db: jobs
query: SELECT source, count(*) as count FROM events WHERE TRUE [[ AND date >= date(:date_start) ]] [[ AND date <= date(:date_end) ]] GROUP BY source ORDER BY count DESC
library: vega
display:
mark: { type: bar, tooltip: true }
encoding:
color: { field: source, type: nominal }
theta: { field: count, type: quantitative }
```
A new menu entry is now available, pointing at `/-/dashboards` to access all defined dashboards.
### Properties
Dashboard properties:
| Property | Type | Description |
| ------------- | -------- | --------------------- |
| `title` | `string` | Dashboard title |
| `description` | `string` | Dashboard description |
| `layout` | `array` | Dashboard layout |
| `filters` | `object` | Dashboard filters |
Dashboard filters:
| Property | Type | Description |
| --------- | ------------------ | -------------------------------------- |
| `name` | `string` | Filter display name |
| `type` | `string` | Filter type (`text`, `date`, `number`) |
| `default` | `string`, `number` | (optional) Filter default value |
| `min` | `number` | (optional) Filter minimum value |
| `max` | `number` | (optional) Filter maximum value |
| `step` | `number` | (optional) Filter stepping value |
Common chart properties for all chart types:
| Property | Type | Description |
| --------- | -------- | -------------------------------------------------------- |
| `title` | `string` | Chart title |
| `db` | `string` | Database name against which to run the query |
| `query` | `string` | SQL query to run and extract data from |
| `library` | `string` | One of supported libraries: `vega`, `markdown`, `metric` |
| `display` | `object` | Chart display specification (depend on the used library) |
To define SQL queries using dashboard filters:
```sql
SELECT * FROM mytable [[ WHERE col >= :my_filter ]]
```
```sql
SELECT * FROM mytable WHERE TRUE [[ AND col1 = :my_filter_1 ]] [[ AND col2 = :my_filter_2 ]]
```
#### Vega properties
Available configuration for `vega` charts:
| Property | Type | Description |
| --------- | -------- | ------------------------- |
| `library` | `string` | Must be set to `vega` |
| `display` | `object` | Vega specification object |
Notes about the `display` property:
- Requires a valid [Vega specification object](https://vega.github.io/vega-lite/docs/)
- Some fields are pre-defined: `$schema`, `title`, `width`, `view`, `config`, `data`
- All fields are passed along as-is (overriding pre-defined fields if any)
- Only `mark` and `encoding` fields are required as the bare-minimum
#### Markdown properties
Available configuration for `markdown` chart:
| Property | Type | Description |
| --------- | -------- | ------------------------------------------------- |
| `library` | `string` | Must be set to `markdown` |
| `display` | `string` | Multi-line string containing the Markdown content |
Note :
- Some common properties do not apply and can be omitted: `title`, `db`, `query`
- Markdown rendering is done by [`datasette-render-markdown`](https://datasette.io/plugins/datasette-render-markdown)
- To configure Markdown rendering, extensions can be enabled in [metadata](https://datasette.io/plugins/datasette-render-markdown#user-content-markdown-extensions)
#### Metric properties
Available configuration for `metric` chart:
| Property | Type | Description |
| ---------------- | -------- | ----------------------------------------- |
| `library` | `string` | Must be set to `metric` |
| `display.field` | `string` | Numerical field to be displayed as metric |
| `display.prefix` | `string` | Prefix to be displayed before metric |
| `display.suffix` | `string` | Prefix to be displayed after metric |
Note:
- The `display.field` must reference a single-numerical value from the SQL query
(e.g. numerical `number` field in `SELECT count(*) as number FROM events`)
### Dashboard layout
The default dashboard layout will present two charts per row (one per row on mobile).
To make use of custom dashboard layout using [CSS Grid Layout](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Grid_Layout),
define the `layout` array property as a grid / matrix:
- Each entry represents a row of charts
- Each column is referring a chart by its property name
## Development
To set up this plugin locally, first checkout the code.
Then create a new virtual environment and the required dependencies:
```bash
pipenv install -d
pipenv shell
```
To run the tests:
```bash
pytest
```
## Demo
With the developmnent environment setup, you can run the demo locally:
```bash
datasette --metadata demo/metadata.yml demo/jobs.db
```
## License
Licensed under Apache License, Version 2.0
Copyright (c) 2021 - present Romain Clement
","
datasette-dashboards
Datasette plugin providing data dashboards from metadata
WARNING: this plugin is still experimental and not ready for production.
Some breaking changes might happen between releases before reaching a stable version.
Use it at your own risks!
Installation
Install this plugin in the same environment as Datasette:
$ datasette install datasette-dashboards
Usage
Define dashboards within metadata.yml / metadata.json:
plugins:
datasette-dashboards:
my-dashboard:
title: My Dashboarddescription: Showing some nice metricslayout:
- [analysis-note, events-count]
- [analysis-note, events-source]filters:
date_start:
name: Date Starttype: datedefault: ""2021-01-01""date_end:
name: Date Endtype: datecharts:
analysis-note:
library: markdowndisplay: |- # Analysis notes > A quick rundown of events statistics and KPIsevents-count:
title: Total number of eventsdb: jobsquery: SELECT count(*) as count FROM eventslibrary: metricdisplay:
field: countprefix:
suffix:
events-source:
title: Number of events by sourcedb: jobsquery: SELECT source, count(*) as count FROM events WHERE TRUE [[ AND date >= date(:date_start) ]] [[ AND date <= date(:date_end) ]] GROUP BY source ORDER BY count DESClibrary: vegadisplay:
mark: { type: bar, tooltip: true }encoding:
color: { field: source, type: nominal }theta: { field: count, type: quantitative }
A new menu entry is now available, pointing at /-/dashboards to access all defined dashboards.
Properties
Dashboard properties:
Property
Type
Description
title
string
Dashboard title
description
string
Dashboard description
layout
array
Dashboard layout
filters
object
Dashboard filters
Dashboard filters:
Property
Type
Description
name
string
Filter display name
type
string
Filter type (text, date, number)
default
string, number
(optional) Filter default value
min
number
(optional) Filter minimum value
max
number
(optional) Filter maximum value
step
number
(optional) Filter stepping value
Common chart properties for all chart types:
Property
Type
Description
title
string
Chart title
db
string
Database name against which to run the query
query
string
SQL query to run and extract data from
library
string
One of supported libraries: vega, markdown, metric
display
object
Chart display specification (depend on the used library)
To define SQL queries using dashboard filters:
SELECT*FROM mytable [[ WHERE col >= :my_filter ]]
SELECT*FROM mytable WHERE TRUE [[ AND col1 = :my_filter_1 ]] [[ AND col2 = :my_filter_2 ]]
To configure Markdown rendering, extensions can be enabled in metadata
Metric properties
Available configuration for metric chart:
Property
Type
Description
library
string
Must be set to metric
display.field
string
Numerical field to be displayed as metric
display.prefix
string
Prefix to be displayed before metric
display.suffix
string
Prefix to be displayed after metric
Note:
The display.field must reference a single-numerical value from the SQL query
(e.g. numerical number field in SELECT count(*) as number FROM events)
Dashboard layout
The default dashboard layout will present two charts per row (one per row on mobile).
To make use of custom dashboard layout using CSS Grid Layout,
define the layout array property as a grid / matrix:
Each entry represents a row of charts
Each column is referring a chart by its property name
Development
To set up this plugin locally, first checkout the code.
Then create a new virtual environment and the required dependencies:
pipenv install -d
pipenv shell
To run the tests:
pytest
Demo
With the developmnent environment setup, you can run the demo locally:
",1,public,0,,0,
399308604,MDEwOlJlcG9zaXRvcnkzOTkzMDg2MDQ=,datasette-app,simonw/datasette-app,0,9599,https://github.com/simonw/datasette-app,The Datasette macOS application,0,2021-08-24T02:21:37Z,2022-11-15T18:57:26Z,2022-09-09T04:55:47Z,https://datasette.io/desktop,897,92,92,JavaScript,1,1,1,1,0,6,0,0,32,,"[""datasette""]",6,32,92,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,6,8,,,1,public,0,,0,1
400678317,MDEwOlJlcG9zaXRvcnk0MDA2NzgzMTc=,datasette-verify,simonw/datasette-verify,0,9599,https://github.com/simonw/datasette-verify,Verify that files can be opened by Datasette,0,2021-08-28T01:59:12Z,2021-08-28T02:37:03Z,2021-08-28T02:31:34Z,https://datasette.io/tools/datasette-verify,0,1,1,Python,1,1,1,1,0,0,0,0,0,,"[""datasette"", ""datasette-io"", ""datasette-plugin""]",0,0,1,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-verify
[](https://pypi.org/project/datasette-verify/)
[](https://github.com/simonw/datasette-verify/releases)
[](https://github.com/simonw/datasette-verify/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-verify/blob/main/LICENSE)
Verify that SQLite files can be opened using Datasette
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-verify
This plugin depends on [Datasette 0.59a2](https://github.com/simonw/datasette/releases/tag/0.59a2) or higher, as it uses the [register_commands()](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hook-register-commands) plugin hook.
## Usage
To confirm that files can be opened by Datasette, run the following:
datasette verify file1.db file2.db
You can pass one or more file paths.
The command will exit silently with a 0 exit code if the files are all valid SQLite databases that Datasette can open.
It will exit with a 1 exit code and display an error for the first file it finds that is not valid.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-verify
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-verify
Verify that SQLite files can be opened using Datasette
Installation
Install this plugin in the same environment as Datasette.
To confirm that files can be opened by Datasette, run the following:
datasette verify file1.db file2.db
You can pass one or more file paths.
The command will exit silently with a 0 exit code if the files are all valid SQLite databases that Datasette can open.
It will exit with a 1 exit code and display an error for the first file it finds that is not valid.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-verify
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
409678203,R_kgDOGGsxew,datasette-template-request,simonw/datasette-template-request,0,9599,https://github.com/simonw/datasette-template-request,Expose the Datasette request object to custom templates,0,2021-09-23T17:07:00Z,2021-09-23T17:29:08Z,2021-09-23T17:29:36Z,https://datasette.io/plugins/datasette-template-request,0,0,0,Python,1,1,1,1,0,0,0,0,0,,"[""datasette"", ""datasette-io"", ""datasette-plugin""]",0,0,0,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-template-request
[](https://pypi.org/project/datasette-template-request/)
[](https://github.com/simonw/datasette-template-request/releases)
[](https://github.com/simonw/datasette-template-request/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-template-request/blob/main/LICENSE)
Expose the Datasette request object to custom templates
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-template-request
## Usage
Once this plugin is installed, Datasette [custom templates](https://docs.datasette.io/en/stable/custom_templates.html) can use `{{ request }}` to access the current [request object](https://docs.datasette.io/en/stable/internals.html#request-object). For example, to access `?name=Cleo` in the query string a template could use this:
Name: {{ request.args.name }}
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-template-request
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-template-request
Expose the Datasette request object to custom templates
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-template-request
Usage
Once this plugin is installed, Datasette custom templates can use {{ request }} to access the current request object. For example, to access ?name=Cleo in the query string a template could use this:
Name: {{ request.args.name }}
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-template-request
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",1,,,,,
430224716,R_kgDOGaS1TA,datasette-redirect-to-https,simonw/datasette-redirect-to-https,0,9599,https://github.com/simonw/datasette-redirect-to-https,Datasette plugin that redirects all non-https requests to https,0,2021-11-20T22:43:33Z,2022-04-24T03:48:01Z,2022-07-07T17:38:32Z,,12,1,1,Python,1,1,1,1,0,0,0,0,0,,"[""asgi"", ""datasette"", ""datasette-io"", ""datasette-plugin""]",0,0,1,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-redirect-to-https
[](https://pypi.org/project/datasette-redirect-to-https/)
[](https://github.com/simonw/datasette-redirect-to-https/releases)
[](https://github.com/simonw/datasette-redirect-to-https/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-redirect-to-https/blob/main/LICENSE)
Datasette plugin that redirects all non-https requests to https
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-redirect-to-https
## Usage
Once installed, incoming GET requests to the `http://` protocol will be 301 redirected to the `https://` equivalent page.
HTTP verbs other than GET will get a 405 Method Not Allowed HTTP error.
## Configuration
Some hosting providers handle HTTPS for you, passing requests back to your application server over HTTP.
For this plugin to work correctly, it needs to detect that the original incoming request came in over HTTP.
Hosting providers like this often set an additional HTTP header such as `x-forwarded-proto: http` identifying the original protocol.
You can configure `datasette-redirect-to-https` to respect this header using the following plugin configuration in `metadata.json`:
```json
{
""plugins"": {
""datasette-redirect-to-https"": {
""if_headers"": {
""x-forwarded-proto"": ""http""
}
}
}
}
```
The above example will redirect to `https://` if the incoming request has a `x-forwarded-proto: http` request header.
If multiple `if_headers` are listed, the redirect will occur if any of them match.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-redirect-to-https
python3 -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-redirect-to-https
Datasette plugin that redirects all non-https requests to https
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-redirect-to-https
Usage
Once installed, incoming GET requests to the http:// protocol will be 301 redirected to the https:// equivalent page.
HTTP verbs other than GET will get a 405 Method Not Allowed HTTP error.
Configuration
Some hosting providers handle HTTPS for you, passing requests back to your application server over HTTP.
For this plugin to work correctly, it needs to detect that the original incoming request came in over HTTP.
Hosting providers like this often set an additional HTTP header such as x-forwarded-proto: http identifying the original protocol.
You can configure datasette-redirect-to-https to respect this header using the following plugin configuration in metadata.json:
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-pretty-traces
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
474468776,R_kgDOHEfRqA,datasette-auth0,simonw/datasette-auth0,0,9599,https://github.com/simonw/datasette-auth0,Datasette plugin that authenticates users using Auth0,0,2022-03-26T21:19:31Z,2022-03-27T17:59:49Z,2022-03-28T03:04:52Z,,11,3,3,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""auth0"", ""datasette"", ""datasette-plugin""]",0,0,3,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-auth0
[](https://pypi.org/project/datasette-auth0/)
[](https://github.com/simonw/datasette-auth0/releases)
[](https://github.com/simonw/datasette-auth0/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-auth0/blob/main/LICENSE)
Datasette plugin that authenticates users using [Auth0](https://auth0.com/)
See [Simplest possible OAuth authentication with Auth0](https://til.simonwillison.net/auth0/oauth-with-auth0) for more about how this plugin works.
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-auth0
## Demo
You can try this out at [datasette-auth0-demo.datasette.io](https://datasette-auth0-demo.datasette.io/) - click on the top right menu icon and select ""Sign in with Auth0"".
## Initial configuration
First, create a new application in Auth0. You will need the domain, client ID and client secret for that application.
The domain should be something like `mysite.us.auth0.com`.
Add `http://127.0.0.1:8001/-/auth0-callback` to the list of Allowed Callback URLs.
Then configure these plugin secrets using `metadata.yml`:
```yaml
plugins:
datasette-auth0:
domain:
""$env"": AUTH0_DOMAIN
client_id:
""$env"": AUTH0_CLIENT_ID
client_secret:
""$env"": AUTH0_CLIENT_SECRET
```
Only the `client_secret` needs to be kept secret, but for consistency I recommend using the `$env` mechanism for all three.
In development, you can run Datasette and pass in environment variables like this:
```
AUTH0_DOMAIN=""your-domain.us.auth0.com"" \
AUTH0_CLIENT_ID=""...client-id-goes-here..."" \
AUTH0_CLIENT_SECRET=""...secret-goes-here..."" \
datasette -m metadata.yml
```
If you are deploying using `datasette publish` you can pass these using `--plugin-secret`. For example, to deploy using Cloud Run you might run the following:
```
datasette publish cloudrun mydatabase.db \
--install datasette-auth0 \
--plugin-secret datasette-auth0 domain ""your-domain.us.auth0.com"" \
--plugin-secret datasette-auth0 client_id ""your-client-id"" \
--plugin-secret datasette-auth0 client_secret ""your-client-secret"" \
--service datasette-auth0-demo
```
Once your Datasette instance is deployed, you will need to add its callback URL to the ""Allowed Callback URLs"" list in Auth0.
The callback URL should be something like:
https://url-to-your-datasette/-/auth0-callback
## Usage
Once installed, a ""Sign in with Auth0"" menu item will appear in the Datasette main menu.
You can sign in and then visit the `/-/actor` page to see full details of the `auth0` profile that has been authenticated.
You can then use [Datasette permissions](https://docs.datasette.io/en/stable/authentication.html#configuring-permissions-in-metadata-json) to grant or deny access to different parts of Datasette based on the authenticated user.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-auth0
python3 -mvenv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-auth0
Datasette plugin that authenticates users using Auth0
If you are deploying using datasette publish you can pass these using --plugin-secret. For example, to deploy using Cloud Run you might run the following:
Once your Datasette instance is deployed, you will need to add its callback URL to the ""Allowed Callback URLs"" list in Auth0.
The callback URL should be something like:
https://url-to-your-datasette/-/auth0-callback
Usage
Once installed, a ""Sign in with Auth0"" menu item will appear in the Datasette main menu.
You can sign in and then visit the /-/actor page to see full details of the auth0 profile that has been authenticated.
You can then use Datasette permissions to grant or deny access to different parts of Datasette based on the authenticated user.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-auth0
python3 -mvenv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
474473836,R_kgDOHEflbA,datasette-nteract-data-explorer,hydrosquall/datasette-nteract-data-explorer,0,9020979,https://github.com/hydrosquall/datasette-nteract-data-explorer,automatic visual data explorer for datasette,0,2022-03-26T21:47:17Z,2022-10-04T03:28:02Z,2022-10-19T00:35:29Z,https://datasette-nteract-data-explorer.vercel.app/,137,8,8,TypeScript,1,1,1,1,0,1,0,0,8,apache-2.0,"[""automatic-viz"", ""datasette"", ""datasette-plugin"", ""dataviz""]",1,8,8,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,1,1,,,1,public,0,,0,
486732669,R_kgDOHQLzfQ,datasette-copy-to-memory,simonw/datasette-copy-to-memory,0,9599,https://github.com/simonw/datasette-copy-to-memory,Copy database files into an in-memory database on startup,0,2022-04-28T20:02:21Z,2022-04-30T21:32:54Z,2022-04-30T19:49:29Z,,19,2,2,Python,1,1,1,1,0,0,0,0,2,apache-2.0,"[""datasette"", ""datasette-plugin""]",0,2,2,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-copy-to-memory
[](https://pypi.org/project/datasette-copy-to-memory/)
[](https://github.com/simonw/datasette-copy-to-memory/releases)
[](https://github.com/simonw/datasette-copy-to-memory/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-copy-to-memory/blob/main/LICENSE)
Copy database files into an in-memory database on startup
This plugin is **highly experimental**. It currently exists to support Datasette performance research, and is not designed for actual production usage.
## Installation
Install this plugin in the same environment as Datasette.
datasette install datasette-copy-to-memory
## Usage
On startup, Datasette will create an in-memory named database for each attached database. This database will have the same name but with `_memory` at the end.
So running this:
datasette fixtures.db
Will serve two databases: the original at `/fixtures` and the in-memory copy at `/fixtures_memory`.
## Demo
A demo is running on [latest-with-plugins.datasette.io](https://latest-with-plugins.datasette.io/) - the [/fixtures_memory](https://latest-with-plugins.datasette.io/fixtures_memory) table there is provided by this plugin.
## Configuration
By default every attached database file will be loaded into a `_memory` copy.
You can use plugin configuration to specify just a subset of the database. For example, to create `github_memory` but not `fixtures_memory` you would use the following `metadata.yml` file:
```yaml
plugins:
datasette-copy-to-memory:
databases:
- github
```
Then start Datasette like this:
datasette github.db fixtures.db -m metadata.yml
If you don't want to have a `fixtures` and `fixtures_memory` database, you can use `replace: true` to have the plugin replace the file-backed database with the new in-memory one, reusing the same database name:
```yaml
plugins:
datasette-copy-to-memory:
replace: true
```
Then:
datasette github.db fixtures.db -m metadata.yml
This will result in both `/github` and `/fixtures` but no `/github_memory` or `/fixtures_memory`.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-copy-to-memory
python3 -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-copy-to-memory
Copy database files into an in-memory database on startup
This plugin is highly experimental. It currently exists to support Datasette performance research, and is not designed for actual production usage.
Installation
Install this plugin in the same environment as Datasette.
datasette install datasette-copy-to-memory
Usage
On startup, Datasette will create an in-memory named database for each attached database. This database will have the same name but with _memory at the end.
So running this:
datasette fixtures.db
Will serve two databases: the original at /fixtures and the in-memory copy at /fixtures_memory.
By default every attached database file will be loaded into a _memory copy.
You can use plugin configuration to specify just a subset of the database. For example, to create github_memory but not fixtures_memory you would use the following metadata.yml file:
If you don't want to have a fixtures and fixtures_memory database, you can use replace: true to have the plugin replace the file-backed database with the new in-memory one, reusing the same database name:
plugins:
datasette-copy-to-memory:
replace: true
Then:
datasette github.db fixtures.db -m metadata.yml
This will result in both /github and /fixtures but no /github_memory or /fixtures_memory.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-copy-to-memory
python3 -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
499911426,R_kgDOHcwLAg,datasette-query-files,eyeseast/datasette-query-files,0,25778,https://github.com/eyeseast/datasette-query-files,Write Datasette canned queries as plain SQL files,0,2022-06-04T18:52:07Z,2022-07-02T19:46:52Z,2022-07-02T20:40:51Z,,24,8,8,Python,1,1,1,1,0,0,0,0,2,apache-2.0,"[""datasette"", ""datasette-plugin"", ""python"", ""sql"", ""sqlite""]",0,2,8,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-query-files
[](https://pypi.org/project/datasette-query-files/)
[](https://github.com/eyeseast/datasette-query-files/releases)
[](https://github.com/eyeseast/datasette-query-files/actions?query=workflow%3ATest)
[](https://github.com/eyeseast/datasette-query-files/blob/main/LICENSE)
Write Datasette canned queries as plain SQL files.
## Installation
Install this plugin in the same environment as Datasette.
datasette install datasette-query-files
Or using `pip` or `pipenv`:
pip install datasette-query-files
pipenv install datasette-query-files
## Usage
This plugin will look for [canned queries](https://docs.datasette.io/en/stable/sql_queries.html#canned-queries) in the filesystem, in addition any defined in metadata.
Let's say you're working in a directory called `project-directory`, with a database file called `my-project.db`. Start by creating a `queries` directory with a `my-project` directory inside it. Any SQL file inside that `my-project` folder will become a canned query that can be run on the `my-project` database. If you have a `query-name.sql` file and a `query-name.json` (or `query-name.yml`) file in the same directory, the JSON file will be used as query metadata.
```
project-directory/
my-project.db
queries/
my-project/
query-name.sql # a query
query-name.yml # query metadata
```
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-query-files
python3 -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-query-files
Write Datasette canned queries as plain SQL files.
Installation
Install this plugin in the same environment as Datasette.
This plugin will look for canned queries in the filesystem, in addition any defined in metadata.
Let's say you're working in a directory called project-directory, with a database file called my-project.db. Start by creating a queries directory with a my-project directory inside it. Any SQL file inside that my-project folder will become a canned query that can be run on the my-project database. If you have a query-name.sql file and a query-name.json (or query-name.yml) file in the same directory, the JSON file will be used as query metadata.
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-query-files
python3 -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,"{""id"": 400878073, ""node_id"": ""MDEwOlJlcG9zaXRvcnk0MDA4NzgwNzM="", ""name"": ""datasette-plugin-template-repository"", ""full_name"": ""simonw/datasette-plugin-template-repository"", ""private"": false, ""owner"": {""login"": ""simonw"", ""id"": 9599, ""node_id"": ""MDQ6VXNlcjk1OTk="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/9599?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/simonw"", ""html_url"": ""https://github.com/simonw"", ""followers_url"": ""https://api.github.com/users/simonw/followers"", ""following_url"": ""https://api.github.com/users/simonw/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/simonw/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/simonw/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/simonw/subscriptions"", ""organizations_url"": ""https://api.github.com/users/simonw/orgs"", ""repos_url"": ""https://api.github.com/users/simonw/repos"", ""events_url"": ""https://api.github.com/users/simonw/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/simonw/received_events"", ""type"": ""User"", ""site_admin"": false}, ""html_url"": ""https://github.com/simonw/datasette-plugin-template-repository"", ""description"": ""GitHub template repository for creating new Datasette plugins, using the simonw/datasette-plugin cookiecutter template"", ""fork"": false, ""url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository"", ""forks_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/forks"", ""keys_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/keys{/key_id}"", ""collaborators_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/collaborators{/collaborator}"", ""teams_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/teams"", ""hooks_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/hooks"", ""issue_events_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/events{/number}"", ""events_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/events"", ""assignees_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/assignees{/user}"", ""branches_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/branches{/branch}"", ""tags_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/tags"", ""blobs_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/blobs{/sha}"", ""git_tags_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/tags{/sha}"", ""git_refs_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/refs{/sha}"", ""trees_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/trees{/sha}"", ""statuses_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/statuses/{sha}"", ""languages_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/languages"", ""stargazers_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/stargazers"", ""contributors_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/contributors"", ""subscribers_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscribers"", ""subscription_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscription"", ""commits_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/commits{/sha}"", ""git_commits_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/commits{/sha}"", ""comments_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/comments{/number}"", ""issue_comment_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/comments{/number}"", ""contents_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/contents/{+path}"", ""compare_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/compare/{base}...{head}"", ""merges_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/merges"", ""archive_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/{archive_format}{/ref}"", ""downloads_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/downloads"", ""issues_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues{/number}"", ""pulls_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/pulls{/number}"", ""milestones_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/milestones{/number}"", ""notifications_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/notifications{?since,all,participating}"", ""labels_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/labels{/name}"", ""releases_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/releases{/id}"", ""deployments_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/deployments"", ""created_at"": ""2021-08-28T19:50:28Z"", ""updated_at"": ""2022-06-10T13:28:46Z"", ""pushed_at"": ""2022-03-16T23:42:16Z"", ""git_url"": ""git://github.com/simonw/datasette-plugin-template-repository.git"", ""ssh_url"": ""git@github.com:simonw/datasette-plugin-template-repository.git"", ""clone_url"": ""https://github.com/simonw/datasette-plugin-template-repository.git"", ""svn_url"": ""https://github.com/simonw/datasette-plugin-template-repository"", ""homepage"": """", ""size"": 9, ""stargazers_count"": 15, ""watchers_count"": 15, ""language"": null, ""has_issues"": true, ""has_projects"": true, ""has_downloads"": true, ""has_wiki"": true, ""has_pages"": false, ""forks_count"": 0, ""mirror_url"": null, ""archived"": false, ""disabled"": false, ""open_issues_count"": 0, ""license"": null, ""allow_forking"": true, ""is_template"": true, ""web_commit_signoff_required"": false, ""topics"": [], ""visibility"": ""public"", ""forks"": 0, ""open_issues"": 0, ""watchers"": 15, ""default_branch"": ""main"", ""permissions"": {""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}, ""temp_clone_token"": """"}",0,
506026919,R_kgDOHilbpw,datasette-scale-to-zero,simonw/datasette-scale-to-zero,0,9599,https://github.com/simonw/datasette-scale-to-zero,Quit Datasette if it has not received traffic for a specified time period,0,2022-06-21T22:49:04Z,2022-07-13T14:02:31Z,2022-08-05T22:29:17Z,https://datasette.io/plugins/datasette-scale-to-zero,22,8,8,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin""]",0,0,8,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-scale-to-zero
[](https://pypi.org/project/datasette-scale-to-zero/)
[](https://github.com/simonw/datasette-scale-to-zero/releases)
[](https://github.com/simonw/datasette-scale-to-zero/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-scale-to-zero/blob/main/LICENSE)
Quit Datasette if it has not received traffic for a specified time period
Some hosting providers such as [Fly](https://fly.io/) offer a scale to zero mechanism, where servers can shut down and will be automatically started when new traffic arrives.
This plugin can be used to configure Datasette to quit X minutes (or seconds, or hours) after the last request it received. It can also cause the Datasette server to exit after a configured maximum time whether or not it is receiving traffic.
## Installation
Install this plugin in the same environment as Datasette.
datasette install datasette-scale-to-zero
## Configuration
This plugin will only take effect if it has been configured.
Add the following to your ``metadata.json`` or ``metadata.yml`` configuration file:
```json
{
""plugins"": {
""datasette-scale-to-zero"": {
""duration"": ""10m""
}
}
}
```
This will cause Datasette to quit if it has not received any HTTP traffic for 10 minutes.
You can set this value using a suffix of `m` for minutes, `h` for hours or `s` for seconds.
To cause Datasette to exit if the server has been running for longer than a specific time, use `""max-age""`:
```json
{
""plugins"": {
""datasette-scale-to-zero"": {
""max-age"": ""10h""
}
}
}
```
This example will exit the Datasette server if it has been running for more than ten hours.
You can use `""duration""` and `""max-age""` together in the same configuration file:
```json
{
""plugins"": {
""datasette-scale-to-zero"": {
""max-age"": ""10h"",
""duration"": ""5m""
}
}
}
```
This example will quit if no traffic has been received in five minutes, or if the server has been running for ten hours.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-scale-to-zero
python3 -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-scale-to-zero
Quit Datasette if it has not received traffic for a specified time period
Some hosting providers such as Fly offer a scale to zero mechanism, where servers can shut down and will be automatically started when new traffic arrives.
This plugin can be used to configure Datasette to quit X minutes (or seconds, or hours) after the last request it received. It can also cause the Datasette server to exit after a configured maximum time whether or not it is receiving traffic.
Installation
Install this plugin in the same environment as Datasette.
datasette install datasette-scale-to-zero
Configuration
This plugin will only take effect if it has been configured.
Add the following to your metadata.json or metadata.yml configuration file:
This example will quit if no traffic has been received in five minutes, or if the server has been running for ten hours.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-scale-to-zero
python3 -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,0,
510136835,R_kgDOHmgSAw,datasette-expose-env,simonw/datasette-expose-env,0,9599,https://github.com/simonw/datasette-expose-env,Datasette plugin to expose selected environment variables at /-/env for debugging,0,2022-07-03T21:14:29Z,2022-07-03T21:27:12Z,2022-07-03T21:28:55Z,https://datasette.io/plugins/datasette-expose-env,0,0,0,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin""]",0,0,0,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-expose-env
[](https://pypi.org/project/datasette-expose-env/)
[](https://github.com/simonw/datasette-expose-env/releases)
[](https://github.com/simonw/datasette-expose-env/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-expose-env/blob/main/LICENSE)
Datasette plugin to expose selected environment variables at `/-/env` for debugging
## Installation
Install this plugin in the same environment as Datasette.
datasette install datasette-expose-env
## Configuration
Decide on a list of environment variables you would like to expose, then add the following to your `metadata.yml` configuration:
```yaml
plugins:
datasette-expose-env:
- ENV_VAR_1
- ENV_VAR_2
- ENV_VAR_3
```
If you are using JSON in a `metadata.json` file use the following:
```json
{
""plugins"": {
""datasette-expose-env"": [
""ENV_VAR_1"",
""ENV_VAR_2"",
""ENV_VAR_3""
]
}
}
```
Visit `/-/env` on your Datasette instance to see the values of the environment variables.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-expose-env
python3 -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-expose-env
Datasette plugin to expose selected environment variables at /-/env for debugging
Installation
Install this plugin in the same environment as Datasette.
datasette install datasette-expose-env
Configuration
Decide on a list of environment variables you would like to expose, then add the following to your metadata.yml configuration:
Visit /-/env on your Datasette instance to see the values of the environment variables.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-expose-env
python3 -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,"{""id"": 400878073, ""node_id"": ""MDEwOlJlcG9zaXRvcnk0MDA4NzgwNzM="", ""name"": ""datasette-plugin-template-repository"", ""full_name"": ""simonw/datasette-plugin-template-repository"", ""private"": false, ""owner"": {""login"": ""simonw"", ""id"": 9599, ""node_id"": ""MDQ6VXNlcjk1OTk="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/9599?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/simonw"", ""html_url"": ""https://github.com/simonw"", ""followers_url"": ""https://api.github.com/users/simonw/followers"", ""following_url"": ""https://api.github.com/users/simonw/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/simonw/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/simonw/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/simonw/subscriptions"", ""organizations_url"": ""https://api.github.com/users/simonw/orgs"", ""repos_url"": ""https://api.github.com/users/simonw/repos"", ""events_url"": ""https://api.github.com/users/simonw/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/simonw/received_events"", ""type"": ""User"", ""site_admin"": false}, ""html_url"": ""https://github.com/simonw/datasette-plugin-template-repository"", ""description"": ""GitHub template repository for creating new Datasette plugins, using the simonw/datasette-plugin cookiecutter template"", ""fork"": false, ""url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository"", ""forks_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/forks"", ""keys_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/keys{/key_id}"", ""collaborators_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/collaborators{/collaborator}"", ""teams_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/teams"", ""hooks_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/hooks"", ""issue_events_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/events{/number}"", ""events_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/events"", ""assignees_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/assignees{/user}"", ""branches_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/branches{/branch}"", ""tags_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/tags"", ""blobs_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/blobs{/sha}"", ""git_tags_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/tags{/sha}"", ""git_refs_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/refs{/sha}"", ""trees_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/trees{/sha}"", ""statuses_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/statuses/{sha}"", ""languages_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/languages"", ""stargazers_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/stargazers"", ""contributors_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/contributors"", ""subscribers_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscribers"", ""subscription_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscription"", ""commits_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/commits{/sha}"", ""git_commits_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/commits{/sha}"", ""comments_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/comments{/number}"", ""issue_comment_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/comments{/number}"", ""contents_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/contents/{+path}"", ""compare_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/compare/{base}...{head}"", ""merges_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/merges"", ""archive_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/{archive_format}{/ref}"", ""downloads_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/downloads"", ""issues_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues{/number}"", ""pulls_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/pulls{/number}"", ""milestones_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/milestones{/number}"", ""notifications_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/notifications{?since,all,participating}"", ""labels_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/labels{/name}"", ""releases_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/releases{/id}"", ""deployments_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/deployments"", ""created_at"": ""2021-08-28T19:50:28Z"", ""updated_at"": ""2022-06-10T13:28:46Z"", ""pushed_at"": ""2022-03-16T23:42:16Z"", ""git_url"": ""git://github.com/simonw/datasette-plugin-template-repository.git"", ""ssh_url"": ""git@github.com:simonw/datasette-plugin-template-repository.git"", ""clone_url"": ""https://github.com/simonw/datasette-plugin-template-repository.git"", ""svn_url"": ""https://github.com/simonw/datasette-plugin-template-repository"", ""homepage"": """", ""size"": 9, ""stargazers_count"": 15, ""watchers_count"": 15, ""language"": null, ""has_issues"": true, ""has_projects"": true, ""has_downloads"": true, ""has_wiki"": true, ""has_pages"": false, ""forks_count"": 0, ""mirror_url"": null, ""archived"": false, ""disabled"": false, ""open_issues_count"": 0, ""license"": null, ""allow_forking"": true, ""is_template"": true, ""web_commit_signoff_required"": false, ""topics"": [], ""visibility"": ""public"", ""forks"": 0, ""open_issues"": 0, ""watchers"": 15, ""default_branch"": ""main"", ""permissions"": {""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}, ""temp_clone_token"": """"}",0,
555992212,R_kgDOISPElA,datasette-gunicorn,simonw/datasette-gunicorn,0,9599,https://github.com/simonw/datasette-gunicorn,Plugin for running Datasette using Gunicorn,0,2022-10-22T20:23:57Z,2022-10-22T21:00:11Z,2022-10-23T16:00:49Z,https://datasette.io/plugins/datasette-gunicorn,13,0,0,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin""]",0,0,0,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-gunicorn
[](https://pypi.org/project/datasette-gunicorn/)
[](https://github.com/simonw/datasette-gunicorn/releases)
[](https://github.com/simonw/datasette-gunicorn/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-gunicorn/blob/main/LICENSE)
Run a [Datasette](https://datasette.io/) server using [Gunicorn](https://gunicorn.org/)
## Installation
Install this plugin in the same environment as Datasette.
datasette install datasette-gunicorn
## Usage
The plugin adds a new `datasette gunicorn` command. This takes most of the same options as `datasette serve`, plus one more option for setting the number of Gunicorn workers to start:
`-w/--workers X` - set the number of workers. Defaults to 1.
To start serving a database using 4 workers, run the following:
datasette gunicorn fixtures.db -w 4
It is advisable to switch your datasette [into WAL mode](https://til.simonwillison.net/sqlite/enabling-wal-mode) to get the best performance out of this configuration:
sqlite3 fixtures.db 'PRAGMA journal_mode=WAL;'
Run `datasette gunicorn --help` for a full list of options (which are the same as `datasette serve --help`, with the addition of the new `-w` option).
## datasette gunicorn --help
Not all of the options to `datasette serve` are supported. Here's the full list of available options:
```
Usage: datasette gunicorn [OPTIONS] [FILES]...
Start a Gunicorn server running to serve Datasette
Options:
-i, --immutable PATH Database files to open in immutable mode
-h, --host TEXT Host for server. Defaults to 127.0.0.1 which means
only connections from the local machine will be
allowed. Use 0.0.0.0 to listen to all IPs and allow
access from other machines.
-p, --port INTEGER RANGE Port for server, defaults to 8001. Use -p 0 to
automatically assign an available port.
[0<=x<=65535]
--cors Enable CORS by serving Access-Control-Allow-Origin:
*
--load-extension TEXT Path to a SQLite extension to load
--inspect-file TEXT Path to JSON file created using ""datasette inspect""
-m, --metadata FILENAME Path to JSON/YAML file containing license/source
metadata
--template-dir DIRECTORY Path to directory containing custom templates
--plugins-dir DIRECTORY Path to directory containing custom plugins
--static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/...
--memory Make /_memory database available
--config CONFIG Deprecated: set config option using
configname:value. Use --setting instead.
--setting SETTING... Setting, see
docs.datasette.io/en/stable/settings.html
--secret TEXT Secret used for signing secure values, such as
signed cookies
--version-note TEXT Additional note to show on /-/versions
--help-settings Show available settings
--create Create database files if they do not exist
--crossdb Enable cross-database joins using the /_memory
database
--nolock Ignore locking, open locked files in read-only mode
-w, --workers INTEGER Number of Gunicorn workers [default: 1]
--help Show this message and exit.
```
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-gunicorn
python3 -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
Install this plugin in the same environment as Datasette.
datasette install datasette-gunicorn
Usage
The plugin adds a new datasette gunicorn command. This takes most of the same options as datasette serve, plus one more option for setting the number of Gunicorn workers to start:
-w/--workers X - set the number of workers. Defaults to 1.
To start serving a database using 4 workers, run the following:
datasette gunicorn fixtures.db -w 4
It is advisable to switch your datasette into WAL mode to get the best performance out of this configuration:
sqlite3 fixtures.db 'PRAGMA journal_mode=WAL;'
Run datasette gunicorn --help for a full list of options (which are the same as datasette serve --help, with the addition of the new -w option).
datasette gunicorn --help
Not all of the options to datasette serve are supported. Here's the full list of available options:
Usage: datasette gunicorn [OPTIONS] [FILES]...
Start a Gunicorn server running to serve Datasette
Options:
-i, --immutable PATH Database files to open in immutable mode
-h, --host TEXT Host for server. Defaults to 127.0.0.1 which means
only connections from the local machine will be
allowed. Use 0.0.0.0 to listen to all IPs and allow
access from other machines.
-p, --port INTEGER RANGE Port for server, defaults to 8001. Use -p 0 to
automatically assign an available port.
[0<=x<=65535]
--cors Enable CORS by serving Access-Control-Allow-Origin:
*
--load-extension TEXT Path to a SQLite extension to load
--inspect-file TEXT Path to JSON file created using ""datasette inspect""
-m, --metadata FILENAME Path to JSON/YAML file containing license/source
metadata
--template-dir DIRECTORY Path to directory containing custom templates
--plugins-dir DIRECTORY Path to directory containing custom plugins
--static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/...
--memory Make /_memory database available
--config CONFIG Deprecated: set config option using
configname:value. Use --setting instead.
--setting SETTING... Setting, see
docs.datasette.io/en/stable/settings.html
--secret TEXT Secret used for signing secure values, such as
signed cookies
--version-note TEXT Additional note to show on /-/versions
--help-settings Show available settings
--create Create database files if they do not exist
--crossdb Enable cross-database joins using the /_memory
database
--nolock Ignore locking, open locked files in read-only mode
-w, --workers INTEGER Number of Gunicorn workers [default: 1]
--help Show this message and exit.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-gunicorn
python3 -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies: