id,node_id,name,full_name,private,owner,html_url,description,fork,created_at,updated_at,pushed_at,homepage,size,stargazers_count,watchers_count,language,has_issues,has_projects,has_downloads,has_wiki,has_pages,forks_count,archived,disabled,open_issues_count,license,topics,forks,open_issues,watchers,default_branch,permissions,temp_clone_token,organization,network_count,subscribers_count,readme,readme_html,allow_forking,visibility,is_template,template_repository,web_commit_signoff_required,has_discussions
107914493,MDEwOlJlcG9zaXRvcnkxMDc5MTQ0OTM=,datasette,simonw/datasette,0,9599,https://github.com/simonw/datasette,An open source multi-tool for exploring and publishing data,0,2017-10-23T00:39:03Z,2022-11-15T23:16:27Z,2022-11-16T03:47:14Z,https://datasette.io,5770,6628,6628,Python,1,0,1,1,0,463,0,0,435,apache-2.0,"[""asgi"", ""automatic-api"", ""csv"", ""datasets"", ""datasette"", ""datasette-io"", ""docker"", ""json"", ""python"", ""sql"", ""sqlite""]",463,435,6628,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,463,97,,,1,public,0,,0,1
110509816,MDEwOlJlcG9zaXRvcnkxMTA1MDk4MTY=,csvs-to-sqlite,simonw/csvs-to-sqlite,0,9599,https://github.com/simonw/csvs-to-sqlite,Convert CSV files into a SQLite database,0,2017-11-13T06:38:21Z,2021-11-18T16:33:39Z,2021-11-18T16:35:33Z,,138,655,655,Python,1,1,1,1,0,50,0,0,34,apache-2.0,"[""click"", ""csv"", ""datasette"", ""datasette-io"", ""datasette-tool"", ""pandas"", ""python"", ""sqlite""]",50,34,655,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,50,17,"# csvs-to-sqlite
[](https://pypi.org/project/csvs-to-sqlite/)
[](https://github.com/simonw/csvs-to-sqlite/releases)
[](https://github.com/simonw/csvs-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/csvs-to-sqlite/blob/main/LICENSE)
Convert CSV files into a SQLite database. Browse and publish that SQLite database with [Datasette](https://github.com/simonw/datasette).
Basic usage:
csvs-to-sqlite myfile.csv mydatabase.db
This will create a new SQLite database called `mydatabase.db` containing a
single table, `myfile`, containing the CSV content.
You can provide multiple CSV files:
csvs-to-sqlite one.csv two.csv bundle.db
The `bundle.db` database will contain two tables, `one` and `two`.
This means you can use wildcards:
csvs-to-sqlite ~/Downloads/*.csv my-downloads.db
If you pass a path to one or more directories, the script will recursively
search those directories for CSV files and create tables for each one.
csvs-to-sqlite ~/path/to/directory all-my-csvs.db
## Handling TSV (tab-separated values)
You can use the `-s` option to specify a different delimiter. If you want
to use a tab character you'll need to apply shell escaping like so:
csvs-to-sqlite my-file.tsv my-file.db -s $'\t'
## Refactoring columns into separate lookup tables
Let's say you have a CSV file that looks like this:
county,precinct,office,district,party,candidate,votes
Clark,1,President,,REP,John R. Kasich,5
Clark,2,President,,REP,John R. Kasich,0
Clark,3,President,,REP,John R. Kasich,7
([Real example taken from the Open Elections project](https://github.com/openelections/openelections-data-sd/blob/master/2016/20160607__sd__primary__clark__precinct.csv))
You can now convert selected columns into separate lookup tables using the new
`--extract-column` option (shortname: `-c`) - for example:
csvs-to-sqlite openelections-data-*/*.csv \
-c county:County:name \
-c precinct:Precinct:name \
-c office -c district -c party -c candidate \
openelections.db
The format is as follows:
column_name:optional_table_name:optional_table_value_column_name
If you just specify the column name e.g. `-c office`, the following table will
be created:
CREATE TABLE ""office"" (
""id"" INTEGER PRIMARY KEY,
""value"" TEXT
);
If you specify all three options, e.g. `-c precinct:Precinct:name` the table
will look like this:
CREATE TABLE ""Precinct"" (
""id"" INTEGER PRIMARY KEY,
""name"" TEXT
);
The original tables will be created like this:
CREATE TABLE ""ca__primary__san_francisco__precinct"" (
""county"" INTEGER,
""precinct"" INTEGER,
""office"" INTEGER,
""district"" INTEGER,
""party"" INTEGER,
""candidate"" INTEGER,
""votes"" INTEGER,
FOREIGN KEY (county) REFERENCES County(id),
FOREIGN KEY (party) REFERENCES party(id),
FOREIGN KEY (precinct) REFERENCES Precinct(id),
FOREIGN KEY (office) REFERENCES office(id),
FOREIGN KEY (candidate) REFERENCES candidate(id)
);
They will be populated with IDs that reference the new derived tables.
## Installation
$ pip install csvs-to-sqlite
`csvs-to-sqlite` now requires Python 3. If you are running Python 2 you can install the last version to support Python 2:
$ pip install csvs-to-sqlite==0.9.2
## csvs-to-sqlite --help
```
Usage: csvs-to-sqlite [OPTIONS] PATHS... DBNAME
PATHS: paths to individual .csv files or to directories containing .csvs
DBNAME: name of the SQLite database file to create
Options:
-s, --separator TEXT Field separator in input .csv
-q, --quoting INTEGER Control field quoting behavior per csv.QUOTE_*
constants. Use one of QUOTE_MINIMAL (0),
QUOTE_ALL (1), QUOTE_NONNUMERIC (2) or
QUOTE_NONE (3).
--skip-errors Skip lines with too many fields instead of
stopping the import
--replace-tables Replace tables if they already exist
-t, --table TEXT Table to use (instead of using CSV filename)
-c, --extract-column TEXT One or more columns to 'extract' into a
separate lookup table. If you pass a simple
column name that column will be replaced with
integer foreign key references to a new table
of that name. You can customize the name of
the table like so: state:States:state_name
This will pull unique values from the 'state'
column and use them to populate a new 'States'
table, with an id column primary key and a
state_name column containing the strings from
the original column.
-d, --date TEXT One or more columns to parse into ISO
formatted dates
-dt, --datetime TEXT One or more columns to parse into ISO
formatted datetimes
-df, --datetime-format TEXT One or more custom date format strings to try
when parsing dates/datetimes
-pk, --primary-key TEXT One or more columns to use as the primary key
-f, --fts TEXT One or more columns to use to populate a full-
text index
-i, --index TEXT Add index on this column (or a compound index
with -i col1,col2)
--shape TEXT Custom shape for the DB table - format is
csvcol:dbcol(TYPE),...
--filename-column TEXT Add a column with this name and populate with
CSV file name
--fixed-column ... Populate column with a fixed string
--fixed-column-int ...
Populate column with a fixed integer
--fixed-column-float ...
Populate column with a fixed float
--no-index-fks Skip adding index to foreign key columns
created using --extract-column (default is to
add them)
--no-fulltext-fks Skip adding full-text index on values
extracted using --extract-column (default is
to add them)
--just-strings Import all columns as text strings by default
(and, if specified, still obey --shape,
--date/datetime, and --datetime-format)
--version Show the version and exit.
--help Show this message and exit.
```
","
csvs-to-sqlite
Convert CSV files into a SQLite database. Browse and publish that SQLite database with Datasette.
Basic usage:
csvs-to-sqlite myfile.csv mydatabase.db
This will create a new SQLite database called mydatabase.db containing a
single table, myfile, containing the CSV content.
You can provide multiple CSV files:
csvs-to-sqlite one.csv two.csv bundle.db
The bundle.db database will contain two tables, one and two.
This means you can use wildcards:
csvs-to-sqlite ~/Downloads/*.csv my-downloads.db
If you pass a path to one or more directories, the script will recursively
search those directories for CSV files and create tables for each one.
csvs-to-sqlite ~/path/to/directory all-my-csvs.db
Handling TSV (tab-separated values)
You can use the -s option to specify a different delimiter. If you want
to use a tab character you'll need to apply shell escaping like so:
csvs-to-sqlite my-file.tsv my-file.db -s $'\t'
Refactoring columns into separate lookup tables
Let's say you have a CSV file that looks like this:
county,precinct,office,district,party,candidate,votes
Clark,1,President,,REP,John R. Kasich,5
Clark,2,President,,REP,John R. Kasich,0
Clark,3,President,,REP,John R. Kasich,7
They will be populated with IDs that reference the new derived tables.
Installation
$ pip install csvs-to-sqlite
csvs-to-sqlite now requires Python 3. If you are running Python 2 you can install the last version to support Python 2:
$ pip install csvs-to-sqlite==0.9.2
csvs-to-sqlite --help
Usage: csvs-to-sqlite [OPTIONS] PATHS... DBNAME
PATHS: paths to individual .csv files or to directories containing .csvs
DBNAME: name of the SQLite database file to create
Options:
-s, --separator TEXT Field separator in input .csv
-q, --quoting INTEGER Control field quoting behavior per csv.QUOTE_*
constants. Use one of QUOTE_MINIMAL (0),
QUOTE_ALL (1), QUOTE_NONNUMERIC (2) or
QUOTE_NONE (3).
--skip-errors Skip lines with too many fields instead of
stopping the import
--replace-tables Replace tables if they already exist
-t, --table TEXT Table to use (instead of using CSV filename)
-c, --extract-column TEXT One or more columns to 'extract' into a
separate lookup table. If you pass a simple
column name that column will be replaced with
integer foreign key references to a new table
of that name. You can customize the name of
the table like so: state:States:state_name
This will pull unique values from the 'state'
column and use them to populate a new 'States'
table, with an id column primary key and a
state_name column containing the strings from
the original column.
-d, --date TEXT One or more columns to parse into ISO
formatted dates
-dt, --datetime TEXT One or more columns to parse into ISO
formatted datetimes
-df, --datetime-format TEXT One or more custom date format strings to try
when parsing dates/datetimes
-pk, --primary-key TEXT One or more columns to use as the primary key
-f, --fts TEXT One or more columns to use to populate a full-
text index
-i, --index TEXT Add index on this column (or a compound index
with -i col1,col2)
--shape TEXT Custom shape for the DB table - format is
csvcol:dbcol(TYPE),...
--filename-column TEXT Add a column with this name and populate with
CSV file name
--fixed-column <TEXT TEXT>... Populate column with a fixed string
--fixed-column-int <TEXT INTEGER>...
Populate column with a fixed integer
--fixed-column-float <TEXT FLOAT>...
Populate column with a fixed float
--no-index-fks Skip adding index to foreign key columns
created using --extract-column (default is to
add them)
--no-fulltext-fks Skip adding full-text index on values
extracted using --extract-column (default is
to add them)
--just-strings Import all columns as text strings by default
(and, if specified, still obey --shape,
--date/datetime, and --datetime-format)
--version Show the version and exit.
--help Show this message and exit.
Run datasette install datasette-cluster-map to add this plugin to your Datasette virtual environment. Datasette will automatically load the plugin if it is installed in this way.
If you are deploying using the datasette publish command you can use the --install option:
If any of your tables have a latitude and longitude column, a map will be automatically displayed.
Configuration
If your columns are called something else you can configure the column names using plugin configuration in a metadata.json file. For example, if all of your columns are called xlat and xlng you can create a metadata.json file like this:
{
""title"": ""Regular metadata keys can go here too"",
""plugins"": {
""datasette-cluster-map"": {
""latitude_column"": ""xlat"",
""longitude_column"": ""xlng""
}
}
}
Then run Datasette like this:
datasette mydata.db -m metadata.json
This will configure the required column names for every database loaded by that Datasette instance.
If you want to customize the column names for just one table in one database, you can do something like this:
You can also use a custom SQL query to rename those columns to latitude and longitude, for example:
select*,
""Capture Latitude""as latitude,
""Capture Longitude""as longitude
from [USGS_WC_eartag_deployments_2009-2011]
The map defaults to being displayed above the main results table on the page. You can use the ""container"" plugin setting to provide a CSS selector indicating an element that the map should be appended to instead.
Custom tile layers
You can customize the tile layer used by the maps using the tile_layer and tile_layer_options configuration settings. For example, to use the Stamen Watercolor tiles you can use these settings:
The marker popup defaults to displaying the data for the underlying database row.
You can customize this by including a popup column in your results containing JSON that defines a more useful popup.
The JSON in the popup column should look something like this:
{
""image"": ""https://niche-museums.imgix.net/dodgems.heic?w=800&h=400&fit=crop"",
""alt"": ""Dingles Fairground Heritage Centre"",
""title"": ""Dingles Fairground Heritage Centre"",
""description"": ""Home of the National Fairground Collection, Dingles has over 45,000 indoor square feet of vintage fairground rides... and you can go on them! Highlights include the last complete surviving and opera"",
""link"": ""/browse/museums/26""
}
Each of these columns is optional.
title is the title to show at the top of the popup
image is the URL to an image to display in the popup
alt is the alt attribute to use for that image
description is a longer string of text to use as a description
link is a URL that the marker content should link to
You can use the SQLite json_object() function to construct this data dynamically as part of your SQL query. Here's an example:
select json_object(
'image', photo_url ||'?w=800&h=400&fit=crop',
'title', name,
'description', substr(description, 0, 200),
'link', '/browse/museums/'|| id
) as popup,
latitude, longitude from museums
where id in (26, 27) order by id
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-cluster-map
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
138669673,MDEwOlJlcG9zaXRvcnkxMzg2Njk2NzM=,datasette-vega,simonw/datasette-vega,0,9599,https://github.com/simonw/datasette-vega,Datasette plugin for visualizing data using Vega,0,2018-06-26T01:40:54Z,2021-12-10T22:20:46Z,2021-12-10T22:20:43Z,,59,42,42,JavaScript,1,1,1,1,0,2,0,0,31,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""plugin"", ""react"", ""vega""]",2,31,42,master,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,2,2,"# datasette-vega
[](https://pypi.org/project/datasette-vega/)
[](https://github.com/simonw/datasette-vega/blob/master/LICENSE)
A [Datasette](https://github.com/simonw/datasette) plugin that provides tools
for generating charts using [Vega](https://vega.github.io/).
Try out the latest master build as a live demo at https://datasette-vega-latest.datasette.io/ or try the latest release installed as a plugin at https://fivethirtyeight.datasettes.com/
To add this to your Datasette installation, install the plugin like so:
pip install datasette-vega
The plugin will then add itself to every Datasette table view.
If you are publishing data using the `datasette publish` command, you can
include this plugin like so:
datasette publish now mydatabase.db --install=datasette-vega
","
datasette-vega
A Datasette plugin that provides tools
for generating charts using Vega.
To add this to your Datasette installation, install the plugin like so:
pip install datasette-vega
The plugin will then add itself to every Datasette table view.
If you are publishing data using the datasette publish command, you can
include this plugin like so:
datasette publish now mydatabase.db --install=datasette-vega
",1,public,0,,,
140912432,MDEwOlJlcG9zaXRvcnkxNDA5MTI0MzI=,sqlite-utils,simonw/sqlite-utils,0,9599,https://github.com/simonw/sqlite-utils,Python CLI utility and library for manipulating SQLite databases,0,2018-07-14T03:21:46Z,2022-11-15T18:12:16Z,2022-11-15T15:53:38Z,https://sqlite-utils.datasette.io,1437,1029,1029,Python,1,1,1,1,0,79,0,0,72,apache-2.0,"[""cli"", ""click"", ""datasette"", ""datasette-io"", ""datasette-tool"", ""python"", ""sqlite"", ""sqlite-database""]",79,72,1029,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,79,16,"# sqlite-utils
[](https://pypi.org/project/sqlite-utils/)
[](https://sqlite-utils.datasette.io/en/stable/changelog.html)
[](https://pypi.org/project/sqlite-utils/)
[](https://github.com/simonw/sqlite-utils/actions?query=workflow%3ATest)
[](http://sqlite-utils.datasette.io/en/stable/?badge=stable)
[](https://codecov.io/gh/simonw/sqlite-utils)
[](https://github.com/simonw/sqlite-utils/blob/main/LICENSE)
[](https://discord.gg/Ass7bCAMDw)
Python CLI utility and library for manipulating SQLite databases.
## Some feature highlights
- [Pipe JSON](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data) (or [CSV or TSV](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-csv-or-tsv-data)) directly into a new SQLite database file, automatically creating a table with the appropriate schema
- [Run in-memory SQL queries](https://sqlite-utils.datasette.io/en/stable/cli.html#querying-data-directly-using-an-in-memory-database), including joins, directly against data in CSV, TSV or JSON files and view the results
- [Configure SQLite full-text search](https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search) against your database tables and run search queries against them, ordered by relevance
- Run [transformations against your tables](https://sqlite-utils.datasette.io/en/stable/cli.html#transforming-tables) to make schema changes that SQLite `ALTER TABLE` does not directly support, such as changing the type of a column
- [Extract columns](https://sqlite-utils.datasette.io/en/stable/cli.html#extracting-columns-into-a-separate-table) into separate tables to better normalize your existing data
Read more on my blog, in this series of posts on [New features in sqlite-utils](https://simonwillison.net/series/sqlite-utils-features/) and other [entries tagged sqliteutils](https://simonwillison.net/tags/sqliteutils/).
## Installation
pip install sqlite-utils
Or if you use [Homebrew](https://brew.sh/) for macOS:
brew install sqlite-utils
## Using as a CLI tool
Now you can do things with the CLI utility like this:
$ sqlite-utils memory dogs.csv ""select * from t""
[{""id"": 1, ""age"": 4, ""name"": ""Cleo""},
{""id"": 2, ""age"": 2, ""name"": ""Pancakes""}]
$ sqlite-utils insert dogs.db dogs dogs.csv --csv
[####################################] 100%
$ sqlite-utils tables dogs.db --counts
[{""table"": ""dogs"", ""count"": 2}]
$ sqlite-utils dogs.db ""select id, name from dogs""
[{""id"": 1, ""name"": ""Cleo""},
{""id"": 2, ""name"": ""Pancakes""}]
$ sqlite-utils dogs.db ""select * from dogs"" --csv
id,age,name
1,4,Cleo
2,2,Pancakes
$ sqlite-utils dogs.db ""select * from dogs"" --table
id age name
---- ----- --------
1 4 Cleo
2 2 Pancakes
You can import JSON data into a new database table like this:
$ curl https://api.github.com/repos/simonw/sqlite-utils/releases \
| sqlite-utils insert releases.db releases - --pk id
Or for data in a CSV file:
$ sqlite-utils insert dogs.db dogs dogs.csv --csv
`sqlite-utils memory` lets you import CSV or JSON data into an in-memory database and run SQL queries against it in a single command:
$ cat dogs.csv | sqlite-utils memory - ""select name, age from stdin""
See the [full CLI documentation](https://sqlite-utils.datasette.io/en/stable/cli.html) for comprehensive coverage of many more commands.
## Using as a library
You can also `import sqlite_utils` and use it as a Python library like this:
```python
import sqlite_utils
db = sqlite_utils.Database(""demo_database.db"")
# This line creates a ""dogs"" table if one does not already exist:
db[""dogs""].insert_all([
{""id"": 1, ""age"": 4, ""name"": ""Cleo""},
{""id"": 2, ""age"": 2, ""name"": ""Pancakes""}
], pk=""id"")
```
Check out the [full library documentation](https://sqlite-utils.datasette.io/en/stable/python-api.html) for everything else you can do with the Python library.
## Related projects
* [Datasette](https://datasette.io/): A tool for exploring and publishing data
* [csvs-to-sqlite](https://github.com/simonw/csvs-to-sqlite): Convert CSV files into a SQLite database
* [db-to-sqlite](https://github.com/simonw/db-to-sqlite): CLI tool for exporting a MySQL or PostgreSQL database as a SQLite file
* [dogsheep](https://dogsheep.github.io/): A family of tools for personal analytics, built on top of `sqlite-utils`
","
sqlite-utils
Python CLI utility and library for manipulating SQLite databases.
Some feature highlights
Pipe JSON (or CSV or TSV) directly into a new SQLite database file, automatically creating a table with the appropriate schema
Run in-memory SQL queries, including joins, directly against data in CSV, TSV or JSON files and view the results
Run transformations against your tables to make schema changes that SQLite ALTER TABLE does not directly support, such as changing the type of a column
Extract columns into separate tables to better normalize your existing data
You can also import sqlite_utils and use it as a Python library like this:
importsqlite_utilsdb=sqlite_utils.Database(""demo_database.db"")
# This line creates a ""dogs"" table if one does not already exist:db[""dogs""].insert_all([
{""id"": 1, ""age"": 4, ""name"": ""Cleo""},
{""id"": 2, ""age"": 2, ""name"": ""Pancakes""}
], pk=""id"")
Datasette: A tool for exploring and publishing data
csvs-to-sqlite: Convert CSV files into a SQLite database
db-to-sqlite: CLI tool for exporting a MySQL or PostgreSQL database as a SQLite file
dogsheep: A family of tools for personal analytics, built on top of sqlite-utils
",1,public,0,,0,0
142967347,MDEwOlJlcG9zaXRvcnkxNDI5NjczNDc=,datasette-json-html,simonw/datasette-json-html,0,9599,https://github.com/simonw/datasette-json-html,Datasette plugin for rendering HTML based on JSON values,0,2018-07-31T05:41:39Z,2022-03-15T04:54:15Z,2022-03-22T01:43:59Z,,46,19,19,Python,1,1,1,1,0,1,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""plugin""]",1,0,19,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,1,4,"# datasette-json-html
[](https://pypi.org/project/datasette-json-html/)
[](https://github.com/simonw/datasette-json-html/releases)
[](https://github.com/simonw/datasette-remote-metadata/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-json-html/blob/main/LICENSE)
Datasette plugin for rendering HTML based on JSON values, using the [render_cell plugin hook](https://docs.datasette.io/en/stable/plugin_hooks.html#render-cell-value-column-table-database-datasette).
This plugin looks for cell values that match a very specific JSON format and converts them into HTML when they are rendered by the Datasette interface.
## Links
{
""href"": ""https://simonwillison.net/"",
""label"": ""Simon Willison""
}
Will be rendered as an `` link:
Simon Willison
You can set a tooltip on the link using a `""title""` key:
{
""href"": ""https://simonwillison.net/"",
""label"": ""Simon Willison"",
""title"": ""My blog""
}
Produces:
Simon Willison
You can also include a description, which will be displayed below the link. If descriptions include newlines they will be converted to ` ` elements:
select json_object(
""href"", ""https://simonwillison.net/"",
""label"", ""Simon Willison"",
""description"", ""This can contain"" || x'0a' || ""newlines""
)
Produces:
Simon Willison This can contain newlines
* [Literal JSON link demo](https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22href%22%3A+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++%22label%22%3A+%22Simon+Willison%22%2C%0D%0A++++%22title%22%3A+%22My+blog%22%0D%0A%7D%27)
## List of links
[
{
""href"": ""https://simonwillison.net/"",
""label"": ""Simon Willison""
},
{
""href"": ""https://github.com/simonw/datasette"",
""label"": ""Datasette""
}
]
Will be rendered as a comma-separated list of `` links:
Simon Willison,
Datasette
The `href` property must begin with `https://` or `http://` or `/`, to avoid potential XSS injection attacks (for example URLs that begin with `javascript:`).
Lists of links cannot include `""description""` keys.
* [Literal list of links demo](https://datasette-json-html.datasette.io/demo?sql=select+%27%5B%0D%0A++++%7B%0D%0A++++++++%22href%22%3A+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++++++%22label%22%3A+%22Simon+Willison%22%0D%0A++++%7D%2C%0D%0A++++%7B%0D%0A++++++++%22href%22%3A+%22https%3A%2F%2Fgithub.com%2Fsimonw%2Fdatasette%22%2C%0D%0A++++++++%22label%22%3A+%22Datasette%22%0D%0A++++%7D%0D%0A%5D%27)
## Images
The image tag is more complex. The most basic version looks like this:
{
""img_src"": ""https://placekitten.com/200/300""
}
This will render as:
But you can also include one or more of `alt`, `caption`, `width` and `href`.
If you include width or alt, they will be added as attributes:
{
""img_src"": ""https://placekitten.com/200/300"",
""alt"": ""Kitten"",
""width"": 200
}
Produces:
* [Literal image demo](https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22img_src%22%3A+%22https%3A%2F%2Fplacekitten.com%2F200%2F300%22%2C%0D%0A++++%22alt%22%3A+%22Kitten%22%2C%0D%0A++++%22width%22%3A+200%0D%0A%7D%27)
The `href` key will cause the image to be wrapped in a link:
{
""img_src"": ""https://placekitten.com/200/300"",
""href"": ""http://www.example.com""
}
Produces:
The `caption` key wraps everything in a fancy figure/figcaption block:
{
""img_src"": ""https://placekitten.com/200/300"",
""caption"": ""Kitten caption""
}
Produces:
Kitten caption
## Preformatted text
You can use `{""pre"": ""text""}` to render text in a `
` HTML tag:
{
""pre"": ""This\nhas\nnewlines""
}
Produces:
This
has
newlines
If the value attached to the `""pre""` key is itself a JSON object, that JSON will be pretty-printed:
{
""pre"": {
""this"": {
""object"": [""is"", ""nested""]
}
}
}
Produces:
{
"this": {
"object": [
"is",
"nested"
]
}
}
* [Preformatted text with JSON demo](https://datasette-json-html.datasette.io/demo?sql=select+%27%7B%0D%0A++++%22pre%22%3A+%7B%0D%0A++++++++%22this%22%3A+%7B%0D%0A++++++++++++%22object%22%3A+%5B%22is%22%2C+%22nested%22%5D%0D%0A++++++++%7D%0D%0A++++%7D%0D%0A%7D%27)
* [Preformatted text demo showing the Mandelbrot Set](https://datasette-json-html.datasette.io/demo?sql=WITH+RECURSIVE%0D%0A++xaxis%28x%29+AS+%28VALUES%28-2.0%29+UNION+ALL+SELECT+x%2B0.05+FROM+xaxis+WHERE+x%3C1.2%29%2C%0D%0A++yaxis%28y%29+AS+%28VALUES%28-1.0%29+UNION+ALL+SELECT+y%2B0.1+FROM+yaxis+WHERE+y%3C1.0%29%2C%0D%0A++m%28iter%2C+cx%2C+cy%2C+x%2C+y%29+AS+%28%0D%0A++++SELECT+0%2C+x%2C+y%2C+0.0%2C+0.0+FROM+xaxis%2C+yaxis%0D%0A++++UNION+ALL%0D%0A++++SELECT+iter%2B1%2C+cx%2C+cy%2C+x*x-y*y+%2B+cx%2C+2.0*x*y+%2B+cy+FROM+m+%0D%0A+++++WHERE+%28x*x+%2B+y*y%29+%3C+4.0+AND+iter%3C28%0D%0A++%29%2C%0D%0A++m2%28iter%2C+cx%2C+cy%29+AS+%28%0D%0A++++SELECT+max%28iter%29%2C+cx%2C+cy+FROM+m+GROUP+BY+cx%2C+cy%0D%0A++%29%2C%0D%0A++a%28t%29+AS+%28%0D%0A++++SELECT+group_concat%28+substr%28%27+.%2B*%23%27%2C+1%2Bmin%28iter%2F7%2C4%29%2C+1%29%2C+%27%27%29+%0D%0A++++FROM+m2+GROUP+BY+cy%0D%0A++%29%0D%0ASELECT+json_object%28%27pre%27%2C+group_concat%28rtrim%28t%29%2Cx%270a%27%29%29+FROM+a%3B) using [this example](https://www.sqlite.org/lang_with.html#outlandish_recursive_query_examples) from the SQLite documentation
## Using these with SQLite JSON functions
The most powerful way to make use of this plugin is in conjunction with SQLite's [JSON functions](https://www.sqlite.org/json1.html). For example:
select json_object(
""href"", ""https://simonwillison.net/"",
""label"", ""Simon Willison""
);
* [json_object() link demo](https://datasette-json-html.datasette.io/demo?sql=select+json_object%28%0D%0A++++%22href%22%2C+%22https%3A%2F%2Fsimonwillison.net%2F%22%2C%0D%0A++++%22label%22%2C+%22Simon+Willison%22%0D%0A%29%3B)
You can use these functions to construct JSON objects that work with the plugin from data in a table:
select id, json_object(
""href"", url, ""label"", text
) from mytable;
* [Demo that builds links against a table](https://datasette-json-html.datasette.io/demo?sql=select+json_object%28%22href%22%2C+url%2C+%22label%22%2C+package%2C+%22title%22%2C+package+%7C%7C+%22+%22+%7C%7C+url%29+as+package+from+packages)
The `json_group_array()` function is an aggregate function similar to `group_concat()` - it allows you to construct lists of JSON objects in conjunction with a `GROUP BY` clause.
This means you can use it to construct dynamic lists of links, for example:
select
substr(package, 0, 12) as prefix,
json_group_array(
json_object(
""href"", url,
""label"", package
)
) as package_links
from packages
group by prefix
* [Demo of json_group_array()](https://datasette-json-html.datasette.io/demo?sql=select%0D%0A++++substr%28package%2C+0%2C+12%29+as+prefix%2C%0D%0A++++json_group_array%28%0D%0A++++++++json_object%28%0D%0A++++++++++++%22href%22%2C+url%2C%0D%0A++++++++++++%22label%22%2C+package%0D%0A++++++++%29%0D%0A++++%29+as+package_links%0D%0Afrom+packages%0D%0Agroup+by+prefix)
## The `urllib_quote_plus()` SQL function
Since this plugin is designed to be used with SQL that constructs the underlying JSON structure, it is likely you will need to construct dynamic URLs from results returned by a SQL query.
This plugin registers a custom SQLite function called `urllib_quote_plus()` to help you do that. It lets you use Python's [urllib.parse.quote\_plus() function](https://docs.python.org/3/library/urllib.parse.html#urllib.parse.quote_plus) from within a SQL query.
Here's an example of how you might use it:
select id, json_object(
""href"",
""/mydatabase/other_table?_search="" || urllib_quote_plus(text),
""label"", text
) from mytable;
","
datasette-json-html
Datasette plugin for rendering HTML based on JSON values, using the render_cell plugin hook.
This plugin looks for cell values that match a very specific JSON format and converts them into HTML when they are rendered by the Datasette interface.
The json_group_array() function is an aggregate function similar to group_concat() - it allows you to construct lists of JSON objects in conjunction with a GROUP BY clause.
This means you can use it to construct dynamic lists of links, for example:
select
substr(package, 0, 12) as prefix,
json_group_array(
json_object(
""href"", url,
""label"", package
)
) as package_links
from packages
group by prefix
Since this plugin is designed to be used with SQL that constructs the underlying JSON structure, it is likely you will need to construct dynamic URLs from results returned by a SQL query.
This plugin registers a custom SQLite function called urllib_quote_plus() to help you do that. It lets you use Python's urllib.parse.quote_plus() function from within a SQL query.
Here's an example of how you might use it:
select id, json_object(
""href"",
""/mydatabase/other_table?_search="" || urllib_quote_plus(text),
""label"", text
) from mytable;
",1,public,0,,,
163790822,MDEwOlJlcG9zaXRvcnkxNjM3OTA4MjI=,datasette-sqlite-fts4,simonw/datasette-sqlite-fts4,0,9599,https://github.com/simonw/datasette-sqlite-fts4,Datasette plugin that adds custom SQL functions for working with SQLite FTS4,0,2019-01-02T03:40:41Z,2022-07-31T16:33:25Z,2022-07-31T14:46:26Z,https://datasette.io/plugins/datasette-sqlite-fts4,14,3,3,Python,1,1,1,1,0,1,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""plugin""]",1,0,3,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,1,2,"# datasette-sqlite-fts4
[](https://pypi.org/project/datasette-sqlite-fts4/)
[](https://github.com/simonw/datasette-sqlite-fts4/releases)
[](https://github.com/simonw/datasette-sqlite-fts4/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-sqlite-fts4/blob/main/LICENSE)
Datasette plugin that exposes the custom SQL functions from [sqlite-fts4](https://github.com/simonw/sqlite-fts4).
[Interactive demo](https://datasette-sqlite-fts4.datasette.io/24ways-fts4?sql=select%0D%0A++++json_object%28%0D%0A++++++++""label""%2C+articles.title%2C+""href""%2C+articles.url%0D%0A++++%29+as+article%2C%0D%0A++++articles.author%2C%0D%0A++++rank_score%28matchinfo%28articles_fts%2C+""pcx""%29%29+as+score%2C%0D%0A++++rank_bm25%28matchinfo%28articles_fts%2C+""pcnalx""%29%29+as+bm25%2C%0D%0A++++json_object%28%0D%0A++++++++""pre""%2C+annotate_matchinfo%28matchinfo%28articles_fts%2C+""pcxnalyb""%29%2C+""pcxnalyb""%29%0D%0A++++%29+as+annotated_matchinfo%2C%0D%0A++++matchinfo%28articles_fts%2C+""pcxnalyb""%29+as+matchinfo%2C%0D%0A++++decode_matchinfo%28matchinfo%28articles_fts%2C+""pcxnalyb""%29%29+as+decoded_matchinfo%0D%0Afrom%0D%0A++++articles_fts+join+articles+on+articles.rowid+%3D+articles_fts.rowid%0D%0Awhere%0D%0A++++articles_fts+match+%3Asearch%0D%0Aorder+by+bm25&search=jquery+maps). Read [Exploring search relevance algorithms with SQLite](https://simonwillison.net/2019/Jan/7/exploring-search-relevance-algorithms-sqlite/) for further details on this project.
## Installation
pip install datasette-sqlite-fts4
If you are deploying a database using `datasette publish` you can include this plugin using the `--install` option:
datasette publish now mydb.db --install=datasette-sqlite-fts4
","
datasette-sqlite-fts4
Datasette plugin that exposes the custom SQL functions from sqlite-fts4.
If you are deploying a database using datasette publish you can include this plugin using the --install option:
datasette publish now mydb.db --install=datasette-sqlite-fts4
",1,public,0,,0,
166159072,MDEwOlJlcG9zaXRvcnkxNjYxNTkwNzI=,db-to-sqlite,simonw/db-to-sqlite,0,9599,https://github.com/simonw/db-to-sqlite,CLI tool for exporting tables or queries from any SQL database to a SQLite file,0,2019-01-17T04:16:48Z,2021-06-11T22:52:12Z,2021-06-11T22:55:56Z,,77,226,226,Python,1,1,1,1,0,12,0,0,2,apache-2.0,"[""sqlalchemy"", ""sqlite"", ""datasette"", ""datasette-io"", ""datasette-tool""]",12,2,226,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,12,4,"# db-to-sqlite
[](https://pypi.python.org/pypi/db-to-sqlite)
[](https://github.com/simonw/db-to-sqlite/releases)
[](https://github.com/simonw/db-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/db-to-sqlite/blob/main/LICENSE)
CLI tool for exporting tables or queries from any SQL database to a SQLite file.
## Installation
Install from PyPI like so:
pip install db-to-sqlite
If you want to use it with MySQL, you can install the extra dependency like this:
pip install 'db-to-sqlite[mysql]'
Installing the `mysqlclient` library on OS X can be tricky - I've found [this recipe](https://gist.github.com/simonw/90ac0afd204cd0d6d9c3135c3888d116) to work (run that before installing `db-to-sqlite`).
For PostgreSQL, use this:
pip install 'db-to-sqlite[postgresql]'
## Usage
Usage: db-to-sqlite [OPTIONS] CONNECTION PATH
Load data from any database into SQLite.
PATH is a path to the SQLite file to create, e.c. /tmp/my_database.db
CONNECTION is a SQLAlchemy connection string, for example:
postgresql://localhost/my_database
postgresql://username:passwd@localhost/my_database
mysql://root@localhost/my_database
mysql://username:passwd@localhost/my_database
More: https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls
Options:
--version Show the version and exit.
--all Detect and copy all tables
--table TEXT Specific tables to copy
--skip TEXT When using --all skip these tables
--redact TEXT... (table, column) pairs to redact with ***
--sql TEXT Optional SQL query to run
--output TEXT Table in which to save --sql query results
--pk TEXT Optional column to use as a primary key
--index-fks / --no-index-fks Should foreign keys have indexes? Default on
-p, --progress Show progress bar
--postgres-schema TEXT PostgreSQL schema to use
--help Show this message and exit.
For example, to save the content of the `blog_entry` table from a PostgreSQL database to a local file called `blog.db` you could do this:
db-to-sqlite ""postgresql://localhost/myblog"" blog.db \
--table=blog_entry
You can specify `--table` more than once.
You can also save the data from all of your tables, effectively creating a SQLite copy of your entire database. Any foreign key relationships will be detected and added to the SQLite database. For example:
db-to-sqlite ""postgresql://localhost/myblog"" blog.db \
--all
When running `--all` you can specify tables to skip using `--skip`:
db-to-sqlite ""postgresql://localhost/myblog"" blog.db \
--all \
--skip=django_migrations
If you want to save the results of a custom SQL query, do this:
db-to-sqlite ""postgresql://localhost/myblog"" output.db \
--output=query_results \
--sql=""select id, title, created from blog_entry"" \
--pk=id
The `--output` option specifies the table that should contain the results of the query.
## Using db-to-sqlite with PostgreSQL schemas
If the tables you want to copy from your PostgreSQL database aren't in the default schema, you can specify an alternate one with the `--postgres-schema` option:
db-to-sqlite ""postgresql://localhost/myblog"" blog.db \
--all \
--postgres-schema my_schema
## Using db-to-sqlite with Heroku Postgres
If you run an application on [Heroku](https://www.heroku.com/) using their [Postgres database product](https://www.heroku.com/postgres), you can use the `heroku config` command to access a compatible connection string:
$ heroku config --app myappname | grep HEROKU_POSTG
HEROKU_POSTGRESQL_OLIVE_URL: postgres://username:password@ec2-xxx-xxx-xxx-x.compute-1.amazonaws.com:5432/dbname
You can pass this to `db-to-sqlite` to create a local SQLite database with the data from your Heroku instance.
You can even do this using a bash one-liner:
$ db-to-sqlite $(heroku config --app myappname | grep HEROKU_POSTG | cut -d: -f 2-) \
/tmp/heroku.db --all -p
1/23: django_migrations
...
17/23: blog_blogmark
[####################################] 100%
...
## Related projects
* [Datasette](https://github.com/simonw/datasette): A tool for exploring and publishing data. Works great with SQLite files generated using `db-to-sqlite`.
* [sqlite-utils](https://github.com/simonw/sqlite-utils): Python CLI utility and library for manipulating SQLite databases.
* [csvs-to-sqlite](https://github.com/simonw/csvs-to-sqlite): Convert CSV files into a SQLite database.
## Development
To set up this tool locally, first checkout the code. Then create a new virtual environment:
cd db-to-sqlite
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
This will skip tests against MySQL or PostgreSQL if you do not have their additional dependencies installed.
You can install those extra dependencies like so:
pip install -e '.[test_mysql,test_postgresql]'
You can alternative use `pip install psycopg2-binary` if you cannot install the `psycopg2` dependency used by the `test_postgresql` extra.
See [Running a MySQL server using Homebrew](https://til.simonwillison.net/homebrew/mysql-homebrew) for tips on running the tests against MySQL on macOS, including how to install the `mysqlclient` dependency.
The PostgreSQL and MySQL tests default to expecting to run against servers on localhost. You can use environment variables to point them at different test database servers:
- `MYSQL_TEST_DB_CONNECTION` - defaults to `mysql://root@localhost/test_db_to_sqlite`
- `POSTGRESQL_TEST_DB_CONNECTION` - defaults to `postgresql://localhost/test_db_to_sqlite`
The database you indicate in the environment variable - `test_db_to_sqlite` by default - will be deleted and recreated on every test run.
","
db-to-sqlite
CLI tool for exporting tables or queries from any SQL database to a SQLite file.
Installation
Install from PyPI like so:
pip install db-to-sqlite
If you want to use it with MySQL, you can install the extra dependency like this:
pip install 'db-to-sqlite[mysql]'
Installing the mysqlclient library on OS X can be tricky - I've found this recipe to work (run that before installing db-to-sqlite).
For PostgreSQL, use this:
pip install 'db-to-sqlite[postgresql]'
Usage
Usage: db-to-sqlite [OPTIONS] CONNECTION PATH
Load data from any database into SQLite.
PATH is a path to the SQLite file to create, e.c. /tmp/my_database.db
CONNECTION is a SQLAlchemy connection string, for example:
postgresql://localhost/my_database
postgresql://username:passwd@localhost/my_database
mysql://root@localhost/my_database
mysql://username:passwd@localhost/my_database
More: https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls
Options:
--version Show the version and exit.
--all Detect and copy all tables
--table TEXT Specific tables to copy
--skip TEXT When using --all skip these tables
--redact TEXT... (table, column) pairs to redact with ***
--sql TEXT Optional SQL query to run
--output TEXT Table in which to save --sql query results
--pk TEXT Optional column to use as a primary key
--index-fks / --no-index-fks Should foreign keys have indexes? Default on
-p, --progress Show progress bar
--postgres-schema TEXT PostgreSQL schema to use
--help Show this message and exit.
For example, to save the content of the blog_entry table from a PostgreSQL database to a local file called blog.db you could do this:
You can also save the data from all of your tables, effectively creating a SQLite copy of your entire database. Any foreign key relationships will be detected and added to the SQLite database. For example:
If you want to save the results of a custom SQL query, do this:
db-to-sqlite ""postgresql://localhost/myblog"" output.db \
--output=query_results \
--sql=""select id, title, created from blog_entry"" \
--pk=id
The --output option specifies the table that should contain the results of the query.
Using db-to-sqlite with PostgreSQL schemas
If the tables you want to copy from your PostgreSQL database aren't in the default schema, you can specify an alternate one with the --postgres-schema option:
If you run an application on Heroku using their Postgres database product, you can use the heroku config command to access a compatible connection string:
Datasette: A tool for exploring and publishing data. Works great with SQLite files generated using db-to-sqlite.
sqlite-utils: Python CLI utility and library for manipulating SQLite databases.
csvs-to-sqlite: Convert CSV files into a SQLite database.
Development
To set up this tool locally, first checkout the code. Then create a new virtual environment:
cd db-to-sqlite
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
This will skip tests against MySQL or PostgreSQL if you do not have their additional dependencies installed.
You can install those extra dependencies like so:
pip install -e '.[test_mysql,test_postgresql]'
You can alternative use pip install psycopg2-binary if you cannot install the psycopg2 dependency used by the test_postgresql extra.
See Running a MySQL server using Homebrew for tips on running the tests against MySQL on macOS, including how to install the mysqlclient dependency.
The PostgreSQL and MySQL tests default to expecting to run against servers on localhost. You can use environment variables to point them at different test database servers:
MYSQL_TEST_DB_CONNECTION - defaults to mysql://root@localhost/test_db_to_sqlite
POSTGRESQL_TEST_DB_CONNECTION - defaults to postgresql://localhost/test_db_to_sqlite
The database you indicate in the environment variable - test_db_to_sqlite by default - will be deleted and recreated on every test run.
",,,,,,
167730071,MDEwOlJlcG9zaXRvcnkxNjc3MzAwNzE=,datasette-pretty-json,simonw/datasette-pretty-json,0,9599,https://github.com/simonw/datasette-pretty-json,Datasette plugin that pretty-prints any column values that are valid JSON objects or arrays,0,2019-01-26T19:30:43Z,2022-09-24T06:13:11Z,2022-09-28T21:06:31Z,,14,8,8,Python,1,1,1,1,0,0,0,0,1,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""json""]",0,1,8,master,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-pretty-json
[](https://pypi.org/project/datasette-pretty-json/)
[](https://github.com/simonw/datasette-pretty-json/releases)
[](https://github.com/simonw/datasette-pretty-json/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-pretty-json/blob/main/LICENSE)
[Datasette](https://github.com/simonw/datasette) plugin that pretty-prints any column values that are valid JSON objects or arrays.
You may also be interested in [datasette-json-html](https://github.com/simonw/datasette-json-html).
","
datasette-pretty-json
Datasette plugin that pretty-prints any column values that are valid JSON objects or arrays.
",1,public,0,,0,
167759846,MDEwOlJlcG9zaXRvcnkxNjc3NTk4NDY=,markdown-to-sqlite,simonw/markdown-to-sqlite,0,9599,https://github.com/simonw/markdown-to-sqlite,CLI tool for loading markdown files into a SQLite database,0,2019-01-27T02:04:54Z,2022-05-13T18:09:26Z,2022-05-13T18:09:22Z,,13,49,49,Python,1,1,1,1,0,2,0,0,2,apache-2.0,"[""datasette-io"", ""datasette-tool"", ""markdown"", ""sqlite"", ""yaml""]",2,2,49,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,2,3,"# markdown-to-sqlite
[](https://pypi.python.org/pypi/markdown-to-sqlite)
[](https://github.com/simonw/markdown-to-sqlite/releases)
[](https://github.com/simonw/markdown-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/markdown-to-sqlite/blob/main/LICENSE)
CLI tool for loading markdown files into a SQLite database.
YAML embedded in the markdown files will be used to populate additional columns.
Usage: markdown-to-sqlite [OPTIONS] DBNAME TABLE PATHS...
For example:
$ markdown-to-sqlite docs.db documents file1.md file2.md
## Breaking change
Prior to version 1.0 this argument order was different - markdown files were listed before the database and table.
","
markdown-to-sqlite
CLI tool for loading markdown files into a SQLite database.
YAML embedded in the markdown files will be used to populate additional columns.
Prior to version 1.0 this argument order was different - markdown files were listed before the database and table.
",1,public,0,,,
168474970,MDEwOlJlcG9zaXRvcnkxNjg0NzQ5NzA=,dbf-to-sqlite,simonw/dbf-to-sqlite,0,9599,https://github.com/simonw/dbf-to-sqlite,"CLI tool for converting DBF files (dBase, FoxPro etc) to SQLite",0,2019-01-31T06:30:46Z,2021-03-23T01:29:41Z,2020-02-16T00:41:20Z,,8,25,25,Python,1,1,1,1,0,8,0,0,3,apache-2.0,"[""sqlite"", ""foxpro"", ""dbf"", ""dbase"", ""datasette-io"", ""datasette-tool""]",8,3,25,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,8,2,"# dbf-to-sqlite
[](https://pypi.python.org/pypi/dbf-to-sqlite)
[](https://travis-ci.com/simonw/dbf-to-sqlite)
[](https://github.com/simonw/dbf-to-sqlite/blob/master/LICENSE)
CLI tool for converting DBF files (dBase, FoxPro etc) to SQLite.
## Installation
pip install dbf-to-sqlite
## Usage
$ dbf-to-sqlite --help
Usage: dbf-to-sqlite [OPTIONS] DBF_PATHS... SQLITE_DB
Convert DBF files (dBase, FoxPro etc) to SQLite
https://github.com/simonw/dbf-to-sqlite
Options:
--version Show the version and exit.
--table TEXT Table name to use (only valid for single files)
-v, --verbose Show what's going on
--help Show this message and exit.
Example usage:
$ dbf-to-sqlite *.DBF database.db
This will create a new SQLite database called `database.db` containing one table for each of the `DBF` files in the current directory.
Looking for DBF files to try this out on? Try downloading the [Himalayan Database](http://himalayandatabase.com/) of all expeditions that have climbed in the Nepal Himalaya.
","
dbf-to-sqlite
CLI tool for converting DBF files (dBase, FoxPro etc) to SQLite.
Installation
pip install dbf-to-sqlite
Usage
$ dbf-to-sqlite --help
Usage: dbf-to-sqlite [OPTIONS] DBF_PATHS... SQLITE_DB
Convert DBF files (dBase, FoxPro etc) to SQLite
https://github.com/simonw/dbf-to-sqlite
Options:
--version Show the version and exit.
--table TEXT Table name to use (only valid for single files)
-v, --verbose Show what's going on
--help Show this message and exit.
Example usage:
$ dbf-to-sqlite *.DBF database.db
This will create a new SQLite database called database.db containing one table for each of the DBF files in the current directory.
Looking for DBF files to try this out on? Try downloading the Himalayan Database of all expeditions that have climbed in the Nepal Himalaya.
",,,,,,
174715153,MDEwOlJlcG9zaXRvcnkxNzQ3MTUxNTM=,datasette-jellyfish,simonw/datasette-jellyfish,0,9599,https://github.com/simonw/datasette-jellyfish,Datasette plugin adding SQL functions for fuzzy text matching powered by Jellyfish,0,2019-03-09T16:02:01Z,2021-02-06T02:33:49Z,2021-02-06T02:34:18Z,https://datasette.io/plugins/datasette-jellyfish,15,9,9,Python,1,1,1,1,0,2,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",2,0,9,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,2,1,"# datasette-jellyfish
[](https://pypi.org/project/datasette-jellyfish/)
[](https://github.com/simonw/datasette-jellyfish/releases)
[](https://github.com/simonw/datasette-jellyfish/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-jellyfish/blob/main/LICENSE)
Datasette plugin that adds custom SQL functions for fuzzy string matching, built on top of the [Jellyfish](https://github.com/jamesturk/jellyfish) Python library by James Turk and Michael Stephens.
Interactive demos:
* [soundex, metaphone, nysiis, match_rating_codex comparison](https://latest-with-plugins.datasette.io/fixtures?sql=SELECT%0D%0A++++soundex%28%3As%29%2C+%0D%0A++++metaphone%28%3As%29%2C+%0D%0A++++nysiis%28%3As%29%2C+%0D%0A++++match_rating_codex%28%3As%29&s=demo).
* [distance functions comparison](https://latest-with-plugins.datasette.io/fixtures?sql=SELECT%0D%0A++++levenshtein_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++damerau_levenshtein_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++hamming_distance%28%3As1%2C+%3As2%29%2C%0D%0A++++jaro_similarity%28%3As1%2C+%3As2%29%2C%0D%0A++++jaro_winkler_similarity%28%3As1%2C+%3As2%29%2C%0D%0A++++match_rating_comparison%28%3As1%2C+%3As2%29%3B&s1=barrack+obama&s2=barrack+h+obama)
Examples:
SELECT soundex(""hello"");
-- Outputs H400
SELECT metaphone(""hello"");
-- Outputs HL
SELECT nysiis(""hello"");
-- Outputs HAL
SELECT match_rating_codex(""hello"");
-- Outputs HLL
SELECT porter_stem(""running"");
-- Outputs run
SELECT levenshtein_distance(""hello"", ""hello world"");
-- Outputs 6
SELECT damerau_levenshtein_distance(""hello"", ""hello world"");
-- Outputs 6
SELECT hamming_distance(""hello"", ""hello world"");
-- Outputs 6
SELECT jaro_similarity(""hello"", ""hello world"");
-- Outputs 0.8181818181818182
SELECT jaro_winkler_similarity(""hello"", ""hello world"");
-- Outputs 0.890909090909091
SELECT match_rating_comparison(""hello"", ""helloo"");
-- Outputs 1
See [the Jellyfish documentation](https://jellyfish.readthedocs.io/en/latest/) for an explanation of each of these functions.","
datasette-jellyfish
Datasette plugin that adds custom SQL functions for fuzzy string matching, built on top of the Jellyfish Python library by James Turk and Michael Stephens.
",,,,,,
175321497,MDEwOlJlcG9zaXRvcnkxNzUzMjE0OTc=,csv-diff,simonw/csv-diff,0,9599,https://github.com/simonw/csv-diff,Python CLI tool and library for diffing CSV and JSON files,0,2019-03-13T01:11:26Z,2022-07-29T20:01:02Z,2022-07-29T20:00:59Z,,34,198,198,Python,1,1,1,1,0,29,0,0,18,apache-2.0,"[""click"", ""csv"", ""datasette-io"", ""datasette-tool"", ""diff"", ""git-scraping""]",29,18,198,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,29,7,"# csv-diff
[](https://pypi.org/project/csv-diff/)
[](https://github.com/simonw/csv-diff/releases)
[](https://github.com/simonw/csv-diff/actions?query=workflow%3ATest)
[](https://github.com/simonw/csv-diff/blob/main/LICENSE)
Tool for viewing the difference between two CSV, TSV or JSON files. See [Generating a commit log for San Francisco’s official list of trees](https://simonwillison.net/2019/Mar/13/tree-history/) (and the [sf-tree-history repo commit log](https://github.com/simonw/sf-tree-history/commits)) for background information on this project.
## Installation
pip install csv-diff
## Usage
Consider two CSV files:
`one.csv`
id,name,age
1,Cleo,4
2,Pancakes,2
`two.csv`
id,name,age
1,Cleo,5
3,Bailey,1
`csv-diff` can show a human-readable summary of differences between the files:
$ csv-diff one.csv two.csv --key=id
1 row changed, 1 row added, 1 row removed
1 row changed
Row 1
age: ""4"" => ""5""
1 row added
id: 3
name: Bailey
age: 1
1 row removed
id: 2
name: Pancakes
age: 2
The `--key=id` option means that the `id` column should be treated as the unique key, to identify which records have changed.
The tool will automatically detect if your files are comma- or tab-separated. You can over-ride this automatic detection and force the tool to use a specific format using `--format=tsv` or `--format=csv`.
You can also feed it JSON files, provided they are a JSON array of objects where each object has the same keys. Use `--format=json` if your input files are JSON.
Use `--show-unchanged` to include full details of the unchanged values for rows with at least one change in the diff output:
% csv-diff one.csv two.csv --key=id --show-unchanged
1 row changed
id: 1
age: ""4"" => ""5""
Unchanged:
name: ""Cleo""
You can use the `--json` option to get a machine-readable difference:
$ csv-diff one.csv two.csv --key=id --json
{
""added"": [
{
""id"": ""3"",
""name"": ""Bailey"",
""age"": ""1""
}
],
""removed"": [
{
""id"": ""2"",
""name"": ""Pancakes"",
""age"": ""2""
}
],
""changed"": [
{
""key"": ""1"",
""changes"": {
""age"": [
""4"",
""5""
]
}
}
],
""columns_added"": [],
""columns_removed"": []
}
## As a Python library
You can also import the Python library into your own code like so:
from csv_diff import load_csv, compare
diff = compare(
load_csv(open(""one.csv""), key=""id""),
load_csv(open(""two.csv""), key=""id"")
)
`diff` will now contain the same data structure as the output in the `--json` example above.
If the columns in the CSV have changed, those added or removed columns will be ignored when calculating changes made to specific rows.
## As a Docker container
### Build the image
$ docker build -t csvdiff .
### Run the container
$ docker run --rm -v $(pwd):/files csvdiff
Suppose current directory contains two csv files : one.csv two.csv
$ docker run --rm -v $(pwd):/files csvdiff one.csv two.csv
## Alternatives
- [csvdiff](https://github.com/aswinkarthik/csvdiff) is a ""fast diff tool for comparing CSV files"" - you may get better results from this than from `csv-diff` against larger files.
","
The --key=id option means that the id column should be treated as the unique key, to identify which records have changed.
The tool will automatically detect if your files are comma- or tab-separated. You can over-ride this automatic detection and force the tool to use a specific format using --format=tsv or --format=csv.
You can also feed it JSON files, provided they are a JSON array of objects where each object has the same keys. Use --format=json if your input files are JSON.
Use --show-unchanged to include full details of the unchanged values for rows with at least one change in the diff output:
diff will now contain the same data structure as the output in the --json example above.
If the columns in the CSV have changed, those added or removed columns will be ignored when calculating changes made to specific rows.
As a Docker container
Build the image
$ docker build -t csvdiff .
Run the container
$ docker run --rm -v $(pwd):/files csvdiff
Suppose current directory contains two csv files : one.csv two.csv
$ docker run --rm -v $(pwd):/files csvdiff one.csv two.csv
Alternatives
csvdiff is a ""fast diff tool for comparing CSV files"" - you may get better results from this than from csv-diff against larger files.
",1,public,0,,0,
175550127,MDEwOlJlcG9zaXRvcnkxNzU1NTAxMjc=,yaml-to-sqlite,simonw/yaml-to-sqlite,0,9599,https://github.com/simonw/yaml-to-sqlite,Utility for converting YAML files to SQLite,0,2019-03-14T04:49:08Z,2021-06-13T09:04:40Z,2021-06-13T04:45:52Z,,19,36,36,Python,1,1,1,1,0,2,0,0,0,apache-2.0,"[""yaml"", ""sqlite"", ""datasette-io"", ""datasette-tool""]",2,0,36,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,2,1,"# yaml-to-sqlite
[](https://pypi.org/project/yaml-to-sqlite/)
[](https://github.com/simonw/yaml-to-sqlite/releases)
[](https://github.com/simonw/yaml-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/yaml-to-sqlite/blob/main/LICENSE)
Load the contents of a YAML file into a SQLite database table.
```
$ yaml-to-sqlite --help
Usage: yaml-to-sqlite [OPTIONS] DB_PATH TABLE YAML_FILE
Convert YAML files to SQLite
Options:
--version Show the version and exit.
--pk TEXT Column to use as a primary key
--single-column TEXT If YAML file is a list of values, populate this column
--help Show this message and exit.
```
## Usage
Given a `news.yml` file containing the following:
```yaml
- date: 2021-06-05
body: |-
[Datasette 0.57](https://docs.datasette.io/en/stable/changelog.html#v0-57) is out with an important security patch.
- date: 2021-05-10
body: |-
[Django SQL Dashboard](https://simonwillison.net/2021/May/10/django-sql-dashboard/) is a new tool that brings a useful authenticated subset of Datasette to Django projects that are built on top of PostgreSQL.
```
Running this command:
```bash
$ yaml-to-sqlite news.db stories news.yml
```
Will create a database file with this schema:
```bash
$ sqlite-utils schema news.db
CREATE TABLE [stories] (
[date] TEXT,
[body] TEXT
);
```
The `--pk` option can be used to set a column as the primary key for the table:
```bash
$ yaml-to-sqlite news.db stories news.yml --pk date
$ sqlite-utils schema news.db
CREATE TABLE [stories] (
[date] TEXT PRIMARY KEY,
[body] TEXT
);
```
## Single column YAML lists
The `--single-column` option can be used when the YAML file is a list of values, for example a file called `dogs.yml` containing the following:
```yaml
- Cleo
- Pancakes
- Nixie
```
Running this command:
```bash
$ yaml-to-sqlite dogs.db dogs.yaml --single-column=name
```
Will create a single `dogs` table with a single `name` column that is the primary key:
```bash
$ sqlite-utils schema dogs.db
CREATE TABLE [dogs] (
[name] TEXT PRIMARY KEY
);
$ sqlite-utils dogs.db 'select * from dogs' -t
name
--------
Cleo
Pancakes
Nixie
```
","
yaml-to-sqlite
Load the contents of a YAML file into a SQLite database table.
$ yaml-to-sqlite --help
Usage: yaml-to-sqlite [OPTIONS] DB_PATH TABLE YAML_FILE
Convert YAML files to SQLite
Options:
--version Show the version and exit.
--pk TEXT Column to use as a primary key
--single-column TEXT If YAML file is a list of values, populate this column
--help Show this message and exit.
Usage
Given a news.yml file containing the following:
- date: 2021-06-05body: |- [Datasette 0.57](https://docs.datasette.io/en/stable/changelog.html#v0-57) is out with an important security patch.
- date: 2021-05-10body: |- [Django SQL Dashboard](https://simonwillison.net/2021/May/10/django-sql-dashboard/) is a new tool that brings a useful authenticated subset of Datasette to Django projects that are built on top of PostgreSQL.
Will create a single dogs table with a single name column that is the primary key:
$ sqlite-utils schema dogs.db
CREATE TABLE [dogs] (
[name] TEXT PRIMARY KEY
);
$ sqlite-utils dogs.db 'select * from dogs' -t
name
--------
Cleo
Pancakes
Nixie
",,,,,,
189321671,MDEwOlJlcG9zaXRvcnkxODkzMjE2NzE=,datasette-jq,simonw/datasette-jq,0,9599,https://github.com/simonw/datasette-jq,Datasette plugin that adds a custom SQL function for executing jq expressions against JSON values,0,2019-05-30T01:06:31Z,2020-12-24T17:35:27Z,2020-04-09T05:43:43Z,,11,10,10,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""jq"", ""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,10,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,2,"# datasette-jq
[](https://pypi.org/project/datasette-jq/)
[](https://circleci.com/gh/simonw/datasette-jq)
[](https://github.com/simonw/datasette-jq/blob/master/LICENSE)
Datasette plugin that adds custom SQL functions for executing [jq](https://stedolan.github.io/jq/) expressions against JSON values.
Install this plugin in the same environment as Datasette to enable the `jq()` SQL function.
Usage:
select jq(
column_with_json,
""{top_3: .classifiers[:3], v: .version}""
)
See [the jq manual](https://stedolan.github.io/jq/manual/#Basicfilters) for full details of supported expression syntax.
## Interactive demo
You can try this plugin out at [datasette-jq-demo.datasette.io](https://datasette-jq-demo.datasette.io/)
Sample query:
select package, ""https://pypi.org/project/"" || package || ""/"" as url,
jq(info, ""{summary: .info.summary, author: .info.author, versions: .releases|keys|reverse}"")
from packages
[Try this query out](https://datasette-jq-demo.datasette.io/demo?sql=select+package%2C+%22https%3A%2F%2Fpypi.org%2Fproject%2F%22+%7C%7C+package+%7C%7C+%22%2F%22+as+url%2C%0D%0Ajq%28info%2C+%22%7Bsummary%3A+.info.summary%2C+author%3A+.info.author%2C+versions%3A+.releases%7Ckeys%7Creverse%7D%22%29%0D%0Afrom+packages) in the interactive demo.
","
datasette-jq
Datasette plugin that adds custom SQL functions for executing jq expressions against JSON values.
Install this plugin in the same environment as Datasette to enable the jq() SQL function.
",,,,,,
190950781,MDEwOlJlcG9zaXRvcnkxOTA5NTA3ODE=,datasette-bplist,simonw/datasette-bplist,0,9599,https://github.com/simonw/datasette-bplist,Datasette plugin for working with Apple's binary plist format,0,2019-06-09T01:15:01Z,2021-06-07T18:05:00Z,2019-06-09T01:17:19Z,,7,9,9,Python,1,1,1,1,0,0,0,0,1,apache-2.0,"[""bplist"", ""datasette"", ""datasette-plugin"", ""datasette-io""]",0,1,9,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,0,"# datasette-bplist
[](https://pypi.org/project/datasette-bplist/)
[](https://circleci.com/gh/simonw/datasette-bplist)
[](https://github.com/simonw/datasette-bplist/blob/master/LICENSE)
Datasette plugin for working with Apple's [binary plist](https://en.wikipedia.org/wiki/Property_list) format.
This plugin adds two features: a display hook and a SQL function.
The display hook will detect any database values that are encoded using the binary plist format. It will decode them, convert them into JSON and display them pretty-printed in the Datasette UI.
The SQL function `bplist_to_json(value)` can be used inside a SQL query to convert a binary plist value into a JSON string. This can then be used with SQLite's `json_extract()` function or with the [datasette-jq](https://github.com/simonw/datasette-jq) plugin to further analyze that data as part of a SQL query.
Install this plugin in the same environment as Datasette to enable this new functionality:
pip install datasette-bplist
## Trying it out
If you use a Mac you already have plenty of SQLite databases that contain binary plist data.
One example is the database that powers the Apple Photos app.
This database tends to be locked, so you will need to create a copy of the database in order to run queries against it:
cp ~/Pictures/Photos\ Library.photoslibrary/database/photos.db /tmp/photos.db
The database also makes use of custom SQLite extensions which prevent it from opening in Datasette.
You can work around this by exporting the data that you want to experiment with into a new SQLite file.
I recommend trying this plugin against the `RKMaster_dataNote` table, which contains plist-encoded EXIF metadata about the photos you have taken.
You can export that table into a fresh database like so:
sqlite3 /tmp/photos.db "".dump RKMaster_dataNote"" | sqlite3 /tmp/exif.db
Now run `datasette /tmp/exif.db` and you can start trying out the plugin.
## Using the bplist_to_json() SQL function
Once you have the `exif.db` demo working, you can try the `bplist_to_json()` SQL function.
Here's a query that shows the camera lenses you have used the most often to take photos:
select
json_extract(
bplist_to_json(value),
""$.{Exif}.LensModel""
) as lens,
count(*) as n
from RKMaster_dataNote
group by lens
order by n desc;
If you have a large number of photos this query can take a long time to execute, so you may need to increase the SQL time limit enforced by Datasette like so:
$ datasette /tmp/exif.db \
--config sql_time_limit_ms:10000
Here's another query, showing the time at which you took every photo in your library which is classified as as screenshot:
select
attachedToId,
json_extract(
bplist_to_json(value),
""$.{Exif}.DateTimeOriginal""
)
from RKMaster_dataNote
where
json_extract(
bplist_to_json(value),
""$.{Exif}.UserComment""
) = ""Screenshot""
And if you install the [datasette-cluster-map](https://github.com/simonw/datasette-cluster-map) plugin, this query will show you a map of your most recent 1000 photos:
select
*,
json_extract(
bplist_to_json(value),
""$.{GPS}.Latitude""
) as latitude,
-json_extract(
bplist_to_json(value),
""$.{GPS}.Longitude""
) as longitude,
json_extract(
bplist_to_json(value),
""$.{Exif}.DateTimeOriginal""
) as datetime
from
RKMaster_dataNote
where
latitude is not null
order by
attachedToId desc
","
datasette-bplist
Datasette plugin for working with Apple's binary plist format.
This plugin adds two features: a display hook and a SQL function.
The display hook will detect any database values that are encoded using the binary plist format. It will decode them, convert them into JSON and display them pretty-printed in the Datasette UI.
The SQL function bplist_to_json(value) can be used inside a SQL query to convert a binary plist value into a JSON string. This can then be used with SQLite's json_extract() function or with the datasette-jq plugin to further analyze that data as part of a SQL query.
Install this plugin in the same environment as Datasette to enable this new functionality:
pip install datasette-bplist
Trying it out
If you use a Mac you already have plenty of SQLite databases that contain binary plist data.
One example is the database that powers the Apple Photos app.
This database tends to be locked, so you will need to create a copy of the database in order to run queries against it:
Now run datasette /tmp/exif.db and you can start trying out the plugin.
Using the bplist_to_json() SQL function
Once you have the exif.db demo working, you can try the bplist_to_json() SQL function.
Here's a query that shows the camera lenses you have used the most often to take photos:
select
json_extract(
bplist_to_json(value),
""$.{Exif}.LensModel""
) as lens,
count(*) as n
from RKMaster_dataNote
group by lens
order by n desc;
If you have a large number of photos this query can take a long time to execute, so you may need to increase the SQL time limit enforced by Datasette like so:
Here's another query, showing the time at which you took every photo in your library which is classified as as screenshot:
select
attachedToId,
json_extract(
bplist_to_json(value),
""$.{Exif}.DateTimeOriginal""
)
from RKMaster_dataNote
where
json_extract(
bplist_to_json(value),
""$.{Exif}.UserComment""
) = ""Screenshot""
And if you install the datasette-cluster-map plugin, this query will show you a map of your most recent 1000 photos:
select
*,
json_extract(
bplist_to_json(value),
""$.{GPS}.Latitude""
) as latitude,
-json_extract(
bplist_to_json(value),
""$.{GPS}.Longitude""
) as longitude,
json_extract(
bplist_to_json(value),
""$.{Exif}.DateTimeOriginal""
) as datetime
from
RKMaster_dataNote
where
latitude is not null
order by
attachedToId desc
",,,,,,
191022928,MDEwOlJlcG9zaXRvcnkxOTEwMjI5Mjg=,datasette-render-binary,simonw/datasette-render-binary,0,9599,https://github.com/simonw/datasette-render-binary,Datasette plugin for rendering binary data,0,2019-06-09T15:25:52Z,2021-06-02T09:29:20Z,2019-06-13T16:14:31Z,,62,7,7,Python,1,1,1,1,0,0,0,0,1,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,1,7,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-render-binary
[](https://pypi.org/project/datasette-render-binary/)
[](https://circleci.com/gh/simonw/datasette-render-binary)
[](https://github.com/simonw/datasette-render-binary/blob/master/LICENSE)
Datasette plugin for rendering binary data.
Install this plugin in the same environment as Datasette to enable this new functionality:
pip install datasette-render-binary
Binary data in cells will now be rendered as a mixture of characters and octets.
","
datasette-render-binary
Datasette plugin for rendering binary data.
Install this plugin in the same environment as Datasette to enable this new functionality:
pip install datasette-render-binary
Binary data in cells will now be rendered as a mixture of characters and octets.
",,,,,,
195087137,MDEwOlJlcG9zaXRvcnkxOTUwODcxMzc=,datasette-auth-github,simonw/datasette-auth-github,0,9599,https://github.com/simonw/datasette-auth-github,Datasette plugin that authenticates users against GitHub,0,2019-07-03T16:02:53Z,2021-06-03T11:42:54Z,2021-02-25T06:40:17Z,https://datasette-auth-github-demo.datasette.io/,119,34,34,Python,1,1,1,1,0,4,0,0,3,apache-2.0,"[""asgi"", ""datasette"", ""datasette-plugin"", ""datasette-io""]",4,3,34,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,4,1,"# datasette-auth-github
[](https://pypi.org/project/datasette-auth-github/)
[](https://github.com/simonw/datasette-auth-github/releases)
[](https://github.com/simonw/datasette-auth-github/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-auth-github/blob/main/LICENSE)
Datasette plugin that authenticates users against GitHub.
- [Setup instructions](#setup-instructions)
- [The authenticated actor](#the-authenticated-actor)
- [Restricting access to specific users](#restricting-access-to-specific-users)
- [Restricting access to specific GitHub organizations or teams](#restricting-access-to-specific-github-organizations-or-teams)
- [What to do if a user is removed from an organization or team](#what-to-do-if-a-user-is-removed-from-an-organization-or-team)
## Setup instructions
* Install the plugin: `datasette install datasette-auth-github`
* Create a GitHub OAuth app: https://github.com/settings/applications/new
* Set the Authorization callback URL to `http://127.0.0.1:8001/-/github-auth-callback`
* Create a `metadata.json` file with the following structure:
```json
{
""title"": ""datasette-auth-github demo"",
""plugins"": {
""datasette-auth-github"": {
""client_id"": {""$env"": ""GITHUB_CLIENT_ID""},
""client_secret"": {""$env"": ""GITHUB_CLIENT_SECRET""}
}
}
}
```
Now you can start Datasette like this, passing in the secrets as environment variables:
$ GITHUB_CLIENT_ID=XXX GITHUB_CLIENT_SECRET=YYY datasette \
fixtures.db -m metadata.json
Note that hard-coding secrets in `metadata.json` is a bad idea as they will be visible to anyone who can navigate to `/-/metadata`. Instead, we use Datasette's mechanism for [adding secret plugin configuration options](https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values).
By default anonymous users will still be able to interact with Datasette. If you wish all users to have to sign in with a GitHub account first, add this to your ``metadata.json``:
```json
{
""allow"": {
""id"": ""*""
},
""plugins"": {
""datasette-auth-github"": {
""..."": ""...""
}
}
}
```
## The authenticated actor
Visit `/-/actor` when signed in to see the shape of the authenticated actor. It should look something like this:
```json
{
""actor"": {
""display"": ""simonw"",
""gh_id"": ""9599"",
""gh_name"": ""Simon Willison"",
""gh_login"": ""simonw"",
""gh_email"": ""..."",
""gh_orgs"": [
""dogsheep"",
""datasette-project""
],
""gh_teams"": [
""dogsheep/test""
]
}
}
```
The `gh_orgs` and `gh_teams` properties will only be present if you used `load_teams` or `load_orgs`, documented below.
## Restricting access to specific users
You can use Datasette's [permissions mechanism](https://docs.datasette.io/en/stable/authentication.html) to specify which user or users are allowed to access your instance. Here's how to restrict access to just GitHub user `simonw`:
```json
{
""allow"": {
""gh_login"": ""simonw""
},
""plugins"": {
""datasette-auth-github"": {
""..."": ""...""
}
}
}
```
This `""allow""` block can be positioned at the database, table or query level instead: see [Configuring permissions in metadata.json](https://docs.datasette.io/en/stable/authentication.html#configuring-permissions-in-metadata-json) for details.
Note that GitHub allows users to change their username, and it is possible for other people to claim old usernames. If you are concerned that your users may change their usernames you can key the allow blocks against GitHub user IDs instead, which do not change:
```json
{
""allow"": {
""gh_id"": ""9599""
}
}
```
## Restricting access to specific GitHub organizations or teams
You can also restrict access to users who are members of a specific GitHub organization.
You'll need to configure the plugin to check if the user is a member of that organization when they first sign in. You can do that using the `""load_orgs""` plugin configuration option.
Then you can use `""allow"": {""gh_orgs"": [...]}` to specify which organizations are allowed access.
```json
{
""plugins"": {
""datasette-auth-github"": {
""..."": ""..."",
""load_orgs"": [""your-organization""]
}
},
""allow"": {
""gh_orgs"": ""your-organization""
}
}
```
If your organization is [arranged into teams](https://help.github.com/en/articles/organizing-members-into-teams) you can restrict access to a specific team like this:
```json
{
""plugins"": {
""datasette-auth-github"": {
""..."": ""..."",
""load_teams"": [
""your-organization/staff"",
""your-organization/engineering"",
]
}
},
""allows"": {
""gh_team"": ""your-organization/engineering""
}
}
```
## What to do if a user is removed from an organization or team
A user's organization and team memberships are checked once, when they first sign in. Those teams and organizations are then persisted in the user's signed `ds_actor` cookie.
This means that if a user is removed from an organization or team but still has a Datasette cookie, they will still be able to access that Datasette instance.
You can remedy this by rotating the `DATASETTE_SECRET` environment variable any time you make changes to your GitHub organization members.
Changing this value will cause all of your existing users to be signed out, by invalidating their cookies. When they sign back in again their new memberships will be recorded in a new cookie.
See [Configuring the secret](https://docs.datasette.io/en/stable/settings.html?highlight=secret#configuring-the-secret) in the Datasette documentation for more details.
","
datasette-auth-github
Datasette plugin that authenticates users against GitHub.
Note that hard-coding secrets in metadata.json is a bad idea as they will be visible to anyone who can navigate to /-/metadata. Instead, we use Datasette's mechanism for adding secret plugin configuration options.
By default anonymous users will still be able to interact with Datasette. If you wish all users to have to sign in with a GitHub account first, add this to your metadata.json:
The gh_orgs and gh_teams properties will only be present if you used load_teams or load_orgs, documented below.
Restricting access to specific users
You can use Datasette's permissions mechanism to specify which user or users are allowed to access your instance. Here's how to restrict access to just GitHub user simonw:
Note that GitHub allows users to change their username, and it is possible for other people to claim old usernames. If you are concerned that your users may change their usernames you can key the allow blocks against GitHub user IDs instead, which do not change:
{
""allow"": {
""gh_id"": ""9599""
}
}
Restricting access to specific GitHub organizations or teams
You can also restrict access to users who are members of a specific GitHub organization.
You'll need to configure the plugin to check if the user is a member of that organization when they first sign in. You can do that using the ""load_orgs"" plugin configuration option.
Then you can use ""allow"": {""gh_orgs"": [...]} to specify which organizations are allowed access.
What to do if a user is removed from an organization or team
A user's organization and team memberships are checked once, when they first sign in. Those teams and organizations are then persisted in the user's signed ds_actor cookie.
This means that if a user is removed from an organization or team but still has a Datasette cookie, they will still be able to access that Datasette instance.
You can remedy this by rotating the DATASETTE_SECRET environment variable any time you make changes to your GitHub organization members.
Changing this value will cause all of your existing users to be signed out, by invalidating their cookies. When they sign back in again their new memberships will be recorded in a new cookie.
",,,,,,
195145678,MDEwOlJlcG9zaXRvcnkxOTUxNDU2Nzg=,sqlite-diffable,simonw/sqlite-diffable,0,9599,https://github.com/simonw/sqlite-diffable,Tools for dumping/loading a SQLite database to diffable directory structure,0,2019-07-04T00:58:46Z,2022-07-12T17:00:19Z,2022-08-18T22:49:29Z,,30,42,42,Python,1,1,1,1,0,3,0,0,3,apache-2.0,"[""datasette-io"", ""datasette-tool"", ""sqlite""]",3,3,42,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,3,1,"# sqlite-diffable
[](https://pypi.org/project/sqlite-diffable/)
[](https://github.com/simonw/sqlite-diffable/releases)
[](https://github.com/simonw/sqlite-diffable/blob/main/LICENSE)
Tools for dumping/loading a SQLite database to diffable directory structure
## Installation
pip install sqlite-diffable
## Demo
The repository at [simonw/simonwillisonblog-backup](https://github.com/simonw/simonwillisonblog-backup) contains a backup of the database on my blog, https://simonwillison.net/ - created using this tool.
## Dumping a database
Given a SQLite database called `fixtures.db` containing a table `facetable`, the following will dump out that table to the `dump/` directory:
sqlite-diffable dump fixtures.db dump/ facetable
To dump out every table in that database, use `--all`:
sqlite-diffable dump fixtures.db dump/ --all
## Loading a database
To load a previously dumped database, run the following:
sqlite-diffable load restored.db dump/
This will show an error if any of the tables that are being restored already exist in the database file.
You can replace those tables (dropping them before restoring them) using the `--replace` option:
sqlite-diffable load restored.db dump/ --replace
## Converting to JSON objects
Table rows are stored in the `.ndjson` files as newline-delimited JSON arrays, like this:
```
[""a"", ""a"", ""a-a"", 63, null, 0.7364712141640124, ""$null""]
[""a"", ""b"", ""a-b"", 51, null, 0.6020187290499803, ""$null""]
```
Sometimes it can be more convenient to work with a list of JSON objects.
The `sqlite-diffable objects` command can read a `.ndjson` file and its accompanying `.metadata.json` file and output JSON objects to standard output:
sqlite-diffable objects fixtures.db dump/sortable.ndjson
The output of that command looks something like this:
```
{""pk1"": ""a"", ""pk2"": ""a"", ""content"": ""a-a"", ""sortable"": 63, ""sortable_with_nulls"": null, ""sortable_with_nulls_2"": 0.7364712141640124, ""text"": ""$null""}
{""pk1"": ""a"", ""pk2"": ""b"", ""content"": ""a-b"", ""sortable"": 51, ""sortable_with_nulls"": null, ""sortable_with_nulls_2"": 0.6020187290499803, ""text"": ""$null""}
```
Add `-o` to write that output to a file:
sqlite-diffable objects fixtures.db dump/sortable.ndjson -o output.txt
Add `--array` to output a JSON array of objects, as opposed to a newline-delimited file:
sqlite-diffable objects fixtures.db dump/sortable.ndjson --array
Output:
```
[
{""pk1"": ""a"", ""pk2"": ""a"", ""content"": ""a-a"", ""sortable"": 63, ""sortable_with_nulls"": null, ""sortable_with_nulls_2"": 0.7364712141640124, ""text"": ""$null""},
{""pk1"": ""a"", ""pk2"": ""b"", ""content"": ""a-b"", ""sortable"": 51, ""sortable_with_nulls"": null, ""sortable_with_nulls_2"": 0.6020187290499803, ""text"": ""$null""}
]
```
## Storage format
Each table is represented as two files. The first, `table_name.metadata.json`, contains metadata describing the structure of the table. For a table called `redirects_redirect` that file might look like this:
```json
{
""name"": ""redirects_redirect"",
""columns"": [
""id"",
""domain"",
""path"",
""target"",
""created""
],
""schema"": ""CREATE TABLE [redirects_redirect] (\n [id] INTEGER PRIMARY KEY,\n [domain] TEXT,\n [path] TEXT,\n [target] TEXT,\n [created] TEXT\n)""
}
```
It is an object with three keys: `name` is the name of the table, `columns` is an array of column strings and `schema` is the SQL schema text used for tha table.
The second file, `table_name.ndjson`, contains [newline-delimited JSON](http://ndjson.org/) for every row in the table. Each row is represented as a JSON array with items corresponding to each of the columns defined in the metadata.
That file for the `redirects_redirect.ndjson` table might look like this:
```
[1, ""feeds.simonwillison.net"", ""swn-everything"", ""https://simonwillison.net/atom/everything/"", ""2017-10-01T21:11:36.440537+00:00""]
[2, ""feeds.simonwillison.net"", ""swn-entries"", ""https://simonwillison.net/atom/entries/"", ""2017-10-01T21:12:32.478849+00:00""]
[3, ""feeds.simonwillison.net"", ""swn-links"", ""https://simonwillison.net/atom/links/"", ""2017-10-01T21:12:54.820729+00:00""]
```
","
sqlite-diffable
Tools for dumping/loading a SQLite database to diffable directory structure
Each table is represented as two files. The first, table_name.metadata.json, contains metadata describing the structure of the table. For a table called redirects_redirect that file might look like this:
It is an object with three keys: name is the name of the table, columns is an array of column strings and schema is the SQL schema text used for tha table.
The second file, table_name.ndjson, contains newline-delimited JSON for every row in the table. Each row is represented as a JSON array with items corresponding to each of the columns defined in the metadata.
That file for the redirects_redirect.ndjson table might look like this:
",1,public,0,,0,
195696804,MDEwOlJlcG9zaXRvcnkxOTU2OTY4MDQ=,datasette-cors,simonw/datasette-cors,0,9599,https://github.com/simonw/datasette-cors,Datasette plugin for configuring CORS headers,0,2019-07-07T21:03:11Z,2021-02-27T00:31:13Z,2019-07-11T04:40:57Z,,11,9,9,Python,1,1,1,1,0,0,0,0,1,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,1,9,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,3,"# datasette-cors
[](https://pypi.org/project/datasette-cors/)
[](https://circleci.com/gh/simonw/datasette-cors)
[](https://github.com/simonw/datasette-cors/blob/master/LICENSE)
Datasette plugin for configuring CORS headers, based on https://github.com/simonw/asgi-cors
You can use this plugin to allow JavaScript running on a whitelisted set of domains to make `fetch()` calls to the JSON API provided by your Datasette instance.
## Installation
pip install datasette-cors
## Configuration
You need to add some configuration to your Datasette `metadata.json` file for this plugin to take effect.
To whitelist specific domains, use this:
```json
{
""plugins"": {
""datasette-cors"": {
""hosts"": [""https://www.example.com""]
}
}
}
```
You can also whitelist patterns like this:
```json
{
""plugins"": {
""datasette-cors"": {
""host_wildcards"": [""https://*.example.com""]
}
}
}
```
## Testing it
To test this plugin out, run it locally by saving one of the above examples as `metadata.json` and running this:
$ datasette --memory -m metadata.json
Now visit https://www.example.com/ in your browser, open the browser developer console and paste in the following:
```javascript
fetch(""http://127.0.0.1:8001/:memory:.json?sql=select+sqlite_version%28%29"").then(r => r.json()).then(console.log)
```
If the plugin is running correctly, you will see the JSON response output to the console.
","
You can use this plugin to allow JavaScript running on a whitelisted set of domains to make fetch() calls to the JSON API provided by your Datasette instance.
Installation
pip install datasette-cors
Configuration
You need to add some configuration to your Datasette metadata.json file for this plugin to take effect.
If the plugin is running correctly, you will see the JSON response output to the console.
",,,,,,
197882382,MDEwOlJlcG9zaXRvcnkxOTc4ODIzODI=,healthkit-to-sqlite,dogsheep/healthkit-to-sqlite,0,53015001,https://github.com/dogsheep/healthkit-to-sqlite,Convert an Apple Healthkit export zip to a SQLite database,0,2019-07-20T05:03:12Z,2021-08-20T00:55:34Z,2021-08-20T00:56:17Z,https://datasette.io/tools/healthkit-to-sqlite,29,91,91,Python,1,1,1,1,0,4,0,0,8,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-tool"", ""dogsheep"", ""healthkit"", ""sqlite""]",4,8,91,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,53015001,4,3,"# healthkit-to-sqlite
[](https://pypi.org/project/healthkit-to-sqlite/)
[](https://github.com/dogsheep/healthkit-to-sqlite/releases)
[](https://github.com/dogsheep/healthkit-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/dogsheep/healthkit-to-sqlite/blob/main/LICENSE)
Convert an Apple Healthkit export zip to a SQLite database
## How to install
$ pip install healthkit-to-sqlite
## How to use
First you need to export your Apple HealthKit data.
1. On your iPhone, open the ""Health"" app
2. Click the profile icon in the top right
3. Click ""Export Health Data"" at the bottom of that page
4. Save the resulting file somewhere you can access it, or AirDrop it directly to your laptop.
Now you can convert the resulting `export.zip` file to SQLite like so:
$ healthkit-to-sqlite export.zip healthkit.db
A progress bar will be displayed. You can disable this using `--silent`.
```
Importing from HealthKit [#-------------] 5% 00:01:33
```
You can explore the resulting data using [Datasette](https://datasette.readthedocs.io/) like this:
$ datasette healthkit.db
","
healthkit-to-sqlite
Convert an Apple Healthkit export zip to a SQLite database
How to install
$ pip install healthkit-to-sqlite
How to use
First you need to export your Apple HealthKit data.
On your iPhone, open the ""Health"" app
Click the profile icon in the top right
Click ""Export Health Data"" at the bottom of that page
Save the resulting file somewhere you can access it, or AirDrop it directly to your laptop.
Now you can convert the resulting export.zip file to SQLite like so:
$ healthkit-to-sqlite export.zip healthkit.db
A progress bar will be displayed. You can disable this using --silent.
Importing from HealthKit [#-------------] 5% 00:01:33
You can explore the resulting data using Datasette like this:
$ datasette healthkit.db
",,,,,,
205429375,MDEwOlJlcG9zaXRvcnkyMDU0MjkzNzU=,swarm-to-sqlite,dogsheep/swarm-to-sqlite,0,53015001,https://github.com/dogsheep/swarm-to-sqlite,Create a SQLite database containing your checkin history from Foursquare Swarm,0,2019-08-30T17:37:29Z,2021-02-22T07:58:39Z,2021-01-18T04:36:03Z,,49,37,37,Python,1,1,1,1,0,1,0,0,1,apache-2.0,"[""sqlite"", ""foursquare"", ""swarm"", ""foursquare-api"", ""datasette"", ""dogsheep"", ""datasette-io"", ""datasette-tool""]",1,1,37,main,"{""admin"": false, ""push"": false, ""pull"": false}",,53015001,1,3,"# swarm-to-sqlite
[](https://pypi.org/project/swarm-to-sqlite/)
[](https://github.com/dogsheep/swarm-to-sqlite/releases)
[](https://github.com/dogsheep/swarm-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/dogsheep/swarm-to-sqlite/blob/main/LICENSE)
Create a SQLite database containing your checkin history from Foursquare Swarm.
## How to install
$ pip install swarm-to-sqlite
## Usage
You will need to first obtain a valid OAuth token for your Foursquare account. You can do so using this tool: https://your-foursquare-oauth-token.glitch.me/
Simplest usage is to simply provide the name of the database file you wish to write to. The tool will prompt you to paste in your token, and will then download your checkins and store them in the specified database file.
$ swarm-to-sqlite checkins.db
Please provide your Foursquare OAuth token:
Importing 3699 checkins [#########-----------------------] 27% 00:02:31
You can also pass the token as a command-line option:
$ swarm-to-sqlite checkins.db --token=XXX
Or as an environment variable:
$ export FOURSQUARE_TOKEN=XXX
$ swarm-to-sqlite checkins.db
To retrieve just checkins within the past X hours, days or weeks, use the `--since=` option. For example, to pull only checkins that happened within the last 10 days use:
$ swarm-to-sqlite checkins.db --token=XXX --since=10d
Use `2w` for two weeks, `10h` for ten hours, `3d` for three days.
In addition to saving the checkins to a database, you can also write them to a JSON file using the `--save` option:
$ swarm-to-sqlite checkins.db --save=checkins.json
Having done this, you can re-import checkins directly from that file (rather than making API calls to fetch data from Foursquare) like this:
$ swarm-to-sqlite checkins.db --load=checkins.json
## Using with Datasette
The SQLite database produced by this tool is designed to be browsed using [Datasette](https://datasette.io/).
You can install the [datasette-cluster-map](https://datasette.io/plugins/datasette-cluster-map) plugin to view your checkins on a map.
","
swarm-to-sqlite
Create a SQLite database containing your checkin history from Foursquare Swarm.
Simplest usage is to simply provide the name of the database file you wish to write to. The tool will prompt you to paste in your token, and will then download your checkins and store them in the specified database file.
$ swarm-to-sqlite checkins.db
Please provide your Foursquare OAuth token:
Importing 3699 checkins [#########-----------------------] 27% 00:02:31
You can also pass the token as a command-line option:
To retrieve just checkins within the past X hours, days or weeks, use the --since= option. For example, to pull only checkins that happened within the last 10 days use:
",,,,,,
206156866,MDEwOlJlcG9zaXRvcnkyMDYxNTY4NjY=,twitter-to-sqlite,dogsheep/twitter-to-sqlite,0,53015001,https://github.com/dogsheep/twitter-to-sqlite,Save data from Twitter to a SQLite database,0,2019-09-03T19:30:08Z,2021-12-26T18:08:43Z,2021-12-26T18:08:40Z,,298,269,269,Python,1,1,1,1,0,13,0,0,10,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-tool"", ""dogsheep"", ""sqlite"", ""twitter"", ""twitter-api""]",13,10,269,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,53015001,13,5,"# twitter-to-sqlite
[](https://pypi.org/project/twitter-to-sqlite/)
[](https://github.com/dogsheep/twitter-to-sqlite/releases)
[](https://github.com/dogsheep/twitter-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/dogsheep/twitter-to-sqlite/blob/main/LICENSE)
Save data from Twitter to a SQLite database.
**This tool currently uses Twitter API v1**. You may be unable to use it if you do not have an API key for that version of the API.
- [How to install](#how-to-install)
- [Authentication](#authentication)
- [Retrieving tweets by specific accounts](#retrieving-tweets-by-specific-accounts)
- [Retrieve user profiles in bulk](#retrieve-user-profiles-in-bulk)
- [Retrieve tweets in bulk](#retrieve-tweets-in-bulk)
- [Retrieving Twitter followers](#retrieving-twitter-followers)
- [Retrieving friends](#retrieving-friends)
- [Retrieving favorited tweets](#retrieving-favorited-tweets)
- [Retrieving Twitter lists](#retrieving-twitter-lists)
- [Retrieving Twitter list memberships](#retrieving-twitter-list-memberships)
- [Retrieving just follower and friend IDs](#retrieving-just-follower-and-friend-ids)
- [Retrieving tweets from your home timeline](#retrieving-tweets-from-your-home-timeline)
- [Retrieving your mentions](#retrieving-your-mentions)
- [Providing input from a SQL query with --sql and --attach](#providing-input-from-a-sql-query-with---sql-and---attach)
- [Running searches](#running-searches)
- [Capturing tweets in real-time with track and follow](#capturing-tweets-in-real-time-with-track-and-follow)
* [track](#track)
* [follow](#follow)
- [Importing data from your Twitter archive](#importing-data-from-your-twitter-archive)
- [Design notes](#design-notes)
## How to install
$ pip install twitter-to-sqlite
## Authentication
First, you will need to create a Twitter application at https://developer.twitter.com/en/apps. You may need to apply for a Twitter developer account - if so, you may find this [example of an email application](https://raw.githubusercontent.com/dogsheep/twitter-to-sqlite/main/email.png) useful that has been approved in the past.
Once you have created your application, navigate to the ""Keys and tokens"" page and make note of the following:
* Your API key
* Your API secret key
* Your access token
* Your access token secret
You will need to save all four of these values to a JSON file in order to use this tool.
You can create that JSON file by running the following command and pasting in the values at the prompts:
$ twitter-to-sqlite auth
Create an app here: https://developer.twitter.com/en/apps
Then navigate to 'Keys and tokens' and paste in the following:
API key: xxx
API secret key: xxx
Access token: xxx
Access token secret: xxx
This will create a file called `auth.json` in your current directory containing the required values. To save the file at a different path or filename, use the `--auth=myauth.json` option.
## Retrieving tweets by specific accounts
The `user-timeline` command retrieves all of the tweets posted by the specified user accounts. It defaults to the account belonging to the authenticated user:
$ twitter-to-sqlite user-timeline twitter.db
Importing tweets [#####-------------------------------] 2799/17780 00:01:39
All of these commands assume that there is an `auth.json` file in the current directory. You can provide the path to your `auth.json` file using `-a`:
$ twitter-to-sqlite user-timeline twitter.db -a /path/to/auth.json
To load tweets for other users, pass their screen names as arguments:
$ twitter-to-sqlite user-timeline twitter.db cleopaws nichemuseums
Twitter's API only returns up to around 3,200 tweets for most user accounts, but you may find that it returns all available tweets for your own user account.
You can pass numeric Twitter user IDs instead of screen names using the `--ids` parameter.
You can use `--since` to retrieve every tweet since the last time you imported for that user, or `--since_id=xxx` to retrieve every tweet since a specific tweet ID.
This command also accepts `--sql` and `--attach` options, documented below.
## Retrieve user profiles in bulk
If you have a list of Twitter screen names (or user IDs) you can bulk fetch their fully inflated Twitter profiles using the `users-lookup` command:
$ twitter-to-sqlite users-lookup users.db simonw cleopaws
You can pass user IDs instead using the `--ids` option:
$ twitter-to-sqlite users-lookup users.db 12497 3166449535 --ids
This command also accepts `--sql` and `--attach` options, documented below.
## Retrieve tweets in bulk
If you have a list of tweet IDS you can bulk fetch them using the `statuses-lookup` command:
$ twitter-to-sqlite statuses-lookup tweets.db 1122154819815239680 1122154178493575169
The `--sql` and `--attach` options are supported.
Here's a recipe to retrieve any tweets that existing tweets are in-reply-to which have not yet been stored in your database:
$ twitter-to-sqlite statuses-lookup tweets.db \
--sql='
select in_reply_to_status_id
from tweets
where in_reply_to_status_id is not null' \
--skip-existing
The `--skip-existing` option means that tweets that have already been stored in the database will not be fetched again.
## Retrieving Twitter followers
The `followers` command retrieves details of every follower of the specified accounts. You can use it to retrieve your own followers, or you can pass one or more screen names to pull the followers for other accounts.
The following command pulls your followers and saves them in a SQLite database file called `twitter.db`:
$ twitter-to-sqlite followers twitter.db
This command is **extremely slow**, because Twitter impose a rate limit of no more than one request per minute to this endpoint! If you are running it against an account with thousands of followers you should expect this to take several hours.
To retrieve followers for another account, use:
$ twitter-to-sqlite followers twitter.db cleopaws
This command also accepts the `--ids`, `--sql` and `--attach` options.
See [Analyzing my Twitter followers with Datasette](https://simonwillison.net/2018/Jan/28/analyzing-my-twitter-followers/) for the original inspiration for this command.
## Retrieving friends
The `friends` command works like the `followers` command, but retrieves the specified (or currently authenticated) user's friends - defined as accounts that the user is following.
$ twitter-to-sqlite friends twitter.db
It takes the same options as the `followers` command.
## Retrieving favorited tweets
The `favorites` command retrieves tweets that have been favorited by a specified user. Called without any extra arguments it retrieves tweets favorited by the currently authenticated user:
$ twitter-to-sqlite favorites faves.db
You can also use the `--screen_name` or `--user_id` arguments to retrieve favorite tweets for another user:
$ twitter-to-sqlite favorites faves-obama.db --screen_name=BarackObama
Use the `--stop_after=xxx` argument to retrieve only the most recent number of favorites, e.g. to get the authenticated user's 50 most recent favorites:
$ twitter-to-sqlite favorites faves.db --stop_after=50
## Retrieving Twitter lists
The `lists` command retrieves all of the lists belonging to one or more users.
$ twitter-to-sqlite lists lists.db simonw dogsheep
This command also accepts the `--sql` and `--attach` and `--ids` options.
To additionally fetch the list of members for each list, use `--members`.
## Retrieving Twitter list memberships
The `list-members` command can be used to retrieve details of one or more Twitter lists, including all of their members.
$ twitter-to-sqlite list-members members.db simonw/the-good-place
You can pass multiple `screen_name/list_slug` identifiers.
If you know the numeric IDs of the lists instead, you can use `--ids`:
$ twitter-to-sqlite list-members members.db 927913322841653248 --ids
## Retrieving just follower and friend IDs
It's also possible to retrieve just the numeric Twitter IDs of the accounts that specific users are following (""friends"" in Twitter's API terminology) or followed-by:
$ twitter-to-sqlite followers-ids members.db simonw cleopaws
This will populate the `following` table with `followed_id`/`follower_id` pairs for the two specified accounts, listing every account ID that is following either of those two accounts.
$ twitter-to-sqlite friends-ids members.db simonw cleopaws
This will do the same thing but pull the IDs that those accounts are following.
Both of these commands also support `--sql` and `--attach` as an alternative to passing screen names as direct command-line arguments. You can use `--ids` to process the inputs as user IDs rather than screen names.
The underlying Twitter APIs have a rate limit of 15 requests every 15 minutes - though they do return up to 5,000 IDs in each call. By default both of these subcommands will wait for 61 seconds between API calls in order to stay within the rate limit - you can adjust this behaviour down to just one second delay if you know you will not be making many calls using `--sleep=1`.
## Retrieving tweets from your home timeline
The `home-timeline` command retrieves up to 800 tweets from the home timeline of the authenticated user - generally this means tweets from people you follow.
$ twitter-to-sqlite home-timeline twitter.db
Importing timeline [#################--------] 591/800 00:01:14
The tweets are stored in the `tweets` table, and a record is added to the `timeline_tweets` table noting that this tweet came in due to being spotted in the timeline of your user.
You can use `--since` to retrieve just tweets that have been posted since the last time this command was run, or `--since_id=xxx` to explicitly pass in a tweet ID to use as the last position.
You can then view your timeline in Datasette using the following URL:
`/tweets/tweets?_where=id+in+(select+tweet+from+[timeline_tweets])&_sort_desc=id&_facet=user`
This will filter your tweets table to just tweets that appear in your timeline, ordered by most recent first and use faceting to show you which users are responsible for the most tweets.
## Retrieving your mentions
The `mentions-timeline` command works like `home-timeline` except it retrieves tweets that mention the authenticated user's account. It records the user account that was mentioned in a `mentions_tweets` table.
It supports `--since` and `--since_id` in the same was as `home-timeline` does.
## Providing input from a SQL query with --sql and --attach
This option is available for some subcommands - run `twitter-to-sqlite command-name --help` to check.
You can provide Twitter screen names (or user IDs or tweet IDs) directly as command-line arguments, or you can provide those screen names or IDs by executing a SQL query.
For example: consider a SQLite database with an `attendees` table listing names and Twitter accounts - something like this:
| First | Last | Twitter |
|---------|------------|--------------|
| Simon | Willison | simonw |
| Avril | Lavigne | AvrilLavigne |
You can run the `users-lookup` command to pull the Twitter profile of every user listed in that database by loading the screen names using a `--sql` query:
$ twitter-to-sqlite users-lookup my.db --sql=""select Twitter from attendees""
If your database table contains Twitter IDs, you can select those IDs and pass the `--ids` argument. For example, to fetch the profiles of users who have had their user IDs inserted into the `following` table using the `twitter-to-sqlite friends-ids` command:
$ twitter-to-sqlite users-lookup my.db --sql=""select follower_id from following"" --ids
Or to avoid re-fetching users that have already been fetched:
$ twitter-to-sqlite users-lookup my.db \
--sql=""select followed_id from following where followed_id not in (
select id from users)"" --ids
If your data lives in a separate database file you can attach it using `--attach`. For example, consider the attendees example above but the data lives in an `attendees.db` file, and you want to fetch the user profiles into a `tweets.db` file. You could do that like this:
$ twitter-to-sqlite users-lookup tweets.db \
--attach=attendees.db \
--sql=""select Twitter from attendees.attendees""
The filename (without the extension) will be used as the database alias within SQLite. If you want a different alias for some reason you can specify that with a colon like this:
$ twitter-to-sqlite users-lookup tweets.db \
--attach=foo:attendees.db \
--sql=""select Twitter from foo.attendees""
## Running searches
The `search` command runs a search against the Twitter [standard search API](https://developer.twitter.com/en/docs/tweets/search/api-reference/get-search-tweets).
$ twitter-to-sqlite search tweets.db ""dogsheep""
This will import up to around 320 tweets that match that search term into the `tweets` table. It will also create a record in the `search_runs` table recording that the search took place, and many-to-many records in the `search_runs_tweets` table recording which tweets were seen for that search at that time.
You can use the `--since` parameter to check for previous search runs with the same arguments and only retrieve tweets that were posted since the last retrieved matching tweet.
The following additional options for `search` are supported:
* `--geocode`: `latitude,longitude,radius` where radius is a number followed by mi or km
* `--lang`: ISO 639-1 language code e.g. `en` or `es`
* `--locale`: Locale: only `ja` is currently effective
* `--result_type`: `mixed`, `recent` or `popular`. Defaults to `mixed`
* `--count`: Number of results per page, defaults to the maximum of 100
* `--stop_after`: Stop after this many results
* `--since_id`: Pull tweets since this Tweet ID. You probably want to use `--since` instead of this.
## Capturing tweets in real-time with track and follow
This functionality is **experimental**. Please [file bug reports](https://github.com/dogsheep/twitter-to-sqlite/issues) if you find any!
Twitter provides a real-time API which can be used to subscribe to tweets as they happen. `twitter-to-sqlite` can use this API to continually update a SQLite database with tweets matching certain keywords, or referencing specific users.
### track
To track keywords, use the `track` command:
$ twitter-to-sqlite track tweets.db kakapo
This command will continue to run until you hit Ctrl+C. It will capture any tweets mentioning the keyword [kakapo](https://en.wikipedia.org/wiki/Kakapo) and store them in the `tweets.db` database file.
You can pass multiple keywords as a space separated list. This will capture tweets matching either of those keywords:
$ twitter-to-sqlite track tweets.db kakapo raccoon
You can enclose phrases in quotes to search for tweets matching both of those keywords:
$ twitter-to-sqlite track tweets.db 'trash panda'
See [the Twitter track documentation](https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/basic-stream-parameters#track) for advanced tips on using this command.
Add the `--verbose` option to see matching tweets (in their verbose JSON form) displayed to the terminal as they are captured:
$ twitter-to-sqlite track tweets.db raccoon --verbose
### follow
The `follow` command will capture all tweets that are relevant to one or more specific Twitter users.
$ twitter-to-sqlite follow tweets.db nytimes
This includes tweets by those users, tweets that reply to or quote those users and retweets by that user. See [the Twitter follow documentation](https://developer.twitter.com/en/docs/tweets/filter-realtime/guides/basic-stream-parameters#follow) for full details.
The command accepts one or more screen names.
You can feed it numeric Twitter user IDs instead of screen names by using the `--ids` flag.
The command also supports the `--sql` and `--attach` options, and the `--verbose` option for displaying tweets as they are captured.
Here's how to start following tweets from every user ID currently represented as being followed in the `following` table (populated using the `friends-ids` command):
$ twitter-to-sqlite follow tweets.db \
--sql=""select distinct followed_id from following"" \
--ids
## Importing data from your Twitter archive
You can request an archive of your Twitter data by [following these instructions](https://help.twitter.com/en/managing-your-account/how-to-download-your-twitter-archive).
Twitter will send you a link to download a `.zip` file. You can import the contents of that file into a set of tables in a new database file called `archive.db` (each table beginning with the `archive_` prefix) using the `import` command:
$ twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2.zip
This command does not populate any of the regular tables, since Twitter's export data does not exactly match the schema returned by the Twitter API.
It will delete and recreate the corresponding `archive_*` tables every time you run it. If this is not what you want, run the command against a new SQLite database file name rather than running it against one that already exists.
If you have already decompressed your archive, you can run this against the directory that you decompressed it to:
$ twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2/
You can also run it against one or more specific files within that folder. For example, to import just the follower.js and following.js files:
$ twitter-to-sqlite import archive.db \
~/Downloads/twitter-2019-06-25-b31f2/follower.js \
~/Downloads/twitter-2019-06-25-b31f2/following.js
You may want to use other commands to populate tables based on data from the archive. For example, to retrieve full API versions of each of the tweets you have favourited in your archive, you could run the following:
$ twitter-to-sqlite statuses-lookup archive.db \
--sql='select tweetId from archive_like' \
--skip-existing
If you want these imported tweets to then be reflected in the `favorited_by` table, you can do so by applying the following SQL query:
$ sqlite3 archive.db
SQLite version 3.22.0 2018-01-22 18:45:57
Enter "".help"" for usage hints.
sqlite> INSERT OR IGNORE INTO favorited_by (tweet, user)
...> SELECT tweetId, 'YOUR_TWITTER_ID' FROM archive_like;
Replace YOUR_TWITTER_ID with your numeric Twitter ID. If you don't know that ID you can find it out by running the following:
$ twitter-to-sqlite fetch \
""https://api.twitter.com/1.1/account/verify_credentials.json"" \
| grep '""id""' | head -n 1
## Design notes
* Tweet IDs are stored as integers, to afford sorting by ID in a sensible way
* While we configure foreign key relationships between tables, we do not ask SQLite to enforce them. This is used by the `following` table to allow the `followers-ids` and `friends-ids` commands to populate it with user IDs even if the user accounts themselves are not yet present in the `users` table.
","
twitter-to-sqlite
Save data from Twitter to a SQLite database.
This tool currently uses Twitter API v1. You may be unable to use it if you do not have an API key for that version of the API.
Once you have created your application, navigate to the ""Keys and tokens"" page and make note of the following:
Your API key
Your API secret key
Your access token
Your access token secret
You will need to save all four of these values to a JSON file in order to use this tool.
You can create that JSON file by running the following command and pasting in the values at the prompts:
$ twitter-to-sqlite auth
Create an app here: https://developer.twitter.com/en/apps
Then navigate to 'Keys and tokens' and paste in the following:
API key: xxx
API secret key: xxx
Access token: xxx
Access token secret: xxx
This will create a file called auth.json in your current directory containing the required values. To save the file at a different path or filename, use the --auth=myauth.json option.
Retrieving tweets by specific accounts
The user-timeline command retrieves all of the tweets posted by the specified user accounts. It defaults to the account belonging to the authenticated user:
Twitter's API only returns up to around 3,200 tweets for most user accounts, but you may find that it returns all available tweets for your own user account.
You can pass numeric Twitter user IDs instead of screen names using the --ids parameter.
You can use --since to retrieve every tweet since the last time you imported for that user, or --since_id=xxx to retrieve every tweet since a specific tweet ID.
This command also accepts --sql and --attach options, documented below.
Retrieve user profiles in bulk
If you have a list of Twitter screen names (or user IDs) you can bulk fetch their fully inflated Twitter profiles using the users-lookup command:
Here's a recipe to retrieve any tweets that existing tweets are in-reply-to which have not yet been stored in your database:
$ twitter-to-sqlite statuses-lookup tweets.db \
--sql='
select in_reply_to_status_id
from tweets
where in_reply_to_status_id is not null' \
--skip-existing
The --skip-existing option means that tweets that have already been stored in the database will not be fetched again.
Retrieving Twitter followers
The followers command retrieves details of every follower of the specified accounts. You can use it to retrieve your own followers, or you can pass one or more screen names to pull the followers for other accounts.
The following command pulls your followers and saves them in a SQLite database file called twitter.db:
$ twitter-to-sqlite followers twitter.db
This command is extremely slow, because Twitter impose a rate limit of no more than one request per minute to this endpoint! If you are running it against an account with thousands of followers you should expect this to take several hours.
To retrieve followers for another account, use:
$ twitter-to-sqlite followers twitter.db cleopaws
This command also accepts the --ids, --sql and --attach options.
The friends command works like the followers command, but retrieves the specified (or currently authenticated) user's friends - defined as accounts that the user is following.
$ twitter-to-sqlite friends twitter.db
It takes the same options as the followers command.
Retrieving favorited tweets
The favorites command retrieves tweets that have been favorited by a specified user. Called without any extra arguments it retrieves tweets favorited by the currently authenticated user:
$ twitter-to-sqlite favorites faves.db
You can also use the --screen_name or --user_id arguments to retrieve favorite tweets for another user:
It's also possible to retrieve just the numeric Twitter IDs of the accounts that specific users are following (""friends"" in Twitter's API terminology) or followed-by:
This will populate the following table with followed_id/follower_id pairs for the two specified accounts, listing every account ID that is following either of those two accounts.
This will do the same thing but pull the IDs that those accounts are following.
Both of these commands also support --sql and --attach as an alternative to passing screen names as direct command-line arguments. You can use --ids to process the inputs as user IDs rather than screen names.
The underlying Twitter APIs have a rate limit of 15 requests every 15 minutes - though they do return up to 5,000 IDs in each call. By default both of these subcommands will wait for 61 seconds between API calls in order to stay within the rate limit - you can adjust this behaviour down to just one second delay if you know you will not be making many calls using --sleep=1.
Retrieving tweets from your home timeline
The home-timeline command retrieves up to 800 tweets from the home timeline of the authenticated user - generally this means tweets from people you follow.
The tweets are stored in the tweets table, and a record is added to the timeline_tweets table noting that this tweet came in due to being spotted in the timeline of your user.
You can use --since to retrieve just tweets that have been posted since the last time this command was run, or --since_id=xxx to explicitly pass in a tweet ID to use as the last position.
You can then view your timeline in Datasette using the following URL:
This will filter your tweets table to just tweets that appear in your timeline, ordered by most recent first and use faceting to show you which users are responsible for the most tweets.
Retrieving your mentions
The mentions-timeline command works like home-timeline except it retrieves tweets that mention the authenticated user's account. It records the user account that was mentioned in a mentions_tweets table.
It supports --since and --since_id in the same was as home-timeline does.
Providing input from a SQL query with --sql and --attach
This option is available for some subcommands - run twitter-to-sqlite command-name --help to check.
You can provide Twitter screen names (or user IDs or tweet IDs) directly as command-line arguments, or you can provide those screen names or IDs by executing a SQL query.
For example: consider a SQLite database with an attendees table listing names and Twitter accounts - something like this:
First
Last
Twitter
Simon
Willison
simonw
Avril
Lavigne
AvrilLavigne
You can run the users-lookup command to pull the Twitter profile of every user listed in that database by loading the screen names using a --sql query:
$ twitter-to-sqlite users-lookup my.db --sql=""select Twitter from attendees""
If your database table contains Twitter IDs, you can select those IDs and pass the --ids argument. For example, to fetch the profiles of users who have had their user IDs inserted into the following table using the twitter-to-sqlite friends-ids command:
$ twitter-to-sqlite users-lookup my.db --sql=""select follower_id from following"" --ids
Or to avoid re-fetching users that have already been fetched:
$ twitter-to-sqlite users-lookup my.db \
--sql=""select followed_id from following where followed_id not in (
select id from users)"" --ids
If your data lives in a separate database file you can attach it using --attach. For example, consider the attendees example above but the data lives in an attendees.db file, and you want to fetch the user profiles into a tweets.db file. You could do that like this:
The filename (without the extension) will be used as the database alias within SQLite. If you want a different alias for some reason you can specify that with a colon like this:
This will import up to around 320 tweets that match that search term into the tweets table. It will also create a record in the search_runs table recording that the search took place, and many-to-many records in the search_runs_tweets table recording which tweets were seen for that search at that time.
You can use the --since parameter to check for previous search runs with the same arguments and only retrieve tweets that were posted since the last retrieved matching tweet.
The following additional options for search are supported:
--geocode: latitude,longitude,radius where radius is a number followed by mi or km
--lang: ISO 639-1 language code e.g. en or es
--locale: Locale: only ja is currently effective
--result_type: mixed, recent or popular. Defaults to mixed
--count: Number of results per page, defaults to the maximum of 100
--stop_after: Stop after this many results
--since_id: Pull tweets since this Tweet ID. You probably want to use --since instead of this.
Capturing tweets in real-time with track and follow
This functionality is experimental. Please file bug reports if you find any!
Twitter provides a real-time API which can be used to subscribe to tweets as they happen. twitter-to-sqlite can use this API to continually update a SQLite database with tweets matching certain keywords, or referencing specific users.
track
To track keywords, use the track command:
$ twitter-to-sqlite track tweets.db kakapo
This command will continue to run until you hit Ctrl+C. It will capture any tweets mentioning the keyword kakapo and store them in the tweets.db database file.
You can pass multiple keywords as a space separated list. This will capture tweets matching either of those keywords:
The follow command will capture all tweets that are relevant to one or more specific Twitter users.
$ twitter-to-sqlite follow tweets.db nytimes
This includes tweets by those users, tweets that reply to or quote those users and retweets by that user. See the Twitter follow documentation for full details.
The command accepts one or more screen names.
You can feed it numeric Twitter user IDs instead of screen names by using the --ids flag.
The command also supports the --sql and --attach options, and the --verbose option for displaying tweets as they are captured.
Here's how to start following tweets from every user ID currently represented as being followed in the following table (populated using the friends-ids command):
Twitter will send you a link to download a .zip file. You can import the contents of that file into a set of tables in a new database file called archive.db (each table beginning with the archive_ prefix) using the import command:
This command does not populate any of the regular tables, since Twitter's export data does not exactly match the schema returned by the Twitter API.
It will delete and recreate the corresponding archive_* tables every time you run it. If this is not what you want, run the command against a new SQLite database file name rather than running it against one that already exists.
If you have already decompressed your archive, you can run this against the directory that you decompressed it to:
You may want to use other commands to populate tables based on data from the archive. For example, to retrieve full API versions of each of the tweets you have favourited in your archive, you could run the following:
If you want these imported tweets to then be reflected in the favorited_by table, you can do so by applying the following SQL query:
$ sqlite3 archive.db
SQLite version 3.22.0 2018-01-22 18:45:57
Enter "".help"" for usage hints.
sqlite> INSERT OR IGNORE INTO favorited_by (tweet, user)
...> SELECT tweetId, 'YOUR_TWITTER_ID' FROM archive_like;
<Ctrl+D>
Replace YOUR_TWITTER_ID with your numeric Twitter ID. If you don't know that ID you can find it out by running the following:
Tweet IDs are stored as integers, to afford sorting by ID in a sensible way
While we configure foreign key relationships between tables, we do not ask SQLite to enforce them. This is used by the following table to allow the followers-ids and friends-ids commands to populate it with user IDs even if the user accounts themselves are not yet present in the users table.
",1,public,0,,,
206202864,MDEwOlJlcG9zaXRvcnkyMDYyMDI4NjQ=,inaturalist-to-sqlite,dogsheep/inaturalist-to-sqlite,0,53015001,https://github.com/dogsheep/inaturalist-to-sqlite,Create a SQLite database containing your observation history from iNaturalist,0,2019-09-04T01:21:21Z,2020-12-19T05:18:38Z,2020-10-22T00:08:58Z,,17,2,2,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""sqlite"", ""inaturalist"", ""datasette"", ""dogsheep"", ""datasette-io"", ""datasette-tool""]",0,0,2,master,"{""admin"": false, ""push"": false, ""pull"": false}",,53015001,0,1,"# inaturalist-to-sqlite
[](https://pypi.org/project/inaturalist-to-sqlite/)
[](https://circleci.com/gh/dogsheep/inaturalist-to-sqlite)
[](https://github.com/dogsheep/inaturalist-to-sqlite/blob/master/LICENSE)
Create a SQLite database containing your observation history from [iNaturalist](https://www.inaturalist.org/).
## How to install
$ pip install inaturalist-to-sqlite
## Usage
$ inaturalist-to-sqlite inaturalist.db yourusername
(Or try `simonw` if you don't yet have an iNaturalist account)
This will import all of your iNaturalist observations into a SQLite database called `inaturalist.db`.","
inaturalist-to-sqlite
Create a SQLite database containing your observation history from iNaturalist.
(Or try simonw if you don't yet have an iNaturalist account)
This will import all of your iNaturalist observations into a SQLite database called inaturalist.db.
",,,,,,
206649770,MDEwOlJlcG9zaXRvcnkyMDY2NDk3NzA=,google-takeout-to-sqlite,dogsheep/google-takeout-to-sqlite,0,53015001,https://github.com/dogsheep/google-takeout-to-sqlite,Save data from Google Takeout to a SQLite database,0,2019-09-05T20:15:15Z,2021-06-08T15:31:47Z,2021-02-24T00:34:55Z,,14,51,51,Python,1,1,1,1,0,4,0,0,6,apache-2.0,"[""google"", ""sqlite"", ""datasette"", ""dogsheep"", ""datasette-io"", ""datasette-tool""]",4,6,51,master,"{""admin"": false, ""push"": false, ""pull"": false}",,53015001,4,3,"# google-takeout-to-sqlite
[](https://pypi.org/project/google-takeout-to-sqlite/)
[](https://circleci.com/gh/dogsheep/google-takeout-to-sqlite)
[](https://github.com/dogsheep/google-takeout-to-sqlite/blob/master/LICENSE)
Save data from google-takeout to a SQLite database.
## How to install
$ pip install google-takeout-to-sqlite
Request your Google data from https://takeout.google.com/ - wait for the email and download the zip file.
This tool only supports a subset of the available options. More will be added over time.
## My Activity
You can request the ""My Activity"" export and then import it with the following command:
$ google-takeout-to-sqlite my-activity takeout.db ~/Downloads/takeout-20190530.zip
This will create a database file called `takeout.db` if one does not already exist.
## Location History
Your location history records latitude, longitude and timestame for where Google has tracked your location. You can import it using this command:
$ google-takeout-to-sqlite location-history takeout.db ~/Downloads/takeout-20190530.zip
## Browsing your data with Datasette
Once you have imported Google data into a SQLite database file you can browse your data using [Datasette](https://github.com/simonw/datasette). Install Datasette like so:
$ pip install datasette
Now browse your data by running this and then visiting `http://localhost:8001/`
$ datasette takeout.db
Install the [datasette-cluster-map](https://github.com/simonw/datasette-cluster-map) plugin to see your location history on a map:
$ pip install datasette-cluster-map
","
google-takeout-to-sqlite
Save data from google-takeout to a SQLite database.
",,,,,,
207052882,MDEwOlJlcG9zaXRvcnkyMDcwNTI4ODI=,github-to-sqlite,dogsheep/github-to-sqlite,0,53015001,https://github.com/dogsheep/github-to-sqlite,Save data from GitHub to a SQLite database,0,2019-09-08T02:50:28Z,2022-09-20T04:36:37Z,2022-09-28T21:07:54Z,https://github-to-sqlite.dogsheep.net/,143,235,235,Python,1,1,1,1,0,32,0,0,20,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-tool"", ""dogsheep"", ""github-api"", ""sqlite""]",32,20,235,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,53015001,32,6,"# github-to-sqlite
[](https://pypi.org/project/github-to-sqlite/)
[](https://github.com/dogsheep/github-to-sqlite/releases)
[](https://github.com/dogsheep/github-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/dogsheep/github-to-sqlite/blob/main/LICENSE)
Save data from GitHub to a SQLite database.
- [Demo](#demo)
- [How to install](#how-to-install)
- [Authentication](#authentication)
- [Fetching issues for a repository](#fetching-issues-for-a-repository)
- [Fetching pull requests for a repository](#fetching-pull-requests-for-a-repository)
- [Fetching issue comments for a repository](#fetching-issue-comments-for-a-repository)
- [Fetching commits for a repository](#fetching-commits-for-a-repository)
- [Fetching releases for a repository](#fetching-releases-for-a-repository)
- [Fetching tags for a repository](#fetching-tags-for-a-repository)
- [Fetching contributors to a repository](#fetching-contributors-to-a-repository)
- [Fetching repos belonging to a user or organization](#fetching-repos-belonging-to-a-user-or-organization)
- [Fetching specific repositories](#fetching-specific-repositories)
- [Fetching repos that have been starred by a user](#fetching-repos-that-have-been-starred-by-a-user)
- [Fetching users that have starred specific repos](#fetching-users-that-have-starred-specific-repos)
- [Fetching GitHub Actions workflows](#fetching-github-actions-workflows)
- [Scraping dependents for a repository](#scraping-dependents-for-a-repository)
- [Fetching emojis](#fetching-emojis)
- [Making authenticated API calls](#making-authenticated-api-calls)
## Demo
https://github-to-sqlite.dogsheep.net/ hosts a [Datasette](https://datasette.io/) demo of a database created by [running this tool](https://github.com/dogsheep/github-to-sqlite/blob/main/.github/workflows/deploy-demo.yml#L40-L60) against all of the repositories in the [Dogsheep GitHub organization](https://github.com/dogsheep), plus the [datasette](https://github.com/simonw/datasette) and [sqlite-utils](https://github.com/simonw/sqlite-utils) repositories.
## How to install
$ pip install github-to-sqlite
## Authentication
Create a GitHub personal access token: https://github.com/settings/tokens
Run this command and paste in your new token:
$ github-to-sqlite auth
This will create a file called `auth.json` in your current directory containing the required value. To save the file at a different path or filename, use the `--auth=myauth.json` option.
As an alternative to using an `auth.json` file you can add your access token to an environment variable called `GITHUB_TOKEN`.
## Fetching issues for a repository
The `issues` command retrieves all of the issues belonging to a specified repository.
$ github-to-sqlite issues github.db simonw/datasette
If an `auth.json` file is present it will use the token from that file. It works without authentication for public repositories but you should be aware that GitHub have strict IP-based rate limits for unauthenticated requests.
You can point to a different location of `auth.json` using `-a`:
$ github-to-sqlite issues github.db simonw/datasette -a /path/to/auth.json
You can use the `--issue` option one or more times to load specific issues:
$ github-to-sqlite issues github.db simonw/datasette --issue=1
Example: [issues table](https://github-to-sqlite.dogsheep.net/github/issues)
## Fetching pull requests for a repository
While pull requests are a type of issue, you will get more information on pull requests by pulling them separately. For example, whether a pull request has been merged and when.
Following the API of issues, the `pull-requests` command retrieves all of the pull requests belonging to a specified repository.
$ github-to-sqlite pull-requests github.db simonw/datasette
You can use the `--pull-request` option one or more times to load specific pull request:
$ github-to-sqlite pull-requests github.db simonw/datasette --pull-request=81
Note that the `merged_by` column on the `pull_requests` table will only be populated for pull requests that are loaded using the `--pull-request` option - the GitHub API does not return this field for pull requests that are loaded in bulk.
Example: [pull_requests table](https://github-to-sqlite.dogsheep.net/github/pull_requests)
## Fetching issue comments for a repository
The `issue-comments` command retrieves all of the comments on all of the issues in a repository.
It is recommended you run `issues` first, so that each imported comment can have a foreign key poining to its issue.
$ github-to-sqlite issues github.db simonw/datasette
$ github-to-sqlite issue-comments github.db simonw/datasette
You can use the `--issue` option to only load comments for a specific issue within that repository, for example:
$ github-to-sqlite issue-comments github.db simonw/datasette --issue=1
Example: [issue_comments table](https://github-to-sqlite.dogsheep.net/github/issue_comments)
## Fetching commits for a repository
The `commits` command retrieves details of all of the commits for one or more repositories. It currently fetches the sha, commit message and author and committer details - it does no retrieve the full commit body.
$ github-to-sqlite commits github.db simonw/datasette simonw/sqlite-utils
The command accepts one or more repositories.
By default it will stop as soon as it sees a commit that has previously been retrieved. You can force it to retrieve all commits (including those that have been previously inserted) using `--all`.
Example: [commits table](https://github-to-sqlite.dogsheep.net/github/commits)
## Fetching releases for a repository
The `releases` command retrieves the releases for one or more repositories.
$ github-to-sqlite releases github.db simonw/datasette simonw/sqlite-utils
The command accepts one or more repositories.
Example: [releases table](https://github-to-sqlite.dogsheep.net/github/releases)
## Fetching tags for a repository
The `tags` command retrieves all of the tags for one or more repositories.
$ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utils
Example: [tags table](https://github-to-sqlite.dogsheep.net/github/tags)
## Fetching contributors to a repository
The `contributors` command retrieves details of all of the contributors for one or more repositories.
$ github-to-sqlite contributors github.db simonw/datasette simonw/sqlite-utils
The command accepts one or more repositories. It populates a `contributors` table, with foreign keys to `repos` and `users` and a `contributions` table listing the number of commits to that repository for each contributor.
Example: [contributors table](https://github-to-sqlite.dogsheep.net/github/contributors)
## Fetching repos belonging to a user or organization
The `repos` command fetches repos belonging to a user or organization.
Without any other arguments, this command will fetch all repos that the currently authenticated user owns, collaborates on or can access via one of their organizations:
$ github-to-sqlite repos github.db
To fetch repos belonging to a specific user or organization, provide their username as an argument:
$ github-to-sqlite repos github.db dogsheep # organization
$ github-to-sqlite repos github.db simonw # user
You can pass more than one username to fetch for multiple users or organizations at once:
$ github-to-sqlite repos github.db simonw dogsheep
Add the `--readme` option to save the README for the repo in a column called `readme`. Add `--readme-html` to save the HTML rendered version of the README into a collumn called `readme_html`.
Example: [repos table](https://github-to-sqlite.dogsheep.net/github/repos)
## Fetching specific repositories
You can use `-r` with the `repos` command one or more times to fetch just specific repositories.
$ github-to-sqlite repos github.db -r simonw/datasette -r dogsheep/github-to-sqlite
## Fetching repos that have been starred by a user
The `starred` command fetches the repos that have been starred by a user.
$ github-to-sqlite starred github.db simonw
If you are using an `auth.json` file you can omit the username to retrieve the starred repos for the authenticated user.
Example: [stars table](https://github-to-sqlite.dogsheep.net/github/stars)
## Fetching users that have starred specific repos
The `stargazers` command fetches the users that have starred the specified repos.
$ github-to-sqlite stargazers github.db simonw/datasette dogsheep/github-to-sqlite
You can specify one or more repository using `owner/repo` syntax.
Users fetched using this command will be inserted into the `users` table. Many-to-many records showing which repository they starred will be added to the `stars` table.
## Fetching GitHub Actions workflows
The `workflows` command fetches the YAML workflow configurations from each repository's `.github/workflows` directory and parses them to populate `workflows`, `jobs` and `steps` tables.
$ github-to-sqlite workflows github.db simonw/datasette dogsheep/github-to-sqlite
You can specify one or more repository using `owner/repo` syntax.
Example: [workflows table](https://github-to-sqlite.dogsheep.net/github/workflows), [jobs table](https://github-to-sqlite.dogsheep.net/github/jobs), [steps table](https://github-to-sqlite.dogsheep.net/github/steps)
## Scraping dependents for a repository
The GitHub dependency graph can show other GitHub projects that depend on a specific repo, for example [simonw/datasette/network/dependents](https://github.com/simonw/datasette/network/dependents).
This data is not yet available through the GitHub API. The `scrape-dependents` command scrapes those pages and uses the GitHub API to load full versions of the dependent repositories.
$ github-to-sqlite scrape-dependents github.db simonw/datasette
The command accepts one or more repositories.
Add `-v` for verbose output.
Example: [dependents table](https://github-to-sqlite.dogsheep.net/github/dependents?_sort_desc=first_seen_utc)
## Fetching emojis
You can fetch a list of every emoji supported by GitHub using the `emojis` command:
$ github-to-sqlite emojis github.db
This will create a table callad `emojis` with a primary key `name` and a `url` column.
If you add the `--fetch` option the command will also fetch the binary content of the images and place them in an `image` column:
$ github-to-sqlite emojis emojis.db -f
[########----------------------------] 397/1799 22% 00:03:43
You can then use the [datasette-render-images](https://github.com/simonw/datasette-render-images) plugin to browse them visually.
Example: [emojis table](https://github-to-sqlite.dogsheep.net/github/emojis)
## Making authenticated API calls
The `github-to-sqlite get` command provides a convenient shortcut for making authenticated calls to the API. Once you have created your `auth.json` file (or set a `GITHUB_TOKEN` environment variable) you can use it like this:
$ github-to-sqlite get https://api.github.com/gists
This will make an authenticated call to the URL you provide and pretty-print the resulting JSON to the console.
You can ommit the `https://api.github.com/` prefix, for example:
$ github-to-sqlite get /gists
Many GitHub APIs are [paginated using the HTTP Link header](https://docs.github.com/en/rest/guides/traversing-with-pagination). You can follow this pagination and output a list of all of the resulting items using `--paginate`:
$ github-to-sqlite get /users/simonw/repos --paginate
You can outline newline-delimited JSON for each item using `--nl`. This can be useful for streaming items into another tool.
$ github-to-sqlite get /users/simonw/repos --nl
","
This will create a file called auth.json in your current directory containing the required value. To save the file at a different path or filename, use the --auth=myauth.json option.
As an alternative to using an auth.json file you can add your access token to an environment variable called GITHUB_TOKEN.
Fetching issues for a repository
The issues command retrieves all of the issues belonging to a specified repository.
If an auth.json file is present it will use the token from that file. It works without authentication for public repositories but you should be aware that GitHub have strict IP-based rate limits for unauthenticated requests.
You can point to a different location of auth.json using -a:
$ github-to-sqlite issues github.db simonw/datasette -a /path/to/auth.json
You can use the --issue option one or more times to load specific issues:
While pull requests are a type of issue, you will get more information on pull requests by pulling them separately. For example, whether a pull request has been merged and when.
Following the API of issues, the pull-requests command retrieves all of the pull requests belonging to a specified repository.
Note that the merged_by column on the pull_requests table will only be populated for pull requests that are loaded using the --pull-request option - the GitHub API does not return this field for pull requests that are loaded in bulk.
The commits command retrieves details of all of the commits for one or more repositories. It currently fetches the sha, commit message and author and committer details - it does no retrieve the full commit body.
By default it will stop as soon as it sees a commit that has previously been retrieved. You can force it to retrieve all commits (including those that have been previously inserted) using --all.
The command accepts one or more repositories. It populates a contributors table, with foreign keys to repos and users and a contributions table listing the number of commits to that repository for each contributor.
Fetching repos belonging to a user or organization
The repos command fetches repos belonging to a user or organization.
Without any other arguments, this command will fetch all repos that the currently authenticated user owns, collaborates on or can access via one of their organizations:
$ github-to-sqlite repos github.db
To fetch repos belonging to a specific user or organization, provide their username as an argument:
Add the --readme option to save the README for the repo in a column called readme. Add --readme-html to save the HTML rendered version of the README into a collumn called readme_html.
You can specify one or more repository using owner/repo syntax.
Users fetched using this command will be inserted into the users table. Many-to-many records showing which repository they starred will be added to the stars table.
Fetching GitHub Actions workflows
The workflows command fetches the YAML workflow configurations from each repository's .github/workflows directory and parses them to populate workflows, jobs and steps tables.
This data is not yet available through the GitHub API. The scrape-dependents command scrapes those pages and uses the GitHub API to load full versions of the dependent repositories.
The github-to-sqlite get command provides a convenient shortcut for making authenticated calls to the API. Once you have created your auth.json file (or set a GITHUB_TOKEN environment variable) you can use it like this:
$ github-to-sqlite get https://api.github.com/gists
This will make an authenticated call to the URL you provide and pretty-print the resulting JSON to the console.
You can ommit the https://api.github.com/ prefix, for example:
$ github-to-sqlite get /gists
Many GitHub APIs are paginated using the HTTP Link header. You can follow this pagination and output a list of all of the resulting items using --paginate:
$ github-to-sqlite get /users/simonw/repos --paginate
You can outline newline-delimited JSON for each item using --nl. This can be useful for streaming items into another tool.
$ github-to-sqlite get /users/simonw/repos --nl
",1,public,0,,0,
207630174,MDEwOlJlcG9zaXRvcnkyMDc2MzAxNzQ=,datasette-rure,simonw/datasette-rure,0,9599,https://github.com/simonw/datasette-rure,Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine,0,2019-09-10T18:09:33Z,2020-12-04T04:26:53Z,2019-09-11T22:59:38Z,,19,4,4,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""sqlite"", ""regular-expressions"", ""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,4,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-rure
[](https://pypi.org/project/datasette-rure/)
[](https://circleci.com/gh/simonw/datasette-rure)
[](https://github.com/simonw/datasette-rure/blob/master/LICENSE)
Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine
Install this plugin in the same environment as Datasette to enable the `regexp()` SQL function.
$ pip install datasette-rure
The plugin is built on top of the [rure-python](https://github.com/davidblewett/rure-python) library by David Blewett.
## regexp() to test regular expressions
You can test if a value matches a regular expression like this:
select regexp('hi.*there', 'hi there')
-- returns 1
select regexp('not.*there', 'hi there')
-- returns 0
You can also use SQLite's custom syntax to run matches:
select 'hi there' REGEXP 'hi.*there'
-- returns 1
This means you can select rows based on regular expression matches - for example, to select every article where the title begins with an E or an F:
select * from articles where title REGEXP '^[EF]'
Try this out: [REGEXP interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+*+from+articles+where+title+REGEXP+%27%5E%5BEF%5D%27)
## regexp_match() to extract groups
You can extract captured subsets of a pattern using `regexp_match()`.
select regexp_match('.*( and .*)', title) as n from articles where n is not null
-- Returns the ' and X' component of any matching titles, e.g.
-- and Recognition
-- and Transitions Their Place
-- etc
This will return the first parenthesis match when called with two arguments. You can call it with three arguments to indicate which match you would like to extract:
select regexp_match('.*(and)(.*)', title, 2) as n from articles where n is not null
The function will return `null` for invalid inputs e.g. a pattern without capture groups.
Try this out: [regexp_match() interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+%27WHY+%27+%7C%7C+regexp_match%28%27Why+%28.*%29%27%2C+title%29+as+t+from+articles+where+t+is+not+null)
## regexp_matches() to extract multiple matches at once
The `regexp_matches()` function can be used to extract multiple patterns from a single string. The result is returned as a JSON array, which can then be further processed using SQLite's [JSON functions](https://www.sqlite.org/json1.html).
The first argument is a regular expression with named capture groups. The second argument is the string to be matched.
select regexp_matches(
'hello (?P\w+) the (?P\w+)',
'hello bob the dog, hello maggie the cat, hello tarquin the otter'
)
This will return a list of JSON objects, each one representing the named captures from the original regular expression:
[
{""name"": ""bob"", ""species"": ""dog""},
{""name"": ""maggie"", ""species"": ""cat""},
{""name"": ""tarquin"", ""species"": ""otter""}
]
Try this out: [regexp_matches() interactive demo](https://datasette-rure-demo.datasette.io/24ways?sql=select+regexp_matches%28%0D%0A++++%27hello+%28%3FP%3Cname%3E%5Cw%2B%29+the+%28%3FP%3Cspecies%3E%5Cw%2B%29%27%2C%0D%0A++++%27hello+bob+the+dog%2C+hello+maggie+the+cat%2C+hello+tarquin+the+otter%27%0D%0A%29)
","
datasette-rure
Datasette plugin that adds a custom SQL function for executing matches using the Rust regular expression engine
Install this plugin in the same environment as Datasette to enable the regexp() SQL function.
$ pip install datasette-rure
The plugin is built on top of the rure-python library by David Blewett.
regexp() to test regular expressions
You can test if a value matches a regular expression like this:
You can extract captured subsets of a pattern using regexp_match().
select regexp_match('.*( and .*)', title) as n from articles where n is not null
-- Returns the ' and X' component of any matching titles, e.g.
-- and Recognition
-- and Transitions Their Place
-- etc
This will return the first parenthesis match when called with two arguments. You can call it with three arguments to indicate which match you would like to extract:
select regexp_match('.*(and)(.*)', title, 2) as n from articles where n is not null
The function will return null for invalid inputs e.g. a pattern without capture groups.
regexp_matches() to extract multiple matches at once
The regexp_matches() function can be used to extract multiple patterns from a single string. The result is returned as a JSON array, which can then be further processed using SQLite's JSON functions.
The first argument is a regular expression with named capture groups. The second argument is the string to be matched.
select regexp_matches(
'hello (?P<name>\w+) the (?P<species>\w+)',
'hello bob the dog, hello maggie the cat, hello tarquin the otter'
)
This will return a list of JSON objects, each one representing the named captures from the original regular expression:
",,,,,,
209091256,MDEwOlJlcG9zaXRvcnkyMDkwOTEyNTY=,datasette-atom,simonw/datasette-atom,0,9599,https://github.com/simonw/datasette-atom,Datasette plugin that adds a .atom output format,0,2019-09-17T15:31:01Z,2021-03-26T02:06:51Z,2021-01-24T23:59:36Z,,47,10,10,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,10,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,2,"# datasette-atom
[](https://pypi.org/project/datasette-atom/)
[](https://github.com/simonw/datasette-atom/releases)
[](https://github.com/simonw/datasette-atom/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-atom/blob/main/LICENSE)
Datasette plugin that adds support for generating [Atom feeds](https://validator.w3.org/feed/docs/atom.html) with the results of a SQL query.
## Installation
Install this plugin in the same environment as Datasette to enable the `.atom` output extension.
$ pip install datasette-atom
## Usage
To create an Atom feed you need to define a custom SQL query that returns a required set of columns:
* `atom_id` - a unique ID for each row. [This article](https://web.archive.org/web/20080211143232/http://diveintomark.org/archives/2004/05/28/howto-atom-id) has suggestions about ways to create these IDs.
* `atom_title` - a title for that row.
* `atom_updated` - an [RFC 3339](http://www.faqs.org/rfcs/rfc3339.html) timestamp representing the last time the entry was modified in a significant way. This can usually be the time that the row was created.
The following columns are optional:
* `atom_content` - content that should be shown in the feed. This will be treated as a regular string, so any embedded HTML tags will be escaped when they are displayed.
* `atom_content_html` - content that should be shown in the feed. This will be treated as an HTML string, and will be sanitized using [Bleach](https://github.com/mozilla/bleach) to ensure it does not have any malicious code in it before being returned as part of a `` Atom element. If both are provided, this will be used in place of `atom_content`.
* `atom_link` - a URL that should be used as the link that the feed entry points to.
* `atom_author_name` - the name of the author of the entry. If you provide this you can also provide `atom_author_uri` and `atom_author_email` with a URL and e-mail address for that author.
A query that returns these columns can then be returned as an Atom feed by adding the `.atom` extension.
## Example
Here is an example SQL query which generates an Atom feed for new entries on [www.niche-museums.com](https://www.niche-museums.com/):
```sql
select
'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id,
name as atom_title,
created as atom_updated,
'https://www.niche-museums.com/browse/museums/' || id as atom_link,
coalesce(
'',
''
) || '
' || description || '
' as atom_content_html
from
museums
order by
created desc
limit
15
```
You can try this query by [pasting it in here](https://www.niche-museums.com/browse) - then click the `.atom` link to see it as an Atom feed.
## Using a canned query
Datasette's [canned query mechanism](https://docs.datasette.io/en/stable/sql_queries.html#canned-queries) is a useful way to configure feeds. If a canned query definition has a `title` that will be used as the title of the Atom feed.
Here's an example, defined using a `metadata.yaml` file:
```yaml
databases:
browse:
queries:
feed:
title: Niche Museums
sql: |-
select
'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id,
name as atom_title,
created as atom_updated,
'https://www.niche-museums.com/browse/museums/' || id as atom_link,
coalesce(
'',
''
) || '
' || description || '
' as atom_content_html
from
museums
order by
created desc
limit
15
```
## Disabling HTML filtering
The HTML allow-list used by Bleach for the `atom_content_html` column can be found in the `clean(html)` function at the bottom of [datasette_atom/__init__.py](https://github.com/simonw/datasette-atom/blob/main/datasette_atom/__init__.py).
You can disable Bleach entirely for Atom feeds generated using a canned query. You should only do this if you are certain that no user-provided HTML could be included in that value.
Here's how to do that in `metadata.json`:
```json
{
""plugins"": {
""datasette-atom"": {
""allow_unsafe_html_in_canned_queries"": true
}
}
}
```
Setting this to `true` will disable Bleach filtering for all canned queries across all databases.
You can disable Bleach filtering just for a specific list of canned queries like so:
```json
{
""plugins"": {
""datasette-atom"": {
""allow_unsafe_html_in_canned_queries"": {
""museums"": [""latest"", ""moderation""]
}
}
}
}
```
This will disable Bleach just for the canned queries called `latest` and `moderation` in the `museums.db` database.
","
datasette-atom
Datasette plugin that adds support for generating Atom feeds with the results of a SQL query.
Installation
Install this plugin in the same environment as Datasette to enable the .atom output extension.
$ pip install datasette-atom
Usage
To create an Atom feed you need to define a custom SQL query that returns a required set of columns:
atom_id - a unique ID for each row. This article has suggestions about ways to create these IDs.
atom_title - a title for that row.
atom_updated - an RFC 3339 timestamp representing the last time the entry was modified in a significant way. This can usually be the time that the row was created.
The following columns are optional:
atom_content - content that should be shown in the feed. This will be treated as a regular string, so any embedded HTML tags will be escaped when they are displayed.
atom_content_html - content that should be shown in the feed. This will be treated as an HTML string, and will be sanitized using Bleach to ensure it does not have any malicious code in it before being returned as part of a <content type=""html""> Atom element. If both are provided, this will be used in place of atom_content.
atom_link - a URL that should be used as the link that the feed entry points to.
atom_author_name - the name of the author of the entry. If you provide this you can also provide atom_author_uri and atom_author_email with a URL and e-mail address for that author.
A query that returns these columns can then be returned as an Atom feed by adding the .atom extension.
Example
Here is an example SQL query which generates an Atom feed for new entries on www.niche-museums.com:
select'tag:niche-museums.com,'|| substr(created, 0, 11) ||':'|| id as atom_id,
name as atom_title,
created as atom_updated,
'https://www.niche-museums.com/browse/museums/'|| id as atom_link,
coalesce(
'<img src=""'|| photo_url ||'?w=800&h=400&fit=crop&auto=compress"">',
''
) ||'<p>'|| description ||'</p>'as atom_content_html
from
museums
order by
created desclimit15
You can try this query by pasting it in here - then click the .atom link to see it as an Atom feed.
Using a canned query
Datasette's canned query mechanism is a useful way to configure feeds. If a canned query definition has a title that will be used as the title of the Atom feed.
Here's an example, defined using a metadata.yaml file:
databases:
browse:
queries:
feed:
title: Niche Museumssql: |- select 'tag:niche-museums.com,' || substr(created, 0, 11) || ':' || id as atom_id, name as atom_title, created as atom_updated, 'https://www.niche-museums.com/browse/museums/' || id as atom_link, coalesce( '<img src=""' || photo_url || '?w=800&h=400&fit=crop&auto=compress"">', '' ) || '<p>' || description || '</p>' as atom_content_html from museums order by created desc limit 15
Disabling HTML filtering
The HTML allow-list used by Bleach for the atom_content_html column can be found in the clean(html) function at the bottom of datasette_atom/init.py.
You can disable Bleach entirely for Atom feeds generated using a canned query. You should only do this if you are certain that no user-provided HTML could be included in that value.
This will disable Bleach just for the canned queries called latest and moderation in the museums.db database.
",,,,,,
209590345,MDEwOlJlcG9zaXRvcnkyMDk1OTAzNDU=,genome-to-sqlite,dogsheep/genome-to-sqlite,0,53015001,https://github.com/dogsheep/genome-to-sqlite,Import your genome into a SQLite database,0,2019-09-19T15:38:39Z,2021-01-18T19:39:48Z,2019-09-19T15:41:17Z,,9,13,13,Python,1,1,1,1,0,0,0,0,2,apache-2.0,"[""genetics"", ""sqlite"", ""23andme"", ""personal-analytics"", ""datasette"", ""dogsheep"", ""datasette-io"", ""datasette-tool""]",0,2,13,master,"{""admin"": false, ""push"": false, ""pull"": false}",,53015001,0,2,"# genome-to-sqlite
[](https://pypi.org/project/genome-to-sqlite/)
[](https://circleci.com/gh/dogsheep/genome-to-sqlite)
[](https://github.com/dogsheep/genome-to-sqlite/blob/master/LICENSE)
Import your genome into a SQLite database.
## How to install
$ pip install genome-to-sqlite
## How to use
First, export your genome. This tool has only been tested against 23andMe so far. You can request an export of your genome from https://you.23andme.com/tools/data/download/
Now you can convert the resulting `export.zip` file to SQLite like so:
$ genome-to-sqlite export.zip genome.db
A progress bar will be displayed. You can disable this using `--silent`.
```
Importing genome [#----------------] 5% 00:01:33
```
You can explore the resulting data using [Datasette](https://datasette.readthedocs.io/) like this:
$ datasette genome.db --config facet_time_limit_ms:1000
Bumping up the facet time limit is useful in order to enable faceting by chromosome:
http://127.0.0.1:8001/genome/genome?_facet=chromosome&_sort=position
","
",,,,,,
213286752,MDEwOlJlcG9zaXRvcnkyMTMyODY3NTI=,pocket-to-sqlite,dogsheep/pocket-to-sqlite,0,53015001,https://github.com/dogsheep/pocket-to-sqlite,Create a SQLite database containing data from your Pocket account,0,2019-10-07T03:24:14Z,2022-08-21T21:11:59Z,2022-08-22T16:21:34Z,,20,63,63,Python,1,1,1,1,0,3,0,0,5,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-tool"", ""dogsheep"", ""pocket"", ""pocket-api"", ""sqlite""]",3,5,63,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,53015001,3,4,"# pocket-to-sqlite
[](https://pypi.org/project/pocket-to-sqlite/)
[](https://github.com/dogsheep/pocket-to-sqlite/releases)
[](https://github.com/dogsheep/pocket-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/dogsheep/pocket-to-sqlite/blob/main/LICENSE)
Create a SQLite database containing data from your [Pocket](https://getpocket.com/) account.
## How to install
$ pip install pocket-to-sqlite
## Usage
You will need to first obtain a valid OAuth token for your Pocket account. You can do this by running the `auth` command and following the prompts:
$ pocket-to-sqlite auth
Visit this page and sign in with your Pocket account:
https://getpocket.com/auth/author...
Once you have signed in there, hit to continue
Authentication tokens written to auth.json
Now you can fetch all of your items from Pocket like this:
$ pocket-to-sqlite fetch pocket.db
The first time you run this command it will fetch all of your items, and display a progress bar while it does it.
On subsequent runs it will only fetch new items.
You can force it to fetch everything from the beginning again using `--all`. Use `--silent` to disable the progress bar.
## Using with Datasette
The SQLite database produced by this tool is designed to be browsed using [Datasette](https://datasette.readthedocs.io/). Use the [datasette-render-timestamps](https://github.com/simonw/datasette-render-timestamps) plugin to improve the display of the timestamp values.
","
pocket-to-sqlite
Create a SQLite database containing data from your Pocket account.
How to install
$ pip install pocket-to-sqlite
Usage
You will need to first obtain a valid OAuth token for your Pocket account. You can do this by running the auth command and following the prompts:
$ pocket-to-sqlite auth
Visit this page and sign in with your Pocket account:
https://getpocket.com/auth/author...
Once you have signed in there, hit <enter> to continue
Authentication tokens written to auth.json
Now you can fetch all of your items from Pocket like this:
$ pocket-to-sqlite fetch pocket.db
The first time you run this command it will fetch all of your items, and display a progress bar while it does it.
On subsequent runs it will only fetch new items.
You can force it to fetch everything from the beginning again using --all. Use --silent to disable the progress bar.
Using with Datasette
The SQLite database produced by this tool is designed to be browsed using Datasette. Use the datasette-render-timestamps plugin to improve the display of the timestamp values.
",1,public,0,,0,
214299267,MDEwOlJlcG9zaXRvcnkyMTQyOTkyNjc=,datasette-render-timestamps,simonw/datasette-render-timestamps,0,9599,https://github.com/simonw/datasette-render-timestamps,Datasette plugin for rendering timestamps,0,2019-10-10T22:50:50Z,2020-10-17T11:09:42Z,2020-03-22T17:57:17Z,,17,4,4,Python,1,1,1,1,0,1,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",1,0,4,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,1,2,"# datasette-render-timestamps
[](https://pypi.org/project/datasette-render-timestamps/)
[](https://circleci.com/gh/simonw/datasette-render-timestamps)
[](https://github.com/simonw/datasette-render-timestamps/blob/master/LICENSE)
Datasette plugin for rendering timestamps.
## Installation
Install this plugin in the same environment as Datasette to enable this new functionality:
pip install datasette-render-timestamps
The plugin will then look out for integer numbers that are likely to be timestamps - anything that would be a number of seconds from 5 years ago to 5 years in the future.
These will then be rendered in a more readable format.
## Configuration
You can disable automatic column detection in favour of explicitly listing the columns that you would like to render using [plugin configuration](https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration) in a `metadata.json` file.
Add a `""datasette-render-timestamps""` configuration block and use a `""columns""` key to list the columns you would like to treat as timestamp values:
```json
{
""plugins"": {
""datasette-render-timestamps"": {
""columns"": [""created"", ""updated""]
}
}
}
```
This will cause any `created` or `updated` columns in any table to be treated as timestamps and rendered.
Save this to `metadata.json` and run datasette with the `--metadata` flag to load this configuration:
datasette serve mydata.db --metadata metadata.json
To disable automatic timestamp detection entirely, you can use `""columnns"": []`.
This configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the `entries` table in the `news.db` database:
```json
{
""databases"": {
""news"": {
""tables"": {
""entries"": {
""plugins"": {
""datasette-render-timestamps"": {
""columns"": [""created"", ""updated""]
}
}
}
}
}
}
}
```
And here's how to apply it to every `created` column in every table in the `news.db` database:
```json
{
""databases"": {
""news"": {
""plugins"": {
""datasette-render-timestamps"": {
""columns"": [""created"", ""updated""]
}
}
}
}
}
```
### Customizing the date format
The default format is `%B %d, %Y - %H:%M:%S UTC` which renders for example: `October 10, 2019 - 07:18:29 UTC`. If you want another format, the date format can be customized using plugin configuration. Any format string supported by [strftime](http://strftime.org/) may be used. For example:
```json
{
""plugins"": {
""datasette-render-timestamps"": {
""format"": ""%Y-%m-%d-%H:%M:%S""
}
}
}
```
","
datasette-render-timestamps
Datasette plugin for rendering timestamps.
Installation
Install this plugin in the same environment as Datasette to enable this new functionality:
pip install datasette-render-timestamps
The plugin will then look out for integer numbers that are likely to be timestamps - anything that would be a number of seconds from 5 years ago to 5 years in the future.
These will then be rendered in a more readable format.
Configuration
You can disable automatic column detection in favour of explicitly listing the columns that you would like to render using plugin configuration in a metadata.json file.
Add a ""datasette-render-timestamps"" configuration block and use a ""columns"" key to list the columns you would like to treat as timestamp values:
To disable automatic timestamp detection entirely, you can use ""columnns"": [].
This configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the entries table in the news.db database:
The default format is %B %d, %Y - %H:%M:%S UTC which renders for example: October 10, 2019 - 07:18:29 UTC. If you want another format, the date format can be customized using plugin configuration. Any format string supported by strftime may be used. For example:
",,,,,,
217216787,MDEwOlJlcG9zaXRvcnkyMTcyMTY3ODc=,datasette-haversine,simonw/datasette-haversine,0,9599,https://github.com/simonw/datasette-haversine,Datasette plugin that adds a custom SQL function for haversine distances,0,2019-10-24T05:16:14Z,2021-07-28T20:13:38Z,2021-07-28T20:14:24Z,,8,1,1,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,1,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-haversine
[](https://pypi.org/project/datasette-haversine/)
[](https://github.com/simonw/datasette-haversine/releases)
[](https://github.com/simonw/datasette-haversine/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-haversine/blob/main/LICENSE)
Datasette plugin that adds a custom SQL function for haversine distances
Install this plugin in the same environment as Datasette to enable the `haversine()` SQL function.
$ pip install datasette-haversine
The plugin is built on top of the [haversine](https://github.com/mapado/haversine) library.
## haversine() to calculate distances
```sql
select haversine(lat1, lon1, lat2, lon2);
```
This will return the distance in kilometers between the point defined by `(lat1, lon1)` and the point defined by `(lat2, lon2)`.
## Custom units
By default `haversine()` returns results in km. You can pass an optional third argument to get results in a different unit:
- `ft` for feet
- `m` for meters
- `in` for inches
- `mi` for miles
- `nmi` for nautical miles
- `km` for kilometers (the default)
```sql
select haversine(lat1, lon1, lat2, lon2, 'mi');
```
","
datasette-haversine
Datasette plugin that adds a custom SQL function for haversine distances
Install this plugin in the same environment as Datasette to enable the haversine() SQL function.
$ pip install datasette-haversine
The plugin is built on top of the haversine library.
haversine() to calculate distances
select haversine(lat1, lon1, lat2, lon2);
This will return the distance in kilometers between the point defined by (lat1, lon1) and the point defined by (lat2, lon2).
Custom units
By default haversine() returns results in km. You can pass an optional third argument to get results in a different unit:
ft for feet
m for meters
in for inches
mi for miles
nmi for nautical miles
km for kilometers (the default)
select haversine(lat1, lon1, lat2, lon2, 'mi');
",,,,,,
219372133,MDEwOlJlcG9zaXRvcnkyMTkzNzIxMzM=,sqlite-transform,simonw/sqlite-transform,0,9599,https://github.com/simonw/sqlite-transform,Tool for running transformations on columns in a SQLite database,0,2019-11-03T22:07:53Z,2021-08-02T22:06:23Z,2021-08-02T22:07:57Z,,64,29,29,Python,1,1,1,1,0,1,0,0,0,apache-2.0,"[""sqlite"", ""datasette-io"", ""datasette-tool""]",1,0,29,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,1,1,"# sqlite-transform
[](https://pypi.org/project/sqlite-transform/)
[](https://github.com/simonw/sqlite-transform/releases)
[](https://github.com/simonw/sqlite-transform/actions?query=workflow%3ATest)
[](https://github.com/dogsheep/sqlite-transform/blob/main/LICENSE)
Tool for running transformations on columns in a SQLite database.
> **:warning: This tool is no longer maintained**
>
> I added a new tool to [sqlite-utils](https://sqlite-utils.datasette.io/) called [sqlite-utils convert](https://sqlite-utils.datasette.io/en/stable/cli.html#converting-data-in-columns) which provides a super-set of the functionality originally provided here. `sqlite-transform` is no longer maintained, and I recommend switching to using `sqlite-utils convert` instead.
## How to install
pip install sqlite-transform
## parsedate and parsedatetime
These subcommands will run all values in the specified column through `dateutils.parser.parse()` and replace them with the result, formatted as an ISO timestamp or ISO date.
For example, if a row in the database has an `opened` column which contains `10/10/2019 08:10:00 PM`, running the following command:
sqlite-transform parsedatetime my.db mytable opened
Will result in that value being replaced by `2019-10-10T20:10:00`.
Using the `parsedate` subcommand here would result in `2019-10-10` instead.
In the case of ambiguous dates such as `03/04/05` these commands both default to assuming American-style `mm/dd/yy` format. You can pass `--dayfirst` to specify that the day should be assumed to be first, or `--yearfirst` for the year.
## jsonsplit
The `jsonsplit` subcommand takes columns that contain a comma-separated list, for example a `tags` column containing records like `""trees,park,dogs""` and converts it into a JSON array `[""trees"", ""park"", ""dogs""]`.
This is useful for taking advantage of Datasette's [Facet by JSON array](https://docs.datasette.io/en/stable/facets.html#facet-by-json-array) feature.
sqlite-transform jsonsplit my.db mytable tags
It defaults to splitting on commas, but you can specify a different delimiter character using the `--delimiter` option, for example:
sqlite-transform jsonsplit \
my.db mytable tags --delimiter ';'
Values within the array will be treated as strings, so a column containing `123,552,775` will be converted into the JSON array `[""123"", ""552"", ""775""]`.
You can specify a different type for these values using `--type int` or `--type float`, for example:
sqlite-transform jsonsplit \
my.db mytable tags --type int
This will result in that column being converted into `[123, 552, 775]`.
## lambda for executing your own code
The `lambda` subcommand lets you specify Python code which will be executed against the column.
Here's how to convert a column to uppercase:
sqlite-transform lambda my.db mytable mycolumn --code='str(value).upper()'
The code you provide will be compiled into a function that takes `value` as a single argument. You can break your function body into multiple lines, provided the last line is a `return` statement:
sqlite-transform lambda my.db mytable mycolumn --code='value = str(value)
return value.upper()'
You can also specify Python modules that should be imported and made available to your code using one or more `--import` options:
sqlite-transform lambda my.db mytable mycolumn \
--code='""\n"".join(textwrap.wrap(value, 10))' \
--import=textwrap
The `--dry-run` option will output a preview of the transformation against the first ten rows, without modifying the database.
## Saving the result to a separate column
Each of these commands accepts optional `--output` and `--output-type` options. These can be used to save the result of the transformation to a separate column, which will be created if the column does not already exist.
To save the result of `jsonsplit` to a new column called `json_tags`, use the following:
sqlite-transform jsonsplit my.db mytable tags \
--output json_tags
The type of the created column defaults to `text`, but a different column type can be specified using `--output-type`. This example will create a new floating point column called `float_id` with a copy of each item's ID increased by 0.5:
sqlite-transform lambda my.db mytable id \
--code 'float(value) + 0.5' \
--output float_id \
--output-type float
You can drop the original column at the end of the operation by adding `--drop`.
## Splitting a column into multiple columns
Sometimes you may wish to convert a single column into multiple derived columns. For example, you may have a `location` column containing `latitude,longitude` values which you wish to split out into separate `latitude` and `longitude` columns.
You can achieve this using the `--multi` option to `sqlite-transform lambda`. This option expects your `--code` function to return a Python dictionary: new columns well be created and populated for each of the keys in that dictionary.
For the `latitude,longitude` example you would use the following:
sqlite-transform lambda demo.db places location \
--code 'return {
""latitude"": float(value.split("","")[0]),
""longitude"": float(value.split("","")[1]),
}' --multi
The type of the returned values will be taken into account when creating the new columns. In this example, the resulting database schema will look like this:
```sql
CREATE TABLE [places] (
[location] TEXT,
[latitude] FLOAT,
[longitude] FLOAT
);
```
The code function can also return `None`, in which case its output will be ignored.
You can drop the original column at the end of the operation by adding `--drop`.
## Disabling the progress bar
By default each command will show a progress bar. Pass `-s` or `--silent` to hide that progress bar.
","
sqlite-transform
Tool for running transformations on columns in a SQLite database.
⚠️ This tool is no longer maintained
I added a new tool to sqlite-utils called sqlite-utils convert which provides a super-set of the functionality originally provided here. sqlite-transform is no longer maintained, and I recommend switching to using sqlite-utils convert instead.
How to install
pip install sqlite-transform
parsedate and parsedatetime
These subcommands will run all values in the specified column through dateutils.parser.parse() and replace them with the result, formatted as an ISO timestamp or ISO date.
For example, if a row in the database has an opened column which contains 10/10/2019 08:10:00 PM, running the following command:
sqlite-transform parsedatetime my.db mytable opened
Will result in that value being replaced by 2019-10-10T20:10:00.
Using the parsedate subcommand here would result in 2019-10-10 instead.
In the case of ambiguous dates such as 03/04/05 these commands both default to assuming American-style mm/dd/yy format. You can pass --dayfirst to specify that the day should be assumed to be first, or --yearfirst for the year.
jsonsplit
The jsonsplit subcommand takes columns that contain a comma-separated list, for example a tags column containing records like ""trees,park,dogs"" and converts it into a JSON array [""trees"", ""park"", ""dogs""].
This is useful for taking advantage of Datasette's Facet by JSON array feature.
sqlite-transform jsonsplit my.db mytable tags
It defaults to splitting on commas, but you can specify a different delimiter character using the --delimiter option, for example:
Values within the array will be treated as strings, so a column containing 123,552,775 will be converted into the JSON array [""123"", ""552"", ""775""].
You can specify a different type for these values using --type int or --type float, for example:
sqlite-transform jsonsplit \
my.db mytable tags --type int
This will result in that column being converted into [123, 552, 775].
lambda for executing your own code
The lambda subcommand lets you specify Python code which will be executed against the column.
The code you provide will be compiled into a function that takes value as a single argument. You can break your function body into multiple lines, provided the last line is a return statement:
The --dry-run option will output a preview of the transformation against the first ten rows, without modifying the database.
Saving the result to a separate column
Each of these commands accepts optional --output and --output-type options. These can be used to save the result of the transformation to a separate column, which will be created if the column does not already exist.
To save the result of jsonsplit to a new column called json_tags, use the following:
The type of the created column defaults to text, but a different column type can be specified using --output-type. This example will create a new floating point column called float_id with a copy of each item's ID increased by 0.5:
You can drop the original column at the end of the operation by adding --drop.
Splitting a column into multiple columns
Sometimes you may wish to convert a single column into multiple derived columns. For example, you may have a location column containing latitude,longitude values which you wish to split out into separate latitude and longitude columns.
You can achieve this using the --multi option to sqlite-transform lambda. This option expects your --code function to return a Python dictionary: new columns well be created and populated for each of the keys in that dictionary.
For the latitude,longitude example you would use the following:
The type of the returned values will be taken into account when creating the new columns. In this example, the resulting database schema will look like this:
The code function can also return None, in which case its output will be ignored.
You can drop the original column at the end of the operation by adding --drop.
Disabling the progress bar
By default each command will show a progress bar. Pass -s or --silent to hide that progress bar.
",,,,,,
220716822,MDEwOlJlcG9zaXRvcnkyMjA3MTY4MjI=,datasette-render-markdown,simonw/datasette-render-markdown,0,9599,https://github.com/simonw/datasette-render-markdown,Datasette plugin for rendering Markdown,0,2019-11-09T23:28:31Z,2022-05-26T04:58:56Z,2022-07-18T19:35:10Z,,57,11,11,Python,1,1,1,1,0,0,0,0,1,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""markdown""]",0,1,11,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-render-markdown
[](https://pypi.org/project/datasette-render-markdown/)
[](https://github.com/simonw/datasette-render-markdown/releases)
[](https://github.com/simonw/datasette-render-markdown/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-render-markdown/blob/main/LICENSE)
Datasette plugin for rendering Markdown.
## Installation
Install this plugin in the same environment as Datasette to enable this new functionality:
$ pip install datasette-render-markdown
## Usage
You can explicitly list the columns you would like to treat as Markdown using [plugin configuration](https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration) in a `metadata.json` file.
Add a `""datasette-render-markdown""` configuration block and use a `""columns""` key to list the columns you would like to treat as Markdown values:
```json
{
""plugins"": {
""datasette-render-markdown"": {
""columns"": [""body""]
}
}
}
```
This will cause any `body` column in any table to be treated as markdown and safely rendered using [Python-Markdown](https://python-markdown.github.io/). The resulting HTML is then run through [Bleach](https://bleach.readthedocs.io/) to avoid the risk of XSS security problems.
Save this to `metadata.json` and run Datasette with the `--metadata` flag to load this configuration:
$ datasette serve mydata.db --metadata metadata.json
The configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the `entries` table in the `news.db` database:
```json
{
""databases"": {
""news"": {
""tables"": {
""entries"": {
""plugins"": {
""datasette-render-markdown"": {
""columns"": [""body""]
}
}
}
}
}
}
}
```
And here's how to apply it to every `body` column in every table in the `news.db` database:
```json
{
""databases"": {
""news"": {
""plugins"": {
""datasette-render-markdown"": {
""columns"": [""body""]
}
}
}
}
}
```
## Columns that match a naming convention
This plugin can also render markdown in any columns that match a specific naming convention.
By default, columns that have a name ending in `_markdown` will be rendered.
You can try this out using the following query:
```sql
select '# Hello there
* This is a list
* of items
[And a link](https://github.com/simonw/datasette-render-markdown).'
as demo_markdown
```
You can configure a different list of wildcard patterns using the `""patterns""` configuration key. Here's how to render columns that end in either `_markdown` or `_md`:
```json
{
""plugins"": {
""datasette-render-markdown"": {
""patterns"": [""*_markdown"", ""*_md""]
}
}
}
```
To disable wildcard column matching entirely, set `""patterns"": []` in your plugin metadata configuration.
## Markdown extensions
The [Python-Markdown library](https://python-markdown.github.io/) that powers this plugin supports extensions, both [bundled](https://python-markdown.github.io/extensions/) and [third-party](https://github.com/Python-Markdown/markdown/wiki/Third-Party-Extensions). These can be used to enable additional Markdown features such as [table support](https://python-markdown.github.io/extensions/tables/).
You can configure support for extensions using the `""extensions""` key in your plugin metadata configuration.
Since extensions may introduce new HTML tags, you will also need to add those tags to the list of tags that are allowed by the [Bleach](https://bleach.readthedocs.io/) sanitizer. You can do that using the `""extra_tags""` key, and you can whitelist additional HTML attributes using `""extra_attrs""`. See [the Bleach documentation](https://bleach.readthedocs.io/en/latest/clean.html#allowed-tags-tags) for more information on this.
Here's how to enable support for [Markdown tables](https://python-markdown.github.io/extensions/tables/):
```json
{
""plugins"": {
""datasette-render-markdown"": {
""extensions"": [""tables""],
""extra_tags"": [""table"", ""thead"", ""tr"", ""th"", ""td"", ""tbody""]
}
}
}
```
### GitHub-Flavored Markdown
Enabling [GitHub-Flavored Markdown](https://help.github.com/en/github/writing-on-github) (useful for if you are working with data imported from GitHub using [github-to-sqlite](https://github.com/dogsheep/github-to-sqlite)) is a little more complicated.
First, you will need to install the [py-gfm](https://py-gfm.readthedocs.io) package:
$ pip install py-gfm
Note that `py-gfm` has [a bug](https://github.com/Zopieux/py-gfm/issues/13) that causes it to pin to `Markdown<3.0` - so if you are using it you should install it _before_ installing `datasette-render-markdown` to ensure you get a compatibly version of that dependency.
Now you can configure it like this. Note that the extension name is `mdx_gfm:GithubFlavoredMarkdownExtension` and you need to whitelist several extra HTML tags and attributes:
```json
{
""plugins"": {
""datasette-render-markdown"": {
""extra_tags"": [
""hr"",
""br"",
""details"",
""summary"",
""input""
],
""extra_attrs"": {
""input"": [
""type"",
""disabled"",
""checked""
],
},
""extensions"": [
""mdx_gfm:GithubFlavoredMarkdownExtension""
]
}
}
}
```
The `` attributes are needed to support rendering checkboxes in issue descriptions.
## Markdown in templates
The plugin also adds a new template function: `render_markdown(value)`. You can use this in your templates like so:
```html+jinja
{{ render_markdown(""""""
# This is markdown
* One
* Two
* Three
"""""") }}
```
You can load additional extensions and whitelist tags by passing extra arguments to the function like this:
```html+jinja
{{ render_markdown(""""""
## Markdown table
First Header | Second Header
------------- | -------------
Content Cell | Content Cell
Content Cell | Content Cell
"""""", extensions=[""tables""],
extra_tags=[""table"", ""thead"", ""tr"", ""th"", ""td"", ""tbody""])) }}
```
","
datasette-render-markdown
Datasette plugin for rendering Markdown.
Installation
Install this plugin in the same environment as Datasette to enable this new functionality:
$ pip install datasette-render-markdown
Usage
You can explicitly list the columns you would like to treat as Markdown using plugin configuration in a metadata.json file.
Add a ""datasette-render-markdown"" configuration block and use a ""columns"" key to list the columns you would like to treat as Markdown values:
This will cause any body column in any table to be treated as markdown and safely rendered using Python-Markdown. The resulting HTML is then run through Bleach to avoid the risk of XSS security problems.
Save this to metadata.json and run Datasette with the --metadata flag to load this configuration:
The configuration block can be used at the top level, or it can be applied just to specific databases or tables. Here's how to apply it to just the entries table in the news.db database:
This plugin can also render markdown in any columns that match a specific naming convention.
By default, columns that have a name ending in _markdown will be rendered.
You can try this out using the following query:
select'# Hello there* This is a list* of items[And a link](https://github.com/simonw/datasette-render-markdown).'as demo_markdown
You can configure a different list of wildcard patterns using the ""patterns"" configuration key. Here's how to render columns that end in either _markdown or _md:
You can configure support for extensions using the ""extensions"" key in your plugin metadata configuration.
Since extensions may introduce new HTML tags, you will also need to add those tags to the list of tags that are allowed by the Bleach sanitizer. You can do that using the ""extra_tags"" key, and you can whitelist additional HTML attributes using ""extra_attrs"". See the Bleach documentation for more information on this.
First, you will need to install the py-gfm package:
$ pip install py-gfm
Note that py-gfm has a bug that causes it to pin to Markdown<3.0 - so if you are using it you should install it before installing datasette-render-markdown to ensure you get a compatibly version of that dependency.
Now you can configure it like this. Note that the extension name is mdx_gfm:GithubFlavoredMarkdownExtension and you need to whitelist several extra HTML tags and attributes:
",1,public,0,,0,
221802296,MDEwOlJlcG9zaXRvcnkyMjE4MDIyOTY=,datasette-template-sql,simonw/datasette-template-sql,0,9599,https://github.com/simonw/datasette-template-sql,Datasette plugin for executing SQL queries from templates,0,2019-11-14T23:05:34Z,2021-05-18T17:58:47Z,2021-05-18T17:58:44Z,https://datasette.io/plugins/datasette-template-sql,23,6,6,Python,1,1,1,1,0,0,0,0,1,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,1,6,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-template-sql
[](https://pypi.org/project/datasette-template-sql/)
[](https://github.com/simonw/datasette-template-sql/releases)
[](https://github.com/simonw/datasette-template-sql/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-template-sql/blob/main/LICENSE)
Datasette plugin for executing SQL queries from templates.
## Examples
[datasette.io](https://datasette.io/) uses this plugin extensively with [custom page templates](https://docs.datasette.io/en/stable/custom_templates.html#custom-pages), check out [simonw/datasette.io](https://github.com/simonw/datasette.io) to see how it works.
[www.niche-museums.com](https://www.niche-museums.com/) uses this plugin to run a custom themed website on top of Datasette. The full source code for the site [is here](https://github.com/simonw/museums) - see also [niche-museums.com, powered by Datasette](https://simonwillison.net/2019/Nov/25/niche-museums/).
[simonw/til](https://github.com/simonw/til) is another simple example, described in [Using a self-rewriting README powered by GitHub Actions to track TILs](https://simonwillison.net/2020/Apr/20/self-rewriting-readme/).
## Installation
Run this command to install the plugin in the same environment as Datasette:
$ pip install datasette-template-sql
## Usage
This plugin makes a new function, `sql(sql_query)`, available to your Datasette templates.
You can use it like this:
```html+jinja
{% for row in sql(""select 1 + 1 as two, 2 * 4 as eight"") %}
{% for key in row.keys() %}
{{ key }}: {{ row[key] }}
{% endfor %}
{% endfor %}
```
The plugin will execute SQL against the current database for the page in `database.html`, `table.html` and `row.html` templates. If a template does not have a current database (`index.html` for example) the query will execute against the first attached database.
### Queries with arguments
You can construct a SQL query using `?` or `:name` parameter syntax by passing a list or dictionary as a second argument:
```html+jinja
{% for row in sql(""select distinct topic from til order by topic"") %}
{{ row.topic }}
{% for til in sql(""select * from til where topic = ?"", [row.topic]) %}
{% endfor %}
```
Here's the same example using the `:topic` style of parameters:
```html+jinja
{% for row in sql(""select distinct topic from til order by topic"") %}
{{ row.topic }}
{% for til in sql(""select * from til where topic = :topic"", {""topic"": row.topic}) %}
{% endfor %}
```
### Querying a different database
You can pass an optional `database=` argument to specify a named database to use for the query. For example, if you have attached a `news.db` database you could use this:
```html+jinja
{% for article in sql(
""select headline, date, summary from articles order by date desc limit 5"",
database=""news""
) %}
{{ article.headline }}
{{ article.date }}
{{ article.summary }}
{% endfor %}
```
","
datasette-template-sql
Datasette plugin for executing SQL queries from templates.
Run this command to install the plugin in the same environment as Datasette:
$ pip install datasette-template-sql
Usage
This plugin makes a new function, sql(sql_query), available to your Datasette templates.
You can use it like this:
{%forrowinsql(""select 1 + 1 as two, 2 * 4 as eight"") %}{%forkeyinrow.keys() %}
{{ key }}: {{ row[key] }}<br>
{%endfor%}{%endfor%}
The plugin will execute SQL against the current database for the page in database.html, table.html and row.html templates. If a template does not have a current database (index.html for example) the query will execute against the first attached database.
Queries with arguments
You can construct a SQL query using ? or :name parameter syntax by passing a list or dictionary as a second argument:
{%forrowinsql(""select distinct topic from til order by topic"") %}
<h2>{{ row.topic }}</h2>
<ul>
{%fortilinsql(""select * from til where topic = ?"", [row.topic]) %}
<li><ahref=""{{ til.url }}"">{{ til.title }}</a> - {{ til.created[:10] }}</li>
{%endfor%}
</ul>
{%endfor%}
Here's the same example using the :topic style of parameters:
{%forrowinsql(""select distinct topic from til order by topic"") %}
<h2>{{ row.topic }}</h2>
<ul>
{%fortilinsql(""select * from til where topic = :topic"", {""topic"": row.topic}) %}
<li><ahref=""{{ til.url }}"">{{ til.title }}</a> - {{ til.created[:10] }}</li>
{%endfor%}
</ul>
{%endfor%}
Querying a different database
You can pass an optional database= argument to specify a named database to use for the query. For example, if you have attached a news.db database you could use this:
{%forarticleinsql(
""select headline, date, summary from articles order by date desc limit 5"",
database=""news""
) %}
<h3>{{ article.headline }}</h2>
<pclass=""date"">{{ article.date }}</p>
<p>{{ article.summary }}</p>
{%endfor%}
",,,,,,
228485806,MDEwOlJlcG9zaXRvcnkyMjg0ODU4MDY=,datasette-configure-asgi,simonw/datasette-configure-asgi,0,9599,https://github.com/simonw/datasette-configure-asgi,Datasette plugin for configuring arbitrary ASGI middleware,0,2019-12-16T22:17:10Z,2020-08-25T15:54:32Z,2019-12-16T22:19:49Z,,6,1,1,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""asgi"", ""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,1,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-configure-asgi
[](https://pypi.org/project/datasette-configure-asgi/)
[](https://circleci.com/gh/simonw/datasette-configure-asgi)
[](https://github.com/simonw/datasette-configure-asgi/blob/master/LICENSE)
Datasette plugin for configuring arbitrary ASGI middleware
## Installation
pip install datasette-configure-asgi
## Usage
This plugin only takes effect if your `metadata.json` file contains relevant top-level plugin configuration in a `""datasette-configure-asgi""` configuration key.
For example, to wrap your Datasette instance in the `asgi-log-to-sqlite` middleware configured to write logs to `/tmp/log.db` you would use the following:
```json
{
""plugins"": {
""datasette-configure-asgi"": [
{
""class"": ""asgi_log_to_sqlite.AsgiLogToSqlite"",
""args"": {
""file"": ""/tmp/log.db""
}
}
]
}
}
```
The `""datasette-configure-asgi""` key should be a list of JSON objects. Each object should have a `""class""` key indicating the class to be used, and an optional `""args""` key providing any necessary arguments to be passed to that class constructor.
## Plugin structure
This plugin can be used to wrap your Datasette instance in any ASGI middleware that conforms to the following structure:
```python
class SomeAsgiMiddleware:
def __init__(self, app, arg1, arg2):
self.app = app
self.arg1 = arg1
self.arg2 = arg2
async def __call__(self, scope, receive, send):
start = time.time()
await self.app(scope, receive, send)
end = time.time()
print(""Time taken: {}"".format(end - start))
```
So the middleware is a class with a constructor which takes the wrapped application as a first argument, `app`, followed by further named arguments to configure the middleware. It provides an `async def __call__(self, scope, receive, send)` method to implement the middleware's behavior.
","
datasette-configure-asgi
Datasette plugin for configuring arbitrary ASGI middleware
Installation
pip install datasette-configure-asgi
Usage
This plugin only takes effect if your metadata.json file contains relevant top-level plugin configuration in a ""datasette-configure-asgi"" configuration key.
For example, to wrap your Datasette instance in the asgi-log-to-sqlite middleware configured to write logs to /tmp/log.db you would use the following:
The ""datasette-configure-asgi"" key should be a list of JSON objects. Each object should have a ""class"" key indicating the class to be used, and an optional ""args"" key providing any necessary arguments to be passed to that class constructor.
Plugin structure
This plugin can be used to wrap your Datasette instance in any ASGI middleware that conforms to the following structure:
So the middleware is a class with a constructor which takes the wrapped application as a first argument, app, followed by further named arguments to configure the middleware. It provides an async def __call__(self, scope, receive, send) method to implement the middleware's behavior.
",,,,,,
234825790,MDEwOlJlcG9zaXRvcnkyMzQ4MjU3OTA=,datasette-upload-csvs,simonw/datasette-upload-csvs,0,9599,https://github.com/simonw/datasette-upload-csvs,Datasette plugin for uploading CSV files and converting them to database tables,0,2020-01-19T02:07:05Z,2022-07-03T20:58:20Z,2022-09-09T16:23:59Z,https://datasette.io/plugins/datasette-upload-csvs,58,9,9,Python,1,1,1,1,0,1,0,0,4,apache-2.0,"[""csvs"", ""datasette"", ""datasette-io"", ""datasette-plugin""]",1,4,9,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,1,2,"# datasette-upload-csvs
[](https://pypi.org/project/datasette-upload-csvs/)
[](https://github.com/simonw/datasette-upload-csvs/releases)
[](https://github.com/simonw/datasette-upload-csvs/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-upload-csvs/blob/main/LICENSE)
Datasette plugin for uploading CSV files and converting them to database tables
## Installation
datasette install datasette-upload-csvs
## Usage
The plugin adds an interface at `/-/upload-csvs` for uploading a CSV file and using it to create a new database table.
By default only [the root actor](https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor) can access the page - so you'll need to run Datasette with the `--root` option and click on the link shown in the terminal to sign in and access the page.
The `upload-csvs` permission governs access. You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to the write interface.
","
datasette-upload-csvs
Datasette plugin for uploading CSV files and converting them to database tables
Installation
datasette install datasette-upload-csvs
Usage
The plugin adds an interface at /-/upload-csvs for uploading a CSV file and using it to create a new database table.
By default only the root actor can access the page - so you'll need to run Datasette with the --root option and click on the link shown in the terminal to sign in and access the page.
The upload-csvs permission governs access. You can use permission plugins such as datasette-permissions-sql to grant additional access to the write interface.
",1,public,0,,0,
236110759,MDEwOlJlcG9zaXRvcnkyMzYxMTA3NTk=,datasette-auth-existing-cookies,simonw/datasette-auth-existing-cookies,0,9599,https://github.com/simonw/datasette-auth-existing-cookies,Datasette plugin that authenticates users based on existing domain cookies,0,2020-01-25T01:20:31Z,2022-05-28T01:50:15Z,2022-05-30T17:10:11Z,,54,3,3,Python,1,1,1,1,0,1,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin""]",1,0,3,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,1,3,"# datasette-auth-existing-cookies
[](https://pypi.org/project/datasette-auth-existing-cookies/)
[](https://github.com/simonw/datasette-auth-existing-cookies/releases)
[](https://github.com/simonw/datasette-auth-existing-cookies/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-auth-existing-cookies/blob/master/LICENSE)
Datasette plugin that authenticates users based on existing domain cookies.
## When to use this
This plugin allows you to build custom authentication for Datasette when you are hosting a Datasette instance on the same domain as another, authenticated website.
Consider a website on `www.example.com` which supports user authentication.
You could run Datasette on `data.example.com` in a way that lets it see cookies that were set for the `.example.com` domain.
Using this plugin, you could build an API endpoint at `www.example.com/user-for-cookies` which returns a JSON object representing the currently signed-in user, based on their cookies.
The plugin running on `data.example.com` will then make the `actor` available to the rest of Datasette based on the response from that API.
Read about [Datasette's authentication and permissions system](https://docs.datasette.io/en/stable/authentication.html) for more on how actors and permissions work.
## Configuration
This plugin requires some configuration in the Datasette [metadata.json file](https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration).
The following configuration options are supported:
- `api_url`: this is the API endpoint that Datasette should call with the user's cookies in order to identify the logged in user.
- `cookies`: optional. A list of cookie names that should be passed through to the API endpoint - if left blank, the default is to send all cookies.
- `ttl`: optional. By default Datasette will make a request to the API endpoint for every HTTP request recieved by Datasette itself. A `ttl` value of 5 will cause Datasette to cache the actor associated with the user's cookies for 5 seconds, reducing that API traffic.
- `headers`: an optional list of other headers to forward to the API endpoint as query string parameters.
Here is an example that uses all four of these settings:
```json
{
""plugins"": {
""datasette-auth-existing-cookies"": {
""api_url"": ""http://www.example.com/user-from-cookies"",
""cookies"": [""sessionid""],
""headers"": [""host""],
""ttl"": 10
}
}
}
```
With this configuration any hit to a Datasette hosted at `data.example.com` will result in the following request being made to the `http://www.example.com/user-from-cookies` API endpoint:
```
GET http://www.example.com/user-from-cookies?host=data.example.com
Cookie: sessionid=abc123
```
That API is expected to return a JSON object representing the current user:
```json
{
""id"": 1,
""name"": ""Barry""
}
```
Since `ttl` is set to 10 that actor will be cached for ten seconds against that exact combination of cookies and headers. When that cache expires another hit will be made to the API.
When deciding on a TTL value, take into account that users who lose access to the core site - maybe because their session expires, or their account is disabled - will still be able to access the Datasette instance until that cache expires.
","
datasette-auth-existing-cookies
Datasette plugin that authenticates users based on existing domain cookies.
When to use this
This plugin allows you to build custom authentication for Datasette when you are hosting a Datasette instance on the same domain as another, authenticated website.
Consider a website on www.example.com which supports user authentication.
You could run Datasette on data.example.com in a way that lets it see cookies that were set for the .example.com domain.
Using this plugin, you could build an API endpoint at www.example.com/user-for-cookies which returns a JSON object representing the currently signed-in user, based on their cookies.
The plugin running on data.example.com will then make the actor available to the rest of Datasette based on the response from that API.
This plugin requires some configuration in the Datasette metadata.json file.
The following configuration options are supported:
api_url: this is the API endpoint that Datasette should call with the user's cookies in order to identify the logged in user.
cookies: optional. A list of cookie names that should be passed through to the API endpoint - if left blank, the default is to send all cookies.
ttl: optional. By default Datasette will make a request to the API endpoint for every HTTP request recieved by Datasette itself. A ttl value of 5 will cause Datasette to cache the actor associated with the user's cookies for 5 seconds, reducing that API traffic.
headers: an optional list of other headers to forward to the API endpoint as query string parameters.
Here is an example that uses all four of these settings:
With this configuration any hit to a Datasette hosted at data.example.com will result in the following request being made to the http://www.example.com/user-from-cookies API endpoint:
GET http://www.example.com/user-from-cookies?host=data.example.com
Cookie: sessionid=abc123
That API is expected to return a JSON object representing the current user:
{
""id"": 1,
""name"": ""Barry""
}
Since ttl is set to 10 that actor will be cached for ten seconds against that exact combination of cookies and headers. When that cache expires another hit will be made to the API.
When deciding on a TTL value, take into account that users who lose access to the core site - maybe because their session expires, or their account is disabled - will still be able to access the Datasette instance until that cache expires.
",1,public,0,,,
236867027,MDEwOlJlcG9zaXRvcnkyMzY4NjcwMjc=,datasette-sentry,simonw/datasette-sentry,0,9599,https://github.com/simonw/datasette-sentry,Datasette plugin for configuring Sentry,0,2020-01-28T23:41:27Z,2022-07-18T20:28:25Z,2022-10-06T22:31:29Z,,26,6,6,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""sentry""]",0,0,6,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-sentry
[](https://pypi.org/project/datasette-sentry/)
[](https://github.com/simonw/datasette-sentry/releases)
[](https://github.com/simonw/datasette-sentry/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-sentry/blob/main/LICENSE)
Datasette plugin for configuring Sentry for error reporting
## Installation
pip install datasette-sentry
## Usage
This plugin only takes effect if your `metadata.json` file contains relevant top-level plugin configuration in a `""datasette-sentry""` configuration key.
You will need a Sentry DSN - see their [Getting Started instructions](https://docs.sentry.io/error-reporting/quickstart/?platform=python).
Add it to `metadata.json` like this:
```json
{
""plugins"": {
""datasette-sentry"": {
""dsn"": ""https://KEY@sentry.io/PROJECTID""
}
}
}
```
Settings in `metadata.json` are visible to anyone who visits the `/-/metadata` URL so this is a good place to take advantage of Datasette's [secret configuration values](https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values), in which case your configuration will look more like this:
```json
{
""plugins"": {
""datasette-sentry"": {
""dsn"": {
""$env"": ""SENTRY_DSN""
}
}
}
}
```
Then make a `SENTRY_DSN` environment variable available to Datasette.
## Configuration
In addition to the `dsn` setting, you can also configure the Sentry [sample rate](https://docs.sentry.io/platforms/python/configuration/sampling/) by setting `sample_rate` to a floating point number between 0 and 1.
For example, to capture 25% of errors you would do this:
```json
{
""plugins"": {
""datasette-sentry"": {
""dsn"": {
""$env"": ""SENTRY_DSN""
},
""sample_rate"": 0.25
}
}
}
```
","
datasette-sentry
Datasette plugin for configuring Sentry for error reporting
Installation
pip install datasette-sentry
Usage
This plugin only takes effect if your metadata.json file contains relevant top-level plugin configuration in a ""datasette-sentry"" configuration key.
Settings in metadata.json are visible to anyone who visits the /-/metadata URL so this is a good place to take advantage of Datasette's secret configuration values, in which case your configuration will look more like this:
",1,public,0,,0,
237321267,MDEwOlJlcG9zaXRvcnkyMzczMjEyNjc=,geojson-to-sqlite,simonw/geojson-to-sqlite,0,9599,https://github.com/simonw/geojson-to-sqlite,CLI tool for converting GeoJSON files to SQLite (with SpatiaLite),0,2020-01-30T22:51:05Z,2022-03-05T00:40:56Z,2022-04-13T23:39:25Z,,117,34,34,Python,1,1,1,1,0,3,0,0,4,apache-2.0,"[""datasette-io"", ""datasette-tool"", ""geojson"", ""gis"", ""sqlite""]",3,4,34,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,3,3,"# geojson-to-sqlite
[](https://pypi.org/project/geojson-to-sqlite/)
[](https://github.com/simonw/geojson-to-sqlite/releases)
[](https://github.com/simonw/geojson-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/geojson-to-sqlite/blob/main/LICENSE)
CLI tool for converting GeoJSON to SQLite (optionally with SpatiaLite)
[RFC 7946: The GeoJSON Format](https://tools.ietf.org/html/rfc7946)
## How to install
$ pip install geojson-to-sqlite
## How to use
You can run this tool against a GeoJSON file like so:
$ geojson-to-sqlite my.db features features.geojson
This will load all of the features from the `features.geojson` file into a table called `features`.
Each row will have a `geometry` column containing the feature geometry, and columns for each of the keys found in any `properties` attached to those features. (To bundle all properties into a single JSON object, use the `--properties` flag.)
The table will be created the first time you run the command.
On subsequent runs you can use the `--alter` option to add any new columns that are missing from the table.
You can pass more than one GeoJSON file, in which case the contents of all of the files will be inserted into the same table.
If your features have an `""id""` property it will be used as the primary key for the table. You can also use `--pk=PROPERTY` with the name of a different property to use that as the primary key instead. If you don't want to use the `""id""` as the primary key (maybe it contains duplicate values) you can use `--pk ''` to specify no primary key.
Specifying a primary key also will allow you to upsert data into the rows instead of insert data into new rows.
If no primary key is specified, a SQLite `rowid` column will be used.
You can use `-` as the filename to import from standard input. For example:
$ curl https://eric.clst.org/assets/wiki/uploads/Stuff/gz_2010_us_040_00_20m.json \
| geojson-to-sqlite my.db states - --pk GEO_ID
## Using with SpatiaLite
By default, the `geometry` column will contain JSON.
If you have installed the [SpatiaLite](https://www.gaia-gis.it/fossil/libspatialite/index) module for SQLite you can instead import the geometry into a geospatially indexed column.
You can do this using the `--spatialite` option, like so:
$ geojson-to-sqlite my.db features features.geojson --spatialite
The tool will search for the SpatiaLite module in the following locations:
- `/usr/lib/x86_64-linux-gnu/mod_spatialite.so`
- `/usr/local/lib/mod_spatialite.dylib`
If you have installed the module in another location, you can use the `--spatialite_mod=xxx` option to specify where:
$ geojson-to-sqlite my.db features features.geojson \
--spatialite_mod=/usr/lib/mod_spatialite.dylib
You can create a SpatiaLite spatial index on the `geometry` column using the `--spatial-index` option:
$ geojson-to-sqlite my.db features features.geojson --spatial-index
Using this option implies `--spatialite` so you do not need to add that.
## Streaming large datasets
For large datasets, consider using newline-delimited JSON to stream features into the database without loading the entire feature collection into memory.
For example, to load a day of earthquake reports from USGS:
$ geojson-to-sqlite quakes.db quakes tests/quakes.ndjson \
--nl --pk=id --spatialite
When using newline-delimited JSON, tables will also be created from the first feature, instead of guessing types based on the first 100 features.
If you want to use a larger subset of your data to guess column types (for example, if some fields are inconsistent) you can use [fiona](https://fiona.readthedocs.io/en/latest/cli.html) to collect features into a single collection.
$ head tests/quakes.ndjson | fio collect | \
geojson-to-sqlite quakes.db quakes - --spatialite
This will take the first 10 lines from `tests/quakes.ndjson`, pass them to `fio collect`, which turns them into a single feature collection, and pass that, in turn, to `geojson-to-sqlite`.
## Using this with Datasette
Databases created using this tool can be explored and published using [Datasette](https://datasette.readthedocs.io/).
The Datasette documentation includes a section on [how to use it to browse SpatiaLite databases](https://datasette.readthedocs.io/en/stable/spatialite.html).
The [datasette-leaflet-geojson](https://datasette.io/plugins/datasette-leaflet-geojson) plugin can be used to visualize columns containing GeoJSON geometries on a [Leaflet](https://leafletjs.com/) map.
If you are using SpatiaLite you will need to output the geometry as GeoJSON in order for that plugin to work. You can do that using the SpaitaLite `AsGeoJSON()` function - something like this:
```sql
select rowid, AsGeoJSON(geometry) from mytable limit 10
```
The [datasette-geojson-map](https://datasette.io/plugins/datasette-geojson-map) is an alternative plugin which will automatically render SpatiaLite geometries as a Leaflet map on the corresponding table page, without needing you to call `AsGeoJSON(geometry)`.
","
geojson-to-sqlite
CLI tool for converting GeoJSON to SQLite (optionally with SpatiaLite)
You can run this tool against a GeoJSON file like so:
$ geojson-to-sqlite my.db features features.geojson
This will load all of the features from the features.geojson file into a table called features.
Each row will have a geometry column containing the feature geometry, and columns for each of the keys found in any properties attached to those features. (To bundle all properties into a single JSON object, use the --properties flag.)
The table will be created the first time you run the command.
On subsequent runs you can use the --alter option to add any new columns that are missing from the table.
You can pass more than one GeoJSON file, in which case the contents of all of the files will be inserted into the same table.
If your features have an ""id"" property it will be used as the primary key for the table. You can also use --pk=PROPERTY with the name of a different property to use that as the primary key instead. If you don't want to use the ""id"" as the primary key (maybe it contains duplicate values) you can use --pk '' to specify no primary key.
Specifying a primary key also will allow you to upsert data into the rows instead of insert data into new rows.
If no primary key is specified, a SQLite rowid column will be used.
You can use - as the filename to import from standard input. For example:
By default, the geometry column will contain JSON.
If you have installed the SpatiaLite module for SQLite you can instead import the geometry into a geospatially indexed column.
You can do this using the --spatialite option, like so:
$ geojson-to-sqlite my.db features features.geojson --spatialite
The tool will search for the SpatiaLite module in the following locations:
/usr/lib/x86_64-linux-gnu/mod_spatialite.so
/usr/local/lib/mod_spatialite.dylib
If you have installed the module in another location, you can use the --spatialite_mod=xxx option to specify where:
$ geojson-to-sqlite my.db features features.geojson \
--spatialite_mod=/usr/lib/mod_spatialite.dylib
You can create a SpatiaLite spatial index on the geometry column using the --spatial-index option:
$ geojson-to-sqlite my.db features features.geojson --spatial-index
Using this option implies --spatialite so you do not need to add that.
Streaming large datasets
For large datasets, consider using newline-delimited JSON to stream features into the database without loading the entire feature collection into memory.
For example, to load a day of earthquake reports from USGS:
When using newline-delimited JSON, tables will also be created from the first feature, instead of guessing types based on the first 100 features.
If you want to use a larger subset of your data to guess column types (for example, if some fields are inconsistent) you can use fiona to collect features into a single collection.
This will take the first 10 lines from tests/quakes.ndjson, pass them to fio collect, which turns them into a single feature collection, and pass that, in turn, to geojson-to-sqlite.
Using this with Datasette
Databases created using this tool can be explored and published using Datasette.
If you are using SpatiaLite you will need to output the geometry as GeoJSON in order for that plugin to work. You can do that using the SpaitaLite AsGeoJSON() function - something like this:
select rowid, AsGeoJSON(geometry) from mytable limit10
The datasette-geojson-map is an alternative plugin which will automatically render SpatiaLite geometries as a Leaflet map on the corresponding table page, without needing you to call AsGeoJSON(geometry).
",1,public,0,,,
238339412,MDEwOlJlcG9zaXRvcnkyMzgzMzk0MTI=,datasette-debug-asgi,simonw/datasette-debug-asgi,0,9599,https://github.com/simonw/datasette-debug-asgi,Datasette plugin for dumping out the ASGI scope,0,2020-02-05T00:57:09Z,2021-08-17T23:40:02Z,2021-08-17T23:41:03Z,https://datasette.io/plugins/datasette-debug-asgi,16,1,1,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""asgi"", ""datasette-io"", ""datasette-plugin""]",0,0,1,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-debug-asgi
[](https://pypi.org/project/datasette-debug-asgi/)
[](https://github.com/simonw/datasette-debug-asgi/releases)
[](https://github.com/simonw/datasette-debug-asgi/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-debug-asgi/blob/main/LICENSE)
Datasette plugin for dumping out the ASGI scope.
Adds a new URL at `/-/asgi-scope` which shows the current ASGI scope. Demo here: https://datasette.io/-/asgi-scope
## Installation
pip install datasette-debug-asgi
## Usage
Visit `/-/asgi-scope` to see debug output showing the ASGI scope.
You can add query string parameters such as `/-/asgi-scope?q=hello`.
You can also add extra path components such as `/-/asgi-scope/more/path/here`.
","
Visit /-/asgi-scope to see debug output showing the ASGI scope.
You can add query string parameters such as /-/asgi-scope?q=hello.
You can also add extra path components such as /-/asgi-scope/more/path/here.
",,,,,,
240815938,MDEwOlJlcG9zaXRvcnkyNDA4MTU5Mzg=,shapefile-to-sqlite,simonw/shapefile-to-sqlite,0,9599,https://github.com/simonw/shapefile-to-sqlite,Load shapefiles into a SQLite (optionally SpatiaLite) database,0,2020-02-16T01:55:29Z,2021-03-26T08:39:43Z,2020-08-23T06:00:41Z,,54,15,15,Python,1,1,1,1,0,0,0,0,3,apache-2.0,"[""sqlite"", ""gis"", ""spatialite"", ""shapefiles"", ""datasette"", ""datasette-io"", ""datasette-tool""]",0,3,15,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# shapefile-to-sqlite
[](https://pypi.org/project/shapefile-to-sqlite/)
[](https://circleci.com/gh/simonw/shapefile-to-sqlite)
[](https://github.com/simonw/shapefile-to-sqlite/blob/main/LICENSE)
Load shapefiles into a SQLite (optionally SpatiaLite) database.
Project background: [Things I learned about shapefiles building shapefile-to-sqlite](https://simonwillison.net/2020/Feb/19/shapefile-to-sqlite/)
## How to install
$ pip install shapefile-to-sqlite
## How to use
You can run this tool against a shapefile file like so:
$ shapefile-to-sqlite my.db features.shp
This will load the geometries as GeoJSON in a text column.
## Using with SpatiaLite
If you have [SpatiaLite](https://www.gaia-gis.it/fossil/libspatialite/index) available you can load them as SpatiaLite geometries like this:
$ shapefile-to-sqlite my.db features.shp --spatialite
The data will be loaded into a table called `features` - based on the name of the shapefile. You can specify an alternative table name using `--table`:
$ shapefile-to-sqlite my.db features.shp --table=places --spatialite
The tool will search for the SpatiaLite module in the following locations:
- `/usr/lib/x86_64-linux-gnu/mod_spatialite.so`
- `/usr/local/lib/mod_spatialite.dylib`
If you have installed the module in another location, you can use the `--spatialite_mod=xxx` option to specify where:
$ shapefile-to-sqlite my.db features.shp \
--spatialite_mod=/usr/lib/mod_spatialite.dylib
You can use the `--spatial-index` option to create a spatial index on the `geometry` column:
$ shapefile-to-sqlite my.db features.shp --spatial-index
You can omit `--spatialite` if you use either `--spatialite-mod` or `--spatial-index`.
## Projections
By default, this tool will attempt to convert geometries in the shapefile to the WGS 84 projection, for best conformance with the [GeoJSON specification](https://tools.ietf.org/html/rfc7946).
If you want it to leave the data in whatever projection was used by the shapefile, use the `--crs=keep` option.
You can convert the data to another output projection by passing it to the `--crs` option. For example, to convert to [EPSG:2227](https://epsg.io/2227) (California zone 3) use `--crs=espg:2227`.
The full list of formats accepted by the `--crs` option is [documented here](https://pyproj4.github.io/pyproj/stable/api/crs.html#pyproj.crs.CRS.__init__).
## Extracting columns
If your data contains columns with a small number of heavily duplicated values - the names of specific agencies responsible for parcels of land for example - you can extract those columns into separate lookup tables referenced by foreign keys using the `-c` option:
$ shapefile-to-sqlite my.db features.shp -c agency
This will create a `agency` table with `id` and `name` columns, and will create the `agency` column in your main table as an integer foreign key reference to that table.
The `-c` option can be used multiple times.
[CPAD_2020a_Units](https://calands.datasettes.com/calands/CPAD_2020a_Units) is an example of a table created using the `-c` option.
","
shapefile-to-sqlite
Load shapefiles into a SQLite (optionally SpatiaLite) database.
You can omit --spatialite if you use either --spatialite-mod or --spatial-index.
Projections
By default, this tool will attempt to convert geometries in the shapefile to the WGS 84 projection, for best conformance with the GeoJSON specification.
If you want it to leave the data in whatever projection was used by the shapefile, use the --crs=keep option.
You can convert the data to another output projection by passing it to the --crs option. For example, to convert to EPSG:2227 (California zone 3) use --crs=espg:2227.
The full list of formats accepted by the --crs option is documented here.
Extracting columns
If your data contains columns with a small number of heavily duplicated values - the names of specific agencies responsible for parcels of land for example - you can extract those columns into separate lookup tables referenced by foreign keys using the -c option:
This will create a agency table with id and name columns, and will create the agency column in your main table as an integer foreign key reference to that table.
The -c option can be used multiple times.
CPAD_2020a_Units is an example of a table created using the -c option.
",,,,,,
242260583,MDEwOlJlcG9zaXRvcnkyNDIyNjA1ODM=,datasette-mask-columns,simonw/datasette-mask-columns,0,9599,https://github.com/simonw/datasette-mask-columns,Datasette plugin that masks specified database columns,0,2020-02-22T01:29:16Z,2021-06-10T19:50:37Z,2021-06-10T19:51:02Z,https://datasette.io/plugins/datasette-mask-columns,15,2,2,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,2,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-mask-columns
[](https://pypi.org/project/datasette-mask-columns/)
[](https://github.com/simonw/datasette-mask-columns/releases)
[](https://github.com/simonw/datasette-mask-columns/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-mask-columns/blob/main/LICENSE)
Datasette plugin that masks specified database columns
## Installation
pip install datasette-mask-columns
This depends on plugin hook changes in a not-yet released branch of Datasette. See [issue #678](https://github.com/simonw/datasette/issues/678) for details.
## Usage
In your `metadata.json` file add a section like this describing the database and table in which you wish to mask columns:
```json
{
""databases"": {
""my-database"": {
""plugins"": {
""datasette-mask-columns"": {
""users"": [""password""]
}
}
}
}
}
```
All SQL queries against the `users` table in `my-database.db` will now return `null` for the `password` column, no matter what value that column actually holds.
The table page for `users` will display the text `REDACTED` in the masked column. This visual hint will only be available on the table page; it will not display his text for arbitrary queries against the table.
","
datasette-mask-columns
Datasette plugin that masks specified database columns
Installation
pip install datasette-mask-columns
This depends on plugin hook changes in a not-yet released branch of Datasette. See issue #678 for details.
Usage
In your metadata.json file add a section like this describing the database and table in which you wish to mask columns:
All SQL queries against the users table in my-database.db will now return null for the password column, no matter what value that column actually holds.
The table page for users will display the text REDACTED in the masked column. This visual hint will only be available on the table page; it will not display his text for arbitrary queries against the table.
",,,,,,
243710733,MDEwOlJlcG9zaXRvcnkyNDM3MTA3MzM=,datasette-ics,simonw/datasette-ics,0,9599,https://github.com/simonw/datasette-ics,Datasette plugin for outputting iCalendar files,0,2020-02-28T08:11:01Z,2022-07-07T14:11:49Z,2022-07-12T02:08:10Z,https://datasette.io/plugins/datasette-ics,34,13,13,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""icalendar"", ""ics""]",0,0,13,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-ics
[](https://pypi.org/project/datasette-ics/)
[](https://github.com/simonw/datasette-ics/releases)
[](https://github.com/simonw/datasette-ics/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-ics/blob/main/LICENSE)
Datasette plugin that adds support for generating [iCalendar .ics files](https://tools.ietf.org/html/rfc5545) with the results of a SQL query.
## Installation
Install this plugin in the same environment as Datasette to enable the `.ics` output extension.
$ pip install datasette-ics
## Usage
To create an iCalendar file you need to define a custom SQL query that returns a required set of columns:
* `event_name` - the short name for the event
* `event_dtstart` - when the event starts
The following columns are optional:
* `event_dtend` - when the event ends
* `event_duration` - the duration of the event (use instead of `dtend`)
* `event_description` - a longer description of the event
* `event_uid` - a globally unique identifier for this event
* `event_tzid` - the timezone for the event, e.g. `America/Chicago`
A query that returns these columns can then be returned as an ics feed by adding the `.ics` extension.
## Demo
[This SQL query]([https://www.rockybeaches.com/data?sql=with+inner+as+(%0D%0A++select%0D%0A++++datetime%2C%0D%0A++++substr(datetime%2C+0%2C+11)+as+date%2C%0D%0A++++mllw_feet%2C%0D%0A++++lag(mllw_feet)+over+win+as+previous_mllw_feet%2C%0D%0A++++lead(mllw_feet)+over+win+as+next_mllw_feet%0D%0A++from%0D%0A++++tide_predictions%0D%0A++where%0D%0A++++station_id+%3D+%3Astation_id%0D%0A++++and+datetime+%3E%3D+date()%0D%0A++++window+win+as+(%0D%0A++++++order+by%0D%0A++++++++datetime%0D%0A++++)%0D%0A++order+by%0D%0A++++datetime%0D%0A)%2C%0D%0Alowest_tide_per_day+as+(%0D%0A++select%0D%0A++++date%2C%0D%0A++++datetime%2C%0D%0A++++mllw_feet%0D%0A++from%0D%0A++++inner%0D%0A++where%0D%0A++++mllw_feet+%3C%3D+previous_mllw_feet%0D%0A++++and+mllw_feet+%3C%3D+next_mllw_feet%0D%0A)%0D%0Aselect%0D%0A++min(datetime)+as+event_dtstart%2C%0D%0A++%27Low+tide%3A+%27+||+mllw_feet+||+%27+feet%27+as+event_name%2C%0D%0A++%27America%2FLos_Angeles%27+as+event_tzid%0D%0Afrom%0D%0A++lowest_tide_per_day%0D%0Agroup+by%0D%0A++date%0D%0Aorder+by%0D%0A++date&station_id=9414131) calculates the lowest tide per day at Pillar Point in Half Moon Bay, California.
Since the query returns `event_name`, `event_dtstart` and `event_tzid` columns it produces [this ICS feed](https://www.rockybeaches.com/data.ics?sql=with+inner+as+(%0D%0A++select%0D%0A++++datetime%2C%0D%0A++++substr(datetime%2C+0%2C+11)+as+date%2C%0D%0A++++mllw_feet%2C%0D%0A++++lag(mllw_feet)+over+win+as+previous_mllw_feet%2C%0D%0A++++lead(mllw_feet)+over+win+as+next_mllw_feet%0D%0A++from%0D%0A++++tide_predictions%0D%0A++where%0D%0A++++station_id+%3D+%3Astation_id%0D%0A++++and+datetime+%3E%3D+date()%0D%0A++++window+win+as+(%0D%0A++++++order+by%0D%0A++++++++datetime%0D%0A++++)%0D%0A++order+by%0D%0A++++datetime%0D%0A)%2C%0D%0Alowest_tide_per_day+as+(%0D%0A++select%0D%0A++++date%2C%0D%0A++++datetime%2C%0D%0A++++mllw_feet%0D%0A++from%0D%0A++++inner%0D%0A++where%0D%0A++++mllw_feet+%3C%3D+previous_mllw_feet%0D%0A++++and+mllw_feet+%3C%3D+next_mllw_feet%0D%0A)%0D%0Aselect%0D%0A++min(datetime)+as+event_dtstart%2C%0D%0A++%27Low+tide%3A+%27+||+mllw_feet+||+%27+feet%27+as+event_name%2C%0D%0A++%27America%2FLos_Angeles%27+as+event_tzid%0D%0Afrom%0D%0A++lowest_tide_per_day%0D%0Agroup+by%0D%0A++date%0D%0Aorder+by%0D%0A++date&station_id=9414131). If you subscribe to that in a calendar application such as Apple Calendar you get something that looks like this:
## Using a canned query
Datasette's [canned query mechanism](https://datasette.readthedocs.io/en/stable/sql_queries.html#canned-queries) can be used to configure calendars. If a canned query definition has a `title` that will be used as the title of the calendar.
Here's an example, defined using a `metadata.yaml` file:
```yaml
databases:
mydatabase:
queries:
calendar:
title: My Calendar
sql: |-
select
title as event_name,
start as event_dtstart,
description as event_description
from
events
order by
start
limit
100
```
This will result in a calendar feed at `http://localhost:8001/mydatabase/calendar.ics`
","
datasette-ics
Datasette plugin that adds support for generating iCalendar .ics files with the results of a SQL query.
Installation
Install this plugin in the same environment as Datasette to enable the .ics output extension.
$ pip install datasette-ics
Usage
To create an iCalendar file you need to define a custom SQL query that returns a required set of columns:
event_name - the short name for the event
event_dtstart - when the event starts
The following columns are optional:
event_dtend - when the event ends
event_duration - the duration of the event (use instead of dtend)
event_description - a longer description of the event
event_uid - a globally unique identifier for this event
event_tzid - the timezone for the event, e.g. America/Chicago
A query that returns these columns can then be returned as an ics feed by adding the .ics extension.
Demo
This SQL query calculates the lowest tide per day at Pillar Point in Half Moon Bay, California.
Since the query returns event_name, event_dtstart and event_tzid columns it produces this ICS feed. If you subscribe to that in a calendar application such as Apple Calendar you get something that looks like this:
Using a canned query
Datasette's canned query mechanism can be used to configure calendars. If a canned query definition has a title that will be used as the title of the calendar.
Here's an example, defined using a metadata.yaml file:
databases:
mydatabase:
queries:
calendar:
title: My Calendarsql: |- select title as event_name, start as event_dtstart, description as event_description from events order by start limit 100
This will result in a calendar feed at http://localhost:8001/mydatabase/calendar.ics
",1,public,0,,0,
243887036,MDEwOlJlcG9zaXRvcnkyNDM4ODcwMzY=,datasette-configure-fts,simonw/datasette-configure-fts,0,9599,https://github.com/simonw/datasette-configure-fts,Datasette plugin for enabling full-text search against selected table columns,0,2020-02-29T01:50:57Z,2020-11-01T02:59:12Z,2020-11-01T02:59:10Z,,42,2,2,Python,1,1,1,1,0,0,0,0,2,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,2,2,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-configure-fts
[](https://pypi.org/project/datasette-configure-fts/)
[](https://github.com/simonw/datasette-configure-fts/releases)
[](https://github.com/simonw/datasette-configure-fts/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-configure-fts/blob/main/LICENSE)
Datasette plugin for enabling full-text search against selected table columns
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-configure-fts
## Usage
Having installed the plugin, visit `/-/configure-fts` on your Datasette instance to configure FTS for tables on attached writable databases.
Any time you have permission to configure FTS for a table a menu item will appear in the table actions menu on the table page.
By default only [the root actor](https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor) can access the page - so you'll need to run Datasette with the `--root` option and click on the link shown in the terminal to sign in and access the page.
The `configure-fts` permission governs access. You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to the write interface.
","
datasette-configure-fts
Datasette plugin for enabling full-text search against selected table columns
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-configure-fts
Usage
Having installed the plugin, visit /-/configure-fts on your Datasette instance to configure FTS for tables on attached writable databases.
Any time you have permission to configure FTS for a table a menu item will appear in the table actions menu on the table page.
By default only the root actor can access the page - so you'll need to run Datasette with the --root option and click on the link shown in the terminal to sign in and access the page.
The configure-fts permission governs access. You can use permission plugins such as datasette-permissions-sql to grant additional access to the write interface.
",,,,,,
245670670,MDEwOlJlcG9zaXRvcnkyNDU2NzA2NzA=,fec-to-sqlite,simonw/fec-to-sqlite,0,9599,https://github.com/simonw/fec-to-sqlite,Save FEC campaign finance data to a SQLite database,0,2020-03-07T16:52:49Z,2020-12-19T05:09:05Z,2020-03-07T18:21:48Z,,16,8,8,Python,1,1,1,1,0,0,0,0,1,apache-2.0,"[""sqlite"", ""fec"", ""datasette"", ""datasette-io"", ""datasette-tool""]",0,1,8,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,2,"# fec-to-sqlite
[](https://pypi.org/project/fec-to-sqlite/)
[](https://circleci.com/gh/simonw/fec-to-sqlite)
[](https://github.com/simonw/fec-to-sqlite/blob/master/LICENSE)
Create a SQLite database using FEC campaign contributions data.
This tool builds on [fecfile](https://github.com/esonderegger/) by Evan Sonderegger.
## How to install
$ pip install fec-to-sqlite
## Usage
$ fec-to-sqlite filings filings.db 1146148
This fetches the filing with ID `1146148` and stores it in tables in a SQLite database called `filings.db`. It will create any tables it needs.
You can pass more than one filing ID, separated by spaces.
","
fec-to-sqlite
Create a SQLite database using FEC campaign contributions data.
This fetches the filing with ID 1146148 and stores it in tables in a SQLite database called filings.db. It will create any tables it needs.
You can pass more than one filing ID, separated by spaces.
",,,,,,
245856731,MDEwOlJlcG9zaXRvcnkyNDU4NTY3MzE=,datasette-search-all,simonw/datasette-search-all,0,9599,https://github.com/simonw/datasette-search-all,Datasette plugin for searching all searchable tables at once,0,2020-03-08T17:21:54Z,2021-12-19T04:06:49Z,2022-10-05T01:53:33Z,,186,6,6,Python,1,1,1,1,0,2,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""search""]",2,0,6,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,2,2,"# datasette-search-all
[](https://pypi.org/project/datasette-search-all/)
[](https://github.com/simonw/datasette-search-all/releases)
[](https://github.com/simonw/datasette-search-all/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-search-all/blob/main/LICENSE)
Datasette plugin for searching all searchable tables at once.
## Installation
Install the plugin in the same Python environment as Datasette:
pip install datasette-search-all
## Background
See [datasette-search-all: a new plugin for searching multiple Datasette tables at once](https://simonwillison.net/2020/Mar/9/datasette-search-all/) for background on this project. You can try the plugin out at https://fara.datasettes.com/
## Usage
This plugin only works if at least one of the tables connected to your Datasette instance has been configured for SQLite's full-text search.
The [Datasette search documentation](https://docs.datasette.io/en/stable/full_text_search.html) includes details on how to enable full-text search for a table.
You can also use the following tools:
* [sqlite-utils](https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search) includes a command-line tool for enabling full-text search.
* [datasette-enable-fts](https://github.com/simonw/datasette-enable-fts) is a Datasette plugin that adds a web interface for enabling search for specific columns.
If the plugin detects at least one searchable table it will add a search form to the homepage.
You can also navigate to `/-/search` on your Datasette instance to use the search interface directly.
## Screenshot
","
datasette-search-all
Datasette plugin for searching all searchable tables at once.
Installation
Install the plugin in the same Python environment as Datasette:
sqlite-utils includes a command-line tool for enabling full-text search.
datasette-enable-fts is a Datasette plugin that adds a web interface for enabling search for specific columns.
If the plugin detects at least one searchable table it will add a search form to the homepage.
You can also navigate to /-/search on your Datasette instance to use the search interface directly.
Screenshot
",1,public,0,,0,
246108561,MDEwOlJlcG9zaXRvcnkyNDYxMDg1NjE=,datasette-column-inspect,simonw/datasette-column-inspect,0,9599,https://github.com/simonw/datasette-column-inspect,Experimental plugin that adds a column inspector,0,2020-03-09T18:11:00Z,2020-12-09T21:46:10Z,2020-12-09T21:47:38Z,,15,1,1,HTML,1,1,1,1,0,0,0,0,3,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,3,1,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-column-inspect
[](https://pypi.org/project/datasette-column-inspect/)
[](https://github.com/simonw/datasette-column-inspect/releases)
[](https://github.com/simonw/datasette-column-inspect/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-column-inspect/blob/main/LICENSE)
Highly experimental Datasette plugin for inspecting columns.
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-column-inspect
## Usage
This plugin adds an icon to each column on the table page which opens an inspection side panel.
","
datasette-column-inspect
Highly experimental Datasette plugin for inspecting columns.
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-column-inspect
Usage
This plugin adds an icon to each column on the table page which opens an inspection side panel.
",,,,,,
247527438,MDEwOlJlcG9zaXRvcnkyNDc1Mjc0Mzg=,datasette-edit-schema,simonw/datasette-edit-schema,0,9599,https://github.com/simonw/datasette-edit-schema,Datasette plugin for modifying table schemas,0,2020-03-15T18:34:06Z,2022-07-01T22:20:25Z,2022-08-22T22:45:58Z,,133,6,6,JavaScript,1,1,1,1,0,0,0,0,10,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin""]",0,10,6,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-edit-schema
[](https://pypi.org/project/datasette-edit-schema/)
[](https://github.com/simonw/datasette-edit-schema/releases)
[](https://github.com/simonw/datasette-edit-schema/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-edit-schema/blob/master/LICENSE)
Datasette plugin for modifying table schemas
## Features
* Add new columns to a table
* Rename columns in a table
* Modify the type of columns in a table
* Re-order the columns in a table
* Rename a table
* Delete a table
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-edit-schema
## Usage
Navigate to `/-/edit-schema/dbname/tablename` on your Datasette instance to edit a specific table.
Use `/-/edit-schema/dbname` to create a new table in a specific database.
By default only [the root actor](https://datasette.readthedocs.io/en/stable/authentication.html#using-the-root-actor) can access the page - so you'll need to run Datasette with the `--root` option and click on the link shown in the terminal to sign in and access the page.
## Permissions
The `edit-schema` permission governs access. You can use permission plugins such as [datasette-permissions-sql](https://github.com/simonw/datasette-permissions-sql) to grant additional access to the write interface.
These permission checks will call the `permission_allowed()` plugin hook with three arguments:
- `action` will be the string `""edit-schema""`
- `actor` will be the currently authenticated actor - usually a dictionary
- `resource` will be the string name of the database
## Screenshot
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-edit-schema
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-edit-schema
Datasette plugin for modifying table schemas
Features
Add new columns to a table
Rename columns in a table
Modify the type of columns in a table
Re-order the columns in a table
Rename a table
Delete a table
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-edit-schema
Usage
Navigate to /-/edit-schema/dbname/tablename on your Datasette instance to edit a specific table.
Use /-/edit-schema/dbname to create a new table in a specific database.
By default only the root actor can access the page - so you'll need to run Datasette with the --root option and click on the link shown in the terminal to sign in and access the page.
Permissions
The edit-schema permission governs access. You can use permission plugins such as datasette-permissions-sql to grant additional access to the write interface.
These permission checks will call the permission_allowed() plugin hook with three arguments:
action will be the string ""edit-schema""
actor will be the currently authenticated actor - usually a dictionary
resource will be the string name of the database
Screenshot
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-edit-schema
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,0,
248385299,MDEwOlJlcG9zaXRvcnkyNDgzODUyOTk=,datasette-publish-fly,simonw/datasette-publish-fly,0,9599,https://github.com/simonw/datasette-publish-fly,Datasette plugin for publishing data using Fly,0,2020-03-19T01:47:01Z,2022-09-29T22:28:45Z,2022-09-29T17:25:15Z,,50,10,10,Python,1,1,1,1,0,3,0,0,4,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""fly""]",3,4,10,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,3,3,"# datasette-publish-fly
[](https://pypi.org/project/datasette-publish-fly/)
[](https://github.com/simonw/datasette-publish-fly/releases)
[](https://github.com/simonw/datasette-publish-fly/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-publish-fly/blob/main/LICENSE)
[Datasette](https://datasette.io/) plugin for deploying Datasette instances to [Fly.io](https://fly.io/).
Project background: [Using SQLite and Datasette with Fly Volumes](https://simonwillison.net/2022/Feb/15/fly-volumes/)
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-publish-fly
## Deploying read-only data
First, install the `flyctl` command-line tool by [following their instructions](https://fly.io/docs/getting-started/installing-flyctl/).
Run `flyctl auth signup` to create an account there, or `flyctl auth login` if you already have one.
You can now use `datasette publish fly` to publish one or more SQLite database files:
datasette publish fly my-database.db --app=""my-data-app""
The argument you pass to `--app` will be used for the URL of your application: `my-data-app.fly.dev`.
To update an application, run the publish command passing the same application name to the `--app` option.
Fly have [a free tier](https://fly.io/docs/about/pricing/#free-allowances), beyond which they will charge you monthly for each application you have live. Details of their pricing can be [found on their site](https://fly.io/docs/pricing/).
Your application will be deployed at `https://your-app-name.fly.io/` - be aware that it may take several minutes to start working the first time you deploy it.
## Using Fly volumes for writable databases
Fly [Volumes](https://fly.io/docs/reference/volumes/) provide persistant disk storage for Fly applications. Volumes can be 1GB or more in size and the Fly free tier includes 3GB of volume space.
Datasette plugins such as [datasette-uploads-csvs](https://datasette.io/plugins/datasette-upload-csvs) and [datasette-tiddlywiki](https://datasette.io/plugins/datasette-tiddlywiki) can be deployed to Fly and store their mutable data in a volume.
> :warning: **You should only run a single instance of your application** if your database accepts writes. Fly has excellent support for running multiple instances in different geographical regions, but `datasette-publish-fly` with volumes is not yet compatible with that model. You should probably [use Fly PostgreSQL instead](https://fly.io/blog/globally-distributed-postgres/).
Here's how to deploy `datasette-tiddlywiki` with authentication provided by `datasette-auth-passwords`.
First, you'll need to create a root password hash to use to sign into the instance.
You can do that by installing the plugin and running the `datasette hash-password` command, or by using [this hosted tool](https://datasette-auth-passwords-demo.datasette.io/-/password-tool).
The hash should look like `pbkdf2_sha256$...` - you'll need this for the next step.
In this example we're also deploying a read-only database called `content.db`.
Pick a name for your new application, then run the following:
datasette publish fly \
content.db \
--app your-application-name \
--create-volume 1 \
--create-db tiddlywiki \
--install datasette-auth-passwords \
--install datasette-tiddlywiki \
--plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$...'
This will create the new application, deploy the `content.db` read-only database, create a 1GB volume for that application, create a new database in that volume called `tiddlywiki.db`, then install the two plugins and configure the password you specified.
### Updating applications that use a volume
Once you have deployed an application using a volume, you can update that application without needing the `--create-volume` or `--create-db` options. To add the [datasette-graphq](https://datasette.io/plugins/datasette-graphql) plugin to your deployed application you would run the following:
datasette publish fly \
content.db \
--app your-application-name \
--install datasette-auth-passwords \
--install datasette-tiddlywiki \
--install datasette-graphql \
--plugin-secret datasette-auth-passwords root_password_hash 'pbkdf2_sha256$...' \
Since the application name is the same you don't need the `--create-volume` or `--create-db` options - these are persisted automatically between deploys.
You do need to specify the full list of plugins that you want to have installed, and any plugin secrets.
You also need to include any read-only database files that are part of the instance - `content.db` in this example - otherwise the new deployment will not include them.
### Advanced volume usage
`datasette publish fly` will add a volume called `datasette` to your Fly application. You can customize the name using the `--volume name custom_name` option.
Fly can be used to scale applications to run multiple instances in multiple regions around the world. This works well with read-only Datasette but is not currently recommended using Datasette with volumes, since each Fly replica would need its own volume and data stored in one instance would not be visible in others.
If you want to use multiple instances with volumes you will need to switch to using the `flyctl` command directly. The `--generate-dir` option, described below, can help with this.
## Generating without deploying
Use the `--generate-dir` option to generate a directory that can be deployed to Fly rather than deploying directly:
datasette publish fly my-database.db \
--app=""my-generated-app"" \
--generate-dir /tmp/deploy-this
You can then manually deploy your generated application using the following:
cd /tmp/deploy-this
flyctl apps create my-generated-app
flyctl deploy
## datasette publish fly --help
```
Usage: datasette publish fly [OPTIONS] [FILES]...
Deploy an application to Fly that runs Datasette against the provided database
files.
Usage example:
datasette publish fly my-database.db --app=""my-data-app""
Full documentation: https://datasette.io/plugins/datasette-publish-fly
Options:
-m, --metadata FILENAME Path to JSON/YAML file containing metadata to
publish
--extra-options TEXT Extra options to pass to datasette serve
--branch TEXT Install datasette from a GitHub branch e.g.
main
--template-dir DIRECTORY Path to directory containing custom templates
--plugins-dir DIRECTORY Path to directory containing custom plugins
--static MOUNT:DIRECTORY Serve static files from this directory at
/MOUNT/...
--install TEXT Additional packages (e.g. plugins) to install
--plugin-secret ...
Secrets to pass to plugins, e.g. --plugin-
secret datasette-auth-github client_id xxx
--version-note TEXT Additional note to show on /-/versions
--secret TEXT Secret used for signing secure values, such as
signed cookies
--title TEXT Title for metadata
--license TEXT License label for metadata
--license_url TEXT License URL for metadata
--source TEXT Source label for metadata
--source_url TEXT Source URL for metadata
--about TEXT About label for metadata
--about_url TEXT About URL for metadata
--spatialite Enable SpatialLite extension
--region TEXT Fly region to deploy to, e.g sjc - see
https://fly.io/docs/reference/regions/
--create-volume INTEGER RANGE Create and attach volume of this size in GB
[x>=1]
--create-db TEXT Names of read-write database files to create
--volume-name TEXT Volume name to use
-a, --app TEXT Name of Fly app to deploy [required]
-o, --org TEXT Name of Fly org to deploy to
--generate-dir DIRECTORY Output generated application files and stop
without deploying
--show-files Output the generated Dockerfile, metadata.json
and fly.toml
--help Show this message and exit.
```
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd datasette-publish-fly
python -m venv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
### Integration tests
The tests in `tests/test_integration.py` make actual calls to Fly to deploy a test application.
These tests are skipped by default. If you have `flyctl` installed and configured, you can run the integration tests like this:
pytest --integration -s
The `-s` option here ensures that output from the deploys will be visible to you - otherwise it can look like the tests have hung.
The tests will create applications on Fly that start with the prefix `publish-fly-temp-` and then delete them at the end of the run.
","
datasette-publish-fly
Datasette plugin for deploying Datasette instances to Fly.io.
The argument you pass to --app will be used for the URL of your application: my-data-app.fly.dev.
To update an application, run the publish command passing the same application name to the --app option.
Fly have a free tier, beyond which they will charge you monthly for each application you have live. Details of their pricing can be found on their site.
Your application will be deployed at https://your-app-name.fly.io/ - be aware that it may take several minutes to start working the first time you deploy it.
Using Fly volumes for writable databases
Fly Volumes provide persistant disk storage for Fly applications. Volumes can be 1GB or more in size and the Fly free tier includes 3GB of volume space.
⚠️You should only run a single instance of your application if your database accepts writes. Fly has excellent support for running multiple instances in different geographical regions, but datasette-publish-fly with volumes is not yet compatible with that model. You should probably use Fly PostgreSQL instead.
Here's how to deploy datasette-tiddlywiki with authentication provided by datasette-auth-passwords.
First, you'll need to create a root password hash to use to sign into the instance.
You can do that by installing the plugin and running the datasette hash-password command, or by using this hosted tool.
The hash should look like pbkdf2_sha256$... - you'll need this for the next step.
In this example we're also deploying a read-only database called content.db.
Pick a name for your new application, then run the following:
This will create the new application, deploy the content.db read-only database, create a 1GB volume for that application, create a new database in that volume called tiddlywiki.db, then install the two plugins and configure the password you specified.
Updating applications that use a volume
Once you have deployed an application using a volume, you can update that application without needing the --create-volume or --create-db options. To add the datasette-graphq plugin to your deployed application you would run the following:
Since the application name is the same you don't need the --create-volume or --create-db options - these are persisted automatically between deploys.
You do need to specify the full list of plugins that you want to have installed, and any plugin secrets.
You also need to include any read-only database files that are part of the instance - content.db in this example - otherwise the new deployment will not include them.
Advanced volume usage
datasette publish fly will add a volume called datasette to your Fly application. You can customize the name using the --volume name custom_name option.
Fly can be used to scale applications to run multiple instances in multiple regions around the world. This works well with read-only Datasette but is not currently recommended using Datasette with volumes, since each Fly replica would need its own volume and data stored in one instance would not be visible in others.
If you want to use multiple instances with volumes you will need to switch to using the flyctl command directly. The --generate-dir option, described below, can help with this.
Generating without deploying
Use the --generate-dir option to generate a directory that can be deployed to Fly rather than deploying directly:
You can then manually deploy your generated application using the following:
cd /tmp/deploy-this
flyctl apps create my-generated-app
flyctl deploy
datasette publish fly --help
Usage: datasette publish fly [OPTIONS] [FILES]...
Deploy an application to Fly that runs Datasette against the provided database
files.
Usage example:
datasette publish fly my-database.db --app=""my-data-app""
Full documentation: https://datasette.io/plugins/datasette-publish-fly
Options:
-m, --metadata FILENAME Path to JSON/YAML file containing metadata to
publish
--extra-options TEXT Extra options to pass to datasette serve
--branch TEXT Install datasette from a GitHub branch e.g.
main
--template-dir DIRECTORY Path to directory containing custom templates
--plugins-dir DIRECTORY Path to directory containing custom plugins
--static MOUNT:DIRECTORY Serve static files from this directory at
/MOUNT/...
--install TEXT Additional packages (e.g. plugins) to install
--plugin-secret <TEXT TEXT TEXT>...
Secrets to pass to plugins, e.g. --plugin-
secret datasette-auth-github client_id xxx
--version-note TEXT Additional note to show on /-/versions
--secret TEXT Secret used for signing secure values, such as
signed cookies
--title TEXT Title for metadata
--license TEXT License label for metadata
--license_url TEXT License URL for metadata
--source TEXT Source label for metadata
--source_url TEXT Source URL for metadata
--about TEXT About label for metadata
--about_url TEXT About URL for metadata
--spatialite Enable SpatialLite extension
--region TEXT Fly region to deploy to, e.g sjc - see
https://fly.io/docs/reference/regions/
--create-volume INTEGER RANGE Create and attach volume of this size in GB
[x>=1]
--create-db TEXT Names of read-write database files to create
--volume-name TEXT Volume name to use
-a, --app TEXT Name of Fly app to deploy [required]
-o, --org TEXT Name of Fly org to deploy to
--generate-dir DIRECTORY Output generated application files and stop
without deploying
--show-files Output the generated Dockerfile, metadata.json
and fly.toml
--help Show this message and exit.
Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd datasette-publish-fly
python -m venv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
Integration tests
The tests in tests/test_integration.py make actual calls to Fly to deploy a test application.
These tests are skipped by default. If you have flyctl installed and configured, you can run the integration tests like this:
pytest --integration -s
The -s option here ensures that output from the deploys will be visible to you - otherwise it can look like the tests have hung.
The tests will create applications on Fly that start with the prefix publish-fly-temp- and then delete them at the end of the run.
",1,public,0,,0,
248903544,MDEwOlJlcG9zaXRvcnkyNDg5MDM1NDQ=,hacker-news-to-sqlite,dogsheep/hacker-news-to-sqlite,0,53015001,https://github.com/dogsheep/hacker-news-to-sqlite,Create a SQLite database containing data pulled from Hacker News,0,2020-03-21T04:02:05Z,2021-06-06T22:42:00Z,2021-03-13T19:15:06Z,,19,25,25,Python,1,1,1,1,0,2,0,0,0,apache-2.0,"[""hacker-news"", ""datasette"", ""dogsheep"", ""datasette-io"", ""datasette-tool""]",2,0,25,main,"{""admin"": false, ""push"": false, ""pull"": false}",,53015001,2,1,"# hacker-news-to-sqlite
[](https://pypi.org/project/hacker-news-to-sqlite/)
[](https://github.com/dogsheep/hacker-news-to-sqlite/releases)
[](https://github.com/dogsheep/hacker-news-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/hacker-news-to-sqlite/blob/main/LICENSE)
Create a SQLite database containing data fetched from [Hacker News](https://news.ycombinator.com/).
## How to install
$ pip install hacker-news-to-sqlite
## Usage
$ hacker-news-to-sqlite user hacker-news.db your-username
Importing items: 37%|███████████ | 845/2297 [05:09<11:02, 2.19it/s]
Imports all of your Hacker News submissions and comments into a SQLite database called `hacker-news.db`.
$ hacker-news-to-sqlite trees hacker-news.db 22640038 22643218
Fetches the entire comments tree in which any of those content IDs appears.
## Browsing your data with Datasette
You can use [Datasette](https://datasette.readthedocs.org/) to browse your data. Install Datasette like this:
$ pip install datasette
Now run it against your `hacker-news.db` file like so:
$ datasette hacker-news.db
Visit `https://localhost:8001/` to search and explore your data.
You can improve the display of your data usinng the [datasette-render-timestamps](https://github.com/simonw/datasette-render-timestamps) and [datasette-render-html](https://github.com/simonw/datasette-render-html) plugins. Install them like this:
$ pip install datasette-render-timestamps datasette-render-html
Now save the following configuration in a file called `metadata.json`:
```json
{
""databases"": {
""hacker-news"": {
""tables"": {
""items"": {
""plugins"": {
""datasette-render-html"": {
""columns"": [
""text""
]
},
""datasette-render-timestamps"": {
""columns"": [
""time""
]
}
}
},
""users"": {
""plugins"": {
""datasette-render-timestamps"": {
""columns"": [
""created""
]
}
}
}
}
}
}
}
```
Run Datasette like this:
$ datasette -m metadata.json hacker-news.db
The timestamp columns will now be rendered as human-readable dates, and any HTML in your posts will be displayed as rendered HTML.
","
hacker-news-to-sqlite
Create a SQLite database containing data fetched from Hacker News.
The timestamp columns will now be rendered as human-readable dates, and any HTML in your posts will be displayed as rendered HTML.
",,,,,,
248999994,MDEwOlJlcG9zaXRvcnkyNDg5OTk5OTQ=,datasette-show-errors,simonw/datasette-show-errors,0,9599,https://github.com/simonw/datasette-show-errors,Datasette plugin for displaying error tracebacks,0,2020-03-21T15:06:04Z,2020-09-24T00:17:29Z,2020-09-01T00:32:23Z,,7,1,1,Python,1,1,1,1,0,0,0,0,1,apache-2.0,"[""asgi"", ""datasette"", ""starlette"", ""datasette-plugin"", ""datasette-io""]",0,1,1,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,0,"# datasette-show-errors
[](https://pypi.org/project/datasette-show-errors/)
[](https://circleci.com/gh/simonw/datasette-show-errors)
[](https://github.com/simonw/datasette-show-errors/blob/master/LICENSE)
Datasette plugin for displaying error tracebacks.
**This plugin does not work with current versions of Datasette.** See [issue #2](https://github.com/simonw/datasette-show-errors/issues/2).
## Installation
pip install datasette-show-errors
## Usage
Installing the plugin will cause any internal error to be displayed with a full traceback, rather than just a generic 500 page.
Be careful not to use this in a context that might expose sensitive information.
","
datasette-show-errors
Datasette plugin for displaying error tracebacks.
This plugin does not work with current versions of Datasette. See issue #2.
Installation
pip install datasette-show-errors
Usage
Installing the plugin will cause any internal error to be displayed with a full traceback, rather than just a generic 500 page.
Be careful not to use this in a context that might expose sensitive information.
",,,,,,
253632948,MDEwOlJlcG9zaXRvcnkyNTM2MzI5NDg=,datasette-publish-vercel,simonw/datasette-publish-vercel,0,9599,https://github.com/simonw/datasette-publish-vercel,Datasette plugin for publishing data using Vercel,0,2020-04-06T22:47:13Z,2022-07-29T17:09:47Z,2022-08-24T17:43:41Z,,55,27,27,Python,1,1,1,1,0,5,0,0,17,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""vercel"", ""zeit-now""]",5,17,27,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,5,4,"# datasette-publish-vercel
[](https://pypi.org/project/datasette-publish-vercel/)
[](https://github.com/simonw/datasette-publish-vercel/releases)
[](https://github.com/simonw/datasette-publish-vercel/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-publish-vercel/blob/main/LICENSE)
Datasette plugin for publishing data using [Vercel](https://vercel.com/).
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-publish-vercel
## Usage
First, install the Vercel CLI tool by [following their instructions](https://vercel.com/download).
Run `vercel login` to login to (or create) an account.
Now you can use `datasette publish vercel` to publish your data:
datasette publish vercel my-database.db --project=my-database
The `--project` argument is required - it specifies the project name that should be used for your deployment. This will be used as part of the deployment's URL.
### Other options
* `--no-prod` deploys to the project without updating the ""production"" URL alias to point to that new deployment. Without that option all deploys go directly to production.
* `--debug` enables the Vercel CLI debug output.
* `--token` allows you to pass a Now authentication token, rather than needing to first run `now login` to configure the tool. Tokens can be created in the Vercel web dashboard under Account Settings -> Tokens.
* `--public` runs `vercel --public` to publish the application source code at `/_src` e.g. https://datasette-public.now.sh/_src and make recent logs visible at `/_logs` e.g. https://datasette-public.now.sh/_logs
* `--generate-dir` - by default this tool generates a new Vercel app in a temporary directory, deploys it and then deletes the directory. Use `--generate-dir=my-app` to output the generated application files to a new directory of your choice instead. You can then deploy it by running `vercel` in that directory.
* `--setting default_page_size 10` - use this to set Datasette settings, as described in [the documentation](https://docs.datasette.io/en/stable/settings.html). This is a replacement for the unsupported `--extra-options` option.
### Full help
**Warning:** Some of these options are not yet implemented by this plugin. In particular, the following do not yet work:
* `--extra-options` - use `--setting` described above instead.
* `--plugin-secret`
* `--version-note`
```
$ datasette publish vercel --help
Usage: datasette publish vercel [OPTIONS] [FILES]...
Publish to https://vercel.com/
Options:
-m, --metadata FILENAME Path to JSON/YAML file containing metadata to publish
--extra-options TEXT Extra options to pass to datasette serve
--branch TEXT Install datasette from a GitHub branch e.g. main
--template-dir DIRECTORY Path to directory containing custom templates
--plugins-dir DIRECTORY Path to directory containing custom plugins
--static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/...
--install TEXT Additional packages (e.g. plugins) to install
--plugin-secret ...
Secrets to pass to plugins, e.g. --plugin-secret
datasette-auth-github client_id xxx
--version-note TEXT Additional note to show on /-/versions
--secret TEXT Secret used for signing secure values, such as signed
cookies
--title TEXT Title for metadata
--license TEXT License label for metadata
--license_url TEXT License URL for metadata
--source TEXT Source label for metadata
--source_url TEXT Source URL for metadata
--about TEXT About label for metadata
--about_url TEXT About URL for metadata
--token TEXT Auth token to use for deploy
--project PROJECT Vercel project name to use [required]
--scope TEXT Optional Vercel scope (e.g. a team name)
--no-prod Don't deploy directly to production
--debug Enable Vercel CLI debug output
--public Publish source with Vercel CLI --public
--generate-dir DIRECTORY Output generated application files and stop without
deploying
--generate-vercel-json Output generated vercel.json file and stop without
deploying
--vercel-json FILENAME Custom vercel.json file to use instead of generating
one
--setting SETTING... Setting, see docs.datasette.io/en/stable/settings.html
--crossdb Enable cross-database SQL queries
--help Show this message and exit.
```
## Using a custom `vercel.json` file
If you want to add additional redirects or similar to your Vercel configuration you may want to provide a custom `vercel.json` file.
To do this, first generate a configuration file (without running a deploy) using the `--generate-vercel-json` option:
datasette publish vercel my-database.db \
--project=my-database \
--generate-vercel-json > vercel.json
You can now edit the `vercel.json` file that this creates to add your custom options.
Then run the deploy using:
datasette publish vercel my-database.db \
--project=my-database \
--vercel-json=vercel.json
## Setting a `DATASETTE_SECRET`
Datasette uses [a secret string](https://docs.datasette.io/en/stable/settings.html#configuring-the-secret) for purposes such as signing authentication cookies. This secret is reset when the server restarts, which will sign out any users who are authenticated using a signed cookie.
You can avoid this by generating a `DATASETTE_SECRET` secret string and setting that as a [Vercel environment variable](https://vercel.com/docs/concepts/projects/environment-variables). If you do this the secret will stay consistent and your users will not be signed out.
## Using this with GitHub Actions
This plugin can be used together with [GitHub Actions](https://github.com/features/actions) to deploy Datasette instances automatically on new pushes to a repo, or on a schedule.
The GitHub Actions runners already have the Vercel deployment tool installed. You'll need to create an API token for your account at [vercel.com/account/tokens](https://vercel.com/account/tokens), and store that as a secret in your GitHub repository called `VERCEL_TOKEN`.
Make sure your workflow has installed `datasette` and `datasette-publish-vercel` using `pip`, then add the following step to your GitHub Actions workflow:
```
- name: Deploy Datasette using Vercel
env:
VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
run: |-
datasette publish vercel mydb.db \
--token $VERCEL_TOKEN \
--project my-vercel-project
```
You can see a full example of a workflow that uses Vercel in this way [in the simonw/til repository](https://github.com/simonw/til/blob/12b3f0d3679320cbeafa5df164bbc08ba703625d/.github/workflows/build.yml).
","
datasette-publish-vercel
Datasette plugin for publishing data using Vercel.
Installation
Install this plugin in the same environment as Datasette.
The --project argument is required - it specifies the project name that should be used for your deployment. This will be used as part of the deployment's URL.
Other options
--no-prod deploys to the project without updating the ""production"" URL alias to point to that new deployment. Without that option all deploys go directly to production.
--debug enables the Vercel CLI debug output.
--token allows you to pass a Now authentication token, rather than needing to first run now login to configure the tool. Tokens can be created in the Vercel web dashboard under Account Settings -> Tokens.
--generate-dir - by default this tool generates a new Vercel app in a temporary directory, deploys it and then deletes the directory. Use --generate-dir=my-app to output the generated application files to a new directory of your choice instead. You can then deploy it by running vercel in that directory.
--setting default_page_size 10 - use this to set Datasette settings, as described in the documentation. This is a replacement for the unsupported --extra-options option.
Full help
Warning: Some of these options are not yet implemented by this plugin. In particular, the following do not yet work:
--extra-options - use --setting described above instead.
--plugin-secret
--version-note
$ datasette publish vercel --help
Usage: datasette publish vercel [OPTIONS] [FILES]...
Publish to https://vercel.com/
Options:
-m, --metadata FILENAME Path to JSON/YAML file containing metadata to publish
--extra-options TEXT Extra options to pass to datasette serve
--branch TEXT Install datasette from a GitHub branch e.g. main
--template-dir DIRECTORY Path to directory containing custom templates
--plugins-dir DIRECTORY Path to directory containing custom plugins
--static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/...
--install TEXT Additional packages (e.g. plugins) to install
--plugin-secret <TEXT TEXT TEXT>...
Secrets to pass to plugins, e.g. --plugin-secret
datasette-auth-github client_id xxx
--version-note TEXT Additional note to show on /-/versions
--secret TEXT Secret used for signing secure values, such as signed
cookies
--title TEXT Title for metadata
--license TEXT License label for metadata
--license_url TEXT License URL for metadata
--source TEXT Source label for metadata
--source_url TEXT Source URL for metadata
--about TEXT About label for metadata
--about_url TEXT About URL for metadata
--token TEXT Auth token to use for deploy
--project PROJECT Vercel project name to use [required]
--scope TEXT Optional Vercel scope (e.g. a team name)
--no-prod Don't deploy directly to production
--debug Enable Vercel CLI debug output
--public Publish source with Vercel CLI --public
--generate-dir DIRECTORY Output generated application files and stop without
deploying
--generate-vercel-json Output generated vercel.json file and stop without
deploying
--vercel-json FILENAME Custom vercel.json file to use instead of generating
one
--setting SETTING... Setting, see docs.datasette.io/en/stable/settings.html
--crossdb Enable cross-database SQL queries
--help Show this message and exit.
Using a custom vercel.json file
If you want to add additional redirects or similar to your Vercel configuration you may want to provide a custom vercel.json file.
To do this, first generate a configuration file (without running a deploy) using the --generate-vercel-json option:
Datasette uses a secret string for purposes such as signing authentication cookies. This secret is reset when the server restarts, which will sign out any users who are authenticated using a signed cookie.
You can avoid this by generating a DATASETTE_SECRET secret string and setting that as a Vercel environment variable. If you do this the secret will stay consistent and your users will not be signed out.
Using this with GitHub Actions
This plugin can be used together with GitHub Actions to deploy Datasette instances automatically on new pushes to a repo, or on a schedule.
The GitHub Actions runners already have the Vercel deployment tool installed. You'll need to create an API token for your account at vercel.com/account/tokens, and store that as a secret in your GitHub repository called VERCEL_TOKEN.
Make sure your workflow has installed datasette and datasette-publish-vercel using pip, then add the following step to your GitHub Actions workflow:
",1,public,0,,0,
255460347,MDEwOlJlcG9zaXRvcnkyNTU0NjAzNDc=,datasette-clone,simonw/datasette-clone,0,9599,https://github.com/simonw/datasette-clone,Create a local copy of database files from a Datasette instance,0,2020-04-13T23:05:41Z,2021-06-08T15:33:21Z,2021-02-22T19:32:36Z,,20,2,2,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-tool""]",0,0,2,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-clone
[](https://pypi.org/project/datasette-clone/)
[](https://github.com/simonw/datasette-clone/releases)
[](https://github.com/simonw/datasette-clone/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-clone/blob/main/LICENSE)
Create a local copy of database files from a Datasette instance.
See [datasette-clone](https://simonwillison.net/2020/Apr/14/datasette-clone/) on my blog for background on this project.
## How to install
$ pip install datasette-clone
## Usage
This only works against Datasette instances running immutable databases (with the `-i` option). Databases published using the `datasette publish` command should be compatible with this tool.
To download copies of all `.db` files from an instance, run:
datasette-clone https://latest.datasette.io
You can provide an optional second argument to specify a directory:
datasette-clone https://latest.datasette.io /tmp/here-please
The command stores its own copy of a `databases.json` manifest and uses it to only download databases that have changed the next time you run the command.
It also stores a copy of the instance's `metadata.json` to ensure you have a copy of any source and licensing information for the downloaded databases.
If your instance is protected by an API token, you can use `--token` to provide it:
datasette-clone https://latest.datasette.io --token=xyz
For verbose output showing what the tool is doing, use `-v`.
","
datasette-clone
Create a local copy of database files from a Datasette instance.
See datasette-clone on my blog for background on this project.
How to install
$ pip install datasette-clone
Usage
This only works against Datasette instances running immutable databases (with the -i option). Databases published using the datasette publish command should be compatible with this tool.
To download copies of all .db files from an instance, run:
datasette-clone https://latest.datasette.io
You can provide an optional second argument to specify a directory:
The command stores its own copy of a databases.json manifest and uses it to only download databases that have changed the next time you run the command.
It also stores a copy of the instance's metadata.json to ensure you have a copy of any source and licensing information for the downloaded databases.
If your instance is protected by an API token, you can use --token to provide it:
For verbose output showing what the tool is doing, use -v.
",,,,,,
256834907,MDEwOlJlcG9zaXRvcnkyNTY4MzQ5MDc=,dogsheep-photos,dogsheep/dogsheep-photos,0,53015001,https://github.com/dogsheep/dogsheep-photos,Upload your photos to S3 and import metadata about them into a SQLite database,0,2020-04-18T19:22:13Z,2021-11-04T20:45:03Z,2021-11-04T20:45:00Z,,68,124,124,Python,1,1,1,1,0,7,0,0,19,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-tool"", ""dogsheep"", ""sqlite""]",7,19,124,master,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,53015001,7,10,"# dogsheep-photos
[](https://pypi.org/project/dogsheep-photos/)
[](https://github.com/dogsheep/dogsheep-photos/releases)
[](https://circleci.com/gh/dogsheep/dogsheep-photos)
[](https://github.com/dogsheep/dogsheep-photos/blob/master/LICENSE)
Save details of your photos to a SQLite database and upload them to S3.
See [Using SQL to find my best photo of a pelican according to Apple Photos](https://simonwillison.net/2020/May/21/apple-photos-sqlite/) for background information on this project.
## What these tools do
These tools are a work-in-progress mechanism for taking full ownership of your photos. The core idea is to help implement the following:
* Every photo you have taken lives in a single, private Amazon S3 bucket
* You have a single SQLite database file which stores metadata about those photos - potentially pulled from multiple different places. This may include EXIF data, Apple Photos, the results of running machine learning APIs against photos and much more besides.
* You can then use [Datasette](https://github.com/simonw/datasette) to explore your own photos.
I'm a heavy user of Apple Photos so the initial releases of this tool will have a bias towards that, but ideally I would like a subset of these tools to be useful to people no matter which core photo solution they are using.
## Installation
$ pip install dogsheep-photos
## Authentication (if using S3)
If you want to use S3 to store your photos, you will need to first create S3 credentials for a new, dedicated bucket.
You may find the [s3-credentials tool](https://github.com/simonw/s3-credentials) useful for this.
Run this command and paste in your credentials. You will need three values: the name of your S3 bucket, your Access key ID and your Secret access key.
$ dogsheep-photos s3-auth
This will create a file called `auth.json` in your current directory containing the required values. To save the file at a different path or filename, use the `--auth=myauth.json` option.
## Uploading photos
Run this command to upload every photo in a specific directory to your S3 bucket:
$ dogsheep-photos upload photos.db \
~/Pictures/Photos\ Library.photoslibrary/original
The command will only upload photos that have not yet been uploaded, based on their sha256 hash.
`photos.db` will be created with an `uploads` table containing details of which files were uploaded.
To see what the command would do without uploading any files, use the `--dry-run` option.
The sha256 hash of the photo contents will be used as the name of the file in the bucket, with an extension matching the type of file. This is an implementation of the [Content addressable storage](https://en.wikipedia.org/wiki/Content-addressable_storage) pattern.
## Importing Apple Photos metadata
The `apple-photos` command imports metadata from your Apple Photos library.
$ photo-to-sqlite apple-photos photos.db
Imported metadata includes places, people, albums, quality scores and machine learning labels for the photo contents.
## Creating a subset database
You can create a new, subset database of photos using the `create-subset` command.
This is useful for creating a shareable SQLite database that only contains metadata for a selected set of photos.
Since photo metadata contains latitude and longitude you may not want to share a database that includes photos taken at your home address.
`create-subset` takes three arguments: an existing database file created using the `apple-photos` command, the name of the new, shareable database file you would like to create and a SQL query that returns the `sha256` hash values of the photos you would like to include in that database.
For example, here's how to create a shareable database of just the photos that have been added to albums containing the word ""Public"":
$ dogsheep-photos create-subset \
photos.db \
public.db \
""select sha256 from apple_photos where albums like '%Public%'""
## Serving photos locally with datasette-media
If you don't want to upload your photos to S3 but you still want to browse them using Datasette you can do so using the [datasette-media](https://github.com/simonw/datasette-media) plugin. This plugin adds the ability to serve images and other static files directly from disk, configured using a SQL query.
To use it, first install Datasette and the plugin:
$ pip install datasette datasette-media
If any of your photos are `.HEIC` images taken by an iPhone you should also install the optional `pyheif` dependency:
$ pip install pyheif
Now create a `metadata.yaml` file configuring the plugin:
```yaml
plugins:
datasette-media:
thumbnail:
sql: |-
select path as filepath, 200 as resize_height from apple_photos where uuid = :key
large:
sql: |-
select path as filepath, 1024 as resize_height from apple_photos where uuid = :key
```
This will configure two URL endpoints - one for 200 pixel high thumbnails and one for 1024 pixel high larger images.
Create your `photos.db` database using the `apple-photos` command, then run Datasette like this:
$ datasette -m metadata.yaml
Your photos will be served on URLs that look like this:
http://127.0.0.1:8001/-/media/thumbnail/F4469918-13F3-43D8-9EC1-734C0E6B60AD
http://127.0.0.1:8001/-/media/large/F4469918-13F3-43D8-9EC1-734C0E6B60AD
You can find the UUIDs for use in these URLs by running `select uuid from photos_with_apple_metadata`.
### Displaying images using datasette-json-html
If you are using `datasette-media` to serve photos you can include images directly in Datasette query results using the [datasette-json-html](https://github.com/simonw/datasette-json-html) plugin.
Run `pip install datasette-json-html` to install the plugin, then use queries like this to view your images:
```sql
select
json_object(
'img_src',
'/-/media/thumbnail/' || uuid
) as photo,
uuid,
date
from
apple_photos
order by
date desc
limit 10;
```
The `photo` column returned by this query should render as image tags that display the correct images.
### Displaying images using custom template pages
Datasette's [custom pages](https://datasette.readthedocs.io/en/stable/custom_templates.html#custom-pages) feature lets you create custom pages for a Datasette instance by dropping HTML templates into a `templates/pages` directory and then running Datasette using `datasette --template-dir=templates/`.
You can combine that ability with the [datasette-template-sql](https://github.com/simonw/datasette-template-sql) plugin to create custom template pages that directly display photos served by `datasette-media`.
Install the plugin using `pip install datasette-template-sql`.
Create a `templates/pages` folder and add the following files:
`recent-photos.html`
```html+jinja
Recent photos
{% for photo in sql(""select * from apple_photos order by date desc limit 20"") %}
{% endfor %}
```
`random-photos.html`
```html+jinja
Random photos
{% for photo in sql(""with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 20"") %}
{% endfor %}
```
Now run Datasette like this:
$ datasette photos.db -m metadata.yaml --template-dir=templates/
Visiting `http://localhost:8001/recent-photos` will display 20 recent photos. Visiting `http://localhost:8001/random-photos` will display 20 photos randomly selected from your 5,000 most recent.
","
dogsheep-photos
Save details of your photos to a SQLite database and upload them to S3.
These tools are a work-in-progress mechanism for taking full ownership of your photos. The core idea is to help implement the following:
Every photo you have taken lives in a single, private Amazon S3 bucket
You have a single SQLite database file which stores metadata about those photos - potentially pulled from multiple different places. This may include EXIF data, Apple Photos, the results of running machine learning APIs against photos and much more besides.
You can then use Datasette to explore your own photos.
I'm a heavy user of Apple Photos so the initial releases of this tool will have a bias towards that, but ideally I would like a subset of these tools to be useful to people no matter which core photo solution they are using.
Installation
$ pip install dogsheep-photos
Authentication (if using S3)
If you want to use S3 to store your photos, you will need to first create S3 credentials for a new, dedicated bucket.
Run this command and paste in your credentials. You will need three values: the name of your S3 bucket, your Access key ID and your Secret access key.
$ dogsheep-photos s3-auth
This will create a file called auth.json in your current directory containing the required values. To save the file at a different path or filename, use the --auth=myauth.json option.
Uploading photos
Run this command to upload every photo in a specific directory to your S3 bucket:
The command will only upload photos that have not yet been uploaded, based on their sha256 hash.
photos.db will be created with an uploads table containing details of which files were uploaded.
To see what the command would do without uploading any files, use the --dry-run option.
The sha256 hash of the photo contents will be used as the name of the file in the bucket, with an extension matching the type of file. This is an implementation of the Content addressable storage pattern.
Importing Apple Photos metadata
The apple-photos command imports metadata from your Apple Photos library.
$ photo-to-sqlite apple-photos photos.db
Imported metadata includes places, people, albums, quality scores and machine learning labels for the photo contents.
Creating a subset database
You can create a new, subset database of photos using the create-subset command.
This is useful for creating a shareable SQLite database that only contains metadata for a selected set of photos.
Since photo metadata contains latitude and longitude you may not want to share a database that includes photos taken at your home address.
create-subset takes three arguments: an existing database file created using the apple-photos command, the name of the new, shareable database file you would like to create and a SQL query that returns the sha256 hash values of the photos you would like to include in that database.
For example, here's how to create a shareable database of just the photos that have been added to albums containing the word ""Public"":
$ dogsheep-photos create-subset \
photos.db \
public.db \
""select sha256 from apple_photos where albums like '%Public%'""
Serving photos locally with datasette-media
If you don't want to upload your photos to S3 but you still want to browse them using Datasette you can do so using the datasette-media plugin. This plugin adds the ability to serve images and other static files directly from disk, configured using a SQL query.
To use it, first install Datasette and the plugin:
$ pip install datasette datasette-media
If any of your photos are .HEIC images taken by an iPhone you should also install the optional pyheif dependency:
$ pip install pyheif
Now create a metadata.yaml file configuring the plugin:
plugins:
datasette-media:
thumbnail:
sql: |- select path as filepath, 200 as resize_height from apple_photos where uuid = :keylarge:
sql: |- select path as filepath, 1024 as resize_height from apple_photos where uuid = :key
This will configure two URL endpoints - one for 200 pixel high thumbnails and one for 1024 pixel high larger images.
Create your photos.db database using the apple-photos command, then run Datasette like this:
$ datasette -m metadata.yaml
Your photos will be served on URLs that look like this:
You can find the UUIDs for use in these URLs by running select uuid from photos_with_apple_metadata.
Displaying images using datasette-json-html
If you are using datasette-media to serve photos you can include images directly in Datasette query results using the datasette-json-html plugin.
Run pip install datasette-json-html to install the plugin, then use queries like this to view your images:
select
json_object(
'img_src',
'/-/media/thumbnail/'|| uuid
) as photo,
uuid,
datefrom
apple_photos
order bydatedesclimit10;
The photo column returned by this query should render as image tags that display the correct images.
Displaying images using custom template pages
Datasette's custom pages feature lets you create custom pages for a Datasette instance by dropping HTML templates into a templates/pages directory and then running Datasette using datasette --template-dir=templates/.
You can combine that ability with the datasette-template-sql plugin to create custom template pages that directly display photos served by datasette-media.
Install the plugin using pip install datasette-template-sql.
Create a templates/pages folder and add the following files:
recent-photos.html
<h1>Recent photos</h1>
<div>
{%forphotoinsql(""select * from apple_photos order by date desc limit 20"") %}
<imgsrc=""/-/media/photo/{{ photo['uuid'] }}"">
{%endfor%}
</div>
random-photos.html
<h1>Random photos</h1>
<div>
{%forphotoinsql(""with foo as (select * from apple_photos order by date desc limit 5000) select * from foo order by random() limit 20"") %}
<imgsrc=""/-/media/photo/{{ photo['uuid'] }}"">
{%endfor%}
</div>
Visiting http://localhost:8001/recent-photos will display 20 recent photos. Visiting http://localhost:8001/random-photos will display 20 photos randomly selected from your 5,000 most recent.
",1,public,0,,,
261634807,MDEwOlJlcG9zaXRvcnkyNjE2MzQ4MDc=,datasette-media,simonw/datasette-media,0,9599,https://github.com/simonw/datasette-media,Datasette plugin for serving media based on a SQL query,0,2020-05-06T02:42:57Z,2021-05-03T05:04:39Z,2020-07-30T23:39:29Z,,70,11,11,Python,1,1,1,1,0,0,0,0,8,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,8,11,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-media
[](https://pypi.org/project/datasette-media/)
[](https://github.com/simonw/datasette-media/releases)
[](https://circleci.com/gh/simonw/datasette-media)
[](https://github.com/simonw/datasette-media/blob/master/LICENSE)
Datasette plugin for serving media based on a SQL query.
Use this when you have a database table containing references to files on disk - or binary content stored in BLOB columns - that you would like to be able to serve to your users.
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-media
### HEIC image support
Modern iPhones save their photos using the [HEIC image format](https://en.wikipedia.org/wiki/High_Efficiency_Image_File_Format). Processing these images requires an additional dependency, [pyheif](https://pypi.org/project/pyheif/). You can include this dependency by running:
$ pip install datasette-media[heif]
## Usage
You can use this plugin to configure Datasette to serve static media based on SQL queries to an underlying database table.
Media will be served from URLs that start with `/-/media/`. The full URL to each media asset will look like this:
/-/media/type-of-media/media-key
`type-of-media` will correspond to a configured SQL query, and might be something like `photo`. `media-key` will be an identifier that is used as part of the underlying SQL query to find which file should be served.
### Serving static files from disk
The following ``metadata.json`` configuration will cause this plugin to serve files from disk, based on queries to a database table called `apple_photos`.
```json
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath from apple_photos where uuid=:key""
}
}
}
}
```
A request to `/-/media/photo/CF972D33-5324-44F2-8DAE-22CB3182CD31` will execute the following SQL query:
```sql
select filepath from apple_photos where uuid=:key
```
The value from the URL - in this case `CF972D33-5324-44F2-8DAE-22CB3182CD31` - will be passed as the `:key` parameter to the query.
The query returns a `filepath` value that has been read from the table. The plugin will then read that file from disk and serve it in response to the request.
SQL queries default to running against the first connected database. You can specify a different database to execute the query against using `""database"": ""name_of_db""`. To execute against `photos.db`, use this:
```json
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath from apple_photos where uuid=:key"",
""database"": ""photos""
}
}
}
}
```
See [dogsheep-photos](https://github.com/dogsheep/dogsheep-photos) for an example of an application that can benefit from this plugin.
### Serving binary content from BLOB columns
If your SQL query returns a `content` column, this will be served directly to the user:
```json
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select thumbnail as content from photos where uuid=:key"",
""database"": ""thumbs""
}
}
}
}
```
You can also return a `content_type` column which will be used as the `Content-Type` header served to the user:
```json
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select body as content, 'text/html;charset=utf-8' as content_type from documents where id=:key"",
""database"": ""documents""
}
}
}
}
```
If you do not specify a `content_type` the default of `application/octet-stream` will be used.
### Serving content proxied from a URL
To serve content that is itself fetched from elsewhere, return a `content_url` column. This can be particularly useful when combined with the ability to resize images (described in the next section).
```json
{
""plugins"": {
""datasette-media"": {
""photos"": {
""sql"": ""select photo_url as content_url from photos where id=:key"",
""database"": ""photos"",
""enable_transform"": true
}
}
}
}
```
Now you can access resized versions of images from that URL like so:
/-/media/photos/13?w=200
### Setting a download file name
The `content_filename` column can be returned to force browsers to download the content using a specific file name.
```json
{
""plugins"": {
""datasette-media"": {
""hello"": {
""sql"": ""select 'Hello ' || :key as content, 'hello.txt' as content_filename""
}
}
}
}
```
Visiting `/-/media/hello/Groot` will cause your browser to download a file called `hello.txt` containing the text `Hello Groot`.
### Resizing or transforming images
Your SQL query can specify that an image should be resized and/or converted to another format by returning additional columns. All three are optional.
* `resize_width` - the width to resize the image to
* `resize_width` - the height to resize the image to
* `output_format` - the output format to use (e.g. `jpeg` or `png`) - any output format [supported by Pillow](https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html) is allowed here.
If you specify one but not the other of `resize_width` or `resize_height` the unspecified one will be calculated automatically to maintain the aspect ratio of the image.
Here's an example configuration that will resize all images to be JPEGs that are 200 pixels in height:
```json
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath, 200 as resize_height, 'jpeg' as output_format from apple_photos where uuid=:key"",
""database"": ""photos""
}
}
}
}
```
If you enable the `enable_transform` configuration option you can instead specify transform parameters at runtime using querystring parameters. For example:
- `/-/media/photo/CF972D33?w=200` to resize to a fixed width
- `/-/media/photo/CF972D33?h=200` to resize to a fixed height
- `/-/media/photo/CF972D33?format=jpeg` to convert to JPEG
That option is added like so:
```json
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath from apple_photos where uuid=:key"",
""database"": ""photos"",
""enable_transform"": true
}
}
}
}
```
The maximum allowed height or width is 4000 pixels. You can change this limit using the `""max_width_height""` option:
```json
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath from apple_photos where uuid=:key"",
""database"": ""photos"",
""enable_transform"": true,
""max_width_height"": 1000
}
}
}
}
```
## Configuration
In addition to the different named content types, the following special plugin configuration setting is available:
- `transform_threads` - number of threads to use for running transformations (e.g. resizing). Defaults to 4.
This can be used like this:
```json
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath from apple_photos where uuid=:key"",
""database"": ""photos""
},
""transform_threads"": 8
}
}
}
```
","
datasette-media
Datasette plugin for serving media based on a SQL query.
Use this when you have a database table containing references to files on disk - or binary content stored in BLOB columns - that you would like to be able to serve to your users.
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-media
HEIC image support
Modern iPhones save their photos using the HEIC image format. Processing these images requires an additional dependency, pyheif. You can include this dependency by running:
$ pip install datasette-media[heif]
Usage
You can use this plugin to configure Datasette to serve static media based on SQL queries to an underlying database table.
Media will be served from URLs that start with /-/media/. The full URL to each media asset will look like this:
/-/media/type-of-media/media-key
type-of-media will correspond to a configured SQL query, and might be something like photo. media-key will be an identifier that is used as part of the underlying SQL query to find which file should be served.
Serving static files from disk
The following metadata.json configuration will cause this plugin to serve files from disk, based on queries to a database table called apple_photos.
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath from apple_photos where uuid=:key""
}
}
}
}
A request to /-/media/photo/CF972D33-5324-44F2-8DAE-22CB3182CD31 will execute the following SQL query:
select filepath from apple_photos where uuid=:key
The value from the URL - in this case CF972D33-5324-44F2-8DAE-22CB3182CD31 - will be passed as the :key parameter to the query.
The query returns a filepath value that has been read from the table. The plugin will then read that file from disk and serve it in response to the request.
SQL queries default to running against the first connected database. You can specify a different database to execute the query against using ""database"": ""name_of_db"". To execute against photos.db, use this:
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath from apple_photos where uuid=:key"",
""database"": ""photos""
}
}
}
}
See dogsheep-photos for an example of an application that can benefit from this plugin.
Serving binary content from BLOB columns
If your SQL query returns a content column, this will be served directly to the user:
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select thumbnail as content from photos where uuid=:key"",
""database"": ""thumbs""
}
}
}
}
You can also return a content_type column which will be used as the Content-Type header served to the user:
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select body as content, 'text/html;charset=utf-8' as content_type from documents where id=:key"",
""database"": ""documents""
}
}
}
}
If you do not specify a content_type the default of application/octet-stream will be used.
Serving content proxied from a URL
To serve content that is itself fetched from elsewhere, return a content_url column. This can be particularly useful when combined with the ability to resize images (described in the next section).
{
""plugins"": {
""datasette-media"": {
""photos"": {
""sql"": ""select photo_url as content_url from photos where id=:key"",
""database"": ""photos"",
""enable_transform"": true
}
}
}
}
Now you can access resized versions of images from that URL like so:
/-/media/photos/13?w=200
Setting a download file name
The content_filename column can be returned to force browsers to download the content using a specific file name.
Visiting /-/media/hello/Groot will cause your browser to download a file called hello.txt containing the text Hello Groot.
Resizing or transforming images
Your SQL query can specify that an image should be resized and/or converted to another format by returning additional columns. All three are optional.
resize_width - the width to resize the image to
resize_width - the height to resize the image to
output_format - the output format to use (e.g. jpeg or png) - any output format supported by Pillow is allowed here.
If you specify one but not the other of resize_width or resize_height the unspecified one will be calculated automatically to maintain the aspect ratio of the image.
Here's an example configuration that will resize all images to be JPEGs that are 200 pixels in height:
{
""plugins"": {
""datasette-media"": {
""photo"": {
""sql"": ""select filepath, 200 as resize_height, 'jpeg' as output_format from apple_photos where uuid=:key"",
""database"": ""photos""
}
}
}
}
If you enable the enable_transform configuration option you can instead specify transform parameters at runtime using querystring parameters. For example:
/-/media/photo/CF972D33?w=200 to resize to a fixed width
/-/media/photo/CF972D33?h=200 to resize to a fixed height
/-/media/photo/CF972D33?format=jpeg to convert to JPEG
",,,,,,
271408895,MDEwOlJlcG9zaXRvcnkyNzE0MDg4OTU=,datasette-permissions-sql,simonw/datasette-permissions-sql,0,9599,https://github.com/simonw/datasette-permissions-sql,Datasette plugin for configuring permission checks using SQL queries,0,2020-06-10T23:48:13Z,2020-06-12T07:06:12Z,2020-06-12T07:06:15Z,,25,0,0,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""datasette"", ""datasette-plugin"", ""datasette-io""]",0,0,0,master,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-permissions-sql
[](https://pypi.org/project/datasette-permissions-sql/)
[](https://circleci.com/gh/simonw/datasette-permissions-sql)
[](https://github.com/simonw/datasette-permissions-sql/blob/master/LICENSE)
Datasette plugin for configuring permission checks using SQL queries
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-permissions-sql
## Usage
First, read up on how Datasette's [authentication and permissions system](https://datasette.readthedocs.io/en/latest/authentication.html) works.
This plugin lets you define rules containing SQL queries that are executed to see if the currently authenticated actor has permission to perform certain actions.
Consider a canned query which authenticated users should only be able to execute if a row in the `users` table says that they are a member of staff.
That `users` table in the `mydatabase.db` database could look like this:
| id | username | is_staff |
|--|--------|--------|
| 1 | cleopaws | 0 |
| 2 | simon | 1 |
Authenticated users have an `actor` that looks like this:
```json
{
""id"": 2,
""username"": ""simon""
}
```
To configure the canned query to only be executable by staff users, add the following to your `metadata.json`:
```json
{
""plugins"": {
""datasette-permissions-sql"": [
{
""action"": ""view-query"",
""resource"": [""mydatabase"", ""promote_to_staff""],
""sql"": ""SELECT * FROM users WHERE is_staff = 1 AND id = :actor_id""
}
]
},
""databases"": {
""mydatabase"": {
""queries"": {
""promote_to_staff"": {
""sql"": ""UPDATE users SET is is_staff=1 WHERE id=:id"",
""write"": true
}
}
}
}
}
```
The `""datasette-permissions-sql""` key is a list of rules. Each of those rules has the following shape:
```json
{
""action"": ""name-of-action"",
""resource"": [""resource identifier to run this on""],
""sql"": ""SQL query to execute"",
""database"": ""mydatabase""
}
```
Both `""action""` and `""resource""` are optional. If present, the SQL query will only be executed on permission checks that match the action and, if present, the resource indicators.
`""database""` is also optional: it specifies the named database that the query should be executed against. If it is not present the first connected database will be used.
The Datasette documentation includes a [list of built-in permissions](https://datasette.readthedocs.io/en/stable/authentication.html#built-in-permissions) that you might want to use here.
### The SQL query
If the SQL query returns any rows the action will be allowed. If it returns no rows, the plugin hook will return `False` and deny access to that action.
The SQL query is called with a number of named parameters. You can use any of these as part of the query.
The list of parameters is as follows:
* `action` - the action, e.g. `""view-database""`
* `resource_1` - the first component of the resource, if one was passed
* `resource_2` - the second component of the resource, if available
* `actor_*` - a parameter for every key on the actor. Usually `actor_id` is present.
If any rows are returned, the permission check passes. If no rows are returned the check fails.
Another example table, this time granting explicit access to individual tables. Consider a table called `table_access` that looks like this:
| user_id | database | table |
| - | - | - |
| 1 | mydb | dogs |
| 2 | mydb | dogs |
| 1 | mydb | cats |
The following SQL query would grant access to the `dogs` ttable in the `mydb.db` database to users 1 and 2 - but would forbid access for user 2 to the `cats` table:
```sql
SELECT
*
FROM
table_access
WHERE
user_id = :actor_id
AND ""database"" = :resource_1
AND ""table"" = :resource_2
```
In a `metadata.yaml` configuration file that would look like this:
```yaml
databases:
mydb:
allow_sql: {}
plugins:
datasette-permissions-sql:
- action: view-table
sql: |-
SELECT
*
FROM
table_access
WHERE
user_id = :actor_id
AND ""database"" = :resource_1
AND ""table"" = :resource_2
```
We're using `allow_sql: {}` here to disable arbitrary SQL queries. This prevents users from running `select * from cats` directly to work around the permissions limits.
### Fallback mode
The default behaviour of this plugin is to take full control of specified permissions. The SQL query will directly control if the user is allowed or denied access to the permission.
This means that the default policy for each permission (which in Datasette core is ""allow"" for `view-database` and friends) will be ignored. It also means that any other `permission_allowed` plugins will not get their turn once this plugin has executed.
You can change this on a per-rule basis using ``""fallback"": true``:
```json
{
""action"": ""view-table"",
""resource"": [""mydatabase"", ""mytable""],
""sql"": ""select * from admins where user_id = :actor_id"",
""fallback"": true
}
```
When running in fallback mode, a query result returning no rows will cause the plugin hook to return ``None`` - which means ""I have no opinion on this permission, fall back to other plugins or the default"".
In this mode you can still return `False` (for ""deny access"") by returning a single row with a single value of `-1`. For example:
```json
{
""action"": ""view-table"",
""resource"": [""mydatabase"", ""mytable""],
""sql"": ""select -1 from banned where user_id = :actor_id"",
""fallback"": true
}
```
","
datasette-permissions-sql
Datasette plugin for configuring permission checks using SQL queries
Installation
Install this plugin in the same environment as Datasette.
This plugin lets you define rules containing SQL queries that are executed to see if the currently authenticated actor has permission to perform certain actions.
Consider a canned query which authenticated users should only be able to execute if a row in the users table says that they are a member of staff.
That users table in the mydatabase.db database could look like this:
id
username
is_staff
1
cleopaws
0
2
simon
1
Authenticated users have an actor that looks like this:
{
""id"": 2,
""username"": ""simon""
}
To configure the canned query to only be executable by staff users, add the following to your metadata.json:
{
""plugins"": {
""datasette-permissions-sql"": [
{
""action"": ""view-query"",
""resource"": [""mydatabase"", ""promote_to_staff""],
""sql"": ""SELECT * FROM users WHERE is_staff = 1 AND id = :actor_id""
}
]
},
""databases"": {
""mydatabase"": {
""queries"": {
""promote_to_staff"": {
""sql"": ""UPDATE users SET is is_staff=1 WHERE id=:id"",
""write"": true
}
}
}
}
}
The ""datasette-permissions-sql"" key is a list of rules. Each of those rules has the following shape:
{
""action"": ""name-of-action"",
""resource"": [""resource identifier to run this on""],
""sql"": ""SQL query to execute"",
""database"": ""mydatabase""
}
Both ""action"" and ""resource"" are optional. If present, the SQL query will only be executed on permission checks that match the action and, if present, the resource indicators.
""database"" is also optional: it specifies the named database that the query should be executed against. If it is not present the first connected database will be used.
If the SQL query returns any rows the action will be allowed. If it returns no rows, the plugin hook will return False and deny access to that action.
The SQL query is called with a number of named parameters. You can use any of these as part of the query.
The list of parameters is as follows:
action - the action, e.g. ""view-database""
resource_1 - the first component of the resource, if one was passed
resource_2 - the second component of the resource, if available
actor_* - a parameter for every key on the actor. Usually actor_id is present.
If any rows are returned, the permission check passes. If no rows are returned the check fails.
Another example table, this time granting explicit access to individual tables. Consider a table called table_access that looks like this:
user_id
database
table
1
mydb
dogs
2
mydb
dogs
1
mydb
cats
The following SQL query would grant access to the dogs ttable in the mydb.db database to users 1 and 2 - but would forbid access for user 2 to the cats table:
SELECT*FROM
table_access
WHERE
user_id = :actor_id
AND""database""= :resource_1
AND""table""= :resource_2
In a metadata.yaml configuration file that would look like this:
databases:
mydb:
allow_sql: {}plugins:
datasette-permissions-sql:
- action: view-tablesql: |- SELECT * FROM table_access WHERE user_id = :actor_id AND ""database"" = :resource_1 AND ""table"" = :resource_2
We're using allow_sql: {} here to disable arbitrary SQL queries. This prevents users from running select * from cats directly to work around the permissions limits.
Fallback mode
The default behaviour of this plugin is to take full control of specified permissions. The SQL query will directly control if the user is allowed or denied access to the permission.
This means that the default policy for each permission (which in Datasette core is ""allow"" for view-database and friends) will be ignored. It also means that any other permission_allowed plugins will not get their turn once this plugin has executed.
You can change this on a per-rule basis using ""fallback"": true:
{
""action"": ""view-table"",
""resource"": [""mydatabase"", ""mytable""],
""sql"": ""select * from admins where user_id = :actor_id"",
""fallback"": true
}
When running in fallback mode, a query result returning no rows will cause the plugin hook to return None - which means ""I have no opinion on this permission, fall back to other plugins or the default"".
In this mode you can still return False (for ""deny access"") by returning a single row with a single value of -1. For example:
{
""action"": ""view-table"",
""resource"": [""mydatabase"", ""mytable""],
""sql"": ""select -1 from banned where user_id = :actor_id"",
""fallback"": true
}
",,,,,,
271665336,MDEwOlJlcG9zaXRvcnkyNzE2NjUzMzY=,datasette-auth-tokens,simonw/datasette-auth-tokens,0,9599,https://github.com/simonw/datasette-auth-tokens,Datasette plugin for authenticating access using API tokens,0,2020-06-11T23:23:30Z,2021-10-15T00:52:53Z,2021-10-15T00:54:20Z,,34,4,4,Python,1,1,1,1,0,1,0,0,0,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin""]",1,0,4,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,1,3,"# datasette-auth-tokens
[](https://pypi.org/project/datasette-auth-tokens/)
[](https://github.com/simonw/datasette-auth-tokens/releases)
[](https://github.com/simonw/datasette-auth-tokens/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-auth-tokens/blob/main/LICENSE)
Datasette plugin for authenticating access using API tokens
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-auth-tokens
## Hard-coded tokens
Read about Datasette's [authentication and permissions system](https://datasette.readthedocs.io/en/latest/authentication.html).
This plugin lets you configure secret API tokens which can be used to make authenticated requests to Datasette.
First, create a random API token. A useful recipe for doing that is the following:
$ python -c 'import secrets; print(secrets.token_hex(32))'
5f9a486dd807de632200b17508c75002bb66ca6fde1993db1de6cbd446362589
Decide on the actor that this token should represent, for example:
```json
{
""bot_id"": ""my-bot""
}
```
You can then use `""allow""` blocks to provide that token with permission to access specific actions. To enable access to a configured writable SQL query you could use this in your `metadata.json`:
```json
{
""plugins"": {
""datasette-auth-tokens"": {
""tokens"": [
{
""token"": {
""$env"": ""BOT_TOKEN""
},
""actor"": {
""bot_id"": ""my-bot""
}
}
]
}
},
""databases"": {
"":memory:"": {
""queries"": {
""show_version"": {
""sql"": ""select sqlite_version()"",
""allow"": {
""bot_id"": ""my-bot""
}
}
}
}
}
}
```
This uses Datasette's [secret configuration values mechanism](https://datasette.readthedocs.io/en/stable/plugins.html#secret-configuration-values) to allow the secret token to be passed as an environment variable.
Run Datasette like this:
BOT_TOKEN=""this-is-the-secret-token"" \
datasette -m metadata.json
You can now run authenticated API queries like this:
$ curl -H 'Authorization: Bearer this-is-the-secret-token' \
'http://127.0.0.1:8001/:memory:/show_version.json?_shape=array'
[{""sqlite_version()"": ""3.31.1""}]
Additionally you can allow passing the token as a query string parameter, although that's disabled by default given the security implications of URLs with secret tokens included. This may be useful to easily allow embedding data between different services.
Simply enable it using the `param` config value:
```json
{
""plugins"": {
""datasette-auth-tokens"": {
""tokens"": [
{
""token"": {
""$env"": ""BOT_TOKEN""
},
""actor"": {
""bot_id"": ""my-bot""
},
}
],
""param"": ""_auth_token""
}
},
""databases"": {
"":memory:"": {
""queries"": {
""show_version"": {
""sql"": ""select sqlite_version()"",
""allow"": {
""bot_id"": ""my-bot""
}
}
}
}
}
}
```
You can now run authenticated API queries like this:
$ curl http://127.0.0.1:8001/:memory:/show_version.json?_shape=array&_auth_token=this-is-the-secret-token
[{""sqlite_version()"": ""3.31.1""}]
## Tokens from your database
As an alternative (or in addition) to the hard-coded list of tokens you can store tokens in a database table and configure the plugin to access them using a SQL query.
Your query needs to take a `:token_id` parameter and return at least two columns: one called `token_secret` and one called `actor_*` - usually `actor_id`. Further `actor_` prefixed columns can be returned to provide more details for the authenticated actor.
Here's a simple example of a configuration query:
```sql
select actor_id, actor_name, token_secret from tokens where token_id = :token_id
```
This can run against a table like this one:
| token_id | token_secret | actor_id | actor_name |
| -------- | ------------ | -------- | ---------- |
| 1 | bd3c94f51fcd | 78 | Cleopaws |
| 2 | 86681b4d6f66 | 32 | Pancakes |
The tokens are formed as the token ID, then a hyphen, then the token secret. For example:
- `1-bd3c94f51fcd`
- `2-86681b4d6f66`
The SQL query will be executed with the portion before the hyphen as the `:token_id` parameter.
The `token_secret` value returned by the query will be compared to the portion of the token after the hyphen to check if the token is valid.
Columns with a prefix of `actor_` will be used to populate the actor dictionary. In the above example, a token of `2-86681b4d6f66` will become an actor dictionary of `{""id"": 32, ""name"": ""Pancakes""}`.
To configure this, use a `""query""` block in your plugin configuration like this:
```json
{
""plugins"": {
""datasette-auth-tokens"": {
""query"": {
""sql"": ""select actor_id, actor_name, token_secret from tokens where token_id = :token_id"",
""database"": ""tokens""
}
}
},
""databases"": {
""tokens"": {
""allow"": {}
}
}
}
```
The `""sql""` key here contains the SQL query. The `""database""` key has the name of the attached database file that the query should be executed against - in this case it would execute against `tokens.db`.
### Securing your tokens
Anyone with access to your Datasette instance can use it to read the `token_secret` column in your tokens table. This probably isn't what you want!
To avoid this, you should lock down access to that table. The configuration example above shows how to do this using an `""allow"": {}` block. Consult Datasette's [Permissions documentation](https://datasette.readthedocs.io/en/stable/authentication.html#permissions) for more information about how to lock down this kind of access.
","
datasette-auth-tokens
Datasette plugin for authenticating access using API tokens
Installation
Install this plugin in the same environment as Datasette.
Decide on the actor that this token should represent, for example:
{
""bot_id"": ""my-bot""
}
You can then use ""allow"" blocks to provide that token with permission to access specific actions. To enable access to a configured writable SQL query you could use this in your metadata.json:
Additionally you can allow passing the token as a query string parameter, although that's disabled by default given the security implications of URLs with secret tokens included. This may be useful to easily allow embedding data between different services.
As an alternative (or in addition) to the hard-coded list of tokens you can store tokens in a database table and configure the plugin to access them using a SQL query.
Your query needs to take a :token_id parameter and return at least two columns: one called token_secret and one called actor_* - usually actor_id. Further actor_ prefixed columns can be returned to provide more details for the authenticated actor.
Here's a simple example of a configuration query:
select actor_id, actor_name, token_secret from tokens where token_id = :token_id
This can run against a table like this one:
token_id
token_secret
actor_id
actor_name
1
bd3c94f51fcd
78
Cleopaws
2
86681b4d6f66
32
Pancakes
The tokens are formed as the token ID, then a hyphen, then the token secret. For example:
1-bd3c94f51fcd
2-86681b4d6f66
The SQL query will be executed with the portion before the hyphen as the :token_id parameter.
The token_secret value returned by the query will be compared to the portion of the token after the hyphen to check if the token is valid.
Columns with a prefix of actor_ will be used to populate the actor dictionary. In the above example, a token of 2-86681b4d6f66 will become an actor dictionary of {""id"": 32, ""name"": ""Pancakes""}.
To configure this, use a ""query"" block in your plugin configuration like this:
The ""sql"" key here contains the SQL query. The ""database"" key has the name of the attached database file that the query should be executed against - in this case it would execute against tokens.db.
Securing your tokens
Anyone with access to your Datasette instance can use it to read the token_secret column in your tokens table. This probably isn't what you want!
To avoid this, you should lock down access to that table. The configuration example above shows how to do this using an ""allow"": {} block. Consult Datasette's Permissions documentation for more information about how to lock down this kind of access.
",1,public,0,,,
272098486,MDEwOlJlcG9zaXRvcnkyNzIwOTg0ODY=,datasette-psutil,simonw/datasette-psutil,0,9599,https://github.com/simonw/datasette-psutil,Datasette plugin adding a /-/psutil debugging endpoint,0,2020-06-13T22:57:07Z,2022-03-07T15:36:30Z,2022-03-07T15:35:57Z,https://datasette.io/plugins/datasette-psutil,12,2,2,Python,1,1,1,1,0,0,0,0,1,apache-2.0,"[""datasette"", ""datasette-io"", ""datasette-plugin"", ""psutil""]",0,1,2,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-psutil
[](https://pypi.org/project/datasette-psutil/)
[](https://github.com/simonw/datasette-psutil/releases)
[](https://github.com/simonw/datasette-psutil/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-psutil/blob/main/LICENSE)
Datasette plugin adding a `/-/psutil` debugging endpoint
## Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-psutil
## Usage
Visit `/-/psutil` on your Datasette instance to see various information provided by [psutil](https://psutil.readthedocs.io/).
## Demo
https://latest-with-plugins.datasette.io/-/psutil is a live demo of this plugin, hosted on Google Cloud Run.
","
datasette-psutil
Datasette plugin adding a /-/psutil debugging endpoint
Installation
Install this plugin in the same environment as Datasette.
$ pip install datasette-psutil
Usage
Visit /-/psutil on your Datasette instance to see various information provided by psutil.
",1,public,0,,,
274264484,MDEwOlJlcG9zaXRvcnkyNzQyNjQ0ODQ=,sqlite-generate,simonw/sqlite-generate,0,9599,https://github.com/simonw/sqlite-generate,Tool for generating demo SQLite databases,0,2020-06-22T23:36:44Z,2021-02-27T15:25:26Z,2021-02-27T15:25:24Z,https://sqlite-generate-demo.datasette.io/,56,17,17,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""sqlite"", ""datasette-io"", ""datasette-tool""]",0,0,17,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,2,"# sqlite-generate
[](https://pypi.org/project/sqlite-generate/)
[](https://github.com/simonw/sqlite-generate/releases)
[](https://github.com/simonw/sqlite-generate/blob/master/LICENSE)
Tool for generating demo SQLite databases
## Installation
Install this plugin using `pip`:
$ pip install sqlite-generate
## Demo
You can see a demo of the database generated using this command running in [Datasette](https://github.com/simonw/datasette) at https://sqlite-generate-demo.datasette.io/
The demo is generated using the following command:
sqlite-generate demo.db --seed seed --fts --columns=10 --fks=0,3 --pks=0,2
## Usage
To generate a SQLite database file called `data.db` with 10 randomly named tables in it, run the following:
sqlite-generate data.db
You can use the `--tables` option to generate a different number of tables:
sqlite-generate data.db --tables 20
You can run the command against the same database file multiple times to keep adding new tables, using different settings for each batch of generated tables.
By default each table will contain a random number of rows between 0 and 200. You can customize this with the `--rows` option:
sqlite-generate data.db --rows 20
This will insert 20 rows into each table.
sqlite-generate data.db --rows 500,2000
This inserts a random number of rows between 500 and 2000 into each table.
Each table will have 5 columns. You can change this using `--columns`:
sqlite-generate data.db --columns 10
`--columns` can also accept a range:
sqlite-generate data.db --columns 5,15
You can control the random number seed used with the `--seed` option. This will result in the exact same database file being created by multiple runs of the tool:
sqlite-generate data.db --seed=myseed
By default each table will contain between 0 and 2 foreign key columns to other tables. You can control this using the `--fks` option, with either a single number or a range:
sqlite-generate data.db --columns=20 --fks=5,15
Each table will have a single primary key column called `id`. You can use the `--pks=` option to change the number of primary key columns on each table. Drop it to 0 to generate [rowid tables](https://www.sqlite.org/rowidtable.html). Increase it above 1 to generate tables with compound primary keys. Or use a range to get a random selection of different primary key layouts:
sqlite-generate data.db --pks=0,2
To configure [SQLite full-text search](https://www.sqlite.org/fts5.html) for all columns of type text, use `--fts`:
sqlite-generate data.db --fts
This will use FTS5 by default. To use [FTS4](https://www.sqlite.org/fts3.html) instead, use `--fts4`.
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd sqlite-generate
python -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
To generate a SQLite database file called data.db with 10 randomly named tables in it, run the following:
sqlite-generate data.db
You can use the --tables option to generate a different number of tables:
sqlite-generate data.db --tables 20
You can run the command against the same database file multiple times to keep adding new tables, using different settings for each batch of generated tables.
By default each table will contain a random number of rows between 0 and 200. You can customize this with the --rows option:
sqlite-generate data.db --rows 20
This will insert 20 rows into each table.
sqlite-generate data.db --rows 500,2000
This inserts a random number of rows between 500 and 2000 into each table.
Each table will have 5 columns. You can change this using --columns:
sqlite-generate data.db --columns 10
--columns can also accept a range:
sqlite-generate data.db --columns 5,15
You can control the random number seed used with the --seed option. This will result in the exact same database file being created by multiple runs of the tool:
sqlite-generate data.db --seed=myseed
By default each table will contain between 0 and 2 foreign key columns to other tables. You can control this using the --fks option, with either a single number or a range:
sqlite-generate data.db --columns=20 --fks=5,15
Each table will have a single primary key column called id. You can use the --pks= option to change the number of primary key columns on each table. Drop it to 0 to generate rowid tables. Increase it above 1 to generate tables with compound primary keys. Or use a range to get a random selection of different primary key layouts:
This will use FTS5 by default. To use FTS4 instead, use --fts4.
Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd sqlite-generate
python -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
291339086,MDEwOlJlcG9zaXRvcnkyOTEzMzkwODY=,airtable-export,simonw/airtable-export,0,9599,https://github.com/simonw/airtable-export,"Export Airtable data to YAML, JSON or SQLite files on disk",0,2020-08-29T19:51:37Z,2021-06-08T17:30:30Z,2021-04-09T23:41:52Z,https://datasette.io/tools/airtable-export,41,33,33,Python,1,1,1,1,0,5,0,0,6,apache-2.0,"[""yaml"", ""airtable"", ""airtable-api"", ""datasette-io"", ""datasette-tool""]",5,6,33,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,5,3,"# airtable-export
[](https://pypi.org/project/airtable-export/)
[](https://github.com/simonw/airtable-export/releases)
[](https://github.com/simonw/airtable-export/actions?query=workflow%3ATest)
[](https://github.com/simonw/airtable-export/blob/master/LICENSE)
Export Airtable data to files on disk
## Installation
Install this tool using `pip`:
$ pip install airtable-export
## Usage
You will need to know the following information:
- Your Airtable base ID - this is a string starting with `app...`
- Your Airtable API key - this is a string starting with `key...`
- The names of each of the tables that you wish to export
You can export all of your data to a folder called `export/` by running the following:
airtable-export export base_id table1 table2 --key=key
This example would create two files: `export/table1.yml` and `export/table2.yml`.
Rather than passing the API key using the `--key` option you can set it as an environment variable called `AIRTABLE_KEY`.
## Export options
By default the tool exports your data as YAML.
You can also export as JSON or as [newline delimited JSON](http://ndjson.org/) using the `--json` or `--ndjson` options:
airtable-export export base_id table1 table2 --key=key --ndjson
You can pass multiple format options at once. This command will create a `.json`, `.yml` and `.ndjson` file for each exported table:
airtable-export export base_id table1 table2 \
--key=key --ndjson --yaml --json
### SQLite database export
You can export tables to a SQLite database file using the `--sqlite database.db` option:
airtable-export export base_id table1 table2 \
--key=key --sqlite database.db
This can be combined with other format options. If you only specify `--sqlite` the export directory argument will be ignored.
The SQLite database will have a table created for each table you export. Those tables will have a primary key column called `airtable_id`.
If you run this command against an existing SQLite database records with matching primary keys will be over-written by new records from the export.
## Request options
By default the tool uses [python-httpx](https://www.python-httpx.org)'s default configurations.
You can override the `user-agent` using the `--user-agent` option:
airtable-export export base_id table1 table2 --key=key --user-agent ""Airtable Export Robot""
You can override the [timeout during a network read operation](https://www.python-httpx.org/advanced/#fine-tuning-the-configuration) using the `--http-read-timeout` option. If not set, this defaults to 5s.
airtable-export export base_id table1 table2 --key=key --http-read-timeout 60
## Running this using GitHub Actions
[GitHub Actions](https://github.com/features/actions) is GitHub's workflow automation product. You can use it to run `airtable-export` in order to back up your Airtable data to a GitHub repository. Doing this gives you a visible commit history of changes you make to your Airtable data - like [this one](https://github.com/natbat/rockybeaches/commits/main/airtable).
To run this for your own Airtable database you'll first need to add the following secrets to your GitHub repository:
AIRTABLE_BASE_ID
The base ID, a string beginning `app...`
AIRTABLE_KEY
Your Airtable API key
AIRTABLE_TABLES
A space separated list of the Airtable tables that you want to backup. If any of these contain spaces you will need to enclose them in single quotes, e.g. 'My table with spaces in the name' OtherTableWithNoSpaces
Once you have set those secrets, add the following as a file called `.github/workflows/backup-airtable.yml`:
```yaml
name: Backup Airtable
on:
workflow_dispatch:
schedule:
- cron: '32 0 * * *'
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Check out repo
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.8
- uses: actions/cache@v2
name: Configure pip caching
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-
restore-keys: |
${{ runner.os }}-pip-
- name: Install airtable-export
run: |
pip install airtable-export
- name: Backup Airtable to backups/
env:
AIRTABLE_BASE_ID: ${{ secrets.AIRTABLE_BASE_ID }}
AIRTABLE_KEY: ${{ secrets.AIRTABLE_KEY }}
AIRTABLE_TABLES: ${{ secrets.AIRTABLE_TABLES }}
run: |-
airtable-export backups $AIRTABLE_BASE_ID $AIRTABLE_TABLES -v
- name: Commit and push if it changed
run: |-
git config user.name ""Automated""
git config user.email ""actions@users.noreply.github.com""
git add -A
timestamp=$(date -u)
git commit -m ""Latest data: ${timestamp}"" || exit 0
git push
```
This will run once a day (at 32 minutes past midnight UTC) and will also run if you manually click the ""Run workflow"" button, see [GitHub Actions: Manual triggers with workflow_dispatch](https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/).
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd airtable-export
python -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
airtable-export
Export Airtable data to files on disk
Installation
Install this tool using pip:
$ pip install airtable-export
Usage
You will need to know the following information:
Your Airtable base ID - this is a string starting with app...
Your Airtable API key - this is a string starting with key...
The names of each of the tables that you wish to export
You can export all of your data to a folder called export/ by running the following:
GitHub Actions is GitHub's workflow automation product. You can use it to run airtable-export in order to back up your Airtable data to a GitHub repository. Doing this gives you a visible commit history of changes you make to your Airtable data - like this one.
To run this for your own Airtable database you'll first need to add the following secrets to your GitHub repository:
AIRTABLE_BASE_ID
The base ID, a string beginning `app...`
AIRTABLE_KEY
Your Airtable API key
AIRTABLE_TABLES
A space separated list of the Airtable tables that you want to backup. If any of these contain spaces you will need to enclose them in single quotes, e.g. 'My table with spaces in the name' OtherTableWithNoSpaces
Once you have set those secrets, add the following as a file called .github/workflows/backup-airtable.yml:
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd airtable-export
python -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
293361514,MDEwOlJlcG9zaXRvcnkyOTMzNjE1MTQ=,geocode-sqlite,eyeseast/geocode-sqlite,0,25778,https://github.com/eyeseast/geocode-sqlite,Geocode rows in a SQLite database table,0,2020-09-06T21:05:39Z,2022-11-02T19:19:56Z,2022-11-07T17:31:05Z,,125,223,223,Python,1,1,1,1,0,6,0,0,8,apache-2.0,[],6,8,223,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,6,5,"# geocode-sqlite
[](https://pypi.org/project/geocode-sqlite/)
[](https://github.com/eyeseast/geocode-sqlite/releases)
[](https://github.com/eyeseast/geocode-sqlite/actions?query=workflow%3ATest)
[](https://github.com/eyeseast/geocode-sqlite/blob/master/LICENSE)
Geocode rows from a SQLite table
## Installation
Install this tool using `pip` or `pipx`:
```sh
# install inside a virtualenv
pip install geocode-sqlite
# install globally
pipx install geocode-sqlite
```
## Usage
Let's say you have a spreadsheet with addresses in it, and you'd like to map those locations.
First, create a SQLite database and insert rows from that spreadsheet using `sqlite-utils`.
```sh
sqlite-utils insert data.db data data.csv --csv
```
Now, geocode it using OpenStreetMap's Nominatim geocoder.
```sh
geocode-sqlite nominatim data.db data \
--location=""{address}, {city}, {state} {zip}"" \
--delay=1 \
--user-agent=""this-is-me""
```
In the command above, you're using Nominatim, which is free and only asks for a unique user agent (`--user-agent`).
This will connect to a database (`data.db`) and read all rows from the table `data` (skipping any that already
have both a `latitude` and `longitude` column filled).
You're also telling the geocoder how to extract a location query (`--location`) from a row of data, using Python's
built-in string formatting, and setting a rate limit (`--delay`) of one request per second.
For each row where geocoding succeeds, `latitude` and `longitude` will be populated. If you hit an error, or a rate limit,
run the same query and pick up where you left off.
The resulting table layout can be visualized with [datasette-cluster-map](https://datasette.io/plugins/datasette-cluster-map).
Under the hood, this package uses the excellent [geopy](https://geopy.readthedocs.io/en/latest/) library, which is stable and thoroughly road-tested. If you need help understanding a particular geocoder's options, consult [geopy's documentation](https://geopy.readthedocs.io/en/latest/#module-geopy.geocoders).
### Supported Geocoders
The CLI currently supports these geocoders:
- `bing`
- `googlev3`
- `mapquest` (and `open-mapquest`)
- `mapbox`
- `nominatim`
- `opencage`
#### Adding new geocoders
1. Open an issue with the name of the geocoding service as the ticket title ([example](https://github.com/eyeseast/geocode-sqlite/issues/35)). Put any noteworthy implementation details in the ticket body, like where to get an API key if one is required.
2. Fork the repo and add a geocoder.
3. Add an example to the `Makefile`. Add tests if there's new shared functionality.
### Common arguments and options
Each geocoder needs to know where to find the data it's working with. These are the first two arguments:
- `database`: a path to a SQLite file, which must already exist
- `table`: the name of a table, in that database, which exists and has data to geocode
From there, we have a set of options passed to every geocoder:
- `location`: a [string format](https://docs.python.org/3/library/stdtypes.html#str.format) that will be expanded with each row to build a full query, to be geocoded
- `delay`: a delay between each call (some services require this)
- `latitude`: latitude column name
- `longitude`: longitude column name
- `geojson`: store results as GeoJSON, instead of in latitude and longitude columns
- `spatialite`: store results in a SpatiaLite geometry column, instead of in latitude and longitude columns
- `raw`: store raw geocoding results in a JSON column
Each geocoder takes additional, specific arguments beyond these, such as API keys. Again, [geopy's documentation](https://geopy.readthedocs.io/en/latest/#module-geopy.geocoders) is an excellent resource.
## Using SpatiaLite
The `--spatialite` flag will store results in a [geometry column](https://www.gaia-gis.it/gaia-sins/spatialite-cookbook-5/cookbook_topics.adminstration.html#topic_TABLE_to_SpatialTable), instead of `latitude` and `longitude` columns. This is useful if you're doing other GIS operations, such as using a [spatial index](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=SpatialIndex). See the [SpatiaLite cookbook](https://www.gaia-gis.it/gaia-sins/spatialite-cookbook-5/index.html) and [functions list](https://www.gaia-gis.it/gaia-sins/spatialite-sql-latest.html) for more of what's possible.
## Capturing additional geocoding data
Geocoding services typically return more data than just coordinates. This might include accuracy, normalized addresses or other context. This can be captured using the `--raw` flag. By default, this will add a `raw` column and store the full geocoding response as JSON. If you want to rename that column, pass a value, like `--raw custom_raw`.
The shape of this response object will vary between services. You can query specific values using [SQLite's built-in JSON functions](https://www.sqlite.org/json1.html). For example, this will work with Google's geocoder:
```sql
select
json_extract(raw, '$.formatted_address') as address,
json_extract(raw, '$.geometry.location_type') as location_type
from
innout_test
```
Check each geocoding service's documentation for what's included in the response.
## Python API
The command line interface aims to support the most common options for each geocoder. For more fine-grained control, use the Python API.
As with the CLI, this assumes you already have a SQLite database and a table of location data.
```python
from geocode_sqlite import geocode_table
from geopy.geocoders import Nominatim
# create a geocoder instance, with some extra options
nominatim = Nominatim(user_agent=""this-is-me"", domain=""nominatim.local.dev"", scheme=""http"")
# assuming our database is in the same directory
count = geocode_table(""data.db"", ""data"", query_template=""{address}, {city}, {state} {zip}"")
# when it's done
print(f""Geocoded {count} rows"")
```
Any [geopy geocoder](https://geopy.readthedocs.io/en/latest/#module-geopy.geocoders) can be used with the Python API.
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
```sh
cd geocode-sqlite
python -m venv .venv
source .venv/bin/activate
```
Or if you are using `pipenv`:
```sh
pipenv shell
```
Now install the dependencies and tests:
```sh
pip install -e '.[test]'
```
To run the tests:
```sh
pytest
```
Please remember that this library is mainly glue code between other well-tested projects, specifically: [click](https://click.palletsprojects.com/), [geopy](https://geopy.readthedocs.io/en/stable/) and [sqlite-utils](https://sqlite-utils.datasette.io/en/stable/). Tests should focus on making sure those parts fit together correctly. We can assume the parts themselves already work.
To that end, there is a test geocoder included: `geocode_sqlite.testing.DummyGeocoder`. That geocoder works with an included dataset of In-N-Out Burger locations provided by [AllThePlaces](https://www.alltheplaces.xyz/). It works like a normal GeoPy geocoder, except it will only return results for In-N-Out locations using the included database.
","
Let's say you have a spreadsheet with addresses in it, and you'd like to map those locations.
First, create a SQLite database and insert rows from that spreadsheet using sqlite-utils.
sqlite-utils insert data.db data data.csv --csv
Now, geocode it using OpenStreetMap's Nominatim geocoder.
In the command above, you're using Nominatim, which is free and only asks for a unique user agent (--user-agent).
This will connect to a database (data.db) and read all rows from the table data (skipping any that already
have both a latitude and longitude column filled).
You're also telling the geocoder how to extract a location query (--location) from a row of data, using Python's
built-in string formatting, and setting a rate limit (--delay) of one request per second.
For each row where geocoding succeeds, latitude and longitude will be populated. If you hit an error, or a rate limit,
run the same query and pick up where you left off.
Under the hood, this package uses the excellent geopy library, which is stable and thoroughly road-tested. If you need help understanding a particular geocoder's options, consult geopy's documentation.
Supported Geocoders
The CLI currently supports these geocoders:
bing
googlev3
mapquest (and open-mapquest)
mapbox
nominatim
opencage
Adding new geocoders
Open an issue with the name of the geocoding service as the ticket title (example). Put any noteworthy implementation details in the ticket body, like where to get an API key if one is required.
Fork the repo and add a geocoder.
Add an example to the Makefile. Add tests if there's new shared functionality.
Common arguments and options
Each geocoder needs to know where to find the data it's working with. These are the first two arguments:
database: a path to a SQLite file, which must already exist
table: the name of a table, in that database, which exists and has data to geocode
From there, we have a set of options passed to every geocoder:
location: a string format that will be expanded with each row to build a full query, to be geocoded
delay: a delay between each call (some services require this)
latitude: latitude column name
longitude: longitude column name
geojson: store results as GeoJSON, instead of in latitude and longitude columns
spatialite: store results in a SpatiaLite geometry column, instead of in latitude and longitude columns
raw: store raw geocoding results in a JSON column
Each geocoder takes additional, specific arguments beyond these, such as API keys. Again, geopy's documentation is an excellent resource.
Using SpatiaLite
The --spatialite flag will store results in a geometry column, instead of latitude and longitude columns. This is useful if you're doing other GIS operations, such as using a spatial index. See the SpatiaLite cookbook and functions list for more of what's possible.
Capturing additional geocoding data
Geocoding services typically return more data than just coordinates. This might include accuracy, normalized addresses or other context. This can be captured using the --raw flag. By default, this will add a raw column and store the full geocoding response as JSON. If you want to rename that column, pass a value, like --raw custom_raw.
The shape of this response object will vary between services. You can query specific values using SQLite's built-in JSON functions. For example, this will work with Google's geocoder:
select
json_extract(raw, '$.formatted_address') as address,
json_extract(raw, '$.geometry.location_type') as location_type
from
innout_test
Check each geocoding service's documentation for what's included in the response.
Python API
The command line interface aims to support the most common options for each geocoder. For more fine-grained control, use the Python API.
As with the CLI, this assumes you already have a SQLite database and a table of location data.
fromgeocode_sqliteimportgeocode_tablefromgeopy.geocodersimportNominatim# create a geocoder instance, with some extra optionsnominatim=Nominatim(user_agent=""this-is-me"", domain=""nominatim.local.dev"", scheme=""http"")
# assuming our database is in the same directorycount=geocode_table(""data.db"", ""data"", query_template=""{address}, {city}, {state} {zip}"")
# when it's doneprint(f""Geocoded {count} rows"")
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd geocode-sqlite
python -m venv .venv
source .venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
Please remember that this library is mainly glue code between other well-tested projects, specifically: click, geopy and sqlite-utils. Tests should focus on making sure those parts fit together correctly. We can assume the parts themselves already work.
To that end, there is a test geocoder included: geocode_sqlite.testing.DummyGeocoder. That geocoder works with an included dataset of In-N-Out Burger locations provided by AllThePlaces. It works like a normal GeoPy geocoder, except it will only return results for In-N-Out locations using the included database.
",1,public,0,,0,0
303218369,MDEwOlJlcG9zaXRvcnkzMDMyMTgzNjk=,evernote-to-sqlite,dogsheep/evernote-to-sqlite,0,53015001,https://github.com/dogsheep/evernote-to-sqlite,Tools for converting Evernote content to SQLite,0,2020-10-11T21:45:49Z,2021-08-26T19:01:54Z,2021-08-26T19:02:47Z,,51,24,24,Python,1,1,1,1,0,4,0,0,3,apache-2.0,"[""datasette-io"", ""datasette-tool"", ""dogsheep"", ""evernote"", ""sqlite""]",4,3,24,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,53015001,4,4,"# evernote-to-sqlite
[](https://pypi.org/project/evernote-to-sqlite/)
[](https://github.com/dogsheep/evernote-to-sqlite/releases)
[](https://github.com/dogsheep/evernote-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/dogsheep/evernote-to-sqlite/blob/master/LICENSE)
Tools for converting Evernote content to SQLite. See [Building an Evernote to SQLite exporter](https://simonwillison.net/2020/Oct/16/building-evernote-sqlite-exporter/) for background on this project.
## Installation
Install this tool using `pip`:
$ pip install evernote-to-sqlite
## Usage
Currently the only available command is `evernote-to-sqlite enex`, which converts Evernote's ENEX export files into a SQLite database.
You can create [an ENEX export](https://help.evernote.com/hc/en-us/articles/209005557-Export-notes-and-notebooks-as-ENEX-or-HTML) in the Evernote desktop application by selecting some notes (or all of your notes) and using the `File -> Export Notes...` menu option.
This used to be able to export everything in one go, but it looks like more recent Evernote versions only allow exporting up to fifty notes at a time, or let you export an entire notebook by right-clicking on the notebook and selecting ""Export notebook..."".
You can convert that file to SQLite like so:
$ evernote-to-sqlite enex evernote.db MyNotes.enex
This will display a progress bar and create a SQLite database file called `evernote.db`.
### Limitations
Unfortunately the ENEX export format does not include a unique identifier for each note. This means you cannot use this tool to re-import notes after they have been updated - you should consider this tool to be a one-time transformation of an ENEX file into an equivalent SQLite database.
ENEX exports also do not include details of which notebook a note belongs to.
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd evernote-to-sqlite
python -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
Currently the only available command is evernote-to-sqlite enex, which converts Evernote's ENEX export files into a SQLite database.
You can create an ENEX export in the Evernote desktop application by selecting some notes (or all of your notes) and using the File -> Export Notes... menu option.
This used to be able to export everything in one go, but it looks like more recent Evernote versions only allow exporting up to fifty notes at a time, or let you export an entire notebook by right-clicking on the notebook and selecting ""Export notebook..."".
This will display a progress bar and create a SQLite database file called evernote.db.
Limitations
Unfortunately the ENEX export format does not include a unique identifier for each note. This means you cannot use this tool to re-import notes after they have been updated - you should consider this tool to be a one-time transformation of an ENEX file into an equivalent SQLite database.
ENEX exports also do not include details of which notebook a note belongs to.
Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd evernote-to-sqlite
python -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
305199661,MDEwOlJlcG9zaXRvcnkzMDUxOTk2NjE=,sphinx-to-sqlite,simonw/sphinx-to-sqlite,0,9599,https://github.com/simonw/sphinx-to-sqlite,Create a SQLite database from Sphinx documentation,0,2020-10-18T21:26:55Z,2020-12-19T05:08:12Z,2020-10-22T04:55:45Z,,9,2,2,Python,1,1,1,1,0,0,0,0,2,apache-2.0,"[""sqlite"", ""sphinx"", ""datasette-io"", ""datasette-tool""]",0,2,2,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,2,"# sphinx-to-sqlite
[](https://pypi.org/project/sphinx-to-sqlite/)
[](https://github.com/simonw/sphinx-to-sqlite/releases)
[](https://github.com/simonw/sphinx-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/sphinx-to-sqlite/blob/master/LICENSE)
Create a SQLite database from Sphinx documentation.
## Demo
You can see the results of running this tool against the [Datasette documentation](https://docs.datasette.io/) at https://latest-docs.datasette.io/docs/sections
## Installation
Install this tool using `pip`:
$ pip install sphinx-to-sqlite
## Usage
First run `sphinx-build` with the `-b xml` option to create XML files in your `_build/` directory.
Then run:
$ sphinx-to-sqlite docs.db path/to/_build
To build the SQLite database.
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd sphinx-to-sqlite
python -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
sphinx-to-sqlite
Create a SQLite database from Sphinx documentation.
First run sphinx-build with the -b xml option to create XML files in your _build/ directory.
Then run:
$ sphinx-to-sqlite docs.db path/to/_build
To build the SQLite database.
Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd sphinx-to-sqlite
python -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
315796015,MDEwOlJlcG9zaXRvcnkzMTU3OTYwMTU=,datasette-ripgrep,simonw/datasette-ripgrep,0,9599,https://github.com/simonw/datasette-ripgrep,"Web interface for searching your code using ripgrep, built as a Datasette plugin",0,2020-11-25T01:26:36Z,2022-04-24T03:48:42Z,2022-06-30T22:45:03Z,https://ripgrep.datasette.io,55,58,58,Python,1,1,1,1,0,1,0,0,6,apache-2.0,"[""codesearch"", ""datasette"", ""datasette-io"", ""datasette-plugin"", ""ripgrep""]",1,6,58,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,1,3,"# datasette-ripgrep
[](https://pypi.org/project/datasette-ripgrep/)
[](https://github.com/simonw/datasette-ripgrep/releases)
[](https://github.com/simonw/datasette-ripgrep/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-ripgrep/blob/main/LICENSE)
Web interface for searching your code using [ripgrep](https://github.com/BurntSushi/ripgrep), built as a [Datasette](https://datasette.io/) plugin
For background on this project see [datasette-ripgrep: deploy a regular expression search engine for your source code](https://simonwillison.net/2020/Nov/28/datasette-ripgrep/).
## Demo
Try this plugin out at https://ripgrep.datasette.io/-/ripgrep - where you can run regular expression searches across the source code of Datasette and all of the `datasette-*` plugins belonging to the [simonw GitHub user](https://github.com/simonw).
Some example searches:
- [with.\*AsyncClient](https://ripgrep.datasette.io/-/ripgrep?pattern=with.*AsyncClient) - regular expression search for `with.*AsyncClient`
- [.plugin_config, literal=on](https://ripgrep.datasette.io/-/ripgrep?pattern=.plugin_config\(&literal=on) - a non-regular expression search for `.plugin_config(`
- [with.\*AsyncClient glob=datasette/\*\*](https://ripgrep.datasette.io/-/ripgrep?pattern=with.*AsyncClient&glob=datasette%2F%2A%2A) - search for that pattern only within the `datasette/` top folder
- [""sqlite-utils\["">\] glob=setup.py](https://ripgrep.datasette.io/-/ripgrep?pattern=%22sqlite-utils%5B%22%3E%5D&glob=setup.py) - a regular expression search for packages that depend on either `sqlite-utils` or `sqlite-utils>=some-version`
- [test glob=!\*.html](https://ripgrep.datasette.io/-/ripgrep?pattern=test&glob=%21*.html) - search for the string `test` but exclude results in HTML files
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-ripgrep
The `rg` executable needs to be [installed](https://github.com/BurntSushi/ripgrep/blob/master/README.md#installation) such that it can be run by this tool.
## Usage
This plugin requires configuration: it needs to a `path` setting so that it knows where to run searches.
Create a `metadata.json` file that looks like this:
```json
{
""plugins"": {
""datasette-ripgrep"": {
""path"": ""/path/to/your/files""
}
}
}
```
Now run Datasette using `datasette -m metadata.json`. The plugin will add an interface at `/-/ripgrep` for running searches.
## Plugin configuration
The `""path""` configuration is required. Optional extra configuration options are:
- `time_limit` - floating point number. The `rg` process will be terminated if it takes longer than this limit. The default is one second, `1.0`.
- `max_lines` - integer. The `rg` process will be terminated if it returns more than this number of lines. The default is `2000`.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-ripgrep
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-ripgrep
Web interface for searching your code using ripgrep, built as a Datasette plugin
Now run Datasette using datasette -m metadata.json. The plugin will add an interface at /-/ripgrep for running searches.
Plugin configuration
The ""path"" configuration is required. Optional extra configuration options are:
time_limit - floating point number. The rg process will be terminated if it takes longer than this limit. The default is one second, 1.0.
max_lines - integer. The rg process will be terminated if it returns more than this number of lines. The default is 2000.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-ripgrep
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,0,
330304628,MDEwOlJlcG9zaXRvcnkzMzAzMDQ2Mjg=,datasette-query-history,bretwalker/datasette-query-history,0,181698,https://github.com/bretwalker/datasette-query-history,,0,2021-01-17T03:13:34Z,2021-04-18T03:06:36Z,2021-01-17T23:11:37Z,,75,3,3,JavaScript,1,1,1,1,0,0,0,0,0,apache-2.0,[],0,0,3,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,0,1,"# datasette-query-history
[](https://pypi.org/project/datasette-query-history/)
[](https://github.com/bretwalker/datasette-query-history/releases)
[](https://github.com/bretwalker/datasette-query-history/actions?query=workflow%3ATest)
[](https://github.com/bretwalker/datasette-query-history/blob/main/LICENSE)
Datasette plugin that keeps a list of the queries you've run and lets you rerun them.
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-query-history
## Usage
Click the `Query History` button on the SQL editor page to see previous queries.
Click the ⬆︎ button to replace the current query with a previous query.
Click the `Clear Query History` button to clear the list previous queries.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-query-history
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-query-history
Datasette plugin that keeps a list of the queries you've run and lets you rerun them.
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-query-history
Usage
Click the Query History button on the SQL editor page to see previous queries.
Click the ⬆︎ button to replace the current query with a previous query.
Click the Clear Query History button to clear the list previous queries.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-query-history
python3 -mvenv venv
source venv/bin/activate
Download map tiles and store them in an MBTiles database
Installation
Install this tool using pip:
$ pip install download-tiles
Usage
This tool downloads tiles from a specified TMS (Tile Map Server) server for a specified bounding box and range of zoom levels and stores those tiles in a MBTiles SQLite database. It is a command-line wrapper around the Landez Python libary.
Please use this tool responsibly. Consult the usage policies of the tile servers you are interacting with, for example the OpenStreetMap Tile Usage Policy.
Running the following will download zoom levels 0-3 of OpenStreetMap, 85 tiles total, and store them in a SQLite database called world.mbtiles:
download-tiles world.mbtiles
You can customize which tile and zoom levels are downloaded using command options:
--zoom-levels=0-3 or -z=0-3
The different zoom levels to download. Specify a single number, e.g. 15, or a range of numbers e.g. 0-4. Be careful with this setting as you can easily go over the limits requested by the underlying tile server.
--bbox=3.9,-6.3,14.5,10.2 or -b=3.9,-6.3,14.5,10.2
The bounding box to fetch. Should be specified as min-lon,min-lat,max-lon,max-lat. You can use bboxfinder.com to find these for different areas.
--city=london or --country=madagascar
These options can be used instead of --bbox. The city or country specified will be looked up using the Nominatum API and used to derive a bounding box.
--show-bbox
Use this option to output the bounding box that was retrieved for the --city or --country without downloading any tiles.
--name=Name
A name for this tile collection, used for the name field in the metadata table. If not specified a UUID will be used, or if you used --city or --country the name will be set to the full name of that place.
The tile server URL to use. This should include {z} and {x} and {y} specifiers, and can optionally include {s} for subdomains.
The default URL used here is for OpenStreetMap, http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png
--tiles-subdomains=a,b,c
A comma-separated list of subdomains to use for the {s} parameter.
--verbose
Use this option to turn on verbose logging.
--cache-dir=/tmp/tiles
Provide a directory to cache downloaded tiles between runs. This can be useful if you are worried you might not have used the correct options for the bounding box or zoom levels.
Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd download-tiles
python -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
346597557,MDEwOlJlcG9zaXRvcnkzNDY1OTc1NTc=,tableau-to-sqlite,simonw/tableau-to-sqlite,0,9599,https://github.com/simonw/tableau-to-sqlite,Fetch data from Tableau into a SQLite database,0,2021-03-11T06:12:02Z,2021-06-10T04:40:44Z,2021-04-29T16:11:03Z,,212,8,8,Python,1,1,1,1,0,2,0,0,2,apache-2.0,"[""datasette-io"", ""datasette-tool""]",2,2,8,main,"{""admin"": false, ""push"": false, ""pull"": false}",,,2,1,"# tableau-to-sqlite
[](https://pypi.org/project/tableau-to-sqlite/)
[](https://github.com/simonw/tableau-to-sqlite/releases)
[](https://github.com/simonw/tableau-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/tableau-to-sqlite/blob/master/LICENSE)
Fetch data from Tableau into a SQLite database. A wrapper around [TableauScraper](https://github.com/bertrandmartel/tableau-scraping/).
## Installation
Install this tool using `pip`:
$ pip install tableau-to-sqlite
## Usage
If you have the URL to a Tableau dashboard like this:
https://results.mo.gov/t/COVID19/views/VaccinationsDashboard/Vaccinations
You can pass that directly to the tool:
tableau-to-sqlite tableau.db \
https://results.mo.gov/t/COVID19/views/VaccinationsDashboard/Vaccinations
This will create a SQLite database called `tableau.db` containing one table for each of the worksheets in that dashboard.
If the dashboard is hosted on https://public.tableau.com/ you can instead provide the view name. This will be two strings separated by a `/` symbol - something like this:
OregonCOVID-19VaccineProviderEnrollment/COVID-19VaccineProviderEnrollment
Now run the tool like this:
tableau-to-sqlite tableau.db \
OregonCOVID-19VaccineProviderEnrollment/COVID-19VaccineProviderEnrollment
## Get the data as JSON or CSV
If you're building a [git scraper](https://simonwillison.net/2020/Oct/9/git-scraping/) you may want to convert the data gathered by this tool to CSV or JSON to check into your repository.
You can do that using [sqlite-utils](https://sqlite-utils.datasette.io/). Install it using `pip`:
pip install sqlite-utils
You can dump out a table as JSON like so:
sqlite-utils rows tableau.db \
'Admin Site and County Map Site No Info' > tableau.json
Or as CSV like this:
sqlite-utils rows tableau.db --csv \
'Admin Site and County Map Site No Info' > tableau.csv
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd tableau-to-sqlite
python -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
","
tableau-to-sqlite
Fetch data from Tableau into a SQLite database. A wrapper around TableauScraper.
Installation
Install this tool using pip:
$ pip install tableau-to-sqlite
Usage
If you have the URL to a Tableau dashboard like this:
This will create a SQLite database called tableau.db containing one table for each of the worksheets in that dashboard.
If the dashboard is hosted on https://public.tableau.com/ you can instead provide the view name. This will be two strings separated by a / symbol - something like this:
If you're building a git scraper you may want to convert the data gathered by this tool to CSV or JSON to check into your repository.
You can do that using sqlite-utils. Install it using pip:
pip install sqlite-utils
You can dump out a table as JSON like so:
sqlite-utils rows tableau.db \
'Admin Site and County Map Site No Info' > tableau.json
Or as CSV like this:
sqlite-utils rows tableau.db --csv \
'Admin Site and County Map Site No Info' > tableau.csv
Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd tableau-to-sqlite
python -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",,,,,,
347263722,MDEwOlJlcG9zaXRvcnkzNDcyNjM3MjI=,django-sql-dashboard,simonw/django-sql-dashboard,0,9599,https://github.com/simonw/django-sql-dashboard,Django app for building dashboards using raw SQL queries,0,2021-03-13T03:38:23Z,2022-04-19T01:13:12Z,2022-04-20T00:27:39Z,https://django-sql-dashboard.datasette.io/,513,335,335,Python,1,1,1,1,0,28,0,0,25,apache-2.0,"[""dashboards"", ""datasette-io"", ""datasette-tool"", ""django"", ""sql""]",28,25,335,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,28,9,"# django-sql-dashboard
[](https://pypi.org/project/django-sql-dashboard/)
[](https://github.com/simonw/django-sql-dashboard/releases)
[](https://github.com/simonw/django-sql-dashboard/actions?query=workflow%3ATest)
[](http://django-sql-dashboard.datasette.io/en/latest/?badge=latest)
[](https://github.com/simonw/django-sql-dashboard/blob/main/LICENSE)
Django SQL Dashboard provides an authenticated interface for executing read-only SQL queries directly against your PostgreSQL database, bringing a useful subset of [Datasette](https://datasette.io/) to Django.
Applications include ad-hoc analysis and debugging, plus the creation of reporting dashboards that can be shared with team members or published online.
See my blog for [more about this project](https://simonwillison.net/2021/May/10/django-sql-dashboard/), including [a video demo](https://www.youtube.com/watch?v=ausrmMZkPEY).
Features include:
- Safely run read-only one or more SQL queries against your database and view the results in your browser
- Bookmark queries and share those links with other members of your team
- Create [saved dashboards](https://django-sql-dashboard.datasette.io/en/latest/saved-dashboards.html) from your queries, with full control over who can view and edit them
- [Named parameters](https://django-sql-dashboard.datasette.io/en/latest/sql.html#sql-parameters) such as `select * from entries where id = %(id)s` will be turned into form fields, allowing quick creation of interactive dashboards
- Produce [bar charts](https://django-sql-dashboard.datasette.io/en/latest/widgets.html#bar-label-bar-quantity), [progress bars](https://django-sql-dashboard.datasette.io/en/latest/widgets.html#total-count-completed-count) and more from SQL queries, with the ability to easily create new [custom dashboard widgets](https://django-sql-dashboard.datasette.io/en/latest/widgets.html#custom-widgets) using the Django template system
- Write SQL queries that safely construct and render [markdown](https://django-sql-dashboard.datasette.io/en/latest/widgets.html#markdown) and [HTML](https://django-sql-dashboard.datasette.io/en/latest/widgets.html#html)
- Export the full results of a SQL query as a downloadable CSV or TSV file, using a combination of Django's [streaming HTTP response](https://docs.djangoproject.com/en/3.2/ref/request-response/#django.http.StreamingHttpResponse) mechanism and PostgreSQL [server-side cursors](https://www.psycopg.org/docs/usage.html#server-side-cursors) to efficiently stream large amounts of data without running out of resources
- Copy and paste the results of SQL queries directly into tools such as Google Sheets or Excel
- Uses Django's authentication system, so dashboard accounts can be granted using Django's Admin tools
## Documentation
Full documentation is at [django-sql-dashboard.datasette.io](https://django-sql-dashboard.datasette.io/)
## Screenshot
## Alternatives
- [django-sql-explorer](https://github.com/groveco/django-sql-explorer) provides a related set of functionality that also works against database backends other than PostgreSQL
","
django-sql-dashboard
Django SQL Dashboard provides an authenticated interface for executing read-only SQL queries directly against your PostgreSQL database, bringing a useful subset of Datasette to Django.
Applications include ad-hoc analysis and debugging, plus the creation of reporting dashboards that can be shared with team members or published online.
Write SQL queries that safely construct and render markdown and HTML
Export the full results of a SQL query as a downloadable CSV or TSV file, using a combination of Django's streaming HTTP response mechanism and PostgreSQL server-side cursors to efficiently stream large amounts of data without running out of resources
Copy and paste the results of SQL queries directly into tools such as Google Sheets or Excel
Uses Django's authentication system, so dashboard accounts can be granted using Django's Admin tools
django-sql-explorer provides a related set of functionality that also works against database backends other than PostgreSQL
",1,public,0,,,
361014273,MDEwOlJlcG9zaXRvcnkzNjEwMTQyNzM=,datasette-dashboards,rclement/datasette-dashboards,0,1238873,https://github.com/rclement/datasette-dashboards,Datasette plugin providing data dashboards from metadata,0,2021-04-23T21:56:48Z,2022-09-21T13:03:39Z,2022-10-07T07:18:03Z,https://datasette-dashboards-demo.vercel.app,1746,74,74,Python,1,1,1,1,0,3,0,0,3,apache-2.0,"[""dashboards"", ""data-visualization"", ""datasette"", ""datasette-io"", ""datasette-plugin"", ""sql"", ""vega-lite""]",3,3,74,master,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,3,1,"# datasette-dashboards
> Datasette plugin providing data dashboards from metadata
[](https://pypi.org/project/datasette-dashboards/)
[](https://github.com/rclement/datasette-dashboards/actions/workflows/ci-cd.yml)
[](https://codecov.io/gh/rclement/datasette-dashboards)
[](https://github.com/rclement/datasette-dashboards/blob/master/LICENSE)
Try out a live demo at [https://datasette-dashboards-demo.vercel.app](https://datasette-dashboards-demo.vercel.app/-/dashboards)
**WARNING**: this plugin is still experimental and not ready for production.
Some breaking changes might happen between releases before reaching a stable version.
Use it at your own risks!
## Installation
Install this plugin in the same environment as Datasette:
```bash
$ datasette install datasette-dashboards
```
## Usage
Define dashboards within `metadata.yml` / `metadata.json`:
```yaml
plugins:
datasette-dashboards:
my-dashboard:
title: My Dashboard
description: Showing some nice metrics
layout:
- [analysis-note, events-count]
- [analysis-note, events-source]
filters:
date_start:
name: Date Start
type: date
default: ""2021-01-01""
date_end:
name: Date End
type: date
charts:
analysis-note:
library: markdown
display: |-
# Analysis notes
> A quick rundown of events statistics and KPIs
events-count:
title: Total number of events
db: jobs
query: SELECT count(*) as count FROM events
library: metric
display:
field: count
prefix:
suffix:
events-source:
title: Number of events by source
db: jobs
query: SELECT source, count(*) as count FROM events WHERE TRUE [[ AND date >= date(:date_start) ]] [[ AND date <= date(:date_end) ]] GROUP BY source ORDER BY count DESC
library: vega
display:
mark: { type: bar, tooltip: true }
encoding:
color: { field: source, type: nominal }
theta: { field: count, type: quantitative }
```
A new menu entry is now available, pointing at `/-/dashboards` to access all defined dashboards.
### Properties
Dashboard properties:
| Property | Type | Description |
| ------------- | -------- | --------------------- |
| `title` | `string` | Dashboard title |
| `description` | `string` | Dashboard description |
| `layout` | `array` | Dashboard layout |
| `filters` | `object` | Dashboard filters |
Dashboard filters:
| Property | Type | Description |
| --------- | ------------------ | -------------------------------------- |
| `name` | `string` | Filter display name |
| `type` | `string` | Filter type (`text`, `date`, `number`) |
| `default` | `string`, `number` | (optional) Filter default value |
| `min` | `number` | (optional) Filter minimum value |
| `max` | `number` | (optional) Filter maximum value |
| `step` | `number` | (optional) Filter stepping value |
Common chart properties for all chart types:
| Property | Type | Description |
| --------- | -------- | -------------------------------------------------------- |
| `title` | `string` | Chart title |
| `db` | `string` | Database name against which to run the query |
| `query` | `string` | SQL query to run and extract data from |
| `library` | `string` | One of supported libraries: `vega`, `markdown`, `metric` |
| `display` | `object` | Chart display specification (depend on the used library) |
To define SQL queries using dashboard filters:
```sql
SELECT * FROM mytable [[ WHERE col >= :my_filter ]]
```
```sql
SELECT * FROM mytable WHERE TRUE [[ AND col1 = :my_filter_1 ]] [[ AND col2 = :my_filter_2 ]]
```
#### Vega properties
Available configuration for `vega` charts:
| Property | Type | Description |
| --------- | -------- | ------------------------- |
| `library` | `string` | Must be set to `vega` |
| `display` | `object` | Vega specification object |
Notes about the `display` property:
- Requires a valid [Vega specification object](https://vega.github.io/vega-lite/docs/)
- Some fields are pre-defined: `$schema`, `title`, `width`, `view`, `config`, `data`
- All fields are passed along as-is (overriding pre-defined fields if any)
- Only `mark` and `encoding` fields are required as the bare-minimum
#### Markdown properties
Available configuration for `markdown` chart:
| Property | Type | Description |
| --------- | -------- | ------------------------------------------------- |
| `library` | `string` | Must be set to `markdown` |
| `display` | `string` | Multi-line string containing the Markdown content |
Note :
- Some common properties do not apply and can be omitted: `title`, `db`, `query`
- Markdown rendering is done by [`datasette-render-markdown`](https://datasette.io/plugins/datasette-render-markdown)
- To configure Markdown rendering, extensions can be enabled in [metadata](https://datasette.io/plugins/datasette-render-markdown#user-content-markdown-extensions)
#### Metric properties
Available configuration for `metric` chart:
| Property | Type | Description |
| ---------------- | -------- | ----------------------------------------- |
| `library` | `string` | Must be set to `metric` |
| `display.field` | `string` | Numerical field to be displayed as metric |
| `display.prefix` | `string` | Prefix to be displayed before metric |
| `display.suffix` | `string` | Prefix to be displayed after metric |
Note:
- The `display.field` must reference a single-numerical value from the SQL query
(e.g. numerical `number` field in `SELECT count(*) as number FROM events`)
### Dashboard layout
The default dashboard layout will present two charts per row (one per row on mobile).
To make use of custom dashboard layout using [CSS Grid Layout](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Grid_Layout),
define the `layout` array property as a grid / matrix:
- Each entry represents a row of charts
- Each column is referring a chart by its property name
## Development
To set up this plugin locally, first checkout the code.
Then create a new virtual environment and the required dependencies:
```bash
pipenv install -d
pipenv shell
```
To run the tests:
```bash
pytest
```
## Demo
With the developmnent environment setup, you can run the demo locally:
```bash
datasette --metadata demo/metadata.yml demo/jobs.db
```
## License
Licensed under Apache License, Version 2.0
Copyright (c) 2021 - present Romain Clement
","
datasette-dashboards
Datasette plugin providing data dashboards from metadata
WARNING: this plugin is still experimental and not ready for production.
Some breaking changes might happen between releases before reaching a stable version.
Use it at your own risks!
Installation
Install this plugin in the same environment as Datasette:
$ datasette install datasette-dashboards
Usage
Define dashboards within metadata.yml / metadata.json:
plugins:
datasette-dashboards:
my-dashboard:
title: My Dashboarddescription: Showing some nice metricslayout:
- [analysis-note, events-count]
- [analysis-note, events-source]filters:
date_start:
name: Date Starttype: datedefault: ""2021-01-01""date_end:
name: Date Endtype: datecharts:
analysis-note:
library: markdowndisplay: |- # Analysis notes > A quick rundown of events statistics and KPIsevents-count:
title: Total number of eventsdb: jobsquery: SELECT count(*) as count FROM eventslibrary: metricdisplay:
field: countprefix:
suffix:
events-source:
title: Number of events by sourcedb: jobsquery: SELECT source, count(*) as count FROM events WHERE TRUE [[ AND date >= date(:date_start) ]] [[ AND date <= date(:date_end) ]] GROUP BY source ORDER BY count DESClibrary: vegadisplay:
mark: { type: bar, tooltip: true }encoding:
color: { field: source, type: nominal }theta: { field: count, type: quantitative }
A new menu entry is now available, pointing at /-/dashboards to access all defined dashboards.
Properties
Dashboard properties:
Property
Type
Description
title
string
Dashboard title
description
string
Dashboard description
layout
array
Dashboard layout
filters
object
Dashboard filters
Dashboard filters:
Property
Type
Description
name
string
Filter display name
type
string
Filter type (text, date, number)
default
string, number
(optional) Filter default value
min
number
(optional) Filter minimum value
max
number
(optional) Filter maximum value
step
number
(optional) Filter stepping value
Common chart properties for all chart types:
Property
Type
Description
title
string
Chart title
db
string
Database name against which to run the query
query
string
SQL query to run and extract data from
library
string
One of supported libraries: vega, markdown, metric
display
object
Chart display specification (depend on the used library)
To define SQL queries using dashboard filters:
SELECT*FROM mytable [[ WHERE col >= :my_filter ]]
SELECT*FROM mytable WHERE TRUE [[ AND col1 = :my_filter_1 ]] [[ AND col2 = :my_filter_2 ]]
To configure Markdown rendering, extensions can be enabled in metadata
Metric properties
Available configuration for metric chart:
Property
Type
Description
library
string
Must be set to metric
display.field
string
Numerical field to be displayed as metric
display.prefix
string
Prefix to be displayed before metric
display.suffix
string
Prefix to be displayed after metric
Note:
The display.field must reference a single-numerical value from the SQL query
(e.g. numerical number field in SELECT count(*) as number FROM events)
Dashboard layout
The default dashboard layout will present two charts per row (one per row on mobile).
To make use of custom dashboard layout using CSS Grid Layout,
define the layout array property as a grid / matrix:
Each entry represents a row of charts
Each column is referring a chart by its property name
Development
To set up this plugin locally, first checkout the code.
Then create a new virtual environment and the required dependencies:
pipenv install -d
pipenv shell
To run the tests:
pytest
Demo
With the developmnent environment setup, you can run the demo locally:
",1,public,0,,0,
390535500,MDEwOlJlcG9zaXRvcnkzOTA1MzU1MDA=,datasette-remote-metadata,simonw/datasette-remote-metadata,0,9599,https://github.com/simonw/datasette-remote-metadata,Periodically refresh Datasette metadata from a remote URL,0,2021-07-28T23:17:19Z,2021-12-13T19:40:51Z,2021-12-13T19:40:48Z,,8,3,3,Python,1,1,1,1,0,0,0,0,0,apache-2.0,[],0,0,3,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-remote-metadata
[](https://pypi.org/project/datasette-remote-metadata/)
[](https://github.com/simonw/datasette-remote-metadata/releases)
[](https://github.com/simonw/datasette-remote-metadata/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-remote-metadata/blob/main/LICENSE)
Periodically refresh Datasette metadata from a remote URL
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-remote-metadata
## Usage
Add the following to your `metadata.json`:
```json
{
""plugins"": {
""datasette-remote-metadata"": {
""url"": ""https://example.com/remote-metadata.yml""
}
}
}
```
The plugin will fetch the specified metadata from that URL at startup and combine it with any existing metadata. You can use a URL to either a JSON file or a YAML file.
It will periodically refresh that metadata - by default every 30 seconds, unless you specify an alternative `""ttl""` value in the plugin configuration.
## Configuration
Available configuration options are as follows:
- `""url""` - the URL to retrieve remote metadata from. Can link to a JSON or a YAML file.
- `""ttl""` - integer value in secords: how frequently should the script check for fresh metadata. Defaults to 30 seconds.
- `""headers""` - a dictionary of additional request headers to send.
- `""cachebust""` - if true, a random `?0.29508` value will be added to the query string of the remote metadata to bust any intermediary caches.
This example `metadata.json` configuration refreshes every 10 seconds, uses cache busting and sends an `Authorization: Bearer xyz` header with the request:
```json
{
""plugins"": {
""datasette-remote-metadata"": {
""url"": ""https://example.com/remote-metadata.yml"",
""ttl"": 10,
""cachebust"": true,
""headers"": {
""Authorization"": ""Bearer xyz""
}
}
}
}
```
This example if you are using `metadata.yaml` for configuration:
```yaml
plugins:
datasette-remote-metadata:
url: https://example.com/remote-metadata.yml
ttl: 10
cachebust: true
headers:
Authorization: Bearer xyz
```
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-remote-metadata
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-remote-metadata
Periodically refresh Datasette metadata from a remote URL
Installation
Install this plugin in the same environment as Datasette.
The plugin will fetch the specified metadata from that URL at startup and combine it with any existing metadata. You can use a URL to either a JSON file or a YAML file.
It will periodically refresh that metadata - by default every 30 seconds, unless you specify an alternative ""ttl"" value in the plugin configuration.
Configuration
Available configuration options are as follows:
""url"" - the URL to retrieve remote metadata from. Can link to a JSON or a YAML file.
""ttl"" - integer value in secords: how frequently should the script check for fresh metadata. Defaults to 30 seconds.
""headers"" - a dictionary of additional request headers to send.
""cachebust"" - if true, a random ?0.29508 value will be added to the query string of the remote metadata to bust any intermediary caches.
This example metadata.json configuration refreshes every 10 seconds, uses cache busting and sends an Authorization: Bearer xyz header with the request:
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-remote-metadata
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
423984522,R_kgDOGUV9ig,s3-credentials,simonw/s3-credentials,0,9599,https://github.com/simonw/s3-credentials,A tool for creating credentials for accessing S3 buckets,0,2021-11-02T20:09:50Z,2022-09-05T15:12:46Z,2022-09-15T23:43:10Z,https://s3-credentials.readthedocs.io,204,129,129,Python,1,1,1,1,0,10,0,0,18,apache-2.0,"[""aws"", ""boto3"", ""s3""]",10,18,129,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,10,2,"# s3-credentials
[](https://pypi.org/project/s3-credentials/)
[](https://github.com/simonw/s3-credentials/releases)
[](https://github.com/simonw/s3-credentials/actions?query=workflow%3ATest)
[](https://s3-credentials.readthedocs.org/)
[](https://github.com/simonw/s3-credentials/blob/master/LICENSE)
A tool for creating credentials for accessing S3 buckets
For project background, see [s3-credentials: a tool for creating credentials for S3 buckets](https://simonwillison.net/2021/Nov/3/s3-credentials/) on my blog.
## Installation
pip install s3-credentials
## Basic usage
To create a new S3 bucket and output credentials that can be used with only that bucket:
```
% s3-credentials create my-new-s3-bucket --create-bucket
Created bucket: my-new-s3-bucket
Created user: s3.read-write.my-new-s3-bucket with permissions boundary: arn:aws:iam::aws:policy/AmazonS3FullAccess
Attached policy s3.read-write.my-new-s3-bucket to user s3.read-write.my-new-s3-bucket
Created access key for user: s3.read-write.my-new-s3-bucket
{
""UserName"": ""s3.read-write.my-new-s3-bucket"",
""AccessKeyId"": ""AKIAWXFXAIOZOYLZAEW5"",
""Status"": ""Active"",
""SecretAccessKey"": ""..."",
""CreateDate"": ""2021-11-03 01:38:24+00:00""
}
```
The tool can do a lot more than this. See the [documentation](https://s3-credentials.readthedocs.io/) for details.
## Documentation
- [Full documentation](https://s3-credentials.readthedocs.io/)
- [Command help reference](https://s3-credentials.readthedocs.io/en/stable/help.html)
- [Release notes](https://github.com/simonw/s3-credentials/releases)
","
s3-credentials
A tool for creating credentials for accessing S3 buckets
",1,public,0,,0,
427128866,R_kgDOGXV4Ig,git-history,simonw/git-history,0,9599,https://github.com/simonw/git-history,Tools for analyzing Git history using SQLite,0,2021-11-11T20:07:06Z,2022-10-20T20:33:28Z,2022-10-21T23:06:33Z,,117,114,114,Python,1,1,1,1,0,11,0,0,20,apache-2.0,"[""git"", ""sqlite""]",11,20,114,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,11,3,"# git-history
[](https://pypi.org/project/git-history/)
[](https://github.com/simonw/git-history/releases)
[](https://github.com/simonw/git-history/actions?query=workflow%3ATest)
[](https://github.com/simonw/git-history/blob/master/LICENSE)
Tools for analyzing Git history using SQLite
For background on this project see [git-history: a tool for analyzing scraped data collected using Git and SQLite](https://simonwillison.net/2021/Dec/7/git-history/).
[Measuring traffic during the Half Moon Bay Pumpkin Festival](https://simonwillison.net/2022/Oct/19/measuring-traffic/) describes a project using this tool in detail.
## Installation
Install this tool using `pip`:
$ pip install git-history
## Demos
[git-history-demos.datasette.io](http://git-history-demos.datasette.io/) hosts three example databases created using this tool:
- [pge-outages](https://git-history-demos.datasette.io/pge-outages) shows a history of PG&E (the electricity supplier) [outages](https://pgealerts.alerts.pge.com/outagecenter/), using data collected in [simonw/pge-outages](https://github.com/simonw/pge-outages) converted using [pge-outages.sh](https://github.com/simonw/git-history/blob/main/demos/pge-outages.sh)
- [ca-fires](https://git-history-demos.datasette.io/ca-fires) shows a history of fires in California reported on [fire.ca.gov/incidents](https://www.fire.ca.gov/incidents/), from data in [simonw/ca-fires-history](https://github.com/simonw/ca-fires-history) converted using [ca-fires.sh](https://github.com/simonw/git-history/blob/main/demos/ca-fires.sh)
- [sf-bay-511](https://git-history-demos.datasette.io/sf-bay-511) has records of San Francisco Bay Area traffic and transit incident data from [511.org](https://511.org/), collected in [dbreunig/511-events-history](https://github.com/dbreunig/511-events-history) converted using [sf-bay-511.sh](https://github.com/simonw/git-history/blob/main/demos/sf-bay-511.sh)
The demos are deployed using [Datasette](https://datasette.io/) on [Google Cloud Run](https://cloud.google.com/run/) by [this GitHub Actions workflow](https://github.com/simonw/git-history/blob/main/.github/workflows/deploy-demos.yml).
## Usage
This tool can be run against a Git repository that holds a file that contains JSON, CSV/TSV or some other format and which has multiple versions tracked in the Git history. Read [Git scraping: track changes over time by scraping to a Git repository](https://simonwillison.net/2020/Oct/9/git-scraping/) to understand how you might create such a repository.
The `file` command analyzes the history of an individual file within the repository, and generates a SQLite database table that represents the different versions of that file over time.
The file is assumed to contain multiple objects - for example, the results of scraping an electricity outage map or a CSV file full of records.
Assuming you have a file called `incidents.json` that is a JSON array of objects, with multiple versions of that file recorded in a repository. Each version of that file might look something like this:
```json
[
{
""IncidentID"": ""abc123"",
""Location"": ""Corner of 4th and Vermont"",
""Type"": ""fire""
},
{
""IncidentID"": ""cde448"",
""Location"": ""555 West Example Drive"",
""Type"": ""medical""
}
]
```
Change directory into the GitHub repository in question and run the following:
git-history file incidents.db incidents.json
This will create a new SQLite database in the `incidents.db` file with three tables:
- `commits` containing a row for every commit, with a `hash` column, the `commit_at` date and a foreign key to a `namespace`.
- `item` containing a row for every item in every version of the `filename.json` file - with an extra `_commit` column that is a foreign key back to the `commit` table.
- `namespaces` containing a single row. This allows you to build multiple tables for different files, using the `--namespace` option described below.
The database schema for this example will look like this:
```sql
CREATE TABLE [namespaces] (
[id] INTEGER PRIMARY KEY,
[name] TEXT
);
CREATE UNIQUE INDEX [idx_namespaces_name]
ON [namespaces] ([name]);
CREATE TABLE [commits] (
[id] INTEGER PRIMARY KEY,
[namespace] INTEGER REFERENCES [namespaces]([id]),
[hash] TEXT,
[commit_at] TEXT
);
CREATE UNIQUE INDEX [idx_commits_namespace_hash]
ON [commits] ([namespace], [hash]);
CREATE TABLE [item] (
[IncidentID] TEXT,
[Location] TEXT,
[Type] TEXT
);
```
If you have 10 historic versions of the `incidents.json` file and each one contains 30 incidents, you will end up with 10 * 30 = 300 rows in your `item` table.
### Track the history of individual items using IDs
If your objects have a unique identifier - or multiple columns that together form a unique identifier - you can use the `--id` option to de-duplicate and track changes to each of those items over time.
This provides a much more interesting way to apply this tool.
If there is a unique identifier column called `IncidentID` you could run the following:
git-history file incidents.db incidents.json --id IncidentID
The database schema used here is very different from the one used without the `--id` option.
If you have already imported history, the command will skip any commits that it has seen already and just process new ones. This means that even though an initial import could be slow subsequent imports should run a lot faster.
This command will create six tables - `commits`, `item`, `item_version`, `columns`, `item_changed` and `namespaces`.
Here's the full schema:
```sql
CREATE TABLE [namespaces] (
[id] INTEGER PRIMARY KEY,
[name] TEXT
);
CREATE UNIQUE INDEX [idx_namespaces_name]
ON [namespaces] ([name]);
CREATE TABLE [commits] (
[id] INTEGER PRIMARY KEY,
[namespace] INTEGER REFERENCES [namespaces]([id]),
[hash] TEXT,
[commit_at] TEXT
);
CREATE UNIQUE INDEX [idx_commits_namespace_hash]
ON [commits] ([namespace], [hash]);
CREATE TABLE [item] (
[_id] INTEGER PRIMARY KEY,
[_item_id] TEXT
, [IncidentID] TEXT, [Location] TEXT, [Type] TEXT, [_commit] INTEGER);
CREATE UNIQUE INDEX [idx_item__item_id]
ON [item] ([_item_id]);
CREATE TABLE [item_version] (
[_id] INTEGER PRIMARY KEY,
[_item] INTEGER REFERENCES [item]([_id]),
[_version] INTEGER,
[_commit] INTEGER REFERENCES [commits]([id]),
[IncidentID] TEXT,
[Location] TEXT,
[Type] TEXT,
[_item_full_hash] TEXT
);
CREATE TABLE [columns] (
[id] INTEGER PRIMARY KEY,
[namespace] INTEGER REFERENCES [namespaces]([id]),
[name] TEXT
);
CREATE UNIQUE INDEX [idx_columns_namespace_name]
ON [columns] ([namespace], [name]);
CREATE TABLE [item_changed] (
[item_version] INTEGER REFERENCES [item_version]([_id]),
[column] INTEGER REFERENCES [columns]([id]),
PRIMARY KEY ([item_version], [column])
);
CREATE VIEW item_version_detail AS select
commits.commit_at as _commit_at,
commits.hash as _commit_hash,
item_version.*,
(
select json_group_array(name) from columns
where id in (
select column from item_changed
where item_version = item_version._id
)
) as _changed_columns
from item_version
join commits on commits.id = item_version._commit;
CREATE INDEX [idx_item_version__item]
ON [item_version] ([_item]);
```
#### item table
The `item` table will contain the most recent version of each row, de-duplicated by ID, plus the following additional columns:
- `_id` - a numeric integer primary key, used as a foreign key from the `item_version` table.
- `_item_id` - a hash of the values of the columns specified using the `--id` option to the command. This is used for de-duplication when processing new versions.
- `_commit` - a foreign key to the `commit` table, representing the most recent commit to modify this item.
#### item_version table
The `item_version` table will contain a row for each captured differing version of that item, plus the following columns:
- `_id` - a numeric ID for the item version record.
- `_item` - a foreign key to the `item` table.
- `_version` - the numeric version number, starting at 1 and incrementing for each captured version.
- `_commit` - a foreign key to the `commit` table.
- `_item_full_hash` - a hash of this version of the item. This is used internally by the tool to identify items that have changed between commits.
The other columns in this table represent columns in the original data that have changed since the previous version. If the value has not changed, it will be represented by a `null`.
If a value was previously set but has been changed back to `null` it will still be represented as `null` in the `item_version` row. You can identify these using the `item_changed` many-to-many table described below.
You can use the `--full-versions` option to store full copies of the item at each version, rather than just storing the columns that have changed.
#### item_version_detail view
This SQL view joins `item_version` against `commits` to add three further columns: `_commit_at` with the date of the commit, and `_commit_hash` with the Git commit hash.
#### item_changed
This many-to-many table indicates exactly which columns were changed in an `item_version`.
- `item_version` is a foreign key to a row in the `item_version` table.
- `column` is a foreign key to a row in the `columns` table.
This table with have the largest number of rows, which is why it stores just two integers in order to save space.
#### columns
The `columns` table stores column names. It is referenced by `item_changed`.
- `id` - an integer ID.
- `name` - the name of the column.
- `namespace` - a foreign key to `namespaces`, for if multiple file histories are sharing the same database.
#### Reserved column names
Note that `_id`, `_item_full_hash`, `_item`, `_item_id`, `_version`, `_commit`, `_item_id`, `_commit_at`, `_commit_hash`, `_changed_columns`, `rowid` are considered reserved column names for the purposes of this tool.
If your data contains any of these they will be renamed to add a trailing underscore, for example `_id_`, `_item_`, `_version_`, to avoid clashing with the reserved columns.
If you have a column with a name such as `_commit_` it will be renamed too, adding an additional trailing underscore, so `_commit_` becomes `_commit__` and `_commit__` becomes `_commit___`.
### Additional options
- `--repo DIRECTORY` - the path to the Git repository, if it is not the current working directory.
- `--branch TEXT` - the Git branch to analyze - defaults to `main`.
- `--id TEXT` - as described above: pass one or more columns that uniquely identify a record, so that changes to that record can be calculated over time.
- `--full-versions` - instead of recording just the columns that have changed in the `item_version` table record a full copy of each version of theh item.
- `--ignore TEXT` - one or more columns to ignore - they will not be included in the resulting database.
- `--csv` - treat the data is CSV or TSV rather than JSON, and attempt to guess the correct dialect
- `--dialect` - use a spcific CSV dialect. Options are `excel`, `excel-tab` and `unix` - see [the Python CSV documentation](https://docs.python.org/3/library/csv.html#csv.excel) for details.
- `--skip TEXT` - one or more full Git commit hashes that should be skipped. You can use this if some of the data in your revision history is corrupted in a way that prevents this tool from working.
- `--start-at TEXT` - skip commits prior to the specified commit hash.
- `--start-after TEXT` - skip commits up to and including the specified commit hash, then start processing from the following commit.
- `--convert TEXT` - custom Python code for a conversion, described below.
- `--import TEXT` - additional Python modules to import for `--convert`.
- `--ignore-duplicate-ids` - if a single version of a file has the same ID in it more than once, the tool will exit with an error. Use this option to ignore this and instead pick just the first of the two duplicates.
- `--namespace TEXT` - use this if you wish to include the history of multiple different files in the same database. The default is `item` but you can set it to something else, which will produce tables with names like `yournamespace` and `yournamespace_version`.
- `--wal` - Enable WAL mode on the created database file. Use this if you plan to run queries against the database while `git-history` is creating it.
- `--silent` - don't show the progress bar.
### CSV and TSV data
If the data in your repository is a CSV or TSV file you can process it by adding the `--csv` option. This will attempt to detect which delimiter is used by the file, so the same option works for both comma- and tab-separated values.
git-history file trees.db trees.csv --id TreeID
You can also specify the CSV dialect using the `--dialect` option.
### Custom conversions using --convert
If your data is not already either CSV/TSV or a flat JSON array, you can reshape it using the `--convert` option.
The format needed by this tool is an array of dictionaries, as demonstrated by the `incidents.json` example above.
If your data does not fit this shape, you can provide a snippet of Python code to converts the on-disk content of each stored file into a Python list of dictionaries.
For example, if your stored files each look like this:
```json
{
""incidents"": [
{
""id"": ""552"",
""name"": ""Hawthorne Fire"",
""engines"": 3
},
{
""id"": ""556"",
""name"": ""Merlin Fire"",
""engines"": 1
}
]
}
```
You could use the following Python snippet to convert them to the required format:
```python
json.loads(content)[""incidents""]
```
(The `json` module is exposed to your custom function by default.)
You would then run the tool like this:
git-history file database.db incidents.json \
--id id \
--convert 'json.loads(content)[""incidents""]'
The `content` variable is always a `bytes` object representing the content of the file at a specific moment in the repository's history.
You can import additional modules using `--import`. This example shows how you could read a CSV file that uses `;` as the delimiter:
git-history file trees.db ../sf-tree-history/Street_Tree_List.csv \
--repo ../sf-tree-history \
--import csv \
--import io \
--convert '
fp = io.StringIO(content.decode(""utf-8""))
return list(csv.DictReader(fp, delimiter="";""))
' \
--id TreeID
You can import nested modules such as [ElementTree](https://docs.python.org/3/library/xml.etree.elementtree.html) using `--import xml.etree.ElementTree`, then refer to them in your function body as `xml.etree.ElementTree`. For example, if your tracked data was in an `items.xml` file that looked like this:
```xml
```
You could load it using the following `--convert` script:
```
git-history file items.xml --convert '
tree = xml.etree.ElementTree.fromstring(content)
return [el.attrib for el in tree.iter(""item"")]
' --import xml.etree.ElementTree --id id
```
If your Python code spans more than one line it needs to include a `return` statement.
You can also use Python generators in your `--convert` code, for example:
git-history file stats.db package-stats/stats.json \
--repo package-stats \
--convert '
data = json.loads(content)
for key, counts in data.items():
for date, count in counts.items():
yield {
""package"": key,
""date"": date,
""count"": count
}
' --id package --id date
This conversion function expects data that looks like this:
```json
{
""airtable-export"": {
""2021-05-18"": 66,
""2021-05-19"": 60,
""2021-05-20"": 87
}
}
```
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd git-history
python -m venv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
To update the schema examples in this README file:
cog -r README.md
","
This tool can be run against a Git repository that holds a file that contains JSON, CSV/TSV or some other format and which has multiple versions tracked in the Git history. Read Git scraping: track changes over time by scraping to a Git repository to understand how you might create such a repository.
The file command analyzes the history of an individual file within the repository, and generates a SQLite database table that represents the different versions of that file over time.
The file is assumed to contain multiple objects - for example, the results of scraping an electricity outage map or a CSV file full of records.
Assuming you have a file called incidents.json that is a JSON array of objects, with multiple versions of that file recorded in a repository. Each version of that file might look something like this:
[
{
""IncidentID"": ""abc123"",
""Location"": ""Corner of 4th and Vermont"",
""Type"": ""fire""
},
{
""IncidentID"": ""cde448"",
""Location"": ""555 West Example Drive"",
""Type"": ""medical""
}
]
Change directory into the GitHub repository in question and run the following:
git-history file incidents.db incidents.json
This will create a new SQLite database in the incidents.db file with three tables:
commits containing a row for every commit, with a hash column, the commit_at date and a foreign key to a namespace.
item containing a row for every item in every version of the filename.json file - with an extra _commit column that is a foreign key back to the commit table.
namespaces containing a single row. This allows you to build multiple tables for different files, using the --namespace option described below.
The database schema for this example will look like this:
CREATE TABLE [namespaces] (
[id] INTEGERPRIMARY KEY,
[name] TEXT
);
CREATE UNIQUE INDEX [idx_namespaces_name]
ON [namespaces] ([name]);
CREATE TABLE [commits] (
[id] INTEGERPRIMARY KEY,
[namespace] INTEGERREFERENCES [namespaces]([id]),
[hash] TEXT,
[commit_at] TEXT
);
CREATE UNIQUE INDEX [idx_commits_namespace_hash]
ON [commits] ([namespace], [hash]);
CREATE TABLE [item] (
[IncidentID] TEXT,
[Location] TEXT,
[Type] TEXT
);
If you have 10 historic versions of the incidents.json file and each one contains 30 incidents, you will end up with 10 * 30 = 300 rows in your item table.
Track the history of individual items using IDs
If your objects have a unique identifier - or multiple columns that together form a unique identifier - you can use the --id option to de-duplicate and track changes to each of those items over time.
This provides a much more interesting way to apply this tool.
If there is a unique identifier column called IncidentID you could run the following:
The database schema used here is very different from the one used without the --id option.
If you have already imported history, the command will skip any commits that it has seen already and just process new ones. This means that even though an initial import could be slow subsequent imports should run a lot faster.
This command will create six tables - commits, item, item_version, columns, item_changed and namespaces.
Here's the full schema:
CREATE TABLE [namespaces] (
[id] INTEGERPRIMARY KEY,
[name] TEXT
);
CREATE UNIQUE INDEX [idx_namespaces_name]
ON [namespaces] ([name]);
CREATE TABLE [commits] (
[id] INTEGERPRIMARY KEY,
[namespace] INTEGERREFERENCES [namespaces]([id]),
[hash] TEXT,
[commit_at] TEXT
);
CREATE UNIQUE INDEX [idx_commits_namespace_hash]
ON [commits] ([namespace], [hash]);
CREATE TABLE [item] (
[_id] INTEGERPRIMARY KEY,
[_item_id] TEXT
, [IncidentID] TEXT, [Location] TEXT, [Type] TEXT, [_commit] INTEGER);
CREATE UNIQUE INDEX [idx_item__item_id]
ON [item] ([_item_id]);
CREATE TABLE [item_version] (
[_id] INTEGERPRIMARY KEY,
[_item] INTEGERREFERENCES [item]([_id]),
[_version] INTEGER,
[_commit] INTEGERREFERENCES [commits]([id]),
[IncidentID] TEXT,
[Location] TEXT,
[Type] TEXT,
[_item_full_hash] TEXT
);
CREATE TABLE [columns] (
[id] INTEGERPRIMARY KEY,
[namespace] INTEGERREFERENCES [namespaces]([id]),
[name] TEXT
);
CREATE UNIQUE INDEX [idx_columns_namespace_name]
ON [columns] ([namespace], [name]);
CREATE TABLE [item_changed] (
[item_version] INTEGERREFERENCES [item_version]([_id]),
[column] INTEGERREFERENCES [columns]([id]),
PRIMARY KEY ([item_version], [column])
);
CREATEVIEWitem_version_detailASselectcommits.commit_atas _commit_at,
commits.hashas _commit_hash,
item_version.*,
(
select json_group_array(name) from columns
where id in (
select column from item_changed
where item_version =item_version._id
)
) as _changed_columns
from item_version
join commits oncommits.id=item_version._commit;
CREATE INDEX [idx_item_version__item]
ON [item_version] ([_item]);
item table
The item table will contain the most recent version of each row, de-duplicated by ID, plus the following additional columns:
_id - a numeric integer primary key, used as a foreign key from the item_version table.
_item_id - a hash of the values of the columns specified using the --id option to the command. This is used for de-duplication when processing new versions.
_commit - a foreign key to the commit table, representing the most recent commit to modify this item.
item_version table
The item_version table will contain a row for each captured differing version of that item, plus the following columns:
_id - a numeric ID for the item version record.
_item - a foreign key to the item table.
_version - the numeric version number, starting at 1 and incrementing for each captured version.
_commit - a foreign key to the commit table.
_item_full_hash - a hash of this version of the item. This is used internally by the tool to identify items that have changed between commits.
The other columns in this table represent columns in the original data that have changed since the previous version. If the value has not changed, it will be represented by a null.
If a value was previously set but has been changed back to null it will still be represented as null in the item_version row. You can identify these using the item_changed many-to-many table described below.
You can use the --full-versions option to store full copies of the item at each version, rather than just storing the columns that have changed.
item_version_detail view
This SQL view joins item_version against commits to add three further columns: _commit_at with the date of the commit, and _commit_hash with the Git commit hash.
item_changed
This many-to-many table indicates exactly which columns were changed in an item_version.
item_version is a foreign key to a row in the item_version table.
column is a foreign key to a row in the columns table.
This table with have the largest number of rows, which is why it stores just two integers in order to save space.
columns
The columns table stores column names. It is referenced by item_changed.
id - an integer ID.
name - the name of the column.
namespace - a foreign key to namespaces, for if multiple file histories are sharing the same database.
Reserved column names
Note that _id, _item_full_hash, _item, _item_id, _version, _commit, _item_id, _commit_at, _commit_hash, _changed_columns, rowid are considered reserved column names for the purposes of this tool.
If your data contains any of these they will be renamed to add a trailing underscore, for example _id_, _item_, _version_, to avoid clashing with the reserved columns.
If you have a column with a name such as _commit_ it will be renamed too, adding an additional trailing underscore, so _commit_ becomes _commit__ and _commit__ becomes _commit___.
Additional options
--repo DIRECTORY - the path to the Git repository, if it is not the current working directory.
--branch TEXT - the Git branch to analyze - defaults to main.
--id TEXT - as described above: pass one or more columns that uniquely identify a record, so that changes to that record can be calculated over time.
--full-versions - instead of recording just the columns that have changed in the item_version table record a full copy of each version of theh item.
--ignore TEXT - one or more columns to ignore - they will not be included in the resulting database.
--csv - treat the data is CSV or TSV rather than JSON, and attempt to guess the correct dialect
--dialect - use a spcific CSV dialect. Options are excel, excel-tab and unix - see the Python CSV documentation for details.
--skip TEXT - one or more full Git commit hashes that should be skipped. You can use this if some of the data in your revision history is corrupted in a way that prevents this tool from working.
--start-at TEXT - skip commits prior to the specified commit hash.
--start-after TEXT - skip commits up to and including the specified commit hash, then start processing from the following commit.
--convert TEXT - custom Python code for a conversion, described below.
--import TEXT - additional Python modules to import for --convert.
--ignore-duplicate-ids - if a single version of a file has the same ID in it more than once, the tool will exit with an error. Use this option to ignore this and instead pick just the first of the two duplicates.
--namespace TEXT - use this if you wish to include the history of multiple different files in the same database. The default is item but you can set it to something else, which will produce tables with names like yournamespace and yournamespace_version.
--wal - Enable WAL mode on the created database file. Use this if you plan to run queries against the database while git-history is creating it.
--silent - don't show the progress bar.
CSV and TSV data
If the data in your repository is a CSV or TSV file you can process it by adding the --csv option. This will attempt to detect which delimiter is used by the file, so the same option works for both comma- and tab-separated values.
git-history file trees.db trees.csv --id TreeID
You can also specify the CSV dialect using the --dialect option.
Custom conversions using --convert
If your data is not already either CSV/TSV or a flat JSON array, you can reshape it using the --convert option.
The format needed by this tool is an array of dictionaries, as demonstrated by the incidents.json example above.
If your data does not fit this shape, you can provide a snippet of Python code to converts the on-disk content of each stored file into a Python list of dictionaries.
For example, if your stored files each look like this:
You can import nested modules such as ElementTree using --import xml.etree.ElementTree, then refer to them in your function body as xml.etree.ElementTree. For example, if your tracked data was in an items.xml file that looked like this:
You could load it using the following --convert script:
git-history file items.xml --convert '
tree = xml.etree.ElementTree.fromstring(content)
return [el.attrib for el in tree.iter(""item"")]
' --import xml.etree.ElementTree --id id
If your Python code spans more than one line it needs to include a return statement.
You can also use Python generators in your --convert code, for example:
git-history file stats.db package-stats/stats.json \
--repo package-stats \
--convert '
data = json.loads(content)
for key, counts in data.items():
for date, count in counts.items():
yield {
""package"": key,
""date"": date,
""count"": count
}
' --id package --id date
This conversion function expects data that looks like this:
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-pretty-traces
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
441024802,R_kgDOGkmBIg,datasette-tiddlywiki,simonw/datasette-tiddlywiki,0,9599,https://github.com/simonw/datasette-tiddlywiki,Run TiddlyWiki in Datasette and save Tiddlers to a SQLite database,0,2021-12-23T01:05:56Z,2022-02-14T08:57:33Z,2022-03-08T01:36:10Z,,426,22,22,HTML,1,1,1,1,0,0,0,0,3,apache-2.0,[],0,3,22,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,2,"# datasette-tiddlywiki
[](https://pypi.org/project/datasette-tiddlywiki/)
[](https://github.com/simonw/datasette-tiddlywiki/releases)
[](https://github.com/simonw/datasette-tiddlywiki/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-tiddlywiki/blob/main/LICENSE)
Run [TiddlyWiki](https://tiddlywiki.com/) in Datasette and save Tiddlers to a SQLite database
Read more about this project [on my blog](https://simonwillison.net/2021/Dec/24/datasette-tiddlywiki/).
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-tiddlywiki
## Usage
Start Datasette with a `tiddlywiki.db` database. You can create it if it does not yet exist using `--create`.
You need to be signed in as the `root` user to write to the wiki, so use the `--root` option and click on the link it provides:
% datasette tiddlywiki.db --create --root
http://127.0.0.1:8001/-/auth-token?token=456670f1e8d01a8a33b71e17653130de17387336e29afcdfb4ab3d18261e6630
# ...
Navigate to `/-/tiddlywiki` on your instance to interact with TiddlyWiki.
## Authentication and permissions
By default, the wiki can be read by anyone who has permission to read the `tiddlywiki.db` database. Only the signed in `root` user can write to it.
You can sign in using the `--root` option described above, or you can set a password for that user using the [datasette-auth-passwords](https://datasette.io/plugins/datasette-auth-passwords) plugin and sign in using the `/-/login` page.
You can use the `edit-tiddlywiki` permission to grant edit permisions to other users, using another plugin such as [datasette-permissions-sql](https://datasette.io/plugins/datasette-permissions-sql).
You can use the `view-database` permission against the `tiddlywiki` database to control who can view the wiki.
Datasette's permissions mechanism is described in full in [the Datasette documentation](https://docs.datasette.io/en/stable/authentication.html).
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-tiddlywiki
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-tiddlywiki
Run TiddlyWiki in Datasette and save Tiddlers to a SQLite database
Navigate to /-/tiddlywiki on your instance to interact with TiddlyWiki.
Authentication and permissions
By default, the wiki can be read by anyone who has permission to read the tiddlywiki.db database. Only the signed in root user can write to it.
You can sign in using the --root option described above, or you can set a password for that user using the datasette-auth-passwords plugin and sign in using the /-/login page.
You can use the edit-tiddlywiki permission to grant edit permisions to other users, using another plugin such as datasette-permissions-sql.
You can use the view-database permission against the tiddlywiki database to control who can view the wiki.
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-tiddlywiki
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
459821110,R_kgDOG2hQNg,google-drive-to-sqlite,simonw/google-drive-to-sqlite,0,9599,https://github.com/simonw/google-drive-to-sqlite,Create a SQLite database containing metadata from Google Drive,0,2022-02-16T02:16:29Z,2022-05-17T00:30:43Z,2022-05-21T16:56:11Z,https://datasette.io/tools/google-drive-to-sqlite,74,133,133,Python,1,1,1,1,0,11,0,0,9,apache-2.0,[],11,9,133,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,11,3,"# google-drive-to-sqlite
[](https://pypi.org/project/google-drive-to-sqlite/)
[](https://github.com/simonw/google-drive-to-sqlite/releases)
[](https://github.com/simonw/google-drive-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/google-drive-to-sqlite/blob/master/LICENSE)
Create a SQLite database containing metadata from [Google Drive](https://www.google.com/drive)
For background on this project, see [Google Drive to SQLite](https://simonwillison.net/2022/Feb/20/google-drive-to-sqlite/) on my blog.
If you use Google Drive, and especially if you have shared drives with other people there's a good chance you have hundreds or even thousands of files that you may not be fully aware of.
This tool can download metadata about those files - their names, sizes, folders, content types, permissions, creation dates and more - and store them in a SQLite database.
This lets you use SQL to analyze your Google Drive contents, using [Datasette](https://datasette.io/) or the SQLite command-line tool or any other SQLite database browsing software.
## Installation
Install this tool using `pip`:
pip install google-drive-to-sqlite
## Quickstart
Authenticate with Google Drive by running:
google-drive-to-sqlite auth
Now create a SQLite database with metadata about all of the files you have starred using:
google-drive-to-sqlite files starred.db --starred
You can explore the resulting database using [Datasette](https://datasette.io/):
$ pip install datasette
$ datasette starred.db
INFO: Started server process [24661]
INFO: Uvicorn running on http://127.0.0.1:8001
## Authentication
> :warning: **This application has not yet been verified by Google** - you may find you are unable to authenticate until that verification is complete. [#10](https://github.com/simonw/google-drive-to-sqlite/issues/10)
>
> You can work around this issue by [creating your own OAuth client ID key](https://til.simonwillison.net/googlecloud/google-oauth-cli-application) and passing it to the `auth` command using `--google-client-id` and `--google-client-secret`.
First, authenticate with Google Drive using the `auth` command:
$ google-drive-to-sqlite auth
Visit the following URL to authenticate with Google Drive
https://accounts.google.com/o/oauth2/v2/auth?...
Then return here and paste in the resulting code:
Paste code here:
Follow the link, sign in with Google Drive and then copy and paste the resulting code back into the tool.
This will save an authentication token to the file called `auth.json` in the current directory.
To specify a different location for that file, use the `--auth` option:
google-drive-to-sqlite auth --auth ~/google-drive-auth.json
The `auth` command also provides options for using a different scope, Google client ID and Google client secret. You can use these to create your own custom authentication tokens that can work with other Google APIs, see [issue #5](https://github.com/simonw/google-drive-to-sqlite/issues/5) for details.
Full `--help`:
```
Usage: google-drive-to-sqlite auth [OPTIONS]
Authenticate user and save credentials
Options:
-a, --auth FILE Path to save token, defaults to auth.json
--google-client-id TEXT Custom Google client ID
--google-client-secret TEXT Custom Google client secret
--scope TEXT Custom token scope
--help Show this message and exit.
```
To revoke the token that is stored in `auth.json`, such that it cannot be used to access Google Drive in the future, run the `revoke` command:
google-drive-to-sqlite revoke
Or if your token is stored in another location:
google-drive-to-sqlite revoke -a ~/google-drive-auth.json
You will need to obtain a fresh token using the `auth` command in order to continue using this tool.
## google-drive-to-sqlite files
To retrieve metadata about the files in your Google Drive, or a folder or search within it, use the `google-drive-to-sqlite files` command.
This will default to writing details about every file in your Google Drive to a SQLite database:
google-drive-to-sqlite files files.db
Files and folders will be written to databases tables, which will be created if they do not yet exist. The database schema is [shown below](#database-schema).
If a file or folder already exists, based on a matching `id`, it will be replaced with fresh data.
Instead of writing to SQLite you can use `--json` to output as JSON, or `--nl` to output as newline-delimited JSON:
google-drive-to-sqlite files --nl
Use `--folder ID` to retrieve everything in a specified folder and its sub-folders:
google-drive-to-sqlite files files.db --folder 1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i
Use `--q QUERY` to use a [custom search query](https://developers.google.com/drive/api/v3/reference/query-ref):
google-drive-to-sqlite files files.db -q ""viewedByMeTime > '2022-01-01'""
The following shortcut options help build queries:
- `--full-text TEXT` to search for files where the full text matches a search term
- `--starred` for files and folders you have starred
- `--trashed` for files and folders in the trash
- `--shared-with-me` for files and folders that have been shared with you
- `--apps` for Google Apps documents, spreadsheets, presentations and drawings (equivalent to setting all of the next four options)
- `--docs` for Google Apps documents
- `--sheets` for Google Apps spreadsheets
- `--presentations` for Google Apps presentations
- `--drawings` for Google Apps drawings
You can combine these - for example, this returns all files that you have starred and that were shared with you:
google-drive-to-sqlite files highlights.db \
--starred --shared-with-me
Multiple options are treated as AND, with the exception of the Google Apps options which are treated as OR - so the following would retrieve all spreadsheets and presentations that have also been starred:
google-drive-to-sqlite files highlights.db \
--starred --sheets --presentations
You can use `--stop-after X` to stop after retrieving X files, useful for trying out a new search pattern and seeing results straight away.
The `--import-json` and `--import-nl` options are mainly useful for testing and developing this tool. They allow you to replay the JSON or newline-delimited JSON that was previously fetched using `--json` or `--nl` and use it to create a fresh SQLite database, without needing to make any outbound API calls:
# Fetch all starred files from the API, write to starred.json
google-drive-to-sqlite files -q 'starred = true' --json > starred.json
# Now import that data into a new SQLite database file
google-drive-to-sqlite files starred.db --import-json starred.json
Full `--help`:
```
Usage: google-drive-to-sqlite files [OPTIONS] [DATABASE]
Retrieve metadata for files in Google Drive, and write to a SQLite database or
output as JSON.
google-drive-to-sqlite files files.db
Use --json to output JSON, --nl for newline-delimited JSON:
google-drive-to-sqlite files files.db --json
Use a folder ID to recursively fetch every file in that folder and its sub-
folders:
google-drive-to-sqlite files files.db --folder
1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i
Fetch files you have starred:
google-drive-to-sqlite files starred.db --starred
Options:
-a, --auth FILE Path to auth.json token file
--folder TEXT Files in this folder ID and its sub-folders
-q TEXT Files matching this query
--full-text TEXT Search for files with text match
--starred Files you have starred
--trashed Files in the trash
--shared-with-me Files that have been shared with you
--apps Google Apps docs, spreadsheets, presentations and
drawings
--docs Google Apps docs
--sheets Google Apps spreadsheets
--presentations Google Apps presentations
--drawings Google Apps drawings
--json Output JSON rather than write to DB
--nl Output newline-delimited JSON rather than write to DB
--stop-after INTEGER Stop paginating after X results
--import-json FILE Import from this JSON file instead of the API
--import-nl FILE Import from this newline-delimited JSON file
-v, --verbose Send verbose output to stderr
--help Show this message and exit.
```
## google-drive-to-sqlite download FILE_ID
The `download` command can be used to download files from Google Drive.
You'll need one or more file IDs, which look something like `0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB`.
To download the file, run this:
google-drive-to-sqlite download 0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB
This will detect the content type of the file and use that as the extension - so if this file is a JPEG the file would be downloaded as:
0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB.jpeg
You can pass multiple file IDs to the command at once.
To hide the progress bar and filename output, use `-s` or `--silent`.
If you are downloading a single file you can use the `-o` output to specify a filename and location:
google-drive-to-sqlite download 0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB \
-o my-image.jpeg
Use `-o -` to write the file contents to standard output:
google-drive-to-sqlite download 0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB \
-o - > my-image.jpeg
Full `--help`:
```
Usage: google-drive-to-sqlite download [OPTIONS] FILE_IDS...
Download one or more files to disk, based on their file IDs.
The file content will be saved to a file with the name:
FILE_ID.ext
Where the extension is automatically picked based on the type of file.
If you are downloading a single file you can specify a filename with -o:
google-drive-to-sqlite download MY_FILE_ID -o myfile.txt
Options:
-a, --auth FILE Path to auth.json token file
-o, --output FILE File to write to, or - for standard output
-s, --silent Hide progress bar and filename
--help Show this message and exit.
```
## google-drive-to-sqlite export FORMAT FILE_ID
The `export` command can be used to export Google Docs documents, spreadsheets and presentations in a number of different formats.
You'll need one or more document IDs, which look something like `10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU`. You can find these by looking at the URL of your document on the Google Docs site.
To export that document as PDF, run this:
google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU
The file will be exported as:
10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU-export.pdf
You can pass multiple file IDs to the command at once.
For the `FORMAT` option you can use any of the mime type options listed [on this page](https://developers.google.com/drive/api/v3/ref-export-formats) - for example, to export as an Open Office document you could use:
google-drive-to-sqlite export \
application/vnd.oasis.opendocument.text \
10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU
For convenience the following shortcuts for common file formats are provided:
- Google Docs: `html`, `txt`, `rtf`, `pdf`, `doc`, `zip`, `epub`
- Google Sheets: `xls`, `pdf`, `csv`, `tsv`, `zip`
- Presentations: `ppt`, `pdf`, `txt`
- Drawings: `jpeg`, `png`, `svg`
The `zip` option returns a zip file of HTML. `txt` returns plain text. The others should be self-evident.
To hide the filename output, use `-s` or `--silent`.
If you are exporting a single file you can use the `-o` output to specify a filename and location:
google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU \
-o my-document.pdf
Use `-o -` to write the file contents to standard output:
google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU \
-o - > my-document.pdf
Full `--help`:
```
Usage: google-drive-to-sqlite export [OPTIONS] FORMAT FILE_IDS...
Export one or more files to the specified format.
Usage:
google-drive-to-sqlite export pdf FILE_ID_1 FILE_ID_2
The file content will be saved to a file with the name:
FILE_ID-export.ext
Where the extension is based on the format you specified.
Available export formats can be seen here:
https://developers.google.com/drive/api/v3/ref-export-formats
Or you can use one of the following shortcuts:
- Google Docs: html, txt, rtf, pdf, doc, zip, epub
- Google Sheets: xls, pdf, csv, tsv, zip
- Presentations: ppt, pdf, txt
- Drawings: jpeg, png, svg
""zip"" returns a zip file of HTML.
If you are exporting a single file you can specify a filename with -o:
google-drive-to-sqlite export zip MY_FILE_ID -o myfile.zip
Options:
-a, --auth FILE Path to auth.json token file
-o, --output FILE File to write to, or - for standard output
-s, --silent Hide progress bar and filename
--help Show this message and exit.
```
## google-drive-to-sqlite get URL
The `get` command makes authenticated requests to the specified URL, using credentials derived from the `auth.json` file.
For example:
$ google-drive-to-sqlite get 'https://www.googleapis.com/drive/v3/about?fields=*'
{
""kind"": ""drive#about"",
""user"": {
""kind"": ""drive#user"",
""displayName"": ""Simon Willison"",
# ...
If the resource you are fetching supports pagination you can use `--paginate key` to paginate through all of the rows in a specified key. For example, the following API has a `nextPageToken` key and a `files` list, suggesting it supports pagination:
$ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files
{
""kind"": ""drive#fileList"",
""nextPageToken"": ""~!!~AI9...wogHHYlc="",
""incompleteSearch"": false,
""files"": [
{
""kind"": ""drive#file"",
""id"": ""1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg"",
""name"": ""Title of a spreadsheet"",
""mimeType"": ""application/vnd.google-apps.spreadsheet""
},
To paginate through everything in the `files` list you would use `--paginate files` like this:
$ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files --paginate files
[
{
""kind"": ""drive#file"",
""id"": ""1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg"",
""name"": ""Title of a spreadsheet"",
""mimeType"": ""application/vnd.google-apps.spreadsheet""
},
# ...
Add `--nl` to stream paginated data as newline-delimited JSON:
$ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files --paginate files --nl
{""kind"": ""drive#file"", ""id"": ""1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg"", ""name"": ""Title of a spreadsheet"", ""mimeType"": ""application/vnd.google-apps.spreadsheet""}
{""kind"": ""drive#file"", ""id"": ""1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i"", ""name"": ""Subfolder"", ""mimeType"": ""application/vnd.google-apps.folder""}
Add `--stop-after 5` to stop after 5 records - useful for testing.
Full `--help`:
```
Usage: google-drive-to-sqlite get [OPTIONS] URL
Make an authenticated HTTP GET to the specified URL
Options:
-a, --auth FILE Path to auth.json token file
--paginate TEXT Paginate through all results in this key
--nl Output paginated data as newline-delimited JSON
--stop-after INTEGER Stop paginating after X results
-v, --verbose Send verbose output to stderr
--help Show this message and exit.
```
## Database schema
The database created by this tool has the following schema:
```sql
CREATE TABLE [drive_users] (
[permissionId] TEXT PRIMARY KEY,
[kind] TEXT,
[displayName] TEXT,
[photoLink] TEXT,
[me] INTEGER,
[emailAddress] TEXT
);
CREATE TABLE [drive_folders] (
[id] TEXT PRIMARY KEY,
[_parent] TEXT,
[_owner] TEXT,
[lastModifyingUser] TEXT,
[kind] TEXT,
[name] TEXT,
[mimeType] TEXT,
[starred] INTEGER,
[trashed] INTEGER,
[explicitlyTrashed] INTEGER,
[parents] TEXT,
[spaces] TEXT,
[version] TEXT,
[webViewLink] TEXT,
[iconLink] TEXT,
[hasThumbnail] INTEGER,
[thumbnailVersion] TEXT,
[viewedByMe] INTEGER,
[createdTime] TEXT,
[modifiedTime] TEXT,
[modifiedByMe] INTEGER,
[shared] INTEGER,
[ownedByMe] INTEGER,
[viewersCanCopyContent] INTEGER,
[copyRequiresWriterPermission] INTEGER,
[writersCanShare] INTEGER,
[folderColorRgb] TEXT,
[quotaBytesUsed] TEXT,
[isAppAuthorized] INTEGER,
[linkShareMetadata] TEXT,
FOREIGN KEY([_parent]) REFERENCES [drive_folders]([id]),
FOREIGN KEY([_owner]) REFERENCES [drive_users]([permissionId]),
FOREIGN KEY([lastModifyingUser]) REFERENCES [drive_users]([permissionId])
);
CREATE TABLE [drive_files] (
[id] TEXT PRIMARY KEY,
[_parent] TEXT,
[_owner] TEXT,
[lastModifyingUser] TEXT,
[kind] TEXT,
[name] TEXT,
[mimeType] TEXT,
[starred] INTEGER,
[trashed] INTEGER,
[explicitlyTrashed] INTEGER,
[parents] TEXT,
[spaces] TEXT,
[version] TEXT,
[webViewLink] TEXT,
[iconLink] TEXT,
[hasThumbnail] INTEGER,
[thumbnailVersion] TEXT,
[viewedByMe] INTEGER,
[createdTime] TEXT,
[modifiedTime] TEXT,
[modifiedByMe] INTEGER,
[shared] INTEGER,
[ownedByMe] INTEGER,
[viewersCanCopyContent] INTEGER,
[copyRequiresWriterPermission] INTEGER,
[writersCanShare] INTEGER,
[quotaBytesUsed] TEXT,
[isAppAuthorized] INTEGER,
[linkShareMetadata] TEXT,
FOREIGN KEY([_parent]) REFERENCES [drive_folders]([id]),
FOREIGN KEY([_owner]) REFERENCES [drive_users]([permissionId]),
FOREIGN KEY([lastModifyingUser]) REFERENCES [drive_users]([permissionId])
);
```
## Thumbnails
You can construct a thumbnail image for a known file ID using the following URL:
https://drive.google.com/thumbnail?sz=w800-h800&id=FILE_ID
Users who are signed into Google Drive and have permission to view a file will be redirected to a thumbnail version of that file. You can tweak the `w800` and `h800` parameters to request different thumbnail sizes.
## Privacy policy
This tool requests access to your Google Drive account in order to retrieve metadata about your files there. It also offers a feature that can download the content of those files.
The credentials used to access your account are stored in the `auth.json` file on your computer. The metadata and content retrieved from Google Drive is also stored only on your own personal computer.
At no point do the developers of this tool gain access to any of your data.
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd google-drive-to-sqlite
python -m venv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
google-drive-to-sqlite
Create a SQLite database containing metadata from Google Drive
If you use Google Drive, and especially if you have shared drives with other people there's a good chance you have hundreds or even thousands of files that you may not be fully aware of.
This tool can download metadata about those files - their names, sizes, folders, content types, permissions, creation dates and more - and store them in a SQLite database.
This lets you use SQL to analyze your Google Drive contents, using Datasette or the SQLite command-line tool or any other SQLite database browsing software.
Installation
Install this tool using pip:
pip install google-drive-to-sqlite
Quickstart
Authenticate with Google Drive by running:
google-drive-to-sqlite auth
Now create a SQLite database with metadata about all of the files you have starred using:
google-drive-to-sqlite files starred.db --starred
You can explore the resulting database using Datasette:
$ pip install datasette
$ datasette starred.db
INFO: Started server process [24661]
INFO: Uvicorn running on http://127.0.0.1:8001
Authentication
⚠️This application has not yet been verified by Google - you may find you are unable to authenticate until that verification is complete. #10
You can work around this issue by creating your own OAuth client ID key and passing it to the auth command using --google-client-id and --google-client-secret.
First, authenticate with Google Drive using the auth command:
$ google-drive-to-sqlite auth
Visit the following URL to authenticate with Google Drive
https://accounts.google.com/o/oauth2/v2/auth?...
Then return here and paste in the resulting code:
Paste code here:
Follow the link, sign in with Google Drive and then copy and paste the resulting code back into the tool.
This will save an authentication token to the file called auth.json in the current directory.
To specify a different location for that file, use the --auth option:
The auth command also provides options for using a different scope, Google client ID and Google client secret. You can use these to create your own custom authentication tokens that can work with other Google APIs, see issue #5 for details.
Full --help:
Usage: google-drive-to-sqlite auth [OPTIONS]
Authenticate user and save credentials
Options:
-a, --auth FILE Path to save token, defaults to auth.json
--google-client-id TEXT Custom Google client ID
--google-client-secret TEXT Custom Google client secret
--scope TEXT Custom token scope
--help Show this message and exit.
To revoke the token that is stored in auth.json, such that it cannot be used to access Google Drive in the future, run the revoke command:
google-drive-to-sqlite revoke
Or if your token is stored in another location:
google-drive-to-sqlite revoke -a ~/google-drive-auth.json
You will need to obtain a fresh token using the auth command in order to continue using this tool.
google-drive-to-sqlite files
To retrieve metadata about the files in your Google Drive, or a folder or search within it, use the google-drive-to-sqlite files command.
This will default to writing details about every file in your Google Drive to a SQLite database:
google-drive-to-sqlite files files.db
Files and folders will be written to databases tables, which will be created if they do not yet exist. The database schema is shown below.
If a file or folder already exists, based on a matching id, it will be replaced with fresh data.
Instead of writing to SQLite you can use --json to output as JSON, or --nl to output as newline-delimited JSON:
google-drive-to-sqlite files --nl
Use --folder ID to retrieve everything in a specified folder and its sub-folders:
Multiple options are treated as AND, with the exception of the Google Apps options which are treated as OR - so the following would retrieve all spreadsheets and presentations that have also been starred:
You can use --stop-after X to stop after retrieving X files, useful for trying out a new search pattern and seeing results straight away.
The --import-json and --import-nl options are mainly useful for testing and developing this tool. They allow you to replay the JSON or newline-delimited JSON that was previously fetched using --json or --nl and use it to create a fresh SQLite database, without needing to make any outbound API calls:
# Fetch all starred files from the API, write to starred.json
google-drive-to-sqlite files -q 'starred = true' --json > starred.json
# Now import that data into a new SQLite database file
google-drive-to-sqlite files starred.db --import-json starred.json
Full --help:
Usage: google-drive-to-sqlite files [OPTIONS] [DATABASE]
Retrieve metadata for files in Google Drive, and write to a SQLite database or
output as JSON.
google-drive-to-sqlite files files.db
Use --json to output JSON, --nl for newline-delimited JSON:
google-drive-to-sqlite files files.db --json
Use a folder ID to recursively fetch every file in that folder and its sub-
folders:
google-drive-to-sqlite files files.db --folder
1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i
Fetch files you have starred:
google-drive-to-sqlite files starred.db --starred
Options:
-a, --auth FILE Path to auth.json token file
--folder TEXT Files in this folder ID and its sub-folders
-q TEXT Files matching this query
--full-text TEXT Search for files with text match
--starred Files you have starred
--trashed Files in the trash
--shared-with-me Files that have been shared with you
--apps Google Apps docs, spreadsheets, presentations and
drawings
--docs Google Apps docs
--sheets Google Apps spreadsheets
--presentations Google Apps presentations
--drawings Google Apps drawings
--json Output JSON rather than write to DB
--nl Output newline-delimited JSON rather than write to DB
--stop-after INTEGER Stop paginating after X results
--import-json FILE Import from this JSON file instead of the API
--import-nl FILE Import from this newline-delimited JSON file
-v, --verbose Send verbose output to stderr
--help Show this message and exit.
google-drive-to-sqlite download FILE_ID
The download command can be used to download files from Google Drive.
You'll need one or more file IDs, which look something like 0B32uDVNZfiEKLUtIT1gzYWN2NDI4SzVQYTFWWWxCWUtvVGNB.
Usage: google-drive-to-sqlite download [OPTIONS] FILE_IDS...
Download one or more files to disk, based on their file IDs.
The file content will be saved to a file with the name:
FILE_ID.ext
Where the extension is automatically picked based on the type of file.
If you are downloading a single file you can specify a filename with -o:
google-drive-to-sqlite download MY_FILE_ID -o myfile.txt
Options:
-a, --auth FILE Path to auth.json token file
-o, --output FILE File to write to, or - for standard output
-s, --silent Hide progress bar and filename
--help Show this message and exit.
google-drive-to-sqlite export FORMAT FILE_ID
The export command can be used to export Google Docs documents, spreadsheets and presentations in a number of different formats.
You'll need one or more document IDs, which look something like 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU. You can find these by looking at the URL of your document on the Google Docs site.
To export that document as PDF, run this:
google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU
For convenience the following shortcuts for common file formats are provided:
Google Docs: html, txt, rtf, pdf, doc, zip, epub
Google Sheets: xls, pdf, csv, tsv, zip
Presentations: ppt, pdf, txt
Drawings: jpeg, png, svg
The zip option returns a zip file of HTML. txt returns plain text. The others should be self-evident.
To hide the filename output, use -s or --silent.
If you are exporting a single file you can use the -o output to specify a filename and location:
google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU \
-o my-document.pdf
Use -o - to write the file contents to standard output:
google-drive-to-sqlite export pdf 10BOHGDUYa7lBjUSo26YFCHTpgEmtXabdVFaopCTh1vU \
-o - > my-document.pdf
Full --help:
Usage: google-drive-to-sqlite export [OPTIONS] FORMAT FILE_IDS...
Export one or more files to the specified format.
Usage:
google-drive-to-sqlite export pdf FILE_ID_1 FILE_ID_2
The file content will be saved to a file with the name:
FILE_ID-export.ext
Where the extension is based on the format you specified.
Available export formats can be seen here:
https://developers.google.com/drive/api/v3/ref-export-formats
Or you can use one of the following shortcuts:
- Google Docs: html, txt, rtf, pdf, doc, zip, epub
- Google Sheets: xls, pdf, csv, tsv, zip
- Presentations: ppt, pdf, txt
- Drawings: jpeg, png, svg
""zip"" returns a zip file of HTML.
If you are exporting a single file you can specify a filename with -o:
google-drive-to-sqlite export zip MY_FILE_ID -o myfile.zip
Options:
-a, --auth FILE Path to auth.json token file
-o, --output FILE File to write to, or - for standard output
-s, --silent Hide progress bar and filename
--help Show this message and exit.
google-drive-to-sqlite get URL
The get command makes authenticated requests to the specified URL, using credentials derived from the auth.json file.
If the resource you are fetching supports pagination you can use --paginate key to paginate through all of the rows in a specified key. For example, the following API has a nextPageToken key and a files list, suggesting it supports pagination:
$ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files
{
""kind"": ""drive#fileList"",
""nextPageToken"": ""~!!~AI9...wogHHYlc="",
""incompleteSearch"": false,
""files"": [
{
""kind"": ""drive#file"",
""id"": ""1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg"",
""name"": ""Title of a spreadsheet"",
""mimeType"": ""application/vnd.google-apps.spreadsheet""
},
To paginate through everything in the files list you would use --paginate files like this:
$ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files --paginate files
[
{
""kind"": ""drive#file"",
""id"": ""1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg"",
""name"": ""Title of a spreadsheet"",
""mimeType"": ""application/vnd.google-apps.spreadsheet""
},
# ...
Add --nl to stream paginated data as newline-delimited JSON:
$ google-drive-to-sqlite get https://www.googleapis.com/drive/v3/files --paginate files --nl
{""kind"": ""drive#file"", ""id"": ""1YEsITp_X8PtDUJWHGM0osT-TXAU1nr0e7RSWRM2Jpyg"", ""name"": ""Title of a spreadsheet"", ""mimeType"": ""application/vnd.google-apps.spreadsheet""}
{""kind"": ""drive#file"", ""id"": ""1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i"", ""name"": ""Subfolder"", ""mimeType"": ""application/vnd.google-apps.folder""}
Add --stop-after 5 to stop after 5 records - useful for testing.
Full --help:
Usage: google-drive-to-sqlite get [OPTIONS] URL
Make an authenticated HTTP GET to the specified URL
Options:
-a, --auth FILE Path to auth.json token file
--paginate TEXT Paginate through all results in this key
--nl Output paginated data as newline-delimited JSON
--stop-after INTEGER Stop paginating after X results
-v, --verbose Send verbose output to stderr
--help Show this message and exit.
Database schema
The database created by this tool has the following schema:
Users who are signed into Google Drive and have permission to view a file will be redirected to a thumbnail version of that file. You can tweak the w800 and h800 parameters to request different thumbnail sizes.
Privacy policy
This tool requests access to your Google Drive account in order to retrieve metadata about your files there. It also offers a feature that can download the content of those files.
The credentials used to access your account are stored in the auth.json file on your computer. The metadata and content retrieved from Google Drive is also stored only on your own personal computer.
At no point do the developers of this tool gain access to any of your data.
Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd google-drive-to-sqlite
python -m venv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
461322238,R_kgDOG383_g,sqlite-colorbrewer,eyeseast/sqlite-colorbrewer,0,25778,https://github.com/eyeseast/sqlite-colorbrewer,A custom function to use ColorBrewer scales in SQLite queries,0,2022-02-19T21:53:46Z,2022-03-03T17:16:40Z,2022-03-02T03:04:56Z,,19,4,4,Python,1,1,1,1,0,0,0,0,0,apache-2.0,[],0,0,4,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# sqlite-colorbrewer
[](https://pypi.org/project/sqlite-colorbrewer/)
[](https://github.com/eyeseast/sqlite-colorbrewer/releases)
[](https://github.com/eyeseast/sqlite-colorbrewer/actions?query=workflow%3ATest)
[](https://github.com/eyeseast/sqlite-colorbrewer/blob/main/LICENSE)
A custom function to use [ColorBrewer](https://colorbrewer2.org/) scales in SQLite queries.
Colors are exported from [here](https://colorbrewer2.org/export/colorbrewer.json).
## Installation
To install as a Python library and use with the [standard SQLite3 module](https://docs.python.org/3/library/sqlite3.html):
pip install sqlite-colorbrewer
To install this plugin in the same environment as Datasette.
datasette install sqlite-colorbrewer
## Usage
If you're using this library with Datasette, it will be automatically registered as a plugin and available for use in SQL queries, like so:
```sql
SELECT colorbrewer('Blues', 9, 0);
```
That will return a single value: `""rgb(247,251,255)""`
To use with a SQLite connection outside of Datasette, use the `register` function:
```python
>>> import sqlite3
>>> import sqlite_colorbrewer
>>> conn = sqlite3.connect(':memory')
>>> sqlite_colorbrewer.register(conn)
>>> cursor = conn.execute(""SELECT colorbrewer('Blues', 9, 0);"")
>>> result = next(cursor)
>>> print(result)
rgb(247,251,255)
```
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd sqlite-colorbrewer
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
To build `sqlite_colorbrewer/colorbrewer.py`:
./json_to_python.py
black . # to format the resulting file
## ColorBrewer
Copyright (c) 2002 Cynthia Brewer, Mark Harrower, and The Pennsylvania State University.
Licensed under the Apache License, Version 2.0 (the ""License""); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an ""AS IS"" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
See the [ColorBrewer updates](http://www.personal.psu.edu/cab38/ColorBrewer/ColorBrewer_updates.html) for updates to copyright information.
","
sqlite-colorbrewer
A custom function to use ColorBrewer scales in SQLite queries.
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd sqlite-colorbrewer
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
To build sqlite_colorbrewer/colorbrewer.py:
./json_to_python.py
black . # to format the resulting file
ColorBrewer
Copyright (c) 2002 Cynthia Brewer, Mark Harrower, and The Pennsylvania State University.
Licensed under the Apache License, Version 2.0 (the ""License""); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an ""AS IS"" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
",1,public,0,,,
462903750,R_kgDOG5dZxg,datasette-redirect-forbidden,simonw/datasette-redirect-forbidden,0,9599,https://github.com/simonw/datasette-redirect-forbidden,Redirect forbidden requests to a login page,0,2022-02-23T20:59:26Z,2022-02-23T22:00:12Z,2022-02-23T22:02:38Z,,7,0,0,Python,1,1,1,1,0,0,0,0,1,apache-2.0,[],0,1,0,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-redirect-forbidden
[](https://pypi.org/project/datasette-redirect-forbidden/)
[](https://github.com/simonw/datasette-redirect-forbidden/releases)
[](https://github.com/simonw/datasette-redirect-forbidden/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-redirect-forbidden/blob/main/LICENSE)
Redirect forbidden requests to a login page
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-redirect-forbidden
## Usage
Add the following to your `metadata.yml` (or `metadata.json`) file to configure the plugin:
```yaml
plugins:
datasette-redirect-forbidden:
redirect_to: /-/login
```
Any 403 forbidden pages will redirect to the specified page.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-redirect-forbidden
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-redirect-forbidden
Redirect forbidden requests to a login page
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-redirect-forbidden
Usage
Add the following to your metadata.yml (or metadata.json) file to configure the plugin:
Any 403 forbidden pages will redirect to the specified page.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-redirect-forbidden
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,"{""id"": 400878073, ""node_id"": ""MDEwOlJlcG9zaXRvcnk0MDA4NzgwNzM="", ""name"": ""datasette-plugin-template-repository"", ""full_name"": ""simonw/datasette-plugin-template-repository"", ""private"": false, ""owner"": {""login"": ""simonw"", ""id"": 9599, ""node_id"": ""MDQ6VXNlcjk1OTk="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/9599?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/simonw"", ""html_url"": ""https://github.com/simonw"", ""followers_url"": ""https://api.github.com/users/simonw/followers"", ""following_url"": ""https://api.github.com/users/simonw/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/simonw/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/simonw/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/simonw/subscriptions"", ""organizations_url"": ""https://api.github.com/users/simonw/orgs"", ""repos_url"": ""https://api.github.com/users/simonw/repos"", ""events_url"": ""https://api.github.com/users/simonw/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/simonw/received_events"", ""type"": ""User"", ""site_admin"": false}, ""html_url"": ""https://github.com/simonw/datasette-plugin-template-repository"", ""description"": ""GitHub template repository for creating new Datasette plugins, using the simonw/datasette-plugin cookiecutter template"", ""fork"": false, ""url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository"", ""forks_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/forks"", ""keys_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/keys{/key_id}"", ""collaborators_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/collaborators{/collaborator}"", ""teams_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/teams"", ""hooks_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/hooks"", ""issue_events_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/events{/number}"", ""events_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/events"", ""assignees_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/assignees{/user}"", ""branches_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/branches{/branch}"", ""tags_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/tags"", ""blobs_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/blobs{/sha}"", ""git_tags_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/tags{/sha}"", ""git_refs_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/refs{/sha}"", ""trees_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/trees{/sha}"", ""statuses_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/statuses/{sha}"", ""languages_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/languages"", ""stargazers_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/stargazers"", ""contributors_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/contributors"", ""subscribers_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscribers"", ""subscription_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/subscription"", ""commits_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/commits{/sha}"", ""git_commits_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/git/commits{/sha}"", ""comments_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/comments{/number}"", ""issue_comment_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues/comments{/number}"", ""contents_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/contents/{+path}"", ""compare_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/compare/{base}...{head}"", ""merges_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/merges"", ""archive_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/{archive_format}{/ref}"", ""downloads_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/downloads"", ""issues_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/issues{/number}"", ""pulls_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/pulls{/number}"", ""milestones_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/milestones{/number}"", ""notifications_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/notifications{?since,all,participating}"", ""labels_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/labels{/name}"", ""releases_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/releases{/id}"", ""deployments_url"": ""https://api.github.com/repos/simonw/datasette-plugin-template-repository/deployments"", ""created_at"": ""2021-08-28T19:50:28Z"", ""updated_at"": ""2021-12-21T20:45:06Z"", ""pushed_at"": ""2021-12-21T20:45:02Z"", ""git_url"": ""git://github.com/simonw/datasette-plugin-template-repository.git"", ""ssh_url"": ""git@github.com:simonw/datasette-plugin-template-repository.git"", ""clone_url"": ""https://github.com/simonw/datasette-plugin-template-repository.git"", ""svn_url"": ""https://github.com/simonw/datasette-plugin-template-repository"", ""homepage"": """", ""size"": 6, ""stargazers_count"": 12, ""watchers_count"": 12, ""language"": null, ""has_issues"": true, ""has_projects"": true, ""has_downloads"": true, ""has_wiki"": true, ""has_pages"": false, ""forks_count"": 0, ""mirror_url"": null, ""archived"": false, ""disabled"": false, ""open_issues_count"": 0, ""license"": null, ""allow_forking"": true, ""is_template"": true, ""topics"": [], ""visibility"": ""public"", ""forks"": 0, ""open_issues"": 0, ""watchers"": 12, ""default_branch"": ""main"", ""permissions"": {""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}, ""temp_clone_token"": """"}",,
467679579,R_kgDOG-A5Ww,shot-scraper,simonw/shot-scraper,0,9599,https://github.com/simonw/shot-scraper,A command-line utility for taking automated screenshots of websites,0,2022-03-08T21:21:02Z,2022-11-15T14:38:08Z,2022-11-16T04:28:52Z,https://shot-scraper.datasette.io,188,775,775,Python,1,1,1,1,0,40,0,0,16,apache-2.0,"[""playwright"", ""playwright-python"", ""scraping"", ""screenshot-utility"", ""screenshots""]",40,16,775,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,40,3,"# shot-scraper
[](https://pypi.org/project/shot-scraper/)
[](https://github.com/simonw/shot-scraper/releases)
[](https://github.com/simonw/shot-scraper/actions?query=workflow%3ATest)
[](https://github.com/simonw/shot-scraper/blob/master/LICENSE)
A command-line utility for taking automated screenshots of websites
For background on this project see [shot-scraper: automated screenshots for documentation, built on Playwright](https://simonwillison.net/2022/Mar/10/shot-scraper/).
## Documentation
- [Full documentation for shot-scraper](https://shot-scraper.datasette.io/)
- [Tutorial: Automating screenshots for the Datasette documentation using shot-scraper](https://simonwillison.net/2022/Oct/14/automating-screenshots/)
- [Release notes](https://github.com/simonw/shot-scraper/releases)
## Get started with GitHub Actions
To get started without installing any software, use the [shot-scraper-template](https://github.com/simonw/shot-scraper-template) template to create your own GitHub repository which takes screenshots of a page using `shot-scraper`. See [Instantly create a GitHub repository to take screenshots of a web page](https://simonwillison.net/2022/Mar/14/shot-scraper-template/) for details.
## Quick installation
You can install the `shot-scraper` CLI tool using [pip](https://pip.pypa.io/):
pip install shot-scraper
# Now install the browser it needs:
shot-scraper install
## Taking your first screenshot
You can take a screenshot of a web page like this:
shot-scraper https://datasette.io/
This will create a screenshot in a file called `datasette-io.png`.
Many more options are available, see [Taking a screenshot](https://shot-scraper.datasette.io/en/stable/screenshots.html) for details.
## Examples
- The [shot-scraper-demo](https://github.com/simonw/shot-scraper-demo) repository uses this tool to capture recently spotted owls in El Granada, CA according to [this page](https://www.owlsnearme.com/?place=127871), and to generate an annotated screenshot illustrating a Datasette feature as described [in my blog](https://simonwillison.net/2022/Mar/10/shot-scraper/#a-complex-example).
- The [Datasette Documentation](https://docs.datasette.io/en/latest/) uses screenshots taken by `shot-scraper` running in the [simonw/datasette-screenshots](https://github.com/simonw/datasette-screenshots) GitHub repository, described in detail in [Automating screenshots for the Datasette documentation using shot-scraper](https://simonwillison.net/2022/Oct/14/automating-screenshots/).
- Ben Welsh built [@newshomepages](https://twitter.com/newshomepages), a Twitter bot that uses `shot-scraper` and GitHub Actions to take screenshots of news website homepages and publish them to Twitter. The code for that lives in [palewire/news-homepages](https://github.com/palewire/news-homepages).
- [scrape-hacker-news-by-domain](https://github.com/simonw/scrape-hacker-news-by-domain) uses `shot-scraper javascript` to scrape a web page. See [Scraping web pages from the command-line with shot-scraper](https://simonwillison.net/2022/Mar/14/scraping-web-pages-shot-scraper/) for details of how this works.
","
shot-scraper
A command-line utility for taking automated screenshots of websites
The shot-scraper-demo repository uses this tool to capture recently spotted owls in El Granada, CA according to this page, and to generate an annotated screenshot illustrating a Datasette feature as described in my blog.
Ben Welsh built @newshomepages, a Twitter bot that uses shot-scraper and GitHub Actions to take screenshots of news website homepages and publish them to Twitter. The code for that lives in palewire/news-homepages.
",1,public,0,,0,0
470338069,R_kgDOHAjKFQ,datasette-hashed-urls,simonw/datasette-hashed-urls,0,9599,https://github.com/simonw/datasette-hashed-urls,Optimize Datasette performance behind a caching proxy,0,2022-03-15T21:31:52Z,2022-03-17T03:00:34Z,2022-03-24T17:58:05Z,,38,3,3,Python,1,1,1,1,0,0,0,0,0,apache-2.0,[],0,0,3,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-hashed-urls
[](https://pypi.org/project/datasette-hashed-urls/)
[](https://github.com/simonw/datasette-hashed-urls/releases)
[](https://github.com/simonw/datasette-hashed-urls/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-hashed-urls/blob/main/LICENSE)
Optimize Datasette performance behind a caching proxy
When you open a database file in immutable mode using the `-i` option, Datasette calculates a SHA-256 hash of the contents of that file on startup.
This content hash can then optionally be used to create URLs that are guaranteed to change if the contents of the file changes in the future.
The result is pages that can be cached indefinitely by both browsers and caching proxies - providing a significant performance boost.
## Demo
A demo of this plugin is running at https://datasette-hashed-urls.vercel.app/
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-hashed-urls
## Usage
Once installed, this plugin will act on any immutable database files that are loaded into Datasette:
datasette -i fixtures.db
The database will automatically be renamed to incorporate a hash of the contents of the SQLite file - so the above database would be served as:
http://127.0.0.1:8001/fixtures-aa7318b
Every page that accesss that database, including JSON endpoints, will be served with the following far-future cache expiry header:
cache-control: max-age=31536000, public
Here `max-age=31536000` is the number of seconds in a year.
A caching proxy such as Cloudflare can then be used to cache and accelerate content served by Datasette.
When the database file is updated and the server is restarted, the hash will change and content will be served from a new URL. Any hits to the previous hashed URLs will be automatically redirected.
If you run Datasette using the `--crossdb` option to enable [cross-database queries](https://docs.datasette.io/en/stable/sql_queries.html#cross-database-queries) the `_memory` database will also have a hash added to its URL - in this case, the hash will be a combination of the hashes of the other attached databases.
## Configuration
You can use the `max_age` plugin configuration setting to change the cache duration specified in the `cache-control` HTTP header.
To set the cache expiry time to one hour you would add this to your Datasette `metadata.json` configuration file:
```json
{
""plugins"": {
""datasette-hashed-urls"": {
""max_age"": 3600
}
}
}
```
## History
This functionality used to ship as part of Datasette itself, as a feature called [Hashed URL mode](https://docs.datasette.io/en/0.60.2/performance.html#hashed-url-mode).
That feature has been deprecated and will be removed in Datasette 1.0. This plugin should be used as an alternative.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-hashed-urls
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-hashed-urls
Optimize Datasette performance behind a caching proxy
When you open a database file in immutable mode using the -i option, Datasette calculates a SHA-256 hash of the contents of that file on startup.
This content hash can then optionally be used to create URLs that are guaranteed to change if the contents of the file changes in the future.
The result is pages that can be cached indefinitely by both browsers and caching proxies - providing a significant performance boost.
Install this plugin in the same environment as Datasette.
$ datasette install datasette-hashed-urls
Usage
Once installed, this plugin will act on any immutable database files that are loaded into Datasette:
datasette -i fixtures.db
The database will automatically be renamed to incorporate a hash of the contents of the SQLite file - so the above database would be served as:
http://127.0.0.1:8001/fixtures-aa7318b
Every page that accesss that database, including JSON endpoints, will be served with the following far-future cache expiry header:
cache-control: max-age=31536000, public
Here max-age=31536000 is the number of seconds in a year.
A caching proxy such as Cloudflare can then be used to cache and accelerate content served by Datasette.
When the database file is updated and the server is restarted, the hash will change and content will be served from a new URL. Any hits to the previous hashed URLs will be automatically redirected.
If you run Datasette using the --crossdb option to enable cross-database queries the _memory database will also have a hash added to its URL - in this case, the hash will be a combination of the hashes of the other attached databases.
Configuration
You can use the max_age plugin configuration setting to change the cache duration specified in the cache-control HTTP header.
To set the cache expiry time to one hour you would add this to your Datasette metadata.json configuration file:
This functionality used to ship as part of Datasette itself, as a feature called Hashed URL mode.
That feature has been deprecated and will be removed in Datasette 1.0. This plugin should be used as an alternative.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-hashed-urls
python3 -mvenv venv
source venv/bin/activate
Or if you are using pipenv:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
474176116,R_kgDOHENadA,datasette-packages,simonw/datasette-packages,0,9599,https://github.com/simonw/datasette-packages,Show a list of currently installed Python packages,0,2022-03-25T22:04:23Z,2022-03-25T22:04:45Z,2022-07-03T02:41:55Z,,9,0,0,Python,1,1,1,1,0,0,0,0,1,apache-2.0,[],0,1,0,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-packages
[](https://pypi.org/project/datasette-packages/)
[](https://github.com/simonw/datasette-packages/releases)
[](https://github.com/simonw/datasette-packages/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-packages/blob/main/LICENSE)
Show a list of currently installed Python packages
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-packages
## Usage
Visit `/-/packages` to see a list of installed Python packages.
Visit `/-/packages.json` to get that back as JSON.
## Demo
The output of this plugin can be seen here:
- https://latest-with-plugins.datasette.io/-/packages
- https://latest-with-plugins.datasette.io/-/packages.json
## With datasette-graphql
if you have version 2.1 or higher of the [datasette-graphql](https://datasette.io/plugins/datasette-graphql) plugin installed you can also query the list of packages using this GraphQL query:
```graphql
{
packages {
name
version
}
}
```
[Demo of this query](https://latest-with-plugins.datasette.io/graphql?query=%7B%0A%20%20packages%20%7B%0A%20%20%20%20name%0A%20%20%20%20version%0A%20%20%7D%0A%7D).
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-packages
python3 -mvenv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-packages
Show a list of currently installed Python packages
Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-packages
Usage
Visit /-/packages to see a list of installed Python packages.
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-packages
python3 -mvenv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,0,
474468776,R_kgDOHEfRqA,datasette-auth0,simonw/datasette-auth0,0,9599,https://github.com/simonw/datasette-auth0,Datasette plugin that authenticates users using Auth0,0,2022-03-26T21:19:31Z,2022-03-27T17:59:49Z,2022-03-28T03:04:52Z,,11,3,3,Python,1,1,1,1,0,0,0,0,0,apache-2.0,"[""auth0"", ""datasette"", ""datasette-plugin""]",0,0,3,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-auth0
[](https://pypi.org/project/datasette-auth0/)
[](https://github.com/simonw/datasette-auth0/releases)
[](https://github.com/simonw/datasette-auth0/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-auth0/blob/main/LICENSE)
Datasette plugin that authenticates users using [Auth0](https://auth0.com/)
See [Simplest possible OAuth authentication with Auth0](https://til.simonwillison.net/auth0/oauth-with-auth0) for more about how this plugin works.
## Installation
Install this plugin in the same environment as Datasette.
$ datasette install datasette-auth0
## Demo
You can try this out at [datasette-auth0-demo.datasette.io](https://datasette-auth0-demo.datasette.io/) - click on the top right menu icon and select ""Sign in with Auth0"".
## Initial configuration
First, create a new application in Auth0. You will need the domain, client ID and client secret for that application.
The domain should be something like `mysite.us.auth0.com`.
Add `http://127.0.0.1:8001/-/auth0-callback` to the list of Allowed Callback URLs.
Then configure these plugin secrets using `metadata.yml`:
```yaml
plugins:
datasette-auth0:
domain:
""$env"": AUTH0_DOMAIN
client_id:
""$env"": AUTH0_CLIENT_ID
client_secret:
""$env"": AUTH0_CLIENT_SECRET
```
Only the `client_secret` needs to be kept secret, but for consistency I recommend using the `$env` mechanism for all three.
In development, you can run Datasette and pass in environment variables like this:
```
AUTH0_DOMAIN=""your-domain.us.auth0.com"" \
AUTH0_CLIENT_ID=""...client-id-goes-here..."" \
AUTH0_CLIENT_SECRET=""...secret-goes-here..."" \
datasette -m metadata.yml
```
If you are deploying using `datasette publish` you can pass these using `--plugin-secret`. For example, to deploy using Cloud Run you might run the following:
```
datasette publish cloudrun mydatabase.db \
--install datasette-auth0 \
--plugin-secret datasette-auth0 domain ""your-domain.us.auth0.com"" \
--plugin-secret datasette-auth0 client_id ""your-client-id"" \
--plugin-secret datasette-auth0 client_secret ""your-client-secret"" \
--service datasette-auth0-demo
```
Once your Datasette instance is deployed, you will need to add its callback URL to the ""Allowed Callback URLs"" list in Auth0.
The callback URL should be something like:
https://url-to-your-datasette/-/auth0-callback
## Usage
Once installed, a ""Sign in with Auth0"" menu item will appear in the Datasette main menu.
You can sign in and then visit the `/-/actor` page to see full details of the `auth0` profile that has been authenticated.
You can then use [Datasette permissions](https://docs.datasette.io/en/stable/authentication.html#configuring-permissions-in-metadata-json) to grant or deny access to different parts of Datasette based on the authenticated user.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-auth0
python3 -mvenv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
datasette-auth0
Datasette plugin that authenticates users using Auth0
If you are deploying using datasette publish you can pass these using --plugin-secret. For example, to deploy using Cloud Run you might run the following:
Once your Datasette instance is deployed, you will need to add its callback URL to the ""Allowed Callback URLs"" list in Auth0.
The callback URL should be something like:
https://url-to-your-datasette/-/auth0-callback
Usage
Once installed, a ""Sign in with Auth0"" menu item will appear in the Datasette main menu.
You can sign in and then visit the /-/actor page to see full details of the auth0 profile that has been authenticated.
You can then use Datasette permissions to grant or deny access to different parts of Datasette based on the authenticated user.
Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd datasette-auth0
python3 -mvenv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
474473836,R_kgDOHEflbA,datasette-nteract-data-explorer,hydrosquall/datasette-nteract-data-explorer,0,9020979,https://github.com/hydrosquall/datasette-nteract-data-explorer,automatic visual data explorer for datasette,0,2022-03-26T21:47:17Z,2022-10-04T03:28:02Z,2022-10-19T00:35:29Z,https://datasette-nteract-data-explorer.vercel.app/,137,8,8,TypeScript,1,1,1,1,0,1,0,0,8,apache-2.0,"[""automatic-viz"", ""datasette"", ""datasette-plugin"", ""dataviz""]",1,8,8,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,1,1,,,1,public,0,,0,
479175467,R_kgDOHI-jKw,pypi-to-sqlite,simonw/pypi-to-sqlite,0,9599,https://github.com/simonw/pypi-to-sqlite,Load data about Python packages from PyPI into SQLite,0,2022-04-07T23:09:02Z,2022-04-08T15:16:03Z,2022-04-08T16:29:53Z,,24,2,2,Python,1,1,1,1,0,0,0,0,1,apache-2.0,[],0,1,2,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# pypi-to-sqlite
[](https://pypi.org/project/pypi-to-sqlite/)
[](https://github.com/simonw/pypi-to-sqlite/releases)
[](https://github.com/simonw/pypi-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/pypi-to-sqlite/blob/master/LICENSE)
Load data about Python packages from PyPI into SQLite
## Installation
Install this tool using `pip`:
pip install pypi-to-sqlite
## Usage
To create a SQLite database with details of one or more packages, run:
pypi-to-sqlite pypi.db datasette sqlite-utils
You can also process JSON that you have previously saved to disk like so:
curl -o datasette.json https://pypi.org/pypi/datasette/json
pypi-to-sqlite pypi.db -f datasette.json
The tool will create three tables: `packages`, `versions` and `releases`. The full table schema is shown below.
To create the tables with a prefix, use `--prefix prefix`. For example:
pypi-to-sqlite pypi.db datasette --prefix pypi_
This will create tables called `pypi_packages`, `pypi_versions` and `pypi_releases`.
## Demo
You can see examples of tables created using this tool running in [Datasette](https://datasette.io/) here:
- [packages](https://datasette.io/content/pypi_packages)
- [versions](https://datasette.io/content/pypi_versions)
- [releases](https://datasette.io/content/pypi_releases)
## Database schema
```sql
CREATE TABLE [packages] (
[name] TEXT PRIMARY KEY,
[summary] TEXT,
[classifiers] TEXT,
[description] TEXT,
[author] TEXT,
[author_email] TEXT,
[description_content_type] TEXT,
[home_page] TEXT,
[keywords] TEXT,
[license] TEXT,
[maintainer] TEXT,
[maintainer_email] TEXT,
[package_url] TEXT,
[platform] TEXT,
[project_url] TEXT,
[project_urls] TEXT,
[release_url] TEXT,
[requires_dist] TEXT,
[requires_python] TEXT,
[version] TEXT,
[yanked] INTEGER,
[yanked_reason] TEXT
);
CREATE TABLE [versions] (
[id] TEXT PRIMARY KEY,
[package] TEXT REFERENCES [packages]([name]),
[name] TEXT
);
CREATE TABLE [releases] (
[md5_digest] TEXT PRIMARY KEY,
[package] TEXT REFERENCES [packages]([name]),
[version] TEXT REFERENCES [versions]([id]),
[packagetype] TEXT,
[filename] TEXT,
[comment_text] TEXT,
[digests] TEXT,
[has_sig] INTEGER,
[python_version] TEXT,
[requires_python] TEXT,
[size] INTEGER,
[upload_time] TEXT,
[upload_time_iso_8601] TEXT,
[url] TEXT,
[yanked] INTEGER,
[yanked_reason] TEXT
);
```
## pypi-to-sqlite --help
```
Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]...
Load data about Python packages from PyPI into SQLite
Usage example:
pypi-to-sqlite pypy.db datasette sqlite-utils
Use -f to load data from a JSON file instead:
pypi-to-sqlite pypy.db -f datasette.json
Created tables will be packages, versions and releases
To create tables called pypi_packages, pypi_versions, pypi_releases use
--prefix pypi_:
pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_
Options:
--version Show the version and exit.
-f, --file FILENAME Import JSON from this file
-d, --delay FLOAT Wait this many seconds between requests
--prefix TEXT Prefix to use for the created database tables
--help Show this message and exit.
```
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd pypi-to-sqlite
python -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
","
pypi-to-sqlite
Load data about Python packages from PyPI into SQLite
Installation
Install this tool using pip:
pip install pypi-to-sqlite
Usage
To create a SQLite database with details of one or more packages, run:
pypi-to-sqlite pypi.db datasette sqlite-utils
You can also process JSON that you have previously saved to disk like so:
Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]...
Load data about Python packages from PyPI into SQLite
Usage example:
pypi-to-sqlite pypy.db datasette sqlite-utils
Use -f to load data from a JSON file instead:
pypi-to-sqlite pypy.db -f datasette.json
Created tables will be packages, versions and releases
To create tables called pypi_packages, pypi_versions, pypi_releases use
--prefix pypi_:
pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_
Options:
--version Show the version and exit.
-f, --file FILENAME Import JSON from this file
-d, --delay FLOAT Wait this many seconds between requests
--prefix TEXT Prefix to use for the created database tables
--help Show this message and exit.
Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd pypi-to-sqlite
python -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
",1,public,0,,,
485962807,R_kgDOHPc0Nw,datasette-total-page-time,simonw/datasette-total-page-time,0,9599,https://github.com/simonw/datasette-total-page-time,Add a note to the Datasette footer measuring the total page load time,0,2022-04-26T22:09:58Z,2022-04-26T22:10:27Z,2022-04-26T22:11:48Z,,0,0,0,Python,1,1,1,1,0,0,0,0,0,apache-2.0,[],0,0,0,main,"{""admin"": false, ""maintain"": false, ""push"": false, ""triage"": false, ""pull"": false}",,,0,1,"# datasette-total-page-time
[](https://pypi.org/project/datasette-total-page-time/)
[](https://github.com/simonw/datasette-total-page-time/releases)
[](https://github.com/simonw/datasette-total-page-time/actions?query=workflow%3ATest)
[](https://github.com/simonw/datasette-total-page-time/blob/main/LICENSE)
Add a note to the Datasette footer measuring the total page load time
## Installation
Install this plugin in the same environment as Datasette.
datasette install datasette-total-page-time
## Usage
Once this plugin is installed, a note will appear in the footer of every page showing how long the page took to generate.
> Queries took 326.74ms · Page took 386.310ms
## How it works
Measuring how long a page takes to load and then injecting that note into the page is tricky, because you need to finish generating the page before you know how long it took to load it!
This plugin uses the [asgi_wrapper](https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette) plugin hook to measure the time taken by Datasette and then inject the following JavaScript at the bottom of the response, after the closing `