name,summary,classifiers,description,author,author_email,description_content_type,home_page,keywords,license,maintainer,maintainer_email,package_url,platform,project_url,project_urls,release_url,requires_dist,requires_python,version,yanked,yanked_reason
airtable-export,Export Airtable data to files on disk,[],"# airtable-export
[](https://pypi.org/project/airtable-export/)
[](https://github.com/simonw/airtable-export/releases)
[](https://github.com/simonw/airtable-export/actions?query=workflow%3ATest)
[](https://github.com/simonw/airtable-export/blob/master/LICENSE)
Export Airtable data to files on disk
## Installation
Install this tool using `pip`:
$ pip install airtable-export
## Usage
You will need to know the following information:
- Your Airtable base ID - this is a string starting with `app...`
- Your Airtable API key - this is a string starting with `key...`
- The names of each of the tables that you wish to export
You can export all of your data to a folder called `export/` by running the following:
airtable-export export base_id table1 table2 --key=key
This example would create two files: `export/table1.yml` and `export/table2.yml`.
Rather than passing the API key using the `--key` option you can set it as an environment variable called `AIRTABLE_KEY`.
## Export options
By default the tool exports your data as YAML.
You can also export as JSON or as [newline delimited JSON](http://ndjson.org/) using the `--json` or `--ndjson` options:
airtable-export export base_id table1 table2 --key=key --ndjson
You can pass multiple format options at once. This command will create a `.json`, `.yml` and `.ndjson` file for each exported table:
airtable-export export base_id table1 table2 \
--key=key --ndjson --yaml --json
### SQLite database export
You can export tables to a SQLite database file using the `--sqlite database.db` option:
airtable-export export base_id table1 table2 \
--key=key --sqlite database.db
This can be combined with other format options. If you only specify `--sqlite` the export directory argument will be ignored.
The SQLite database will have a table created for each table you export. Those tables will have a primary key column called `airtable_id`.
If you run this command against an existing SQLite database records with matching primary keys will be over-written by new records from the export.
## Request options
By default the tool uses [python-httpx](https://www.python-httpx.org)'s default configurations.
You can override the `user-agent` using the `--user-agent` option:
airtable-export export base_id table1 table2 --key=key --user-agent ""Airtable Export Robot""
You can override the [timeout during a network read operation](https://www.python-httpx.org/advanced/#fine-tuning-the-configuration) using the `--http-read-timeout` option. If not set, this defaults to 5s.
airtable-export export base_id table1 table2 --key=key --http-read-timeout 60
## Running this using GitHub Actions
[GitHub Actions](https://github.com/features/actions) is GitHub's workflow automation product. You can use it to run `airtable-export` in order to back up your Airtable data to a GitHub repository. Doing this gives you a visible commit history of changes you make to your Airtable data - like [this one](https://github.com/natbat/rockybeaches/commits/main/airtable).
To run this for your own Airtable database you'll first need to add the following secrets to your GitHub repository:
- AIRTABLE_BASE_ID
- The base ID, a string beginning `app...`
- AIRTABLE_KEY
- Your Airtable API key
- AIRTABLE_TABLES
- A space separated list of the Airtable tables that you want to backup. If any of these contain spaces you will need to enclose them in single quotes, e.g. 'My table with spaces in the name' OtherTableWithNoSpaces
Once you have set those secrets, add the following as a file called `.github/workflows/backup-airtable.yml`:
```yaml
name: Backup Airtable
on:
workflow_dispatch:
schedule:
- cron: '32 0 * * *'
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Check out repo
uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.8
- uses: actions/cache@v2
name: Configure pip caching
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-
restore-keys: |
${{ runner.os }}-pip-
- name: Install airtable-export
run: |
pip install airtable-export
- name: Backup Airtable to backups/
env:
AIRTABLE_BASE_ID: ${{ secrets.AIRTABLE_BASE_ID }}
AIRTABLE_KEY: ${{ secrets.AIRTABLE_KEY }}
AIRTABLE_TABLES: ${{ secrets.AIRTABLE_TABLES }}
run: |-
airtable-export backups $AIRTABLE_BASE_ID $AIRTABLE_TABLES -v
- name: Commit and push if it changed
run: |-
git config user.name ""Automated""
git config user.email ""actions@users.noreply.github.com""
git add -A
timestamp=$(date -u)
git commit -m ""Latest data: ${timestamp}"" || exit 0
git push
```
This will run once a day (at 32 minutes past midnight UTC) and will also run if you manually click the ""Run workflow"" button, see [GitHub Actions: Manual triggers with workflow_dispatch](https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/).
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd airtable-export
python -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",Simon Willison,,text/markdown,https://github.com/simonw/airtable-export,,"Apache License, Version 2.0",,,https://pypi.org/project/airtable-export/,,https://pypi.org/project/airtable-export/,"{""CI"": ""https://github.com/simonw/airtable-export/actions"", ""Changelog"": ""https://github.com/simonw/airtable-export/releases"", ""Homepage"": ""https://github.com/simonw/airtable-export"", ""Issues"": ""https://github.com/simonw/airtable-export/issues""}",https://pypi.org/project/airtable-export/0.7.1/,"[""click"", ""PyYAML"", ""httpx"", ""sqlite-utils"", ""pytest ; extra == 'test'"", ""pytest-mock ; extra == 'test'""]",,0.7.1,0,
csv-diff,Python CLI tool and library for diffing CSV and JSON files,"[""Development Status :: 4 - Beta"", ""Intended Audience :: Developers"", ""Intended Audience :: End Users/Desktop"", ""Intended Audience :: Science/Research"", ""License :: OSI Approved :: Apache Software License"", ""Programming Language :: Python :: 3.6"", ""Programming Language :: Python :: 3.7""]","# csv-diff
[](https://pypi.org/project/csv-diff/)
[](https://github.com/simonw/csv-diff/releases)
[](https://github.com/simonw/csv-diff/actions?query=workflow%3ATest)
[](https://github.com/simonw/csv-diff/blob/main/LICENSE)
Tool for viewing the difference between two CSV, TSV or JSON files. See [Generating a commit log for San Francisco’s official list of trees](https://simonwillison.net/2019/Mar/13/tree-history/) (and the [sf-tree-history repo commit log](https://github.com/simonw/sf-tree-history/commits)) for background information on this project.
## Installation
pip install csv-diff
## Usage
Consider two CSV files:
`one.csv`
id,name,age
1,Cleo,4
2,Pancakes,2
`two.csv`
id,name,age
1,Cleo,5
3,Bailey,1
`csv-diff` can show a human-readable summary of differences between the files:
$ csv-diff one.csv two.csv --key=id
1 row changed, 1 row added, 1 row removed
1 row changed
Row 1
age: ""4"" => ""5""
1 row added
id: 3
name: Bailey
age: 1
1 row removed
id: 2
name: Pancakes
age: 2
The `--key=id` option means that the `id` column should be treated as the unique key, to identify which records have changed.
The tool will automatically detect if your files are comma- or tab-separated. You can over-ride this automatic detection and force the tool to use a specific format using `--format=tsv` or `--format=csv`.
You can also feed it JSON files, provided they are a JSON array of objects where each object has the same keys. Use `--format=json` if your input files are JSON.
Use `--show-unchanged` to include full details of the unchanged values for rows with at least one change in the diff output:
% csv-diff one.csv two.csv --key=id --show-unchanged
1 row changed
id: 1
age: ""4"" => ""5""
Unchanged:
name: ""Cleo""
You can use the `--json` option to get a machine-readable difference:
$ csv-diff one.csv two.csv --key=id --json
{
""added"": [
{
""id"": ""3"",
""name"": ""Bailey"",
""age"": ""1""
}
],
""removed"": [
{
""id"": ""2"",
""name"": ""Pancakes"",
""age"": ""2""
}
],
""changed"": [
{
""key"": ""1"",
""changes"": {
""age"": [
""4"",
""5""
]
}
}
],
""columns_added"": [],
""columns_removed"": []
}
## As a Python library
You can also import the Python library into your own code like so:
from csv_diff import load_csv, compare
diff = compare(
load_csv(open(""one.csv""), key=""id""),
load_csv(open(""two.csv""), key=""id"")
)
`diff` will now contain the same data structure as the output in the `--json` example above.
If the columns in the CSV have changed, those added or removed columns will be ignored when calculating changes made to specific rows.
",Simon Willison,,text/markdown,https://github.com/simonw/csv-diff,,"Apache License, Version 2.0",,,https://pypi.org/project/csv-diff/,,https://pypi.org/project/csv-diff/,"{""Homepage"": ""https://github.com/simonw/csv-diff""}",https://pypi.org/project/csv-diff/1.1/,"[""click"", ""dictdiffer"", ""pytest ; extra == 'test'""]",,1.1,0,
datasette-clone,Create a local copy of database files from a Datasette instance,[],"# datasette-clone
[](https://pypi.org/project/datasette-clone/)
[](https://circleci.com/gh/simonw/datasette-clone)
[](https://github.com/simonw/datasette-clone/blob/master/LICENSE)
Create a local copy of database files from a Datasette instance.
See [datasette-clone](https://simonwillison.net/2020/Apr/14/datasette-clone/) on my blog for background on this project.
## How to install
$ pip install datasette-clone
## Usage
This only works against Datasette instances running immutable databases (with the `-i` option). Databases published using the `datasette publish` command should be compatible with this tool.
To download copies of all `.db` files from an instance, run:
datasette-clone https://latest.datasette.io
You can provide an optional second argument to specify a directory:
datasette-clone https://latest.datasette.io /tmp/here-please
The command stores its own copy of a `databases.json` manifest and uses it to only download databases that have changed the next time you run the command.
It also stores a copy of the instance's `metadata.json` to ensure you have a copy of any source and licensing information for the downloaded databases.
If your instance is protected by an API token, you can use `--token` to provide it:
datasette-clone https://latest.datasette.io --token=xyz
For verbose output showing what the tool is doing, use `-v`.
",Simon Willison,,text/markdown,https://github.com/simonw/datasette-clone,,"Apache License, Version 2.0",,,https://pypi.org/project/datasette-clone/,,https://pypi.org/project/datasette-clone/,"{""Homepage"": ""https://github.com/simonw/datasette-clone""}",https://pypi.org/project/datasette-clone/0.5/,"[""requests"", ""click"", ""pytest ; extra == 'test'"", ""requests-mock ; extra == 'test'""]",,0.5,0,
db-to-sqlite,CLI tool for exporting tables or queries from any SQL database to a SQLite file,"[""Development Status :: 3 - Alpha"", ""Intended Audience :: Developers"", ""Intended Audience :: End Users/Desktop"", ""Intended Audience :: Science/Research"", ""License :: OSI Approved :: Apache Software License"", ""Programming Language :: Python :: 3.6"", ""Programming Language :: Python :: 3.7"", ""Topic :: Database""]","# db-to-sqlite
[](https://pypi.python.org/pypi/db-to-sqlite)
[](https://github.com/simonw/db-to-sqlite/releases)
[](https://github.com/simonw/db-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/db-to-sqlite/blob/main/LICENSE)
CLI tool for exporting tables or queries from any SQL database to a SQLite file.
## Installation
Install from PyPI like so:
pip install db-to-sqlite
If you want to use it with MySQL, you can install the extra dependency like this:
pip install 'db-to-sqlite[mysql]'
Installing the `mysqlclient` library on OS X can be tricky - I've found [this recipe](https://gist.github.com/simonw/90ac0afd204cd0d6d9c3135c3888d116) to work (run that before installing `db-to-sqlite`).
For PostgreSQL, use this:
pip install 'db-to-sqlite[postgresql]'
## Usage
Usage: db-to-sqlite [OPTIONS] CONNECTION PATH
Load data from any database into SQLite.
PATH is a path to the SQLite file to create, e.c. /tmp/my_database.db
CONNECTION is a SQLAlchemy connection string, for example:
postgresql://localhost/my_database
postgresql://username:passwd@localhost/my_database
mysql://root@localhost/my_database
mysql://username:passwd@localhost/my_database
More: https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls
Options:
--version Show the version and exit.
--all Detect and copy all tables
--table TEXT Specific tables to copy
--skip TEXT When using --all skip these tables
--redact TEXT... (table, column) pairs to redact with ***
--sql TEXT Optional SQL query to run
--output TEXT Table in which to save --sql query results
--pk TEXT Optional column to use as a primary key
--index-fks / --no-index-fks Should foreign keys have indexes? Default on
-p, --progress Show progress bar
--postgres-schema TEXT PostgreSQL schema to use
--help Show this message and exit.
For example, to save the content of the `blog_entry` table from a PostgreSQL database to a local file called `blog.db` you could do this:
db-to-sqlite ""postgresql://localhost/myblog"" blog.db \
--table=blog_entry
You can specify `--table` more than once.
You can also save the data from all of your tables, effectively creating a SQLite copy of your entire database. Any foreign key relationships will be detected and added to the SQLite database. For example:
db-to-sqlite ""postgresql://localhost/myblog"" blog.db \
--all
When running `--all` you can specify tables to skip using `--skip`:
db-to-sqlite ""postgresql://localhost/myblog"" blog.db \
--all \
--skip=django_migrations
If you want to save the results of a custom SQL query, do this:
db-to-sqlite ""postgresql://localhost/myblog"" output.db \
--output=query_results \
--sql=""select id, title, created from blog_entry"" \
--pk=id
The `--output` option specifies the table that should contain the results of the query.
## Using db-to-sqlite with PostgreSQL schemas
If the tables you want to copy from your PostgreSQL database aren't in the default schema, you can specify an alternate one with the `--postgres-schema` option:
db-to-sqlite ""postgresql://localhost/myblog"" blog.db \
--all \
--postgres-schema my_schema
## Using db-to-sqlite with Heroku Postgres
If you run an application on [Heroku](https://www.heroku.com/) using their [Postgres database product](https://www.heroku.com/postgres), you can use the `heroku config` command to access a compatible connection string:
$ heroku config --app myappname | grep HEROKU_POSTG
HEROKU_POSTGRESQL_OLIVE_URL: postgres://username:password@ec2-xxx-xxx-xxx-x.compute-1.amazonaws.com:5432/dbname
You can pass this to `db-to-sqlite` to create a local SQLite database with the data from your Heroku instance.
You can even do this using a bash one-liner:
$ db-to-sqlite $(heroku config --app myappname | grep HEROKU_POSTG | cut -d: -f 2-) \
/tmp/heroku.db --all -p
1/23: django_migrations
...
17/23: blog_blogmark
[####################################] 100%
...
## Related projects
* [Datasette](https://github.com/simonw/datasette): A tool for exploring and publishing data. Works great with SQLite files generated using `db-to-sqlite`.
* [sqlite-utils](https://github.com/simonw/sqlite-utils): Python CLI utility and library for manipulating SQLite databases.
* [csvs-to-sqlite](https://github.com/simonw/csvs-to-sqlite): Convert CSV files into a SQLite database.
## Development
To set up this tool locally, first checkout the code. Then create a new virtual environment:
cd db-to-sqlite
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
This will skip tests against MySQL or PostgreSQL if you do not have their additional dependencies installed.
You can install those extra dependencies like so:
pip install -e '.[test_mysql,test_postgresql]'
You can alternative use `pip install psycopg2-binary` if you cannot install the `psycopg2` dependency used by the `test_postgresql` extra.
See [Running a MySQL server using Homebrew](https://til.simonwillison.net/homebrew/mysql-homebrew) for tips on running the tests against MySQL on macOS, including how to install the `mysqlclient` dependency.
The PostgreSQL and MySQL tests default to expecting to run against servers on localhost. You can use environment variables to point them at different test database servers:
- `MYSQL_TEST_DB_CONNECTION` - defaults to `mysql://root@localhost/test_db_to_sqlite`
- `POSTGRESQL_TEST_DB_CONNECTION` - defaults to `postgresql://localhost/test_db_to_sqlite`
The database you indicate in the environment variable - `test_db_to_sqlite` by default - will be deleted and recreated on every test run.
",Simon Willison,,text/markdown,https://github.com/simonw/db-to-sqlite,,"Apache License, Version 2.0",,,https://pypi.org/project/db-to-sqlite/,,https://pypi.org/project/db-to-sqlite/,"{""CI"": ""https://travis-ci.com/simonw/db-to-sqlite"", ""Changelog"": ""https://github.com/simonw/db-to-sqlite/releases"", ""Documentation"": ""https://github.com/simonw/db-to-sqlite/blob/main/README.md"", ""Homepage"": ""https://github.com/simonw/db-to-sqlite"", ""Issues"": ""https://github.com/simonw/db-to-sqlite/issues"", ""Source code"": ""https://github.com/simonw/db-to-sqlite""}",https://pypi.org/project/db-to-sqlite/1.4/,"[""sqlalchemy"", ""sqlite-utils (>=2.9.1)"", ""click"", ""mysqlclient ; extra == 'mysql'"", ""psycopg2 ; extra == 'postgresql'"", ""pytest ; extra == 'test'"", ""pytest ; extra == 'test_mysql'"", ""mysqlclient ; extra == 'test_mysql'"", ""pytest ; extra == 'test_postgresql'"", ""psycopg2 ; extra == 'test_postgresql'""]",,1.4,0,
dbf-to-sqlite,"CLCLI tool for converting DBF files (dBase, FoxPro etc) to SQLite","[""Development Status :: 3 - Alpha"", ""Intended Audience :: Developers"", ""Intended Audience :: End Users/Desktop"", ""Intended Audience :: Science/Research"", ""License :: OSI Approved :: Apache Software License"", ""Programming Language :: Python :: 3.6"", ""Programming Language :: Python :: 3.7"", ""Topic :: Database""]","# dbf-to-sqlite
[](https://pypi.python.org/pypi/dbf-to-sqlite)
[](https://travis-ci.com/simonw/dbf-to-sqlite)
[](https://github.com/simonw/dbf-to-sqlite/blob/master/LICENSE)
CLI tool for converting DBF files (dBase, FoxPro etc) to SQLite.
$ dbf-to-sqlite --help
Usage: dbf-to-sqlite [OPTIONS] DBF_PATHS... SQLITE_DB
Convert DBF files (dBase, FoxPro etc) to SQLite
https://github.com/simonw/dbf-to-sqlite
Options:
--version Show the version and exit.
--table TEXT Table name to use (only valid for single files)
-v, --verbose Show what's going on
--help Show this message and exit.
Example usage:
$ dbf-to-sqlite *.DBF database.db
This will create a new SQLite database called `database.db` containing one table for each of the `DBF` files in the current directory.
Looking for DBF files to try this out on? Try downloading the [Himalayan Database](http://himalayandatabase.com/) of all expeditions that have climbed in the Nepal Himalaya.
",Simon Willison,,text/markdown,https://github.com/simonw/dbf-to-sqlite,,"Apache License, Version 2.0",,,https://pypi.org/project/dbf-to-sqlite/,,https://pypi.org/project/dbf-to-sqlite/,"{""Homepage"": ""https://github.com/simonw/dbf-to-sqlite""}",https://pypi.org/project/dbf-to-sqlite/0.1/,"[""dbf (==0.97.11)"", ""click"", ""sqlite-utils""]",,0.1,0,
dogsheep-beta,Build a search index across content from multiple SQLite database tables and run faceted searches against it using Datasette,[],"# dogsheep-beta
[](https://pypi.org/project/dogsheep-beta/)
[](https://github.com/dogsheep/beta/releases)
[](https://github.com/dogsheep/beta/actions?query=workflow%3ATest)
[](https://github.com/dogsheep/beta/blob/main/LICENSE)
Build a search index across content from multiple SQLite database tables and run faceted searches against it using Datasette
## Example
A live example of this plugin is running at https://datasette.io/-/beta - configured using [this YAML file](https://github.com/simonw/datasette.io/blob/main/templates/dogsheep-beta.yml).
Read more about how this example works in [Building a search engine for datasette.io](https://simonwillison.net/2020/Dec/19/dogsheep-beta/).
## Installation
Install this tool like so:
$ pip install dogsheep-beta
## Usage
Run the indexer using the `dogsheep-beta` command-line tool:
$ dogsheep-beta index dogsheep.db config.yml
The `config.yml` file contains details of the databases and document types that should be indexed:
```yaml
twitter.db:
tweets:
sql: |-
select
tweets.id as key,
'Tweet by @' || users.screen_name as title,
tweets.created_at as timestamp,
tweets.full_text as search_1
from tweets join users on tweets.user = users.id
users:
sql: |-
select
id as key,
name || ' @' || screen_name as title,
created_at as timestamp,
description as search_1
from users
```
This will create a `search_index` table in the `dogsheep.db` database populated by data from those SQL queries.
By default the search index that this tool creates will be configured for Porter stemming. This means that searches for words like `run` will match documents containing `runs` or `running`.
If you don't want to use Porter stemming, use the `--tokenize none` option:
$ dogsheep-beta index dogsheep.db config.yml --tokenize none
You can pass other SQLite tokenize argumenst here, see [the SQLite FTS tokenizers documentation](https://www.sqlite.org/fts5.html#tokenizers).
## Columns
The columns that can be returned by our query are:
- `key` - a unique (within that type) primary key
- `title` - the title for the item
- `timestamp` - an ISO8601 timestamp, e.g. `2020-09-02T21:00:21`
- `search_1` - a larger chunk of text to be included in the search index
- `category` - an integer category ID, see below
- `is_public` - an integer (0 or 1, defaults to 0 if not set) specifying if this is public or not
Public records are things like your public tweets, blog posts and GitHub commits.
## Categories
Indexed items can be assigned a category. Categories are integers that correspond to records in the `categories` table, which defaults to containing the following:
| id | name |
|------|------------|
| 1 | created |
| 2 | saved |
| 3 | received |
`created` is for items that have been created by the Dogsheep instance owner.
`saved` is for items that they have saved, liked or favourited.
`received` is for items that have been specifically sent to them by other people - incoming emails or direct messages for example.
## Datasette plugin
Run `datasette install dogsheep-beta` (or use `pip install dogsheep-beta` in the same environment as Datasette) to install the Dogsheep Beta Datasette plugin.
Once installed, a custom search interface will be made available at `/-/beta`. You can use this interface to execute searches.
The Datasette plugin has some configuration options. You can set these by adding the following to your `metadata.json` configuration file:
```json
{
""plugins"": {
""dogsheep-beta"": {
""database"": ""beta"",
""config_file"": ""dogsheep-beta.yml"",
""template_debug"": true
}
}
}
```
The configuration settings for the plugin are:
- `database` - the database file that contains your search index. If the file is `beta.db` you should set `database` to `beta`.
- `config_file` - the YAML file containing your Dogsheep Beta configuration.
- `template_debug` - set this to `true` to enable debugging output if errors occur in your custom templates, see below.
## Custom results display
Each indexed item type can define custom display HTML as part of the `config.yml` file. It can do this using a `display` key containing a fragment of Jinja template, and optionally a `display_sql` key with extra SQL to execute to fetch the data to display.
Here's how to define a custom display template for a tweet:
```yaml
twitter.db:
tweets:
sql: |-
select
tweets.id as key,
'Tweet by @' || users.screen_name as title,
tweets.created_at as timestamp,
tweets.full_text as search_1
from tweets join users on tweets.user = users.id
display: |-
{{ title }} - tweeted at {{ timestamp }}
{{ search_1 }}
```
This example reuses the value that were stored in the `search_index` table when the indexing query was run.
To load in extra values to display in the template, use a `display_sql` query like this:
```yaml
twitter.db:
tweets:
sql: |-
select
tweets.id as key,
'Tweet by @' || users.screen_name as title,
tweets.created_at as timestamp,
tweets.full_text as search_1
from tweets join users on tweets.user = users.id
display_sql: |-
select
users.screen_name,
tweets.full_text,
tweets.created_at
from
tweets join users on tweets.user = users.id
where
tweets.id = :key
display: |-
{{ display.screen_name }} - tweeted at {{ display.created_at }}
{{ display.full_text }}
```
The `display_sql` query will be executed for every search result, passing the key value from the `search_index` table as the `:key` parameter and the user's search term as the `:q` parameter.
This performs well because [many small queries are efficient in SQLite](https://www.sqlite.org/np1queryprob.html).
If an error occurs while rendering one of your templates the search results page will return a 500 error. You can use the `template_debug` configuration setting described above to instead output debugging information for the search results item that experienced the error.
## Displaying maps
This plugin will eventually include a number of useful shortcuts for rendering interesting content.
The first available shortcut is for displaying maps. Make your custom content output something like this:
```html
```
JavaScript on the page will look for any elements with `data-map-latitude` and `data-map-longitude` and, if it finds any, will load Leaflet and convert those elements into maps centered on that location. The default zoom level will be 12, or you can set a `data-map-zoom` attribute to customize this.
## Development
To set up this plugin locally, first checkout the code. Then create a new virtual environment:
cd dogsheep-beta
python3 -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",Simon Willison,,text/markdown,https://github.com/dogsheep/beta,,"Apache License, Version 2.0",,,https://pypi.org/project/dogsheep-beta/,,https://pypi.org/project/dogsheep-beta/,"{""CI"": ""https://github.com/dogsheep/beta/actions"", ""Changelog"": ""https://github.com/dogsheep/beta/releases"", ""Homepage"": ""https://github.com/dogsheep/beta"", ""Issues"": ""https://github.com/dogsheep/beta/issues""}",https://pypi.org/project/dogsheep-beta/0.10.2/,"[""datasette (>=0.50.2)"", ""click"", ""PyYAML"", ""sqlite-utils (>=3.0)"", ""pytest ; extra == 'test'"", ""pytest-asyncio ; extra == 'test'"", ""httpx ; extra == 'test'"", ""beautifulsoup4 ; extra == 'test'"", ""html5lib ; extra == 'test'""]",,0.10.2,0,
download-tiles,Download map tiles and store them in an MBTiles database,[],"# download-tiles
[](https://pypi.org/project/download-tiles/)
[](https://github.com/simonw/download-tiles/releases)
[](https://github.com/simonw/download-tiles/actions?query=workflow%3ATest)
[](https://github.com/simonw/download-tiles/blob/master/LICENSE)
Download map tiles and store them in an MBTiles database
## Installation
Install this tool using `pip`:
$ pip install download-tiles
## Usage
This tool downloads tiles from a specified [TMS (Tile Map Server)](https://wiki.openstreetmap.org/wiki/TMS) server for a specified bounding box and range of zoom levels and stores those tiles in a MBTiles SQLite database. It is a command-line wrapper around the [Landez](https://github.com/makinacorpus/landez) Python libary.
**Please use this tool responsibly**. Consult the usage policies of the tile servers you are interacting with, for example the [OpenStreetMap Tile Usage Policy](https://operations.osmfoundation.org/policies/tiles/).
Running the following will download zoom levels 0-3 of OpenStreetMap, 85 tiles total, and store them in a SQLite database called `world.mbtiles`:
download-tiles world.mbtiles
You can customize which tile and zoom levels are downloaded using command options:
`--zoom-levels=0-3` or `-z=0-3`
The different zoom levels to download. Specify a single number, e.g. `15`, or a range of numbers e.g. `0-4`. Be careful with this setting as you can easily go over the limits requested by the underlying tile server.
`--bbox=3.9,-6.3,14.5,10.2` or `-b=3.9,-6.3,14.5,10.2`
The bounding box to fetch. Should be specified as `min-lon,min-lat,max-lon,max-lat`. You can use [bboxfinder.com](http://bboxfinder.com/) to find these for different areas.
`--city=london` or `--country=madagascar`
These options can be used instead of `--bbox`. The city or country specified will be looked up using the [Nominatum API](https://nominatim.org/release-docs/latest/api/Search/) and used to derive a bounding box.
`--show-bbox`
Use this option to output the bounding box that was retrieved for the `--city` or `--country` without downloading any tiles.
`--name=Name`
A name for this tile collection, used for the `name` field in the `metadata` table. If not specified a UUID will be used, or if you used `--city` or `--country` the name will be set to the full name of that place.
`--attribution=""Attribution string""`
Attribution string to bake into the `metadata` table. This will default to `© OpenStreetMap contributors` unless you use `--tiles-url` to specify an alternative tile server, in which case you should specify a custom attribution string.
You can use the `--attribution=osm` shortcut to specify the `© OpenStreetMap contributors` value without having to type it out in full.
`--tiles-url=https://...`
The tile server URL to use. This should include `{z}` and `{x}` and `{y}` specifiers, and can optionally include `{s}` for subdomains.
The default URL used here is for OpenStreetMap, `http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png`
`--tiles-subdomains=a,b,c`
A comma-separated list of subdomains to use for the `{s}` parameter.
`--verbose`
Use this option to turn on verbose logging.
`--cache-dir=/tmp/tiles`
Provide a directory to cache downloaded tiles between runs. This can be useful if you are worried you might not have used the correct options for the bounding box or zoom levels.
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd download-tiles
python -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",Simon Willison,,text/markdown,https://github.com/simonw/download-tiles,,"Apache License, Version 2.0",,,https://pypi.org/project/download-tiles/,,https://pypi.org/project/download-tiles/,"{""CI"": ""https://github.com/simonw/download-tiles/actions"", ""Changelog"": ""https://github.com/simonw/download-tiles/releases"", ""Homepage"": ""https://github.com/simonw/download-tiles"", ""Issues"": ""https://github.com/simonw/download-tiles/issues""}",https://pypi.org/project/download-tiles/0.4.1/,"[""click"", ""requests"", ""landez (==2.5.0)"", ""pytest ; extra == 'test'"", ""requests-mock ; extra == 'test'""]",>=3.6,0.4.1,0,
evernote-to-sqlite,Tools for converting Evernote content to SQLite,[],"# evernote-to-sqlite
[](https://pypi.org/project/evernote-to-sqlite/)
[](https://github.com/dogsheep/evernote-to-sqlite/releases)
[](https://github.com/dogsheep/evernote-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/dogsheep/evernote-to-sqlite/blob/master/LICENSE)
Tools for converting Evernote content to SQLite. See [Building an Evernote to SQLite exporter](https://simonwillison.net/2020/Oct/16/building-evernote-sqlite-exporter/) for background on this project.
## Installation
Install this tool using `pip`:
$ pip install evernote-to-sqlite
## Usage
Currently the only available command is `evernote-to-sqlite enex`, which converts Evernote's ENEX export files into a SQLite database.
You can create [an ENEX export](https://help.evernote.com/hc/en-us/articles/209005557-Export-notes-and-notebooks-as-ENEX-or-HTML) in the Evernote desktop application by selecting some notes (or all of your notes) and using the `File -> Export Notes...` menu option.
This used to be able to export everything in one go, but it looks like more recent Evernote versions only allow exporting up to fifty notes at a time, or let you export an entire notebook by right-clicking on the notebook and selecting ""Export notebook..."".
You can convert that file to SQLite like so:
$ evernote-to-sqlite enex evernote.db MyNotes.enex
This will display a progress bar and create a SQLite database file called `evernote.db`.
### Limitations
Unfortunately the ENEX export format does not include a unique identifier for each note. This means you cannot use this tool to re-import notes after they have been updated - you should consider this tool to be a one-time transformation of an ENEX file into an equivalent SQLite database.
ENEX exports also do not include details of which notebook a note belongs to.
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd evernote-to-sqlite
python -mvenv venv
source venv/bin/activate
Or if you are using `pipenv`:
pipenv shell
Now install the dependencies and tests:
pip install -e '.[test]'
To run the tests:
pytest
",Simon Willison,,text/markdown,https://github.com/dogsheep/evernote-to-sqlite,,"Apache License, Version 2.0",,,https://pypi.org/project/evernote-to-sqlite/,,https://pypi.org/project/evernote-to-sqlite/,"{""CI"": ""https://github.com/dogsheep/evernote-to-sqlite/actions"", ""Changelog"": ""https://github.com/dogsheep/evernote-to-sqlite/releases"", ""Homepage"": ""https://github.com/dogsheep/evernote-to-sqlite"", ""Issues"": ""https://github.com/dogsheep/evernote-to-sqlite/issues""}",https://pypi.org/project/evernote-to-sqlite/0.3.2/,"[""click"", ""sqlite-utils (>=3.0)"", ""pytest ; extra == 'test'""]",,0.3.2,0,
fec-to-sqlite,Save FEC campaign finance data to a SQLite database,[],"# fec-to-sqlite
[](https://pypi.org/project/fec-to-sqlite/)
[](https://circleci.com/gh/simonw/fec-to-sqlite)
[](https://github.com/simonw/fec-to-sqlite/blob/master/LICENSE)
Create a SQLite database using FEC campaign contributions data.
This tool builds on [fecfile](https://github.com/esonderegger/) by Evan Sonderegger.
## How to install
$ pip install fec-to-sqlite
## Usage
$ fec-to-sqlite filings filings.db 1146148
This fetches the filing with ID `1146148` and stores it in tables in a SQLite database called `filings.db`. It will create any tables it needs.
You can pass more than one filing ID, separated by spaces.
",Simon Willison,,text/markdown,https://github.com/dogsheep/fec-to-sqlite,,"Apache License, Version 2.0",,,https://pypi.org/project/fec-to-sqlite/,,https://pypi.org/project/fec-to-sqlite/,"{""Homepage"": ""https://github.com/dogsheep/fec-to-sqlite""}",https://pypi.org/project/fec-to-sqlite/0.2/,"[""sqlite-utils"", ""click"", ""requests"", ""fecfile"", ""tqdm"", ""pytest ; extra == 'test'""]",,0.2,0,
git-history,Tools for analyzing Git history using SQLite,[],"# git-history
[](https://pypi.org/project/git-history/)
[](https://github.com/simonw/git-history/releases)
[](https://github.com/simonw/git-history/actions?query=workflow%3ATest)
[](https://github.com/simonw/git-history/blob/master/LICENSE)
Tools for analyzing Git history using SQLite
For background on this project see [git-history: a tool for analyzing scraped data collected using Git and SQLite](https://simonwillison.net/2021/Dec/7/git-history/)
## Installation
Install this tool using `pip`:
$ pip install git-history
## Demos
[git-history-demos.datasette.io](http://git-history-demos.datasette.io/) hosts three example databases created using this tool:
- [pge-outages](https://git-history-demos.datasette.io/pge-outages) shows a history of PG&E (the electricity supplier) [outages](https://pgealerts.alerts.pge.com/outagecenter/), using data collected in [simonw/pge-outages](https://github.com/simonw/pge-outages) converted using [pge-outages.sh](https://github.com/simonw/git-history/blob/main/demos/pge-outages.sh)
- [ca-fires](https://git-history-demos.datasette.io/ca-fires) shows a history of fires in California reported on [fire.ca.gov/incidents](https://www.fire.ca.gov/incidents/), from data in [simonw/ca-fires-history](https://github.com/simonw/ca-fires-history) converted using [ca-fires.sh](https://github.com/simonw/git-history/blob/main/demos/ca-fires.sh)
- [sf-bay-511](https://git-history-demos.datasette.io/sf-bay-511) has records of San Francisco Bay Area traffic and transit incident data from [511.org](https://511.org/), collected in [dbreunig/511-events-history](https://github.com/dbreunig/511-events-history) converted using [sf-bay-511.sh](https://github.com/simonw/git-history/blob/main/demos/sf-bay-511.sh)
The demos are deployed using [Datasette](https://datasette.io/) on [Google Cloud Run](https://cloud.google.com/run/) by [this GitHub Actions workflow](https://github.com/simonw/git-history/blob/main/.github/workflows/deploy-demos.yml).
## Usage
This tool can be run against a Git repository that holds a file that contains JSON, CSV/TSV or some other format and which has multiple versions tracked in the Git history. Read [Git scraping: track changes over time by scraping to a Git repository](https://simonwillison.net/2020/Oct/9/git-scraping/) to understand how you might create such a repository.
The `file` command analyzes the history of an individual file within the repository, and generates a SQLite database table that represents the different versions of that file over time.
The file is assumed to contain multiple objects - for example, the results of scraping an electricity outage map or a CSV file full of records.
Assuming you have a file called `incidents.json` that is a JSON array of objects, with multiple versions of that file recorded in a repository. Each version of that file might look something like this:
```json
[
{
""IncidentID"": ""abc123"",
""Location"": ""Corner of 4th and Vermont"",
""Type"": ""fire""
},
{
""IncidentID"": ""cde448"",
""Location"": ""555 West Example Drive"",
""Type"": ""medical""
}
]
```
Change directory into the GitHub repository in question and run the following:
git-history file incidents.db incidents.json
This will create a new SQLite database in the `incidents.db` file with three tables:
- `commits` containing a row for every commit, with a `hash` column, the `commit_at` date and a foreign key to a `namespace`.
- `item` containing a row for every item in every version of the `filename.json` file - with an extra `_commit` column that is a foreign key back to the `commit` table.
- `namespaces` containing a single row. This allows you to build multiple tables for different files, using the `--namespace` option described below.
The database schema for this example will look like this:
```sql
CREATE TABLE [namespaces] (
[id] INTEGER PRIMARY KEY,
[name] TEXT
);
CREATE UNIQUE INDEX [idx_namespaces_name]
ON [namespaces] ([name]);
CREATE TABLE [commits] (
[id] INTEGER PRIMARY KEY,
[namespace] INTEGER REFERENCES [namespaces]([id]),
[hash] TEXT,
[commit_at] TEXT
);
CREATE UNIQUE INDEX [idx_commits_namespace_hash]
ON [commits] ([namespace], [hash]);
CREATE TABLE [item] (
[IncidentID] TEXT,
[Location] TEXT,
[Type] TEXT
);
```
If you have 10 historic versions of the `incidents.json` file and each one contains 30 incidents, you will end up with 10 * 30 = 300 rows in your `item` table.
### Track the history of individual items using IDs
If your objects have a unique identifier - or multiple columns that together form a unique identifier - you can use the `--id` option to de-duplicate and track changes to each of those items over time.
This provides a much more interesting way to apply this tool.
If there is a unique identifier column called `IncidentID` you could run the following:
git-history file incidents.db incidents.json --id IncidentID
The database schema used here is very different from the one used without the `--id` option.
If you have already imported history, the command will skip any commits that it has seen already and just process new ones. This means that even though an initial import could be slow subsequent imports should run a lot faster.
This command will create six tables - `commits`, `item`, `item_version`, `columns`, `item_changed` and `namespaces`.
Here's the full schema:
```sql
CREATE TABLE [namespaces] (
[id] INTEGER PRIMARY KEY,
[name] TEXT
);
CREATE UNIQUE INDEX [idx_namespaces_name]
ON [namespaces] ([name]);
CREATE TABLE [commits] (
[id] INTEGER PRIMARY KEY,
[namespace] INTEGER REFERENCES [namespaces]([id]),
[hash] TEXT,
[commit_at] TEXT
);
CREATE UNIQUE INDEX [idx_commits_namespace_hash]
ON [commits] ([namespace], [hash]);
CREATE TABLE [item] (
[_id] INTEGER PRIMARY KEY,
[_item_id] TEXT
, [IncidentID] TEXT, [Location] TEXT, [Type] TEXT, [_commit] INTEGER);
CREATE UNIQUE INDEX [idx_item__item_id]
ON [item] ([_item_id]);
CREATE TABLE [item_version] (
[_id] INTEGER PRIMARY KEY,
[_item] INTEGER REFERENCES [item]([_id]),
[_version] INTEGER,
[_commit] INTEGER REFERENCES [commits]([id]),
[IncidentID] TEXT,
[Location] TEXT,
[Type] TEXT,
[_item_full_hash] TEXT
);
CREATE TABLE [columns] (
[id] INTEGER PRIMARY KEY,
[namespace] INTEGER REFERENCES [namespaces]([id]),
[name] TEXT
);
CREATE UNIQUE INDEX [idx_columns_namespace_name]
ON [columns] ([namespace], [name]);
CREATE TABLE [item_changed] (
[item_version] INTEGER REFERENCES [item_version]([_id]),
[column] INTEGER REFERENCES [columns]([id]),
PRIMARY KEY ([item_version], [column])
);
CREATE VIEW item_version_detail AS select
commits.commit_at as _commit_at,
commits.hash as _commit_hash,
item_version.*,
(
select json_group_array(name) from columns
where id in (
select column from item_changed
where item_version = item_version._id
)
) as _changed_columns
from item_version
join commits on commits.id = item_version._commit;
CREATE INDEX [idx_item_version__item]
ON [item_version] ([_item]);
```
#### item table
The `item` table will contain the most recent version of each row, de-duplicated by ID, plus the following additional columns:
- `_id` - a numeric integer primary key, used as a foreign key from the `item_version` table.
- `_item_id` - a hash of the values of the columns specified using the `--id` option to the command. This is used for de-duplication when processing new versions.
- `_commit` - a foreign key to the `commit` table, representing the most recent commit to modify this item.
#### item_version table
The `item_version` table will contain a row for each captured differing version of that item, plus the following columns:
- `_id` - a numeric ID for the item version record.
- `_item` - a foreign key to the `item` table.
- `_version` - the numeric version number, starting at 1 and incrementing for each captured version.
- `_commit` - a foreign key to the `commit` table.
- `_item_full_hash` - a hash of this version of the item. This is used internally by the tool to identify items that have changed between commits.
The other columns in this table represent columns in the original data that have changed since the previous version. If the value has not changed, it will be represented by a `null`.
If a value was previously set but has been changed back to `null` it will still be represented as `null` in the `item_version` row. You can identify these using the `item_changed` many-to-many table described below.
You can use the `--full-versions` option to store full copies of the item at each version, rather than just storing the columns that have changed.
#### item_version_detail view
This SQL view joins `item_version` against `commits` to add three further columns: `_commit_at` with the date of the commit, and `_commit_hash` with the Git commit hash.
#### item_changed
This many-to-many table indicates exactly which columns were changed in an `item_version`.
- `item_version` is a foreign key to a row in the `item_version` table.
- `column` is a foreign key to a row in the `columns` table.
This table with have the largest number of rows, which is why it stores just two integers in order to save space.
#### columns
The `columns` table stores column names. It is referenced by `item_changed`.
- `id` - an integer ID.
- `name` - the name of the column.
- `namespace` - a foreign key to `namespaces`, for if multiple file histories are sharing the same database.
#### Reserved column names
Note that `_id`, `_item_full_hash`, `_item`, `_item_id`, `_version`, `_commit`, `_item_id`, `_commit_at`, `_commit_hash`, `_changed_columns`, `rowid` are considered reserved column names for the purposes of this tool.
If your data contains any of these they will be renamed to add a trailing underscore, for example `_id_`, `_item_`, `_version_`, to avoid clashing with the reserved columns.
If you have a column with a name such as `_commit_` it will be renamed too, adding an additional trailing underscore, so `_commit_` becomes `_commit__` and `_commit__` becomes `_commit___`.
### Additional options
- `--repo DIRECTORY` - the path to the Git repository, if it is not the current working directory.
- `--branch TEXT` - the Git branch to analyze - defaults to `main`.
- `--id TEXT` - as described above: pass one or more columns that uniquely identify a record, so that changes to that record can be calculated over time.
- `--full-versions` - instead of recording just the columns that have changed in the `item_version` table record a full copy of each version of theh item.
- `--ignore TEXT` - one or more columns to ignore - they will not be included in the resulting database.
- `--csv` - treat the data is CSV or TSV rather than JSON, and attempt to guess the correct dialect
- `--dialect` - use a spcific CSV dialect. Options are `excel`, `excel-tab` and `unix` - see [the Python CSV documentation](https://docs.python.org/3/library/csv.html#csv.excel) for details.
- `--skip TEXT` - one or more full Git commit hashes that should be skipped. You can use this if some of the data in your revision history is corrupted in a way that prevents this tool from working.
- `--start-at TEXT` - skip commits prior to the specified commit hash.
- `--start-after TEXT` - skip commits up to and including the specified commit hash, then start processing from the following commit.
- `--convert TEXT` - custom Python code for a conversion, described below.
- `--import TEXT` - additional Python modules to import for `--convert`.
- `--ignore-duplicate-ids` - if a single version of a file has the same ID in it more than once, the tool will exit with an error. Use this option to ignore this and instead pick just the first of the two duplicates.
- `--namespace TEXT` - use this if you wish to include the history of multiple different files in the same database. The default is `item` but you can set it to something else, which will produce tables with names like `yournamespace` and `yournamespace_version`.
- `--wal` - Enable WAL mode on the created database file. Use this if you plan to run queries against the database while `git-history` is creating it.
- `--silent` - don't show the progress bar.
### CSV and TSV data
If the data in your repository is a CSV or TSV file you can process it by adding the `--csv` option. This will attempt to detect which delimiter is used by the file, so the same option works for both comma- and tab-separated values.
git-history file trees.db trees.csv --id TreeID
You can also specify the CSV dialect using the `--dialect` option.
### Custom conversions using --convert
If your data is not already either CSV/TSV or a flat JSON array, you can reshape it using the `--convert` option.
The format needed by this tool is an array of dictionaries, as demonstrated by the `incidents.json` example above.
If your data does not fit this shape, you can provide a snippet of Python code to converts the on-disk content of each stored file into a Python list of dictionaries.
For example, if your stored files each look like this:
```json
{
""incidents"": [
{
""id"": ""552"",
""name"": ""Hawthorne Fire"",
""engines"": 3
},
{
""id"": ""556"",
""name"": ""Merlin Fire"",
""engines"": 1
}
]
}
```
You could use the following Python snippet to convert them to the required format:
```python
json.loads(content)[""incidents""]
```
(The `json` module is exposed to your custom function by default.)
You would then run the tool like this:
git-history file database.db incidents.json \
--id id \
--convert 'json.loads(content)[""incidents""]'
The `content` variable is always a `bytes` object representing the content of the file at a specific moment in the repository's history.
You can import additional modules using `--import`. This example shows how you could read a CSV file that uses `;` as the delimiter:
git-history file trees.db ../sf-tree-history/Street_Tree_List.csv \
--repo ../sf-tree-history \
--import csv \
--import io \
--convert '
fp = io.StringIO(content.decode(""utf-8""))
return list(csv.DictReader(fp, delimiter="";""))
' \
--id TreeID
You can import nested modules such as [ElementTree](https://docs.python.org/3/library/xml.etree.elementtree.html) using `--import xml.etree.ElementTree`, then refer to them in your function body as `xml.etree.ElementTree`. For example, if your tracked data was in an `items.xml` file that looked like this:
```xml