_id,_path,text,html,title,summary,order 0574db8535765cbe166676a6b73fd0205331313b,for/rapid-prototyping.md,"Want to build a prototype? Datasette lets you turn any source of data into a powerful JSON or [GraphQL](https://github.com/simonw/datasette-graphql) API, ideal for use as the backend for an interactive prototype. Use Datasette with **[Observable notebooks](https://observablehq.com/)** to quickly try out new visualizations and explore the Datasette API using JavaScript. Datasette works as a great data for **[Jupyter notebooks](https://jupyter.org/)** too. - **[Analyzing ScotRail audio announcements with Datasette—from prototype to production](https://simonwillison.net/2022/Aug/21/scotrail/)** talks through the full process of creating an initial rapid prototype using Datasette Lite and growing that into a full production Datasette instance. - **[Fast Autocomplete Search for Your Website](https://simonwillison.net/2018/Dec/19/fast-autocomplete-search/)** shows how to build a full autocomplete search engine for your website, powered by Datasette and a few dozen lines of custom JavaScript.","

Want to build a prototype? Datasette lets you turn any source of data into a powerful JSON or GraphQL API, ideal for use as the backend for an interactive prototype.

Use Datasette with Observable notebooks to quickly try out new visualizations and explore the Datasette API using JavaScript.

Datasette works as a great data for Jupyter notebooks too.

",Datasette for rapid prototyping,Spin up a JSON (or GraphQL) API for any data in minutes. Use it to prototype and prove your ideas without building a custom backend.,7 1940a8c4df2a2e0e7c95a457ce8dc4e7033c24d9,for/publishing-data.md,"Datasette was [originally inspired](https://simonwillison.net/2018/Aug/19/instantly-publish-datasette/) by work on [the Guardian Data Blog](https://www.theguardian.com/news/datablog/2011/jan/27/data-store-office-for-national-statistics), an initiative to publish the raw data behind the stories in the news. Datasette encourages sharing data as tables that can be explored by users and exported out as JSON or CSV. It's like publishing a CSV file but with much deeper flexibility for visitors to remix, visualize and export data. Data published by Datasette can be deployed to both traditional and serverless hosting platforms. The `datasette publish` command, [documented here](https://docs.datasette.io/en/stable/publish.html), can deploy to [Heroku](https://heroku.com/) and [Google Cloud Run](https://cloud.google.com/run). Plugins can be used to deploy Datasette on [Vercel](https://vercel.com/) or [fly.io](https://fly.io/). Datasette also works great on [Glitch](https://glitch.com/). When combined with GitHub Actions, Datasette can be used to automatically deploy new databases on a scheduled basis. See [Deploying a data API using GitHub Actions and Cloud Run](https://simonwillison.net/2020/Jan/21/github-actions-cloud-run/) for more details on implementing this pattern.","

Datasette was originally inspired by work on the Guardian Data Blog, an initiative to publish the raw data behind the stories in the news.

Datasette encourages sharing data as tables that can be explored by users and exported out as JSON or CSV. It's like publishing a CSV file but with much deeper flexibility for visitors to remix, visualize and export data.

Data published by Datasette can be deployed to both traditional and serverless hosting platforms.

The datasette publish command, documented here, can deploy to Heroku and Google Cloud Run. Plugins can be used to deploy Datasette on Vercel or fly.io.

Datasette also works great on Glitch.

When combined with GitHub Actions, Datasette can be used to automatically deploy new databases on a scheduled basis. See Deploying a data API using GitHub Actions and Cloud Run for more details on implementing this pattern.

",Datasette for publishing data,"The built-in ""datasette publish"" command lets you instantly publish your data to hosting providers like Google Cloud Run, Heroku or Vercel.",2 2a478757dc7f379d19b6c77d1710024b8f02a1e1,for/serverless.md,"Serverless providers such as [Google Cloud Run](https://cloud.google.com/run) and [Vercel](https://vercel.com/) provide robust, scalable, inexpensive hosting for stateless web applications. Databases are usually provided as separate add-ons to serverless hosting. If your data is static or changes infrequently, Datasette can be used to unlock the benefits of serverless hosting without needing a separate hosted database. Datasette can be used to implement the [Baked Data](https://simonwillison.net/2021/Jul/28/baked-data/) pattern, where any structured data needed by your application is bundled and deployed with the application code itself, as part of the same container. This site is an example of Baked Data in action: datasette.io is deployed to Google Cloud Run by build scripts running [on GitHub Actions](https://github.com/simonw/datasette.io/blob/main/.github/workflows/deploy.yml). Many of the [Datasette examples](/examples) listed on this site are also deployed in this manner.","

Serverless providers such as Google Cloud Run and Vercel provide robust, scalable, inexpensive hosting for stateless web applications.

Databases are usually provided as separate add-ons to serverless hosting. If your data is static or changes infrequently, Datasette can be used to unlock the benefits of serverless hosting without needing a separate hosted database.

Datasette can be used to implement the Baked Data pattern, where any structured data needed by your application is bundled and deployed with the application code itself, as part of the same container.

This site is an example of Baked Data in action: datasette.io is deployed to Google Cloud Run by build scripts running on GitHub Actions.

Many of the Datasette examples listed on this site are also deployed in this manner.

",Serverless read-only databases with Datasette,"Serverless hosting usually depends on external hosted databases, but Datasette lets you implement the Baked Data pattern to bundle your data with your application code and deploy it to serverless providers.",8 41624e65228293a1622b9eee3a435699801dead1,for/websites.md,"Datasette templates can be used to build custom websites with Datasette under the hood. Examples include the site you are browsing right now. ## [datasette.io](https://datasette.io/) The official Datasette website runs on top of a customized Datasette instance. - [datasette.io, an official project website for Datasette](https://simonwillison.net/2020/Dec/13/datasette-io/) talks through the basics of how the site works. - [Building a search engine for datasette.io](https://simonwillison.net/2020/Dec/19/dogsheep-beta/) explains how the [site search](https://datasette.io/-/beta) works, using the [dogsheep-beta](https://datasette.io/plugins/dogsheep-beta) search plugin. - [The Baked Data architectural pattern](https://simonwillison.net/2021/Jul/28/baked-data/) describes the Baked Data architecture pattern used by the site, and provides a detailed description of how the build scripts for the site work. - [github.com/simonw/datasette.io](https://github.com/simonw/datasette.io) has the full source code for the site, including the GitHub Actions workflows that build the databases and deploy them to Google Cloud Run. - [datasette.io/content](https://datasette.io/content) is the Datasette interface for the main database that powers the site. ## [til.simonwillison.net](https://til.simonwillison.net/) A blog sharing TILs - Today I Learned snippets. - [Using a self-rewriting README powered by GitHub Actions to track TILs](https://simonwillison.net/2020/Apr/20/self-rewriting-readme/) introduces the site. - The site uses the [datasette-atom](https://datasette.io/plugins/datasette-atom) plugin to provide an Atom feed of new entries. - [til.simonwillison.net/tils](https://til.simonwillison.net/tils) exposes the underlying database. - The site is deployed to [Vercel](https://vercel.com/) using the [datasette-publish-vercel](https://datasette.io/plugins/datasette-publish-vercel) plugin. ## [Niche Museums](https://www.niche-museums.com/) [www.niche-museums.com](https://www.niche-museums.com/) is a directory of small and niche museums, powered by Datasette. Read [niche-museums.com, powered by Datasette](https://simonwillison.net/2019/Nov/25/niche-museums/) for details, or browse [the code](https://github.com/simonw/museums) on GitHub. ## [Rocky Beaches](https://www.rockybeaches.com/us/pillar-point) [Rocky Beaches](https://www.rockybeaches.com/us/pillar-point) by Natalie Downe shows the best times to go tidepooling at Pillar Point near Half Moon Bay, California. It runs on Datasette, using SQL queries to suggest the best times to go based on low tide times and sunrise/sunset and display recent observations recorded on [iNaturalist](https://www.inaturalist.org/). The source code is available at [github.com/natbat/rockybeaches](https://github.com/natbat/rockybeaches). The site uses the [datasette-graphql](https://datasette.io/plugins/datasette-graphql) in some of the templates, for example [here](https://github.com/natbat/rockybeaches/blob/e072d48479f1404baa206ee660ad228d9f95ac7a/templates/row-data-places.html#L273-L290).","

Datasette templates can be used to build custom websites with Datasette under the hood. Examples include the site you are browsing right now.

datasette.io

The official Datasette website runs on top of a customized Datasette instance.

til.simonwillison.net

A blog sharing TILs - Today I Learned snippets.

Niche Museums

www.niche-museums.com is a directory of small and niche museums, powered by Datasette. Read niche-museums.com, powered by Datasette for details, or browse the code on GitHub.

Rocky Beaches

Rocky Beaches by Natalie Downe shows the best times to go tidepooling at Pillar Point near Half Moon Bay, California. It runs on Datasette, using SQL queries to suggest the best times to go based on low tide times and sunrise/sunset and display recent observations recorded on iNaturalist.

The source code is available at github.com/natbat/rockybeaches. The site uses the datasette-graphql in some of the templates, for example here.

",Datasette for websites,Datasette can be used to power dynamic-static websites: sites that run on serverless hosting while providing dynamic data-backed functionality.,3 61a01ca9a0e3de0a0f0af9691661af44ea6e149d,for/data-journalism.md,"Datasette was originally designed as a tool for data journalists, to help report on data-driven stories, crunch through large datasets and [publish the results](/for/publishing-data). [This demo](https://vimeo.com/436903714) from [SRCCON 2020](https://2020.srccon.org/) shows how Datasette can be used to analyze PPP loan data released by the Small Business Administration. You can try the demo [here](https://sba-loans-covid-19.datasettes.com/), or read more about it in [this article](https://simonwillison.net/2020/Jul/9/sba-covid-19-ppp-loans/).","

Datasette was originally designed as a tool for data journalists, to help report on data-driven stories, crunch through large datasets and publish the results.

This demo from SRCCON 2020 shows how Datasette can be used to analyze PPP loan data released by the Small Business Administration.

You can try the demo here, or read more about it in this article.

",Datasette for data journalism,"Datasette was originally designed as a tool for data journalists, to help report on data-driven stories, crunch through large datasets and publish the results.",4 863b689b3625af13bbba9bb9077c997fe3a04d13,for/search.md,"Datasette is built on top of [SQLite](https://sqlite.org/), which includes a robust, full-featured [full-text search](https://www.sqlite.org/fts5.html) implementation. Datasette automatically detects tables which have been configured for full-text search and adds a search box which can be used by human site users or accessed from the JSON or CSV APIs. You can try this out now by [searching FARA records](https://fara.datasettes.com/) or [global power plants](https://global-power-plants.datasettes.com/global-power-plants/global-power-plants). ## Enabling search for a table The [sqlite-utils command-line tool](https://sqlite-utils.datasette.io/en/stable/cli.html#configuring-full-text-search) can be used to enable full-text search on a specific set of columns for a table: % sqlite-utils enable-fts mydb.db documents title summary Alternatively, you can configure search directly within the Datasette interface by installing the [datasette-configure-fts](https://github.com/simonw/datasette-configure-fts) plugin. [Fast Autocomplete Search for Your Website](https://simonwillison.net/2018/Dec/19/fast-autocomplete-search/) is a tutorial that shows how to write a scraper that populates a full-text search index in Datasette, then build a JavaScript autocomplete search interface using the Datasette JSON API. ## Implementing search with a canned query Datasette [canned queries](https://docs.datasette.io/en/stable/sql_queries.html#canned-queries) can be used to configure a SQL query which returns search results for a table, ordered by relevance. The search on [www.niche-museums.com](https://www.niche-museums.com/) uses this approach. Here's an example search, for `bones`: - [www.niche-museums.com/browse/search?q=bones](https://www.niche-museums.com/browse/search?q=bones) This uses the `search` canned query, which is [defined here]([https://github.com/simonw/til/blob/8f961be162868c53b5c484272091bdab703a747a/metadata.yaml#L16-L32](https://github.com/simonw/museums/blob/74e999c0e82781302bf0346a761ee5d88e168863/metadata.yaml#L55-L69)) and looks like this: select museums_fts.rank, museums.* from museums join museums_fts on museums.id = museums_fts.rowid where museums_fts match case :q when '' then '*' else escape_fts_query(:q) end order by museums_fts.rank Try [that query here](https://www.niche-museums.com/browse?sql=select%0D%0A++museums_fts.rank%2C%0D%0A++museums.*%0D%0Afrom%0D%0A++museums%0D%0A++join+museums_fts+on+museums.id+%3D+museums_fts.rowid%0D%0Awhere%0D%0A++museums_fts+match+case%0D%0A++++%3Aq%0D%0A++++when+%27%27+then+%27*%27%0D%0A++++else+escape_fts_query%28%3Aq%29%0D%0A++end%0D%0Aorder+by%0D%0A++museums_fts.rank&q=bones). The results are then rendered by [this custom template](https://github.com/simonw/museums/blob/74e999c0e82781302bf0346a761ee5d88e168863/templates/query-browse-search.html). ## Plugins for search The **[datasette-search-all](https://datasette.io/plugins/datasette-search-all)** plugin adds a search box which runs searches in parallel against all of the FTS-configured tables in all of the databases attached to Datasette. You can see that in action here: [fara.datasettes.com/-/search?q=manafort](https://fara.datasettes.com/-/search?q=manafort) **[dogsheep-beta](https://datasette.io/plugins/dogsheep-beta)** is a plugin which adds a customizable search engine that can create an index to search multiple tables at once with combined relevance scoring. It is used for the search on this site, for example [datasette.io/-/beta?q=fts](https://datasette.io/-/beta?q=fts) - you can read more about how it works in [Building a search engine for datasette.io](https://simonwillison.net/2020/Dec/19/dogsheep-beta/).","

Datasette is built on top of SQLite, which includes a robust, full-featured full-text search implementation.

Datasette automatically detects tables which have been configured for full-text search and adds a search box which can be used by human site users or accessed from the JSON or CSV APIs.

You can try this out now by searching FARA records or global power plants.

Enabling search for a table

The sqlite-utils command-line tool can be used to enable full-text search on a specific set of columns for a table:

% sqlite-utils enable-fts mydb.db documents title summary

Alternatively, you can configure search directly within the Datasette interface by installing the datasette-configure-fts plugin.

Fast Autocomplete Search for Your Website is a tutorial that shows how to write a scraper that populates a full-text search index in Datasette, then build a JavaScript autocomplete search interface using the Datasette JSON API.

Implementing search with a canned query

Datasette canned queries can be used to configure a SQL query which returns search results for a table, ordered by relevance.

The search on www.niche-museums.com uses this approach. Here's an example search, for bones:

This uses the search canned query, which is defined here and looks like this:

select
  museums_fts.rank,
  museums.*
from
  museums
  join museums_fts on museums.id = museums_fts.rowid
where
  museums_fts match case
    :q
    when '' then '*'
    else escape_fts_query(:q)
  end
order by
  museums_fts.rank

Try that query here.

The results are then rendered by this custom template.

Plugins for search

The datasette-search-all plugin adds a search box which runs searches in parallel against all of the FTS-configured tables in all of the databases attached to Datasette.

You can see that in action here: fara.datasettes.com/-/search?q=manafort

dogsheep-beta is a plugin which adds a customizable search engine that can create an index to search multiple tables at once with combined relevance scoring. It is used for the search on this site, for example datasette.io/-/beta?q=fts - you can read more about how it works in Building a search engine for datasette.io.

",Datasette for search,Datasette uses SQLite's built-in full-text search feature to provide faceted search over any correctly configured collection of records.,6 ce0b74d7bd16f128e2bd4664b3aa910162b5373f,for/geospatial.md,"The [SpatiaLite](https://www.gaia-gis.it/fossil/libspatialite/index) extension for SQLite can be used with Datasette to enable a full suite of geospatial SQL functions. ![A screenshot of Datasette running some geospatial plugins](https://camo.githubusercontent.com/56a2ed7e65e28e726d4c3abfbe56bd7960524e6af6512fc2cee083bdd7fcd863/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032312f6461746173657474652d6c6561666c65742d66726565647261772e706e67) Datasette has a number of plugins and tools that can be used to work with geospatial data. Geospatial plugins include: - [datasette-cluster-map](https://datasette.io/plugins/datasette-cluster-map) renders a map of markers for any table or query with `latitude` and `longitude` columns - [datasette-geojson](https://datasette.io/plugins/datasette-geojson) adds a `.geojson` extension which can export SpatiaLite geometries as [GeoJSON](https://geojson.org/) - [datasette-geojson-map](https://datasette.io/plugins/datasette-geojson-map) renders that GeoJSON output on a map - [datasette-leaflet-freedraw](https://datasette.io/plugins/datasette-leaflet-freedraw) adds an interface to filter geometries by drawing a shape on a map - [datasette-leaflet-geojson](https://datasette.io/plugins/datasette-leaflet-geojson) shows maps inline for any column that includes GeoJSON - [datasette-tiles](https://datasette.io/plugins/datasette-tiles) allows Datasette to serve map tile images stored using the [MBTiles](https://github.com/mapbox/mbtiles-spec) format Tools for working with geospatial data include: - [geojson-to-sqlite](https://datasette.io/tools/geojson-to-sqlite), a command line utility for loading GeoJSON data into a SQLite or SpatiaLite database - [shapefile-to-sqlite](https://datasette.io/tools/shapefile-to-sqlite), a command line utility for loading shapefiles into SQLite or SpatiaLite - [download-tiles](https://datasette.io/tools/download-tiles) can be used to download map tiles and store them in MBTiles, suitable for use with [datasette-tiles](https://datasette.io/plugins/datasette-tiles) The following tutorials provide more detail on how Datasette can be used for geospatial processing: - [GUnion to combine geometries in SpatiaLite](https://til.simonwillison.net/spatialite/gunion-to-combine-geometries) shows how to load Amtrak data using `geojson-to-sqlite` and render it using `datasette-geojson-map` - [KNN queries with SpatiaLite](https://til.simonwillison.net/spatialite/knn) shows how to use the KNN feature to run queries showing the nearest geometries to a point - [Natural Earth in SpatiaLite and Datasette](https://til.simonwillison.net/gis/natural-earth-in-spatialite-and-datasette) shows how to use the 791MB [Natural Earth](https://www.naturalearthdata.com/) SQLite vector database with Datasette - [Drawing shapes on a map to query a SpatiaLite database](https://simonwillison.net/2021/Jan/24/drawing-shapes-spatialite/) introduces the `datasette-leaflet-freedraw` plugin","

The SpatiaLite extension for SQLite can be used with Datasette to enable a full suite of geospatial SQL functions.

Datasette has a number of plugins and tools that can be used to work with geospatial data.

Geospatial plugins include:

Tools for working with geospatial data include:

The following tutorials provide more detail on how Datasette can be used for geospatial processing:

",Datasette for geospatial analysis,Combining Datasette with SpatiaLite provides a powerful set of tools for analyzing and visualizing geospatial data.,5 ddae37b01edec73668ab75f7e9799f91b580a554,for/exploratory-analysis.md,"Your first step with any new data set should be to start exploring it. Datasette provides multiple tools for doing this. ## Start browsing the data Datasette's table interface lets you start scrolling through data straight away. ![Screenshot of the table interface](/static/datasette-screenshot.png) ## Facets Datasette automatically identifies columns with less than twenty unique values and gives you the option to facet by them - seeing the most common values, and selecting those to filter the set. This provides a powerful way to start spotting patterns in the data. ![Screenshot of the facet interface](/static/screenshots/facets.png) ## Visualization Plugins such as [datasette-vega](https://github.com/simonw/datasette-vega) and [datasette-cluster-map](https://github.com/simonw/datasette-cluster-map) provide tools for interactively visualizing data directly within the Datasette interface. ![Screenshot of datasette-cluster-map](/static/screenshots/datasette-cluster-map.png)","

Your first step with any new data set should be to start exploring it. Datasette provides multiple tools for doing this.

Start browsing the data

Datasette's table interface lets you start scrolling through data straight away.

Facets

Datasette automatically identifies columns with less than twenty unique values and gives you the option to facet by them - seeing the most common values, and selecting those to filter the set. This provides a powerful way to start spotting patterns in the data.

Visualization

Plugins such as datasette-vega and datasette-cluster-map provide tools for interactively visualizing data directly within the Datasette interface.

",Datasette for exploratory analysis,"Import data from CSVs, JSON, database connections and more. Datasette will automatically show you patterns in your data and help you share your findings with your colleagues.",1