repos: 479175467
This data as json
| id | node_id | name | full_name | private | owner | html_url | description | fork | created_at | updated_at | pushed_at | homepage | size | stargazers_count | watchers_count | language | has_issues | has_projects | has_downloads | has_wiki | has_pages | forks_count | archived | disabled | open_issues_count | license | topics | forks | open_issues | watchers | default_branch | permissions | temp_clone_token | organization | network_count | subscribers_count | readme | readme_html | allow_forking | visibility | is_template | template_repository | web_commit_signoff_required | has_discussions |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 479175467 | R_kgDOHI-jKw | pypi-to-sqlite | simonw/pypi-to-sqlite | 0 | 9599 | https://github.com/simonw/pypi-to-sqlite | Load data about Python packages from PyPI into SQLite | 0 | 2022-04-07T23:09:02Z | 2022-04-08T15:16:03Z | 2022-04-08T16:29:53Z | 24 | 2 | 2 | Python | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | apache-2.0 | [] | 0 | 1 | 2 | main | {"admin": false, "maintain": false, "push": false, "triage": false, "pull": false} | 0 | 1 | # pypi-to-sqlite [](https://pypi.org/project/pypi-to-sqlite/) [](https://github.com/simonw/pypi-to-sqlite/releases) [](https://github.com/simonw/pypi-to-sqlite/actions?query=workflow%3ATest) [](https://github.com/simonw/pypi-to-sqlite/blob/master/LICENSE) Load data about Python packages from PyPI into SQLite ## Installation Install this tool using `pip`: pip install pypi-to-sqlite ## Usage To create a SQLite database with details of one or more packages, run: pypi-to-sqlite pypi.db datasette sqlite-utils You can also process JSON that you have previously saved to disk like so: curl -o datasette.json https://pypi.org/pypi/datasette/json pypi-to-sqlite pypi.db -f datasette.json The tool will create three tables: `packages`, `versions` and `releases`. The full table schema is shown below. To create the tables with a prefix, use `--prefix prefix`. For example: pypi-to-sqlite pypi.db datasette --prefix pypi_ This will create tables called `pypi_packages`, `pypi_versions` and `pypi_releases`. ## Demo You can see examples of tables created using this tool running in [Datasette](https://datasette.io/) here: - [packages](https://datasette.io/content/pypi_packages) - [versions](https://datasette.io/content/pypi_versions) - [releases](https://datasette.io/content/pypi_releases) ## Database schema <!-- [[[cog import cog, json from pypi_to_sqlite import cli from click.testing import CliRunner import sqlite_utils import tempfile, pathlib tmpdir = pathlib.Path(tempfile.mkdtemp()) db_path = str(tmpdir / "pypi.db") runner = CliRunner() result = runner.invoke(cli.cli, [db_path, "-f", "tests/datasette-block.json"]) cog.out("```sql\n") cog.out(sqlite_utils.Database(db_path).schema) cog.out("\n```") ]]] --> ```sql CREATE TABLE [packages] ( [name] TEXT PRIMARY KEY, [summary] TEXT, [classifiers] TEXT, [description] TEXT, [author] TEXT, [author_email] TEXT, [description_content_type] TEXT, [home_page] TEXT, [keywords] TEXT, [license] TEXT, [maintainer] TEXT, [maintainer_email] TEXT, [package_url] TEXT, [platform] TEXT, [project_url] TEXT, [project_urls] TEXT, [release_url] TEXT, [requires_dist] TEXT, [requires_python] TEXT, [version] TEXT, [yanked] INTEGER, [yanked_reason] TEXT ); CREATE TABLE [versions] ( [id] TEXT PRIMARY KEY, [package] TEXT REFERENCES [packages]([name]), [name] TEXT ); CREATE TABLE [releases] ( [md5_digest] TEXT PRIMARY KEY, [package] TEXT REFERENCES [packages]([name]), [version] TEXT REFERENCES [versions]([id]), [packagetype] TEXT, [filename] TEXT, [comment_text] TEXT, [digests] TEXT, [has_sig] INTEGER, [python_version] TEXT, [requires_python] TEXT, [size] INTEGER, [upload_time] TEXT, [upload_time_iso_8601] TEXT, [url] TEXT, [yanked] INTEGER, [yanked_reason] TEXT ); ``` <!-- [[[end]]] --> ## pypi-to-sqlite --help <!-- [[[cog result = runner.invoke(cli.cli, ["--help"]) cog.out("```\n") cog.out(result.output.replace("Usage: cli", "Usage: pypi-to-sqlite")) cog.out("\n```") ]]] --> ``` Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]... Load data about Python packages from PyPI into SQLite Usage example: pypi-to-sqlite pypy.db datasette sqlite-utils Use -f to load data from a JSON file instead: pypi-to-sqlite pypy.db -f datasette.json Created tables will be packages, versions and releases To create tables called pypi_packages, pypi_versions, pypi_releases use --prefix pypi_: pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_ Options: --version Show the version and exit. -f, --file FILENAME Import JSON from this file -d, --delay FLOAT Wait this many seconds between requests --prefix TEXT Prefix to use for the created database tables --help Show this message and exit. ``` <!-- [[[end]]] --> ## Development To contribute to this tool, first checkout the code. Then create a new virtual environment: cd pypi-to-sqlite python -m venv venv source venv/bin/activate Now install the dependencies and test dependencies: pip install -e '.[test]' To run the tests: pytest | <div id="readme" class="md" data-path="README.md"><article class="markdown-body entry-content container-lg" itemprop="text"><h1 dir="auto"><a id="user-content-pypi-to-sqlite" class="anchor" aria-hidden="true" href="#user-content-pypi-to-sqlite"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>pypi-to-sqlite</h1> <p dir="auto"><a href="https://pypi.org/project/pypi-to-sqlite/" rel="nofollow"><img src="https://camo.githubusercontent.com/200eab1d4c07550744081184a86be6c72391ed7061054162919ec2ad2b48fa09/68747470733a2f2f696d672e736869656c64732e696f2f707970692f762f707970692d746f2d73716c6974652e737667" alt="PyPI" data-canonical-src="https://img.shields.io/pypi/v/pypi-to-sqlite.svg" style="max-width: 100%;"></a> <a href="https://github.com/simonw/pypi-to-sqlite/releases"><img src="https://camo.githubusercontent.com/5efc8be881faf986aca6a6396ff257b787a9be964ef6f1328e46568544be0847/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f762f72656c656173652f73696d6f6e772f707970692d746f2d73716c6974653f696e636c7564655f70726572656c6561736573266c6162656c3d6368616e67656c6f67" alt="Changelog" data-canonical-src="https://img.shields.io/github/v/release/simonw/pypi-to-sqlite?include_prereleases&label=changelog" style="max-width: 100%;"></a> <a href="https://github.com/simonw/pypi-to-sqlite/actions?query=workflow%3ATest"><img src="https://github.com/simonw/pypi-to-sqlite/workflows/Test/badge.svg" alt="Tests" style="max-width: 100%;"></a> <a href="https://github.com/simonw/pypi-to-sqlite/blob/master/LICENSE"><img src="https://camo.githubusercontent.com/1698104e976c681143eb0841f9675c6f802bb7aa832afc0c7a4e719b1f3cf955/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d417061636865253230322e302d626c75652e737667" alt="License" data-canonical-src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" style="max-width: 100%;"></a></p> <p dir="auto">Load data about Python packages from PyPI into SQLite</p> <h2 dir="auto"><a id="user-content-installation" class="anchor" aria-hidden="true" href="#user-content-installation"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Installation</h2> <p dir="auto">Install this tool using <code>pip</code>:</p> <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install pypi-to-sqlite"><pre><code>pip install pypi-to-sqlite </code></pre></div> <h2 dir="auto"><a id="user-content-usage" class="anchor" aria-hidden="true" href="#user-content-usage"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Usage</h2> <p dir="auto">To create a SQLite database with details of one or more packages, run:</p> <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pypi-to-sqlite pypi.db datasette sqlite-utils"><pre><code>pypi-to-sqlite pypi.db datasette sqlite-utils </code></pre></div> <p dir="auto">You can also process JSON that you have previously saved to disk like so:</p> <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -o datasette.json https://pypi.org/pypi/datasette/json pypi-to-sqlite pypi.db -f datasette.json"><pre><code>curl -o datasette.json https://pypi.org/pypi/datasette/json pypi-to-sqlite pypi.db -f datasette.json </code></pre></div> <p dir="auto">The tool will create three tables: <code>packages</code>, <code>versions</code> and <code>releases</code>. The full table schema is shown below.</p> <p dir="auto">To create the tables with a prefix, use <code>--prefix prefix</code>. For example:</p> <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pypi-to-sqlite pypi.db datasette --prefix pypi_"><pre><code>pypi-to-sqlite pypi.db datasette --prefix pypi_ </code></pre></div> <p dir="auto">This will create tables called <code>pypi_packages</code>, <code>pypi_versions</code> and <code>pypi_releases</code>.</p> <h2 dir="auto"><a id="user-content-demo" class="anchor" aria-hidden="true" href="#user-content-demo"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Demo</h2> <p dir="auto">You can see examples of tables created using this tool running in <a href="https://datasette.io/" rel="nofollow">Datasette</a> here:</p> <ul dir="auto"> <li><a href="https://datasette.io/content/pypi_packages" rel="nofollow">packages</a></li> <li><a href="https://datasette.io/content/pypi_versions" rel="nofollow">versions</a></li> <li><a href="https://datasette.io/content/pypi_releases" rel="nofollow">releases</a></li> </ul> <h2 dir="auto"><a id="user-content-database-schema" class="anchor" aria-hidden="true" href="#user-content-database-schema"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Database schema</h2> <div class="highlight highlight-source-sql position-relative overflow-auto" data-snippet-clipboard-copy-content="CREATE TABLE [packages] ( [name] TEXT PRIMARY KEY, [summary] TEXT, [classifiers] TEXT, [description] TEXT, [author] TEXT, [author_email] TEXT, [description_content_type] TEXT, [home_page] TEXT, [keywords] TEXT, [license] TEXT, [maintainer] TEXT, [maintainer_email] TEXT, [package_url] TEXT, [platform] TEXT, [project_url] TEXT, [project_urls] TEXT, [release_url] TEXT, [requires_dist] TEXT, [requires_python] TEXT, [version] TEXT, [yanked] INTEGER, [yanked_reason] TEXT ); CREATE TABLE [versions] ( [id] TEXT PRIMARY KEY, [package] TEXT REFERENCES [packages]([name]), [name] TEXT ); CREATE TABLE [releases] ( [md5_digest] TEXT PRIMARY KEY, [package] TEXT REFERENCES [packages]([name]), [version] TEXT REFERENCES [versions]([id]), [packagetype] TEXT, [filename] TEXT, [comment_text] TEXT, [digests] TEXT, [has_sig] INTEGER, [python_version] TEXT, [requires_python] TEXT, [size] INTEGER, [upload_time] TEXT, [upload_time_iso_8601] TEXT, [url] TEXT, [yanked] INTEGER, [yanked_reason] TEXT );"><pre>CREATE TABLE [packages] ( [name] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>, [summary] <span class="pl-k">TEXT</span>, [classifiers] <span class="pl-k">TEXT</span>, [description] <span class="pl-k">TEXT</span>, [author] <span class="pl-k">TEXT</span>, [author_email] <span class="pl-k">TEXT</span>, [description_content_type] <span class="pl-k">TEXT</span>, [home_page] <span class="pl-k">TEXT</span>, [keywords] <span class="pl-k">TEXT</span>, [license] <span class="pl-k">TEXT</span>, [maintainer] <span class="pl-k">TEXT</span>, [maintainer_email] <span class="pl-k">TEXT</span>, [package_url] <span class="pl-k">TEXT</span>, [platform] <span class="pl-k">TEXT</span>, [project_url] <span class="pl-k">TEXT</span>, [project_urls] <span class="pl-k">TEXT</span>, [release_url] <span class="pl-k">TEXT</span>, [requires_dist] <span class="pl-k">TEXT</span>, [requires_python] <span class="pl-k">TEXT</span>, [version] <span class="pl-k">TEXT</span>, [yanked] <span class="pl-k">INTEGER</span>, [yanked_reason] <span class="pl-k">TEXT</span> ); CREATE TABLE [versions] ( [id] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>, [package] <span class="pl-k">TEXT</span> <span class="pl-k">REFERENCES</span> [packages]([name]), [name] <span class="pl-k">TEXT</span> ); CREATE TABLE [releases] ( [md5_digest] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>, [package] <span class="pl-k">TEXT</span> <span class="pl-k">REFERENCES</span> [packages]([name]), [version] <span class="pl-k">TEXT</span> <span class="pl-k">REFERENCES</span> [versions]([id]), [packagetype] <span class="pl-k">TEXT</span>, [filename] <span class="pl-k">TEXT</span>, [comment_text] <span class="pl-k">TEXT</span>, [digests] <span class="pl-k">TEXT</span>, [has_sig] <span class="pl-k">INTEGER</span>, [python_version] <span class="pl-k">TEXT</span>, [requires_python] <span class="pl-k">TEXT</span>, [size] <span class="pl-k">INTEGER</span>, [upload_time] <span class="pl-k">TEXT</span>, [upload_time_iso_8601] <span class="pl-k">TEXT</span>, [url] <span class="pl-k">TEXT</span>, [yanked] <span class="pl-k">INTEGER</span>, [yanked_reason] <span class="pl-k">TEXT</span> );</pre></div> <h2 dir="auto"><a id="user-content-pypi-to-sqlite---help" class="anchor" aria-hidden="true" href="#user-content-pypi-to-sqlite---help"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>pypi-to-sqlite --help</h2> <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]... Load data about Python packages from PyPI into SQLite Usage example: pypi-to-sqlite pypy.db datasette sqlite-utils Use -f to load data from a JSON file instead: pypi-to-sqlite pypy.db -f datasette.json Created tables will be packages, versions and releases To create tables called pypi_packages, pypi_versions, pypi_releases use --prefix pypi_: pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_ Options: --version Show the version and exit. -f, --file FILENAME Import JSON from this file -d, --delay FLOAT Wait this many seconds between requests --prefix TEXT Prefix to use for the created database tables --help Show this message and exit. "><pre><code>Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]... Load data about Python packages from PyPI into SQLite Usage example: pypi-to-sqlite pypy.db datasette sqlite-utils Use -f to load data from a JSON file instead: pypi-to-sqlite pypy.db -f datasette.json Created tables will be packages, versions and releases To create tables called pypi_packages, pypi_versions, pypi_releases use --prefix pypi_: pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_ Options: --version Show the version and exit. -f, --file FILENAME Import JSON from this file -d, --delay FLOAT Wait this many seconds between requests --prefix TEXT Prefix to use for the created database tables --help Show this message and exit. </code></pre></div> <h2 dir="auto"><a id="user-content-development" class="anchor" aria-hidden="true" href="#user-content-development"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Development</h2> <p dir="auto">To contribute to this tool, first checkout the code. Then create a new virtual environment:</p> <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="cd pypi-to-sqlite python -m venv venv source venv/bin/activate"><pre><code>cd pypi-to-sqlite python -m venv venv source venv/bin/activate </code></pre></div> <p dir="auto">Now install the dependencies and test dependencies:</p> <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install -e '.[test]'"><pre><code>pip install -e '.[test]' </code></pre></div> <p dir="auto">To run the tests:</p> <div class="snippet-clipboard-content position-relative overflow-auto" data-snippet-clipboard-copy-content="pytest"><pre><code>pytest </code></pre></div> </article></div> | 1 | public | 0 |
Links from other tables
- 3 rows from repo in releases