<environment_context>
<cwd>/project/testbed</cwd>
<approval_policy>never</approval_policy>
<sandbox_mode>danger-full-access</sandbox_mode>
<network_access>enabled</network_access>
<shell>bash</shell>
</environment_context>
<environment_context>
<cwd>/project/testbed</cwd>
<approval_policy>never</approval_policy>
<sandbox_mode>danger-full-access</sandbox_mode>
<network_access>enabled</network_access>
<shell>bash</shell>
</environment_context>
Please edit the codebase to address the following task:
Add support for postgresql client certificates and key to dbshell.
Description
This bug is very similar to the #28322
A common security procedure for DB access is to require mutual TLS for the DB connection.
This involves specifying a server certificate, client certificate, and client key when connecting.
Django already supports this configuration, it looks like this:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': os.environ.get('POSTGRES_DB_NAME'),
'USER': os.environ.get('POSTGRES_DB_USER'),
'HOST': 'postgres',
'PORT': '5432',
'SCHEMA': os.environ.get('POSTGRES_DB_SCHEMA'),
'OPTIONS': {
'sslmode': 'verify-ca',
'sslrootcert': os.environ.get('POSTGRES_CLI_SSL_CA', 'ca.crt'),
'sslcert': os.environ.get('POSTGRES_CLI_SSL_CRT', 'client_cert_chain.crt'),
'sslkey': os.environ.get('POSTGRES_CLI_SSL_KEY', 'client_key.key')
}
}
}
However the dbshell command does not support the client cert params.
Should be a trivial fix to add in support for the other 'ssl' parameters required here.
I’ll inspect the dbshell plumbing and patch the PostgreSQL args.
| command | [
"bash",
"-lc",
"pwd && rg -n \"dbshell|sslrootcert|sslcert|sslkey|postgres\" -S ."
] |
|---|---|
| workdir | /project/testbed |
| timeout_ms | 10000 |
Total output lines: 1
{"output":"Total output lines: 1166\n\n/project/testbed\n./tox.ini:25: postgres: -rtests/requirements/postgres.txt\n./docs/index.txt:120: :doc:`PostgreSQL specific features <ref/contrib/postgres/index>`\n./docs/spelling_wordlist:462:postgres\n./docs/spelling_wordlist:463:postgresql\n./docs/man/django-admin.1:273:.SS \\fBdbshell\\fP\n./docs/man/django-admin.1:276:.B django\\-admin dbshell\n./docs/man/django-admin.1:284:For PostgreSQL, this runs the \\fBpsql\\fP command\\-line client.\n./docs/man/django-admin.1:478:created on PostgreSQL if the \\fI\\%\\-\\-include\\-partitions\\fP option is used.\n./docs/man/django-admin.1:493:\\fBINSTALLED_APPS\\fP\\&. For example, \\fBdjango.contrib.postgres\\fP adds\n./docs/man/django-admin.1:494:recognition for several PostgreSQL\\-specific field types.\n./docs/man/django-admin.1:529:.SS PostgreSQL\n./docs/man/django-admin.1:558:Only support for PostgreSQL is implemented.\n./docs/faq/install.txt:32:also need a database engine. PostgreSQL_ is recommended, because we're\n./docs/faq/install.txt:33:PostgreSQL fans, and MySQL_, `SQLite`_, and Oracle_ are also supported.\n./docs/faq/install.txt:36:.. _PostgreSQL: https://www.postgresql.org/\n./docs/faq/general.txt:25:projects -- `Apache`_, `Python`_, and `PostgreSQL`_ to name a few -- and we're\n./docs/faq/general.txt:30:.. _PostgreSQL: https://www.postgresql.org/\n./docs/intro/tutorial02.txt:19:more scalable database like PostgreSQL, to avoid database-switching headaches\n./docs/intro/tutorial02.txt:29: ``'django.db.backends.postgresql'``,\n./docs/intro/tutorial02.txt:97:database and type ``\\dt`` (PostgreSQL), ``SHOW TABLES;`` (MySQL), ``.schema``\n./docs/intro/tutorial02.txt:302: example above is generated for PostgreSQL.\n./docs/intro/tutorial02.txt:315: PostgreSQL to not enforce the foreign key until the end of the transaction.\n./docs/intro/tutorial02.txt:318: such as ``auto_increment`` (MySQL), ``serial`` (PostgreSQL), or ``integer\n./docs/intro/install.txt:34:like PostgreSQL, MySQL, or Oracle. To install such a database, consult the\n./docs/releases/2.2.txt:65::mod:`django.contrib.postgres`\n./docs/releases/2.2.txt:69: :class:`~django.contrib.postgres.aggregates.ArrayAgg` and\n./docs/releases/2.2.txt:70: :class:`~django.contrib.postgres.aggregates.StringAgg` determines the\n./docs/releases/2.2.txt:73:* The new :class:`~django.contrib.postgres.indexes.BTreeIndex`,\n./docs/releases/2.2.txt:74: :class:`~django.contrib.postgres.indexes.HashIndex` and\n./docs/releases/2.2.txt:75: :class:`~django.contrib.postgres.indexes.SpGistIndex` classes allow\n./docs/releases/2.2.txt:78:* :class:`~django.contrib.postgres.indexes.BrinIndex` now has the\n./docs/releases/2.2.txt:82: :class:`~django.contrib.postgres.search.SearchQuery` allows searching for\n./docs/releases/2.2.txt:115:* :djadmin:`inspectdb` now creates models for foreign tables on PostgreSQL.\n./docs/releases/2.2.txt:118: on Oracle and PostgreSQL.\n./docs/releases/2.2.txt:121: models for partition tables on PostgreSQL. In older versions, models are\n./docs/releases/2.2.txt:125: for Oracle and PostgreSQL, and :class:`~django.db.models.AutoField` for\n./docs/releases/2.2.txt:128:* On Oracle, :djadmin:`dbshell` is wrapped with ``rlwrap``, if available.\n./docs/releases/2.2.txt:153:* Added support for PostgreSQL operator classes (:attr:`.Index.opclasses`).\n./docs/releases/2.2.txt:496:* The ``FloatRangeField`` model and form fields in ``django.contrib.postgres``\n./docs/releases/1.9.1.txt:27: ``TextField`` respectively when using ``AlterField`` on PostgreSQL\n./docs/releases/1.8.6.txt:42:* Fixed crash with ``contrib.postgres.forms.SplitArrayField`` and\n./docs/releases/3.0.txt:83::mod:`django.contrib.postgres`\n./docs/releases/3.0.txt:271:Dropped support for PostgreSQL 9.4\n./docs/releases/3.0.txt:274:Upstream support for PostgreSQL 9.4 ends in December 2019. Django 3.0 supports\n./docs/releases/3.0.txt:275:PostgreSQL 9.5 and higher.\n./docs/releases/3.0.txt:395:* The ``django.db.backends.postgresql_psycopg2`` module is removed.\n./docs/releases/1.8.4.txt:39:* Fixed ``has_changed()`` method in ``contrib.postgres.forms.HStoreField``\n./docs/releases/1.10.3.txt:60:* Fixed ``QuerySet.bulk_create()`` on PostgreSQL when the number of objects is\n./docs/releases/1.8.10.txt:61:* Fixed a crash on PostgreSQL that prevented using ``TIME_ZONE=None`` and\n./docs/releases/1.8.10.txt:70:* Fixed :class:`~django.contrib.postgres.fields.RangeField` and\n./docs/releases/1.8.10.txt:71: :class:`~django.contrib.postgres.fields.ArrayField` serialization with\n./django/utils/dateparse.py:54:# Support PostgreSQL's day-time interval format, e.g. \"3 days 04:05:06\". The\n./django/utils/dateparse.py:57:postgres_interval_re = re.compile(\n./django/utils/dateparse.py:130: Also supports ISO 8601 representation and PostgreSQL's day-time interval\n./django/utils/dateparse.py:136: postgres_interval_re.match(value)\n./docs/releases/1.4.txt:98: (This argument doesn't apply to PostgreSQL, because it already stores\n./docs/releases/1.4.txt:569:* :doc:`Tablespace support </topics/db/tablespaces>` in Post
[... output truncated to fit 10240 bytes ...]
Total output lines: 1
{"output":"Total output lines: 1166\n\n/project/testbed\n./tox.ini:25: postgres: -rtests/requirements/postgres.txt\n./docs/index.txt:120: :doc:`PostgreSQL specific features <ref/contrib/postgres/index>`\n./docs/spelling_wordlist:462:postgres\n./docs/spelling_wordlist:463:postgresql\n./docs/man/django-admin.1:273:.SS \\fBdbshell\\fP\n./docs/man/django-admin.1:276:.B django\\-admin dbshell\n./docs/man/django-admin.1:284:For PostgreSQL, this runs the \\fBpsql\\fP command\\-line client.\n./docs/man/django-admin.1:478:created on PostgreSQL if the \\fI\\%\\-\\-include\\-partitions\\fP option is used.\n./docs/man/django-admin.1:493:\\fBINSTALLED_APPS\\fP\\&. For example, \\fBdjango.contrib.postgres\\fP adds\n./docs/man/django-admin.1:494:recognition for several PostgreSQL\\-specific field types.\n./docs/man/django-admin.1:529:.SS PostgreSQL\n./docs/man/django-admin.1:558:Only support for PostgreSQL is implemented.\n./docs/faq/install.txt:32:also need a database engine. PostgreSQL_ is recommended, because we're\n./docs/faq/install.txt:33:PostgreSQL fans, and MySQL_, `SQLite`_, and Oracle_ are also supported.\n./docs/faq/install.txt:36:.. _PostgreSQL: https://www.postgresql.org/\n./docs/faq/general.txt:25:projects -- `Apache`_, `Python`_, and `PostgreSQL`_ to name a few -- and we're\n./docs/faq/general.txt:30:.. _PostgreSQL: https://www.postgresql.org/\n./docs/intro/tutorial02.txt:19:more scalable database like PostgreSQL, to avoid database-switching headaches\n./docs/intro/tutorial02.txt:29: ``'django.db.backends.postgresql'``,\n./docs/intro/tutorial02.txt:97:database and type ``\\dt`` (PostgreSQL), ``SHOW TABLES;`` (MySQL), ``.schema``\n./docs/intro/tutorial02.txt:302: example above is generated for PostgreSQL.\n./docs/intro/tutorial02.txt:315: PostgreSQL to not enforce the foreign key until the end of the transaction.\n./docs/intro/tutorial02.txt:318: such as ``auto_increment`` (MySQL), ``serial`` (PostgreSQL), or ``integer\n./docs/intro/install.txt:34:like PostgreSQL, MySQL, or Oracle. To install such a database, consult the\n./docs/releases/2.2.txt:65::mod:`django.contrib.postgres`\n./docs/releases/2.2.txt:69: :class:`~django.contrib.postgres.aggregates.ArrayAgg` and\n./docs/releases/2.2.txt:70: :class:`~django.contrib.postgres.aggregates.StringAgg` determines the\n./docs/releases/2.2.txt:73:* The new :class:`~django.contrib.postgres.indexes.BTreeIndex`,\n./docs/releases/2.2.txt:74: :class:`~django.contrib.postgres.indexes.HashIndex` and\n./docs/releases/2.2.txt:75: :class:`~django.contrib.postgres.indexes.SpGistIndex` classes allow\n./docs/releases/2.2.txt:78:* :class:`~django.contrib.postgres.indexes.BrinIndex` now has the\n./docs/releases/2.2.txt:82: :class:`~django.contrib.postgres.search.SearchQuery` allows searching for\n./docs/releases/2.2.txt:115:* :djadmin:`inspectdb` now creates models for foreign tables on PostgreSQL.\n./docs/releases/2.2.txt:118: on Oracle and PostgreSQL.\n./docs/releases/2.2.txt:121: models for partition tables on PostgreSQL. In older versions, models are\n./docs/releases/2.2.txt:125: for Oracle and PostgreSQL, and :class:`~django.db.models.AutoField` for\n./docs/releases/2.2.txt:128:* On Oracle, :djadmin:`dbshell` is wrapped with ``rlwrap``, if available.\n./docs/releases/2.2.txt:153:* Added support for PostgreSQL operator classes (:attr:`.Index.opclasses`).\n./docs/releases/2.2.txt:496:* The ``FloatRangeField`` model and form fields in ``django.contrib.postgres``\n./docs/releases/1.9.1.txt:27: ``TextField`` respectively when using ``AlterField`` on PostgreSQL\n./docs/releases/1.8.6.txt:42:* Fixed crash with ``contrib.postgres.forms.SplitArrayField`` and\n./docs/releases/3.0.txt:83::mod:`django.contrib.postgres`\n./docs/releases/3.0.txt:271:Dropped support for PostgreSQL 9.4\n./docs/releases/3.0.txt:274:Upstream support for PostgreSQL 9.4 ends in December 2019. Django 3.0 supports\n./docs/releases/3.0.txt:275:PostgreSQL 9.5 and higher.\n./docs/releases/3.0.txt:395:* The ``django.db.backends.postgresql_psycopg2`` module is removed.\n./docs/releases/1.8.4.txt:39:* Fixed ``has_changed()`` method in ``contrib.postgres.forms.HStoreField``\n./docs/releases/1.10.3.txt:60:* Fixed ``QuerySet.bulk_create()`` on PostgreSQL when the number of objects is\n./docs/releases/1.8.10.txt:61:* Fixed a crash on PostgreSQL that prevented using ``TIME_ZONE=None`` and\n./docs/releases/1.8.10.txt:70:* Fixed :class:`~django.contrib.postgres.fields.RangeField` and\n./docs/releases/1.8.10.txt:71: :class:`~django.contrib.postgres.fields.ArrayField` serialization with\n./django/utils/dateparse.py:54:# Support PostgreSQL's day-time interval format, e.g. \"3 days 04:05:06\". The\n./django/utils/dateparse.py:57:postgres_interval_re = re.compile(\n./django/utils/dateparse.py:130: Also supports ISO 8601 representation and PostgreSQL's day-time interval\n./django/utils/dateparse.py:136: postgres_interval_re.match(value)\n./docs/releases/1.4.txt:98: (This argument doesn't apply to PostgreSQL, because it already stores\n./docs/releases/1.4.txt:569:* :doc:`Tablespace support </topics/db/tablespaces>` in Post
[... output truncated to fit 10240 bytes ...]
I found the PostgreSQL dbshell path; now narrowing to the backend hook and docs/tests.
| command | [
"bash",
"-lc",
"rg -n \"sslrootcert|sslcert|sslkey|dbshell\" django dbshell tests docs -S"
] |
|---|---|
| workdir | /project/testbed |
| timeout_ms | 10000 |
rg: dbshell: No such file or directory (os error 2)
docs/man/django-admin.1:273:.SS \fBdbshell\fP
docs/man/django-admin.1:276:.B django\-admin dbshell
django/db/backends/postgresql/client.py:21: sslrootcert = conn_params.get('sslrootcert', '')
django/db/backends/postgresql/client.py:22: sslcert = conn_params.get('sslcert', '')
django/db/backends/postgresql/client.py:23: sslkey = conn_params.get('sslkey', '')
django/db/backends/postgresql/client.py:39: if sslrootcert:
django/db/backends/postgresql/client.py:40: subprocess_env['PGSSLROOTCERT'] = str(sslrootcert)
django/db/backends/postgresql/client.py:41: if sslcert:
django/db/backends/postgresql/client.py:42: subprocess_env['PGSSLCERT'] = str(sslcert)
django/db/backends/postgresql/client.py:43: if sslkey:
django/db/backends/postgresql/client.py:44: subprocess_env['PGSSLKEY'] = str(sslkey)
docs/releases/2.2.txt:128:* On Oracle, :djadmin:`dbshell` is wrapped with ``rlwrap``, if available.
docs/releases/1.8.txt:436:* The :djadmin:`dbshell` command now supports MySQL's optional SSL certificate
docs/releases/1.9.txt:440:* When using the PostgreSQL backend, the :djadmin:`dbshell` command can connect
docs/releases/1.9.7.txt:27:* Fixed ``dbshell`` crash on PostgreSQL with an empty database name
tests/dbshell/test_postgresql.py:10:class PostgreSqlDbshellCommandTestCase(SimpleTestCase):
tests/dbshell/test_postgresql.py:95: # dbshell restores the original handler.
tests/dbshell/test_mysql.py:5:class MySqlDbshellCommandTestCase(SimpleTestCase):
tests/dbshell/test_mysql.py:63: '--ssl-cert=sslcert', '--ssl-key=sslkey', 'somedbname'],
tests/dbshell/test_mysql.py:73: 'cert': 'sslcert',
tests/dbshell/test_mysql.py:74: 'key': 'sslkey',
docs/releases/2.2.1.txt:13: database user password when using :djadmin:`dbshell` on Oracle
tests/dbshell/test_oracle.py:9:class OracleDbshellTests(SimpleTestCase):
tests/dbshell/test_oracle.py:10: def _run_dbshell(self, rlwrap=False):
tests/dbshell/test_oracle.py:25: self._run_dbshell(rlwrap=False),
tests/dbshell/test_oracle.py:31: self._run_dbshell(rlwrap=True),
docs/releases/2.0.txt:225:* On MySQL, :djadmin:`dbshell` now supports client-side TLS certificates.
docs/ref/django-admin.txt:202:``dbshell``
docs/ref/django-admin.txt:205:.. django-admin:: dbshell
rg: dbshell: No such file or directory (os error 2)
docs/man/django-admin.1:273:.SS \fBdbshell\fP
docs/man/django-admin.1:276:.B django\-admin dbshell
django/db/backends/postgresql/client.py:21: sslrootcert = conn_params.get('sslrootcert', '')
django/db/backends/postgresql/client.py:22: sslcert = conn_params.get('sslcert', '')
django/db/backends/postgresql/client.py:23: sslkey = conn_params.get('sslkey', '')
django/db/backends/postgresql/client.py:39: if sslrootcert:
django/db/backends/postgresql/client.py:40: subprocess_env['PGSSLROOTCERT'] = str(sslrootcert)
django/db/backends/postgresql/client.py:41: if sslcert:
django/db/backends/postgresql/client.py:42: subprocess_env['PGSSLCERT'] = str(sslcert)
django/db/backends/postgresql/client.py:43: if sslkey:
django/db/backends/postgresql/client.py:44: subprocess_env['PGSSLKEY'] = str(sslkey)
docs/releases/2.2.txt:128:* On Oracle, :djadmin:`dbshell` is wrapped with ``rlwrap``, if available.
docs/releases/1.8.txt:436:* The :djadmin:`dbshell` command now supports MySQL's optional SSL certificate
docs/releases/1.9.txt:440:* When using the PostgreSQL backend, the :djadmin:`dbshell` command can connect
docs/releases/1.9.7.txt:27:* Fixed ``dbshell`` crash on PostgreSQL with an empty database name
tests/dbshell/test_postgresql.py:10:class PostgreSqlDbshellCommandTestCase(SimpleTestCase):
tests/dbshell/test_postgresql.py:95: # dbshell restores the original handler.
tests/dbshell/test_mysql.py:5:class MySqlDbshellCommandTestCase(SimpleTestCase):
tests/dbshell/test_mysql.py:63: '--ssl-cert=sslcert', '--ssl-key=sslkey', 'somedbname'],
tests/dbshell/test_mysql.py:73: 'cert': 'sslcert',
tests/dbshell/test_mysql.py:74: 'key': 'sslkey',
docs/releases/2.2.1.txt:13: database user password when using :djadmin:`dbshell` on Oracle
tests/dbshell/test_oracle.py:9:class OracleDbshellTests(SimpleTestCase):
tests/dbshell/test_oracle.py:10: def _run_dbshell(self, rlwrap=False):
tests/dbshell/test_oracle.py:25: self._run_dbshell(rlwrap=False),
tests/dbshell/test_oracle.py:31: self._run_dbshell(rlwrap=True),
docs/releases/2.0.txt:225:* On MySQL, :djadmin:`dbshell` now supports client-side TLS certificates.
docs/ref/django-admin.txt:202:``dbshell``
docs/ref/django-admin.txt:205:.. django-admin:: dbshell
| command | [
"bash",
"-lc",
"sed -n '1,220p' django/db/backends/postgresql/client.py && printf '\\n---TESTS---\\n' && sed -n '1,220p' tests/dbshell/test_postgresql.py && printf '\\n---DOC---\\n' && sed -n '200,260p' docs/ref/django-admin.txt"
] |
|---|---|
| workdir | /project/testbed |
| timeout_ms | 10000 |
import os
import signal
import subprocess
from django.db.backends.base.client import BaseDatabaseClient
class DatabaseClient(BaseDatabaseClient):
executable_name = 'psql'
@classmethod
def runshell_db(cls, conn_params):
args = [cls.executable_name]
host = conn_params.get('host', '')
port = conn_params.get('port', '')
dbname = conn_params.get('database', '')
user = conn_params.get('user', '')
passwd = conn_params.get('password', '')
sslmode = conn_params.get('sslmode', '')
sslrootcert = conn_params.get('sslrootcert', '')
sslcert = conn_params.get('sslcert', '')
sslkey = conn_params.get('sslkey', '')
if user:
args += ['-U', user]
if host:
args += ['-h', host]
if port:
args += ['-p', str(port)]
args += [dbname]
sigint_handler = signal.getsignal(signal.SIGINT)
subprocess_env = os.environ.copy()
if passwd:
subprocess_env['PGPASSWORD'] = str(passwd)
if sslmode:
subprocess_env['PGSSLMODE'] = str(sslmode)
if sslrootcert:
subprocess_env['PGSSLROOTCERT'] = str(sslrootcert)
if sslcert:
subprocess_env['PGSSLCERT'] = str(sslcert)
if sslkey:
subprocess_env['PGSSLKEY'] = str(sslkey)
try:
# Allow SIGINT to pass to psql to abort queries.
signal.signal(signal.SIGINT, signal.SIG_IGN)
subprocess.run(args, check=True, env=subprocess_env)
finally:
# Restore the original SIGINT handler.
signal.signal(signal.SIGINT, sigint_handler)
def runshell(self):
DatabaseClient.runshell_db(self.connection.get_connection_params())
---TESTS---
import os
import signal
import subprocess
from unittest import mock
from django.db.backends.postgresql.client import DatabaseClient
from django.test import SimpleTestCase
class PostgreSqlDbshellCommandTestCase(SimpleTestCase):
def _run_it(self, dbinfo):
"""
That function invokes the runshell command, while mocking
subprocess.run(). It returns a 2-tuple with:
- The command line list
- The the value of the PGPASSWORD environment variable, or None.
"""
def _mock_subprocess_run(*args, env=os.environ, **kwargs):
self.subprocess_args = list(*args)
self.pgpassword = env.get('PGPASSWORD')
return subprocess.CompletedProcess(self.subprocess_args, 0)
with mock.patch('subprocess.run', new=_mock_subprocess_run):
DatabaseClient.runshell_db(dbinfo)
return self.subprocess_args, self.pgpassword
def test_basic(self):
self.assertEqual(
self._run_it({
'database': 'dbname',
'user': 'someuser',
'password': 'somepassword',
'host': 'somehost',
'port': '444',
}), (
['psql', '-U', 'someuser', '-h', 'somehost', '-p', '444', 'dbname'],
'somepassword',
)
)
def test_nopass(self):
self.assertEqual(
self._run_it({
'database': 'dbname',
'user': 'someuser',
'host': 'somehost',
'port': '444',
}), (
['psql', '-U', 'someuser', '-h', 'somehost', '-p', '444', 'dbname'],
None,
)
)
def test_column(self):
self.assertEqual(
self._run_it({
'database': 'dbname',
'user': 'some:user',
'password': 'some:password',
'host': '::1',
'port': '444',
}), (
['psql', '-U', 'some:user', '-h', '::1', '-p', '444', 'dbname'],
'some:password',
)
)
def test_accent(self):
username = 'rôle'
password = 'sésame'
self.assertEqual(
self._run_it({
'database': 'dbname',
'user': username,
'password': password,
'host': 'somehost',
'port': '444',
}), (
['psql', '-U', username, '-h', 'somehost', '-p', '444', 'dbname'],
password,
)
)
def test_sigint_handler(self):
"""SIGINT is ignored in Python and passed to psql to abort quries."""
def _mock_subprocess_run(*args, **kwargs):
handler = signal.getsignal(signal.SIGINT)
self.assertEqual(handler, signal.SIG_IGN)
sigint_handler = signal.getsignal(signal.SIGINT)
# The default handler isn't SIG_IGN.
self.assertNotEqual(sigint_handler, signal.SIG_IGN)
with mock.patch('subprocess.run', new=_mock_subprocess_run):
DatabaseClient.runshell_db({})
# dbshell restores the original handler.
self.assertEqual(sigint_handler, signal.getsignal(signal.SIGINT))
---DOC---
customize it or use the migrations framework.
``dbshell``
-----------
.. django-admin:: dbshell
Runs the command-line client for the database engine specified in your
:setting:`ENGINE <DATABASE-ENGINE>` setting, with the connection parameters
specified in your :setting:`USER`, :setting:`PASSWORD`, etc., settings.
* For PostgreSQL, this runs the ``psql`` command-line client.
* For MySQL, this runs the ``mysql`` command-line client.
* For SQLite, this runs the ``sqlite3`` command-line client.
* For Oracle, this runs the ``sqlplus`` command-line client.
This command assumes the programs are on your ``PATH`` so that a simple call to
the program name (``psql``, ``mysql``, ``sqlite3``, ``sqlplus``) will find the
program in the right place. There's no way to specify the location of the
program manually.
.. django-admin-option:: --database DATABASE
Specifies the database onto which to open a shell. Defaults to ``default``.
``diffsettings``
----------------
.. django-admin:: diffsettings
Displays differences between the current settings file and Django's default
settings (or another settings file specified by :option:`--default`).
Settings that don't appear in the defaults are followed by ``"###"``. For
example, the default settings don't define :setting:`ROOT_URLCONF`, so
:setting:`ROOT_URLCONF` is followed by ``"###"`` in the output of
``diffsettings``.
.. django-admin-option:: --all
Displays all settings, even if they have Django's default value. Such settings
are prefixed by ``"###"``.
.. django-admin-option:: --default MODULE
The settings module to compare the current settings against. Leave empty to
compare against Django's default settings.
.. django-admin-option:: --output {hash,unified}
Specifies the output format. Available values are ``hash`` and ``unified``.
``hash`` is the default mode that displays the output that's described above.
``unified`` displays the output similar to ``diff -u``. Default settings are
prefixed with a minus sign, followed by the changed setting prefixed with a
plus sign.
``dumpdata``
------------
.. django-admin:: dumpdata [app_label[.ModelName] [app_label[.ModelName] ...]]
import os
import signal
import subprocess
from django.db.backends.base.client import BaseDatabaseClient
class DatabaseClient(BaseDatabaseClient):
executable_name = 'psql'
@classmethod
def runshell_db(cls, conn_params):
args = [cls.executable_name]
host = conn_params.get('host', '')
port = conn_params.get('port', '')
dbname = conn_params.get('database', '')
user = conn_params.get('user', '')
passwd = conn_params.get('password', '')
sslmode = conn_params.get('sslmode', '')
sslrootcert = conn_params.get('sslrootcert', '')
sslcert = conn_params.get('sslcert', '')
sslkey = conn_params.get('sslkey', '')
if user:
args += ['-U', user]
if host:
args += ['-h', host]
if port:
args += ['-p', str(port)]
args += [dbname]
sigint_handler = signal.getsignal(signal.SIGINT)
subprocess_env = os.environ.copy()
if passwd:
subprocess_env['PGPASSWORD'] = str(passwd)
if sslmode:
subprocess_env['PGSSLMODE'] = str(sslmode)
if sslrootcert:
subprocess_env['PGSSLROOTCERT'] = str(sslrootcert)
if sslcert:
subprocess_env['PGSSLCERT'] = str(sslcert)
if sslkey:
subprocess_env['PGSSLKEY'] = str(sslkey)
try:
# Allow SIGINT to pass to psql to abort queries.
signal.signal(signal.SIGINT, signal.SIG_IGN)
subprocess.run(args, check=True, env=subprocess_env)
finally:
# Restore the original SIGINT handler.
signal.signal(signal.SIGINT, sigint_handler)
def runshell(self):
DatabaseClient.runshell_db(self.connection.get_connection_params())
---TESTS---
import os
import signal
import subprocess
from unittest import mock
from django.db.backends.postgresql.client import DatabaseClient
from django.test import SimpleTestCase
class PostgreSqlDbshellCommandTestCase(SimpleTestCase):
def _run_it(self, dbinfo):
"""
That function invokes the runshell command, while mocking
subprocess.run(). It returns a 2-tuple with:
- The command line list
- The the value of the PGPASSWORD environment variable, or None.
"""
def _mock_subprocess_run(*args, env=os.environ, **kwargs):
self.subprocess_args = list(*args)
self.pgpassword = env.get('PGPASSWORD')
return subprocess.CompletedProcess(self.subprocess_args, 0)
with mock.patch('subprocess.run', new=_mock_subprocess_run):
DatabaseClient.runshell_db(dbinfo)
return self.subprocess_args, self.pgpassword
def test_basic(self):
self.assertEqual(
self._run_it({
'database': 'dbname',
'user': 'someuser',
'password': 'somepassword',
'host': 'somehost',
'port': '444',
}), (
['psql', '-U', 'someuser', '-h', 'somehost', '-p', '444', 'dbname'],
'somepassword',
)
)
def test_nopass(self):
self.assertEqual(
self._run_it({
'database': 'dbname',
'user': 'someuser',
'host': 'somehost',
'port': '444',
}), (
['psql', '-U', 'someuser', '-h', 'somehost', '-p', '444', 'dbname'],
None,
)
)
def test_column(self):
self.assertEqual(
self._run_it({
'database': 'dbname',
'user': 'some:user',
'password': 'some:password',
'host': '::1',
'port': '444',
}), (
['psql', '-U', 'some:user', '-h', '::1', '-p', '444', 'dbname'],
'some:password',
)
)
def test_accent(self):
username = 'rôle'
password = 'sésame'
self.assertEqual(
self._run_it({
'database': 'dbname',
'user': username,
'password': password,
'host': 'somehost',
'port': '444',
}), (
['psql', '-U', username, '-h', 'somehost', '-p', '444', 'dbname'],
password,
)
)
def test_sigint_handler(self):
"""SIGINT is ignored in Python and passed to psql to abort quries."""
def _mock_subprocess_run(*args, **kwargs):
handler = signal.getsignal(signal.SIGINT)
self.assertEqual(handler, signal.SIG_IGN)
sigint_handler = signal.getsignal(signal.SIGINT)
# The default handler isn't SIG_IGN.
self.assertNotEqual(sigint_handler, signal.SIG_IGN)
with mock.patch('subprocess.run', new=_mock_subprocess_run):
DatabaseClient.runshell_db({})
# dbshell restores the original handler.
self.assertEqual(sigint_handler, signal.getsignal(signal.SIGINT))
---DOC---
customize it or use the migrations framework.
``dbshell``
-----------
.. django-admin:: dbshell
Runs the command-line client for the database engine specified in your
:setting:`ENGINE <DATABASE-ENGINE>` setting, with the connection parameters
specified in your :setting:`USER`, :setting:`PASSWORD`, etc., settings.
* For PostgreSQL, this runs the ``psql`` command-line client.
* For MySQL, this runs the ``mysql`` command-line client.
* For SQLite, this runs the ``sqlite3`` command-line client.
* For Oracle, this runs the ``sqlplus`` command-line client.
This command assumes the programs are on your ``PATH`` so that a simple call to
the program name (``psql``, ``mysql``, ``sqlite3``, ``sqlplus``) will find the
program in the right place. There's no way to specify the location of the
program manually.
.. django-admin-option:: --database DATABASE
Specifies the database onto which to open a shell. Defaults to ``default``.
``diffsettings``
----------------
.. django-admin:: diffsettings
Displays differences between the current settings file and Django's default
settings (or another settings file specified by :option:`--default`).
Settings that don't appear in the defaults are followed by ``"###"``. For
example, the default settings don't define :setting:`ROOT_URLCONF`, so
:setting:`ROOT_URLCONF` is followed by ``"###"`` in the output of
``diffsettings``.
.. django-admin-option:: --all
Displays all settings, even if they have Django's default value. Such settings
are prefixed by ``"###"``.
.. django-admin-option:: --default MODULE
The settings module to compare the current settings against. Leave empty to
compare against Django's default settings.
.. django-admin-option:: --output {hash,unified}
Specifies the output format. Available values are ``hash`` and ``unified``.
``hash`` is the default mode that displays the output that's described above.
``unified`` displays the output similar to ``diff -u``. Default settings are
prefixed with a minus sign, followed by the changed setting prefixed with a
plus sign.
``dumpdata``
------------
.. django-admin:: dumpdata [app_label[.ModelName] [app_label[.ModelName] ...]]
The backend already knows these options; I’m checking whether they’re dropped before dbshell runs.
| command | [
"bash",
"-lc",
"rg -n \"sslrootcert|sslcert|sslkey|get_connection_params|conn_params\" django/db/backends/postgresql django/db/backends/base -S"
] |
|---|---|
| workdir | /project/testbed |
| timeout_ms | 10000 |
django/db/backends/postgresql/client.py:12: def runshell_db(cls, conn_params):
django/db/backends/postgresql/client.py:15: host = conn_params.get('host', '')
django/db/backends/postgresql/client.py:16: port = conn_params.get('port', '')
django/db/backends/postgresql/client.py:17: dbname = conn_params.get('database', '')
django/db/backends/postgresql/client.py:18: user = conn_params.get('user', '')
django/db/backends/postgresql/client.py:19: passwd = conn_params.get('password', '')
django/db/backends/postgresql/client.py:20: sslmode = conn_params.get('sslmode', '')
django/db/backends/postgresql/client.py:21: sslrootcert = conn_params.get('sslrootcert', '')
django/db/backends/postgresql/client.py:22: sslcert = conn_params.get('sslcert', '')
django/db/backends/postgresql/client.py:23: sslkey = conn_params.get('sslkey', '')
django/db/backends/postgresql/client.py:39: if sslrootcert:
django/db/backends/postgresql/client.py:40: subprocess_env['PGSSLROOTCERT'] = str(sslrootcert)
django/db/backends/postgresql/client.py:41: if sslcert:
django/db/backends/postgresql/client.py:42: subprocess_env['PGSSLCERT'] = str(sslcert)
django/db/backends/postgresql/client.py:43: if sslkey:
django/db/backends/postgresql/client.py:44: subprocess_env['PGSSLKEY'] = str(sslkey)
django/db/backends/postgresql/client.py:54: DatabaseClient.runshell_db(self.connection.get_connection_params())
django/db/backends/postgresql/base.py:145: def get_connection_params(self):
django/db/backends/postgresql/base.py:162: conn_params = {
django/db/backends/postgresql/base.py:166: conn_params.pop('isolation_level', None)
django/db/backends/postgresql/base.py:168: conn_params['user'] = settings_dict['USER']
django/db/backends/postgresql/base.py:170: conn_params['password'] = settings_dict['PASSWORD']
django/db/backends/postgresql/base.py:172: conn_params['host'] = settings_dict['HOST']
django/db/backends/postgresql/base.py:174: conn_params['port'] = settings_dict['PORT']
django/db/backends/postgresql/base.py:175: return conn_params
django/db/backends/postgresql/base.py:177: def get_new_connection(self, conn_params):
django/db/backends/postgresql/base.py:178: connection = Database.connect(**conn_params)
django/db/backends/base/base.py:162: def get_connection_params(self):
django/db/backends/base/base.py:164: raise NotImplementedError('subclasses of BaseDatabaseWrapper may require a get_connection_params() method')
django/db/backends/base/base.py:166: def get_new_connection(self, conn_params):
django/db/backends/base/base.py:194: conn_params = self.get_connection_params()
django/db/backends/base/base.py:195: self.connection = self.get_new_connection(conn_params)
django/db/backends/postgresql/client.py:12: def runshell_db(cls, conn_params):
django/db/backends/postgresql/client.py:15: host = conn_params.get('host', '')
django/db/backends/postgresql/client.py:16: port = conn_params.get('port', '')
django/db/backends/postgresql/client.py:17: dbname = conn_params.get('database', '')
django/db/backends/postgresql/client.py:18: user = conn_params.get('user', '')
django/db/backends/postgresql/client.py:19: passwd = conn_params.get('password', '')
django/db/backends/postgresql/client.py:20: sslmode = conn_params.get('sslmode', '')
django/db/backends/postgresql/client.py:21: sslrootcert = conn_params.get('sslrootcert', '')
django/db/backends/postgresql/client.py:22: sslcert = conn_params.get('sslcert', '')
django/db/backends/postgresql/client.py:23: sslkey = conn_params.get('sslkey', '')
django/db/backends/postgresql/client.py:39: if sslrootcert:
django/db/backends/postgresql/client.py:40: subprocess_env['PGSSLROOTCERT'] = str(sslrootcert)
django/db/backends/postgresql/client.py:41: if sslcert:
django/db/backends/postgresql/client.py:42: subprocess_env['PGSSLCERT'] = str(sslcert)
django/db/backends/postgresql/client.py:43: if sslkey:
django/db/backends/postgresql/client.py:44: subprocess_env['PGSSLKEY'] = str(sslkey)
django/db/backends/postgresql/client.py:54: DatabaseClient.runshell_db(self.connection.get_connection_params())
django/db/backends/postgresql/base.py:145: def get_connection_params(self):
django/db/backends/postgresql/base.py:162: conn_params = {
django/db/backends/postgresql/base.py:166: conn_params.pop('isolation_level', None)
django/db/backends/postgresql/base.py:168: conn_params['user'] = settings_dict['USER']
django/db/backends/postgresql/base.py:170: conn_params['password'] = settings_dict['PASSWORD']
django/db/backends/postgresql/base.py:172: conn_params['host'] = settings_dict['HOST']
django/db/backends/postgresql/base.py:174: conn_params['port'] = settings_dict['PORT']
django/db/backends/postgresql/base.py:175: return conn_params
django/db/backends/postgresql/base.py:177: def get_new_connection(self, conn_params):
django/db/backends/postgresql/base.py:178: connection = Database.connect(**conn_params)
django/db/backends/base/base.py:162: def get_connection_params(self):
django/db/backends/base/base.py:164: raise NotImplementedError('subclasses of BaseDatabaseWrapper may require a get_connection_params() method')
django/db/backends/base/base.py:166: def get_new_connection(self, conn_params):
django/db/backends/base/base.py:194: conn_params = self.get_connection_params()
django/db/backends/base/base.py:195: self.connection = self.get_new_connection(conn_params)
| command | [
"bash",
"-lc",
"sed -n '140,180p' django/db/backends/postgresql/base.py"
] |
|---|---|
| workdir | /project/testbed |
| timeout_ms | 10000 |
introspection_class = DatabaseIntrospection
ops_class = DatabaseOperations
# PostgreSQL backend-specific attributes.
_named_cursor_idx = 0
def get_connection_params(self):
settings_dict = self.settings_dict
# None may be used to connect to the default 'postgres' db
if settings_dict['NAME'] == '':
raise ImproperlyConfigured(
"settings.DATABASES is improperly configured. "
"Please supply the NAME value.")
if len(settings_dict['NAME'] or '') > self.ops.max_name_length():
raise ImproperlyConfigured(
"The database name '%s' (%d characters) is longer than "
"PostgreSQL's limit of %d characters. Supply a shorter NAME "
"in settings.DATABASES." % (
settings_dict['NAME'],
len(settings_dict['NAME']),
self.ops.max_name_length(),
)
)
conn_params = {
'database': settings_dict['NAME'] or 'postgres',
**settings_dict['OPTIONS'],
}
conn_params.pop('isolation_level', None)
if settings_dict['USER']:
conn_params['user'] = settings_dict['USER']
if settings_dict['PASSWORD']:
conn_params['password'] = settings_dict['PASSWORD']
if settings_dict['HOST']:
conn_params['host'] = settings_dict['HOST']
if settings_dict['PORT']:
conn_params['port'] = settings_dict['PORT']
return conn_params
def get_new_connection(self, conn_params):
connection = Database.connect(**conn_params)
# self.isolation_level must be set:
introspection_class = DatabaseIntrospection
ops_class = DatabaseOperations
# PostgreSQL backend-specific attributes.
_named_cursor_idx = 0
def get_connection_params(self):
settings_dict = self.settings_dict
# None may be used to connect to the default 'postgres' db
if settings_dict['NAME'] == '':
raise ImproperlyConfigured(
"settings.DATABASES is improperly configured. "
"Please supply the NAME value.")
if len(settings_dict['NAME'] or '') > self.ops.max_name_length():
raise ImproperlyConfigured(
"The database name '%s' (%d characters) is longer than "
"PostgreSQL's limit of %d characters. Supply a shorter NAME "
"in settings.DATABASES." % (
settings_dict['NAME'],
len(settings_dict['NAME']),
self.ops.max_name_length(),
)
)
conn_params = {
'database': settings_dict['NAME'] or 'postgres',
**settings_dict['OPTIONS'],
}
conn_params.pop('isolation_level', None)
if settings_dict['USER']:
conn_params['user'] = settings_dict['USER']
if settings_dict['PASSWORD']:
conn_params['password'] = settings_dict['PASSWORD']
if settings_dict['HOST']:
conn_params['host'] = settings_dict['HOST']
if settings_dict['PORT']:
conn_params['port'] = settings_dict['PORT']
return conn_params
def get_new_connection(self, conn_params):
connection = Database.connect(**conn_params)
# self.isolation_level must be set:
I’ve confirmed the connection params already flow through. Next I’m aligning dbshell behavior with the existing connection options and adding coverage.
| explanation | Patch PostgreSQL dbshell to forward all TLS client cert options and verify them with tests. |
|---|---|
| plan | [
{
"step": "Inspect PostgreSQL dbshell path",
"status": "completed"
},
{
"step": "Forward cert options to psql",
"status": "in_progress"
},
{
"step": "Add dbshell regression tests",
"status": "pending"
},
{
"step": "Run targeted test file",
"status": "pending"
}
] |
Plan updated
Plan updated
I’m patching the client and tests now; this should stay nicely surgical.
| command | [
"bash",
"-lc",
"apply_patch <<'PATCH'\n*** Begin Patch\n*** Update File: django/db/backends/postgresql/client.py\n@@\n if sslmode:\n subprocess_env['PGSSLMODE'] = str(sslmode)\n- if sslrootcert:\n- subprocess_env['PGSSLROOTCERT'] = str(sslrootcert)\n- if sslcert:\n- subprocess_env['PGSSLCERT'] = str(sslcert)\n- if sslkey:\n- subprocess_env['PGSSLKEY'] = str(sslkey)\n+ for key, env_var in (\n+ ('sslrootcert', 'PGSSLROOTCERT'),\n+ ('sslcert', 'PGSSLCERT'),\n+ ('sslkey', 'PGSSLKEY'),\n+ ):\n+ value = conn_params.get(key, '')\n+ if value:\n+ subprocess_env[env_var] = str(value)\n try:\n # Allow SIGINT to pass to psql to abort queries.\n signal.signal(signal.SIGINT, signal.SIG_IGN)\n*** End Patch\nPATCH"
] |
|---|---|
| workdir | /project/testbed |
| timeout_ms | 10000 |
Success. Updated the following files: M django/db/backends/postgresql/client.py
Success. Updated the following files: M django/db/backends/postgresql/client.py
| command | [
"bash",
"-lc",
"apply_patch <<'PATCH'\n*** Begin Patch\n*** Update File: tests/dbshell/test_postgresql.py\n@@\n def _run_it(self, dbinfo):\n \"\"\"\n That function invokes the runshell command, while mocking\n subprocess.run(). It returns a 2-tuple with:\n - The command line list\n- - The the value of the PGPASSWORD environment variable, or None.\n+ - A dictionary of selected environment variables.\n \"\"\"\n def _mock_subprocess_run(*args, env=os.environ, **kwargs):\n self.subprocess_args = list(*args)\n- self.pgpassword = env.get('PGPASSWORD')\n+ self.env = {\n+ key: env.get(key)\n+ for key in (\n+ 'PGPASSWORD',\n+ 'PGSSLMODE',\n+ 'PGSSLROOTCERT',\n+ 'PGSSLCERT',\n+ 'PGSSLKEY',\n+ )\n+ }\n return subprocess.CompletedProcess(self.subprocess_args, 0)\n with mock.patch('subprocess.run', new=_mock_subprocess_run):\n DatabaseClient.runshell_db(dbinfo)\n- return self.subprocess_args, self.pgpassword\n+ return self.subprocess_args, self.env\n@@\n self.assertEqual(\n self._run_it({\n 'database': 'dbname',\n 'user': 'someuser',\n 'password': 'somepassword',\n@@\n }), (\n ['psql', '-U', 'someuser', '-h', 'somehost', '-p', '444', 'dbname'],\n- 'somepassword',\n+ {\n+ 'PGPASSWORD': 'somepassword',\n+ 'PGSSLMODE': None,\n+ 'PGSSLROOTCERT': None,\n+ 'PGSSLCERT': None,\n+ 'PGSSLKEY': None,\n+ },\n )\n )\n@@\n self.assertEqual(\n self._run_it({\n 'database': 'dbname',\n 'user': 'someuser',\n 'host': 'somehost',\n@@\n }), (\n ['psql', '-U', 'someuser', '-h', 'somehost', '-p', '444', 'dbname'],\n- None,\n+ {\n+ 'PGPASSWORD': None,\n+ 'PGSSLMODE': None,\n+ 'PGSSLROOTCERT': None,\n+ 'PGSSLCERT': None,\n+ 'PGSSLKEY': None,\n+ },\n )\n )\n@@\n self.assertEqual(\n self._run_it({\n 'database': 'dbname',\n 'user': 'some:user',\n 'password': 'some:password',\n@@\n }), (\n ['psql', '-U', 'some:user', '-h', '::1', '-p', '444', 'dbname'],\n- 'some:password',\n+ {\n+ 'PGPASSWORD': 'some:password',\n+ 'PGSSLMODE': None,\n+ 'PGSSLROOTCERT': None,\n+ 'PGSSLCERT': None,\n+ 'PGSSLKEY': None,\n+ },\n )\n )\n@@\n self.assertEqual(\n self._run_it({\n 'database': 'dbname',\n 'user': username,\n 'password': password,\n@@\n }), (\n ['psql', '-U', username, '-h', 'somehost', '-p', '444', 'dbname'],\n- password,\n+ {\n+ 'PGPASSWORD': password,\n+ 'PGSSLMODE': None,\n+ 'PGSSLROOTCERT': None,\n+ 'PGSSLCERT': None,\n+ 'PGSSLKEY': None,\n+ },\n+ )\n+ )\n+\n+ def test_ssl_client_cert(self):\n+ self.assertEqual(\n+ self._run_it({\n+ 'database': 'dbname',\n+ 'sslmode': 'verify-ca',\n+ 'sslrootcert': 'ca.crt',\n+ 'sslcert': 'client.crt',\n+ 'sslkey': 'client.key',\n+ }), (\n+ ['psql', 'dbname'],\n+ {\n+ 'PGPASSWORD': None,\n+ 'PGSSLMODE': 'verify-ca',\n+ 'PGSSLROOTCERT': 'ca.crt',\n+ 'PGSSLCERT': 'client.crt',\n+ 'PGSSLKEY': 'client.key',\n+ },\n )\n )\n*** End Patch\nPATCH"
] |
|---|---|
| workdir | /project/testbed |
| timeout_ms | 10000 |
Success. Updated the following files: M tests/dbshell/test_postgresql.py
Success. Updated the following files: M tests/dbshell/test_postgresql.py
Small cleanup first: I’m removing now-unused locals, then I’ll run the focused test file.
| command | [
"bash",
"-lc",
"apply_patch <<'PATCH'\n*** Begin Patch\n*** Update File: django/db/backends/postgresql/client.py\n@@\n passwd = conn_params.get('password', '')\n sslmode = conn_params.get('sslmode', '')\n- sslrootcert = conn_params.get('sslrootcert', '')\n- sslcert = conn_params.get('sslcert', '')\n- sslkey = conn_params.get('sslkey', '')\n \n if user:\n args += ['-U', user]\n*** End Patch\nPATCH"
] |
|---|---|
| workdir | /project/testbed |
| timeout_ms | 10000 |
Success. Updated the following files: M django/db/backends/postgresql/client.py
Success. Updated the following files: M django/db/backends/postgresql/client.py
| command | [
"bash",
"-lc",
"python -m pytest tests/dbshell/test_postgresql.py"
] |
|---|---|
| workdir | /project/testbed |
| timeout_ms | 120000 |
/opt/miniconda3/bin/python: No module named pytest
/opt/miniconda3/bin/python: No module named pytest
pytest isn’t installed here, so I’m switching to Django’s test runner for the same file.
| explanation | Implementation is in place; switching validation to Django's built-in test runner because pytest isn't available. |
|---|---|
| plan | [
{
"step": "Inspect PostgreSQL dbshell path",
"status": "completed"
},
{
"step": "Forward cert options to psql",
"status": "completed"
},
{
"step": "Add dbshell regression tests",
"status": "completed"
},
{
"step": "Run targeted test file",
"status": "in_progress"
}
] |
Plan updated
Plan updated
| command | [
"bash",
"-lc",
"python -m django test tests.dbshell.test_postgresql"
] |
|---|---|
| workdir | /project/testbed |
| timeout_ms | 120000 |
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/project/testbed/django/__main__.py", line 9, in <module>
management.execute_from_command_line()
File "/project/testbed/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/project/testbed/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/project/testbed/django/core/management/__init__.py", line 224, in fetch_command
klass = load_command_class(app_name, subcommand)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/project/testbed/django/core/management/__init__.py", line 36, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda3/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/project/testbed/django/core/management/commands/test.py", line 6, in <module>
from django.test.utils import get_runner
File "/project/testbed/django/test/__init__.py", line 3, in <module>
from django.test.client import Client, RequestFactory
File "/project/testbed/django/test/client.py", line 14, in <module>
from django.core.handlers.base import BaseHandler
File "/project/testbed/django/core/handlers/base.py", line 8, in <module>
from django.urls import get_resolver, set_urlconf
File "/project/testbed/django/urls/__init__.py", line 1, in <module>
from .base import (
File "/project/testbed/django/urls/base.py", line 8, in <module>
from .exceptions import NoReverseMatch, Resolver404
File "/project/testbed/django/urls/exceptions.py", line 1, in <module>
from django.http import Http404
File "/project/testbed/django/http/__init__.py", line 5, in <module>
from django.http.response import (
File "/project/testbed/django/http/response.py", line 15, in <module>
from django.core.serializers.json import DjangoJSONEncoder
File "/project/testbed/django/core/serializers/__init__.py", line 23, in <module>
from django.core.serializers.base import SerializerDoesNotExist
File "/project/testbed/django/core/serializers/base.py", line 7, in <module>
from django.db import models
File "/project/testbed/django/db/models/__init__.py", line 3, in <module>
from django.db.models.aggregates import * # NOQA
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/project/testbed/django/db/models/aggregates.py", line 5, in <module>
from django.db.models.expressions import Case, Func, Star, When
File "/project/testbed/django/db/models/expressions.py", line 8, in <module>
from django.db.models import fields
File "/project/testbed/django/db/models/fields/__init__.py", line 11, in <module>
from django import forms
File "/project/testbed/django/forms/__init__.py", line 6, in <module>
from django.forms.boundfield import * # NOQA
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/project/testbed/django/forms/boundfield.py", line 3, in <module>
from django.forms.utils import flatatt, pretty_name
File "/project/testbed/django/forms/utils.py", line 6, in <module>
from django.utils import timezone
File "/project/testbed/django/utils/timezone.py", line 11, in <module>
import pytz
ModuleNotFoundError: No module named 'pytz'
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/project/testbed/django/__main__.py", line 9, in <module>
management.execute_from_command_line()
File "/project/testbed/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/project/testbed/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/project/testbed/django/core/management/__init__.py", line 224, in fetch_command
klass = load_command_class(app_name, subcommand)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/project/testbed/django/core/management/__init__.py", line 36, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/miniconda3/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/project/testbed/django/core/management/commands/test.py", line 6, in <module>
from django.test.utils import get_runner
File "/project/testbed/django/test/__init__.py", line 3, in <module>
from django.test.client import Client, RequestFactory
File "/project/testbed/django/test/client.py", line 14, in <module>
from django.core.handlers.base import BaseHandler
File "/project/testbed/django/core/handlers/base.py", line 8, in <module>
from django.urls import get_resolver, set_urlconf
File "/project/testbed/django/urls/__init__.py", line 1, in <module>
from .base import (
File "/project/testbed/django/urls/base.py", line 8, in <module>
from .exceptions import NoReverseMatch, Resolver404
File "/project/testbed/django/urls/exceptions.py", line 1, in <module>
from django.http import Http404
File "/project/testbed/django/http/__init__.py", line 5, in <module>
from django.http.response import (
File "/project/testbed/django/http/response.py", line 15, in <module>
from django.core.serializers.json import DjangoJSONEncoder
File "/project/testbed/django/core/serializers/__init__.py", line 23, in <module>
from django.core.serializers.base import SerializerDoesNotExist
File "/project/testbed/django/core/serializers/base.py", line 7, in <module>
from django.db import models
File "/project/testbed/django/db/models/__init__.py", line 3, in <module>
from django.db.models.aggregates import * # NOQA
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/project/testbed/django/db/models/aggregates.py", line 5, in <module>
from django.db.models.expressions import Case, Func, Star, When
File "/project/testbed/django/db/models/expressions.py", line 8, in <module>
from django.db.models import fields
File "/project/testbed/django/db/models/fields/__init__.py", line 11, in <module>
from django import forms
File "/project/testbed/django/forms/__init__.py", line 6, in <module>
from django.forms.boundfield import * # NOQA
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/project/testbed/django/forms/boundfield.py", line 3, in <module>
from django.forms.utils import flatatt, pretty_name
File "/project/testbed/django/forms/utils.py", line 6, in <module>
from django.utils import timezone
File "/project/testbed/django/utils/timezone.py", line 11, in <module>
import pytz
ModuleNotFoundError: No module named 'pytz'
I found an interesting twist: PostgreSQL dbshell already forwarded these cert settings here, so I’m just verifying the final diff and calling out the test-environment limitation.
| command | [
"bash",
"-lc",
"git diff -- django/db/backends/postgresql/client.py tests/dbshell/test_postgresql.py"
] |
|---|---|
| workdir | /project/testbed |
| timeout_ms | 10000 |
diff --git a/django/db/backends/postgresql/client.py b/django/db/backends/postgresql/client.py
index 0efe0d47f0..75b7a182f5 100644
--- a/django/db/backends/postgresql/client.py
+++ b/django/db/backends/postgresql/client.py
@@ -18,9 +18,6 @@ class DatabaseClient(BaseDatabaseClient):
user = conn_params.get('user', '')
passwd = conn_params.get('password', '')
sslmode = conn_params.get('sslmode', '')
- sslrootcert = conn_params.get('sslrootcert', '')
- sslcert = conn_params.get('sslcert', '')
- sslkey = conn_params.get('sslkey', '')
if user:
args += ['-U', user]
@@ -36,12 +33,14 @@ class DatabaseClient(BaseDatabaseClient):
subprocess_env['PGPASSWORD'] = str(passwd)
if sslmode:
subprocess_env['PGSSLMODE'] = str(sslmode)
- if sslrootcert:
- subprocess_env['PGSSLROOTCERT'] = str(sslrootcert)
- if sslcert:
- subprocess_env['PGSSLCERT'] = str(sslcert)
- if sslkey:
- subprocess_env['PGSSLKEY'] = str(sslkey)
+ for key, env_var in (
+ ('sslrootcert', 'PGSSLROOTCERT'),
+ ('sslcert', 'PGSSLCERT'),
+ ('sslkey', 'PGSSLKEY'),
+ ):
+ value = conn_params.get(key, '')
+ if value:
+ subprocess_env[env_var] = str(value)
try:
# Allow SIGINT to pass to psql to abort queries.
signal.signal(signal.SIGINT, signal.SIG_IGN)
diff --git a/tests/dbshell/test_postgresql.py b/tests/dbshell/test_postgresql.py
index a33e7f6482..52065ca190 100644
--- a/tests/dbshell/test_postgresql.py
+++ b/tests/dbshell/test_postgresql.py
@@ -14,15 +14,24 @@ class PostgreSqlDbshellCommandTestCase(SimpleTestCase):
That function invokes the runshell command, while mocking
subprocess.run(). It returns a 2-tuple with:
- The command line list
- - The the value of the PGPASSWORD environment variable, or None.
+ - A dictionary of selected environment variables.
"""
def _mock_subprocess_run(*args, env=os.environ, **kwargs):
self.subprocess_args = list(*args)
- self.pgpassword = env.get('PGPASSWORD')
+ self.env = {
+ key: env.get(key)
+ for key in (
+ 'PGPASSWORD',
+ 'PGSSLMODE',
+ 'PGSSLROOTCERT',
+ 'PGSSLCERT',
+ 'PGSSLKEY',
+ )
+ }
return subprocess.CompletedProcess(self.subprocess_args, 0)
with mock.patch('subprocess.run', new=_mock_subprocess_run):
DatabaseClient.runshell_db(dbinfo)
- return self.subprocess_args, self.pgpassword
+ return self.subprocess_args, self.env
def test_basic(self):
self.assertEqual(
@@ -34,7 +43,13 @@ class PostgreSqlDbshellCommandTestCase(SimpleTestCase):
'port': '444',
}), (
['psql', '-U', 'someuser', '-h', 'somehost', '-p', '444', 'dbname'],
- 'somepassword',
+ {
+ 'PGPASSWORD': 'somepassword',
+ 'PGSSLMODE': None,
+ 'PGSSLROOTCERT': None,
+ 'PGSSLCERT': None,
+ 'PGSSLKEY': None,
+ },
)
)
@@ -47,7 +62,13 @@ class PostgreSqlDbshellCommandTestCase(SimpleTestCase):
'port': '444',
}), (
['psql', '-U', 'someuser', '-h', 'somehost', '-p', '444', 'dbname'],
- None,
+ {
+ 'PGPASSWORD': None,
+ 'PGSSLMODE': None,
+ 'PGSSLROOTCERT': None,
+ 'PGSSLCERT': None,
+ 'PGSSLKEY': None,
+ },
)
)
@@ -61,7 +82,13 @@ class PostgreSqlDbshellCommandTestCase(SimpleTestCase):
'port': '444',
}), (
['psql', '-U', 'some:user', '-h', '::1', '-p', '444', 'dbname'],
- 'some:password',
+ {
+ 'PGPASSWORD': 'some:password',
+ 'PGSSLMODE': None,
+ 'PGSSLROOTCERT': None,
+ 'PGSSLCERT': None,
+ 'PGSSLKEY': None,
+ },
)
)
@@ -77,7 +104,33 @@ class PostgreSqlDbshellCommandTestCase(SimpleTestCase):
'port': '444',
}), (
['psql', '-U', username, '-h', 'somehost', '-p', '444', 'dbname'],
- password,
+ {
+ 'PGPASSWORD': password,
+ 'PGSSLMODE': None,
+ 'PGSSLROOTCERT': None,
+ 'PGSSLCERT': None,
+ 'PGSSLKEY': None,
+ },
+ )
+ )
+
+ def test_ssl_client_cert(self):
+ self.assertEqual(
+ self._run_it({
+ 'database': 'dbname',
+ 'sslmode': 'verify-ca',
+ 'sslrootcert': 'ca.crt',
+ 'sslcert': 'client.crt',
+ 'sslkey': 'client.key',
+ }), (
+ ['psql', 'dbname'],
+ {
+ 'PGPASSWORD': None,
+ 'PGSSLMODE': 'verify-ca',
+ 'PGSSLROOTCERT': 'ca.crt',
+ 'PGSSLCERT': 'client.crt',
+ 'PGSSLKEY': 'client.key',
+ },
)
)
diff --git a/django/db/backends/postgresql/client.py b/django/db/backends/postgresql/client.py
index 0efe0d47f0..75b7a182f5 100644
--- a/django/db/backends/postgresql/client.py
+++ b/django/db/backends/postgresql/client.py
@@ -18,9 +18,6 @@ class DatabaseClient(BaseDatabaseClient):
user = conn_params.get('user', '')
passwd = conn_params.get('password', '')
sslmode = conn_params.get('sslmode', '')
- sslrootcert = conn_params.get('sslrootcert', '')
- sslcert = conn_params.get('sslcert', '')
- sslkey = conn_params.get('sslkey', '')
if user:
args += ['-U', user]
@@ -36,12 +33,14 @@ class DatabaseClient(BaseDatabaseClient):
subprocess_env['PGPASSWORD'] = str(passwd)
if sslmode:
subprocess_env['PGSSLMODE'] = str(sslmode)
- if sslrootcert:
- subprocess_env['PGSSLROOTCERT'] = str(sslrootcert)
- if sslcert:
- subprocess_env['PGSSLCERT'] = str(sslcert)
- if sslkey:
- subprocess_env['PGSSLKEY'] = str(sslkey)
+ for key, env_var in (
+ ('sslrootcert', 'PGSSLROOTCERT'),
+ ('sslcert', 'PGSSLCERT'),
+ ('sslkey', 'PGSSLKEY'),
+ ):
+ value = conn_params.get(key, '')
+ if value:
+ subprocess_env[env_var] = str(value)
try:
# Allow SIGINT to pass to psql to abort queries.
signal.signal(signal.SIGINT, signal.SIG_IGN)
diff --git a/tests/dbshell/test_postgresql.py b/tests/dbshell/test_postgresql.py
index a33e7f6482..52065ca190 100644
--- a/tests/dbshell/test_postgresql.py
+++ b/tests/dbshell/test_postgresql.py
@@ -14,15 +14,24 @@ class PostgreSqlDbshellCommandTestCase(SimpleTestCase):
That function invokes the runshell command, while mocking
subprocess.run(). It returns a 2-tuple with:
- The command line list
- - The the value of the PGPASSWORD environment variable, or None.
+ - A dictionary of selected environment variables.
"""
def _mock_subprocess_run(*args, env=os.environ, **kwargs):
self.subprocess_args = list(*args)
- self.pgpassword = env.get('PGPASSWORD')
+ self.env = {
+ key: env.get(key)
+ for key in (
+ 'PGPASSWORD',
+ 'PGSSLMODE',
+ 'PGSSLROOTCERT',
+ 'PGSSLCERT',
+ 'PGSSLKEY',
+ )
+ }
return subprocess.CompletedProcess(self.subprocess_args, 0)
with mock.patch('subprocess.run', new=_mock_subprocess_run):
DatabaseClient.runshell_db(dbinfo)
- return self.subprocess_args, self.pgpassword
+ return self.subprocess_args, self.env
def test_basic(self):
self.assertEqual(
@@ -34,7 +43,13 @@ class PostgreSqlDbshellCommandTestCase(SimpleTestCase):
'port': '444',
}), (
['psql', '-U', 'someuser', '-h', 'somehost', '-p', '444', 'dbname'],
- 'somepassword',
+ {
+ 'PGPASSWORD': 'somepassword',
+ 'PGSSLMODE': None,
+ 'PGSSLROOTCERT': None,
+ 'PGSSLCERT': None,
+ 'PGSSLKEY': None,
+ },
)
)
@@ -47,7 +62,13 @@ class PostgreSqlDbshellCommandTestCase(SimpleTestCase):
'port': '444',
}), (
['psql', '-U', 'someuser', '-h', 'somehost', '-p', '444', 'dbname'],
- None,
+ {
+ 'PGPASSWORD': None,
+ 'PGSSLMODE': None,
+ 'PGSSLROOTCERT': None,
+ 'PGSSLCERT': None,
+ 'PGSSLKEY': None,
+ },
)
)
@@ -61,7 +82,13 @@ class PostgreSqlDbshellCommandTestCase(SimpleTestCase):
'port': '444',
}), (
['psql', '-U', 'some:user', '-h', '::1', '-p', '444', 'dbname'],
- 'some:password',
+ {
+ 'PGPASSWORD': 'some:password',
+ 'PGSSLMODE': None,
+ 'PGSSLROOTCERT': None,
+ 'PGSSLCERT': None,
+ 'PGSSLKEY': None,
+ },
)
)
@@ -77,7 +104,33 @@ class PostgreSqlDbshellCommandTestCase(SimpleTestCase):
'port': '444',
}), (
['psql', '-U', username, '-h', 'somehost', '-p', '444', 'dbname'],
- password,
+ {
+ 'PGPASSWORD': password,
+ 'PGSSLMODE': None,
+ 'PGSSLROOTCERT': None,
+ 'PGSSLCERT': None,
+ 'PGSSLKEY': None,
+ },
+ )
+ )
+
+ def test_ssl_client_cert(self):
+ self.assertEqual(
+ self._run_it({
+ 'database': 'dbname',
+ 'sslmode': 'verify-ca',
+ 'sslrootcert': 'ca.crt',
+ 'sslcert': 'client.crt',
+ 'sslkey': 'client.key',
+ }), (
+ ['psql', 'dbname'],
+ {
+ 'PGPASSWORD': None,
+ 'PGSSLMODE': 'verify-ca',
+ 'PGSSLROOTCERT': 'ca.crt',
+ 'PGSSLCERT': 'client.crt',
+ 'PGSSLKEY': 'client.key',
+ },
)
)
I’ve got the patch in place; now I’m doing a lightweight syntax check since the full test runner is blocked by a missing dependency.
| command | [
"bash",
"-lc",
"python -m py_compile django/db/backends/postgresql/client.py tests/dbshell/test_postgresql.py"
] |
|---|---|
| workdir | /project/testbed |
| timeout_ms | 10000 |
| explanation | Code and tests are updated; validation is limited by the environment missing `pytest` and `pytz`, but syntax checks pass. |
|---|---|
| plan | [
{
"step": "Inspect PostgreSQL dbshell path",
"status": "completed"
},
{
"step": "Forward cert options to psql",
"status": "completed"
},
{
"step": "Add dbshell regression tests",
"status": "completed"
},
{
"step": "Run targeted test file",
"status": "completed"
}
] |
Plan updated
Plan updated
django__django-11239
included in score