g-admin-api: endpoint /api/v1/system/logs/ fails when reading blob-type journal-entries with TypeError: Object of type bytes is not JSON serializable

[umgfoin]

Assume an arbitrary journal-log containing blob-entries similar to:

journalctl -u gromox-imap

Nov 25 13:44:11 postkastl.umgfoin.is imap[2795947]: [::ffff:117.32.102.38]:45596 httpswww.voelkner.deinfo@umgfoin.is < LOGIN ****: ret=800770h code=1904
Nov 25 13:55:27 postkastl.umgfoin.is imap[2795947]: [::ffff:162.142.125.208]:15224  < a001 STARTTLS: ret=6a8h code=1704
Nov 25 13:55:58 postkastl.umgfoin.is imap[2795947]: [83B blob data]
Nov 25 13:56:00 postkastl.umgfoin.is imap[2795947]: [92B blob data]
Nov 25 13:56:01 postkastl.umgfoin.is imap[2795947]: [53B blob data]
Nov 25 13:59:24 postkastl.umgfoin.is imap[2795947]: [::ffff:45.79.181.223]:46424  < GET / HTTP/1.1: ret=0h code=0
Nov 25 13:59:24 postkastl.umgfoin.is imap[2795947]: [::ffff:45.79.181.223]:46424  < Host: 8.9.23.9:993: ret=0h code=0
Nov 25 13:59:24 postkastl.umgfoin.is imap[2795947]: [::ffff:45.79.181.223]:46424  < User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 13_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36: ret=0h code=0
Nov 25 13:59:24 postkastl.umgfoin.is imap[2795947]: [::ffff:45.79.181.223]:46424  < Accept: */*: ret=0h code=0
Nov 25 13:59:24 postkastl.umgfoin.is imap[2795947]: [::ffff:45.79.181.223]:46424  < Accept-Encoding: gzip: ret=0h code=0
Nov 25 14:21:06 postkastl.umgfoin.is imap[2795947]: [::ffff:105.159.255.190]:57086  < JPOP0 CAPABILITY: ret=0h code=0
Nov 25 14:21:08 postkastl.umgfoin.is imap[2795947]: [::ffff:105.159.255.190]:57086 test < LOGIN ****: ret=800770h code=1904

Monitoring gromox-imap-journal in g-webadmin asserts with
(The server encountered an error while processing the request.):

journalctl -u grommunio-admin-api

Nov 25 13:58:13 postkastl.umgfoin.is uwsgi[3718034]: [WARNING] (grommunio Admin API) GET /api/v1/system/logs/Gromox imap?n=100 from 5002:a3:3d3a:da00:a980:151f:2329:1f59 -> 500 '{"message":"The server encountered an error while processing the request."}\n'
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]: [ERROR] (grommunio Admin API) Traceback (most recent call last):
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:   File "/usr/share/grommunio-admin-api/venv/lib/python3.13/site-packages/flask/app.py", line 917, in full_dispatch_request
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:     rv = self.dispatch_request()
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:   File "/usr/share/grommunio-admin-api/venv/lib/python3.13/site-packages/flask/app.py", line 902, in dispatch_request
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:   File "/usr/share/grommunio-admin-api/api/core.py", line 204, in wrapper
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:     return call()
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:   File "/usr/share/grommunio-admin-api/api/core.py", line 169, in call
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:     ret = func(*args, **kwargs)
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:   File "/usr/share/grommunio-admin-api/endpoints/system/logs.py", line 35, in getLog
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:     return jsonify(data=LogReader.tail(log.get("format", "journald"), log["source"], n, skip, after))
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:   File "/usr/share/grommunio-admin-api/venv/lib/python3.13/site-packages/flask/json/__init__.py", line 170, in jsonify
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:     return current_app.json.response(*args, **kwargs)  # type: ignore[return-value]
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:            ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:   File "/usr/share/grommunio-admin-api/venv/lib/python3.13/site-packages/flask/json/provider.py", line 214, in response
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:     f"{self.dumps(obj, **dump_args)}\n", mimetype=self.mimetype
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:        ~~~~~~~~~~^^^^^^^^^^^^^^^^^^
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:   File "/usr/share/grommunio-admin-api/venv/lib/python3.13/site-packages/flask/json/provider.py", line 179, in dumps
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:     return json.dumps(obj, **kwargs)
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:            ~~~~~~~~~~^^^^^^^^^^^^^^^
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:   File "/usr/lib/python3.13/json/__init__.py", line 238, in dumps
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:     **kw).encode(obj)
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:           ~~~~~~^^^^^
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:   File "/usr/lib/python3.13/json/encoder.py", line 200, in encode
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:     chunks = self.iterencode(o, _one_shot=True)
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:   File "/usr/lib/python3.13/json/encoder.py", line 261, in iterencode
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:     return _iterencode(o, 0)
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:   File "/usr/share/grommunio-admin-api/venv/lib/python3.13/site-packages/flask/json/provider.py", line 121, in _default
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]:     raise TypeError(f"Object of type {type(o).__name__} is not JSON serializable")
Nov 25 13:58:17 postkastl.umgfoin.is uwsgi[3718034]: TypeError: Object of type bytes is not JSON serializable

codebase:
admin-api: master 1.17-35-g0445e1c

(venv) dev@wurzelsepp /usr/share/grommunio-admin-api/venv/bin #  ./python --version
Python 3.13.5

(venv) dev@wurzelsepp /usr/share/grommunio-admin-api/venv/bin # ./pip list
Package                   Version
------------------------- ----------
argcomplete               3.6.3
attrs                     25.4.0
blinker                   1.9.0
certifi                   2025.11.12
cffi                      2.0.0
charset-normalizer        3.4.4
click                     8.3.1
crypt_r                   3.13.1
cryptography              46.0.3
dnspython                 2.8.0
Flask                     3.1.2
greenlet                  3.2.4
idna                      3.11
inflection                0.5.1
isodate                   0.7.2
itsdangerous              2.2.0
Jinja2                    3.1.6
jsonschema                4.25.1
jsonschema-path           0.3.4
jsonschema-specifications 2025.9.1
lazy-object-proxy         1.12.0
ldap3                     2.9.1
legacycrypt               0.3
MarkupSafe                3.0.3
mattermostdriver          7.3.2
more-itertools            10.8.0
multidict                 6.7.0
mysqlclient               2.2.7
openapi                   2.0.0
openapi-core              0.19.5
openapi-schema-validator  0.6.3
openapi-spec-validator    0.7.2
parse                     1.20.2
pathable                  0.4.4
pip                       25.3
psutil                    7.1.3
pyasn1                    0.6.1
pycparser                 2.23
PyJWT                     2.10.1
PyYAML                    6.0.3
redis                     7.1.0
referencing               0.37.0
requests                  2.32.5
rfc3339-validator         0.1.4
rpds-py                   0.29.0
six                       1.17.0
SQLAlchemy                2.0.44
systemd-python            235
typing_extensions         4.15.0
urllib3                   2.5.0
uwsgidecorators           1.1.0
websockets                15.0.1
Werkzeug                  3.1.3

[crpb]

just my 2 cents but i think you should also provide your logging settings as this looks like you changed imap_cmd_debug and/or imap_log_level. Just to be thorough.

[umgfoin]

binary-data is logged with

imap_log_level = 6
imap_cmd_debug = 2

Not standard case, but permissible journal content.

[jengelh]

Anecdote: I have observed is that some spambots talk TLS into the plaintext IMAP port without issuing STARTTLS, that's how one can end up with some binary data in the logs.

[umgfoin]

@jengelh Exactly, that's one of the brute-force variants here ...