Skip to content

CallsiteParameterAdder use additional_ignores for foreign events #772

@fraser-langton

Description

@fraser-langton

I was configuring some stdlib logs (eg requests) and I want the log to emit with the callsite from my code not the lib, and so I added additional_ignores=["urllib3", "requests"] alas it doesn't do it if not _from_structlog

https://github.com/hynek/structlog/blob/main/src/structlog/processors.py#L906-L928

    def __call__(
        self, logger: logging.Logger, name: str, event_dict: EventDict
    ) -> EventDict:
        record: logging.LogRecord | None = event_dict.get("_record")
        from_structlog: bool = event_dict.get("_from_structlog", False)

        # If the event dictionary has a record, but it comes from structlog,
        # then the callsite parameters of the record will not be correct.
        if record is not None and not from_structlog:
            for mapping in self._record_mappings:
                event_dict[mapping.event_dict_key] = record.__dict__[
                    mapping.record_attribute
                ]

            return event_dict

        frame, module = _find_first_app_frame_and_name(
            additional_ignores=self._additional_ignores
        )
        for parameter, handler in self._active_handlers:
            event_dict[parameter.value] = handler(module, frame)

        return event_dict

To remedy this I just did my own subclass

class CallsiteParameterAdder(_CallsiteParameterAdder):
    _all_parameters: ClassVar[set[CallsiteParameter]] = set(CallsiteParameter)

    def __init__(
        self,
        parameters: Collection[CallsiteParameter] = _all_parameters,
        additional_ignores: list[str] | None = None,
    ) -> None:
        additional_ignores = additional_ignores or []
        super().__init__(parameters, [__name__, *additional_ignores])

    def __call__(self, logger: logging.Logger, name: str, event_dict: EventDict) -> EventDict:
        frame, module = _find_first_app_frame_and_name(additional_ignores=self._additional_ignores)
        for parameter, handler in self._active_handlers:
            event_dict[parameter.value] = handler(module, frame)

        return event_dict

I am wondering if the comment If the event dictionary has a record, but it comes from structlog, then the callsite parameters of the record will not be correct. in the source is still true, it may be down to how I configure structlog in services but it isn't an issue.

eg here is the stack in structlog.processors.CallsiteParameterAdder

  File "/temp.py", line 16, in <module>
    requests.post("https://google.com")
  File "/.venv/lib/python3.12/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/.venv/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/.venv/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/.venv/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/.venv/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
  File "/.venv/lib/python3.12/site-packages/urllib3/connectionpool.py", line 547, in _make_request
    log.debug(
  File "/python/cpython-3.12.7-macos-aarch64-none/lib/python3.12/logging/__init__.py", line 1527, in debug
    self._log(DEBUG, msg, args, **kwargs)
  File "/python/cpython-3.12.7-macos-aarch64-none/lib/python3.12/logging/__init__.py", line 1684, in _log
    self.handle(record)
  File "/python/cpython-3.12.7-macos-aarch64-none/lib/python3.12/logging/__init__.py", line 1700, in handle
    self.callHandlers(record)
  File "/python/cpython-3.12.7-macos-aarch64-none/lib/python3.12/logging/__init__.py", line 1762, in callHandlers
    hdlr.handle(record)
  File "/python/cpython-3.12.7-macos-aarch64-none/lib/python3.12/logging/__init__.py", line 1028, in handle
    self.emit(record)
  File "/python/cpython-3.12.7-macos-aarch64-none/lib/python3.12/logging/__init__.py", line 1160, in emit
    msg = self.format(record)
  File "/python/cpython-3.12.7-macos-aarch64-none/lib/python3.12/logging/__init__.py", line 999, in format
    return fmt.format(record)
  File "/.venv/lib/python3.12/site-packages/structlog/stdlib.py", line 1160, in format
    ed = p(logger, meth_name, ed)  # type: ignore[arg-type]
  File "/.venv/lib/python3.12/site-packages/structlog/processors.py", line 896, in __call__
    traceback.print_stack()

in condensed form you can see how just ignoring logging + additional_ignores suffices

temp.py
requests
urllib3
logging
structlog

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions