invalid token format' error: MAS <-> Synapse seems to be broken

[alexander-potemkin]

Describe the bug
I can see in the Synapse & MAS logs warning on the tokens issues and Element X stuck with Sync, events get delayed significantly.

To Reproduce
Steps to reproduce the behavior:

1. Install fresh MAS
2. Install fresh Synapse
3. Connect them as per documentation with OpenID provider

Expected behavior
Sync to takes 1 second at least and everything to work.

Screenshots
N/A

Desktop (please complete the following information):
N/A - server side issue.

Smartphone (please complete the following information):
N/A - server side issue.

Additional context
I've been complaining on the matrix.org a bit with additional details, but the key seems to be:

2025-12-02T16:36:39.296649Z DEBUG Metrics.InstrumentCreated POST-8 -  name="Metrics.InstrumentCreated" instrument_name="mas.oauth2.introspection_request" cardinality_limit=2000
2025-12-02T16:36:39.296692Z  INFO http.server.response POST-8 - "POST /oauth2/introspect HTTP/1.1" 200 OK "Synapse/1.143.0" [polls: 5, cpu: 0.3ms, elapsed: 1.6ms]

Messages from the MAS server.

I've adjusted PostgreSQL to log all SQL queries and that way checked the tokens - from what I can see, they are perfectly valid - no problems.

I've spend good two days troubleshooting that but now it seems like some weird bug on MAS side. Happy to provide any additional information required!

[sandhose]

Looks like on the MAS side the introspection is quite quick. From the synapse side, we have a metric which measures how long token introspection takes, could you look at that?

Example query:

histogram_quantile(0.95, 
  sum by(le) (
    rate(
      synapse_api_auth_delegated_introspection_response_bucket{YOUR LABELS}[1m]
    )
  )
)

[alexander-potemkin]

Looks like on the MAS side the introspection is quite quick. From the synapse side, we have a metric which measures how long token introspection takes, could you look at that?

Example query:

histogram_quantile(0.95, 
  sum by(le) (
    rate(
      synapse_api_auth_delegated_introspection_response_bucket{YOUR LABELS}[1m]
    )
  )
)

I'd love to - but I don't have Prometheus... but from the logs timestamps and viewing the logs - the performance doesn't seem to be an issue.