No longer required to wrap the `$ref` key in an `allOf` array with a single `$ref` element

[ollyhensby]

Issue
I've defined the following schema with a single $ref element and resolve_refs is failing to resolve it:

{
    "$defs": {
        "Obj": {
            "properties": {
                "a": {
                    "title": "A",
                    "type": "integer"
                },
                "b": {
                    "title": "B",
                    "type": "string"
                }
            },
            "required": [
                "a",
                "b"
            ],
            "title": "Obj",
            "type": "object"
        },
        "ObjSet": {
            "properties": {
                "obj_set": {
                    "items": {
                        "anyOf": [
                            {
                                "$ref": "#/$defs/Obj"
                            },
                            {
                                "$ref": "#/$defs/ObjSet"
                            }
                        ]
                    },
                    "title": "Obj Set",
                    "type": "array"
                }
            },
            "required": [
                "obj_set"
            ],
            "title": "ObjSet",
            "type": "object"
        }
    },
    "$ref": "#/$defs/ObjSet"
}

I get the following traceback:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/jsonref.py:179](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/jsonref.py#line=178), in JsonRef.resolve_pointer(self, document, pointer)
    178 try:
--> 179     document = document[part]
    180 except (TypeError, LookupError) as e:

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py:190](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py#line=189), in Proxy.add_proxy_meth.<locals>.proxied(self, *args, **kwargs)
    189 args.insert(arg_pos, self.__subject__)
--> 190 result = func(*args, **kwargs)
    191 return result

KeyError: '$defs'

The above exception was the direct cause of the following exception:

JsonRefError                              Traceback (most recent call last)
File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/core/formatters.py:711](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/core/formatters.py#line=710), in PlainTextFormatter.__call__(self, obj)
    704 stream = StringIO()
    705 printer = pretty.RepresentationPrinter(stream, self.verbose,
    706     self.max_width, self.newline,
    707     max_seq_length=self.max_seq_length,
    708     singleton_pprinters=self.singleton_printers,
    709     type_pprinters=self.type_printers,
    710     deferred_pprinters=self.deferred_printers)
--> 711 printer.pretty(obj)
    712 printer.flush()
    713 return stream.getvalue()

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py:394](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py#line=393), in RepresentationPrinter.pretty(self, obj)
    391 for cls in _get_mro(obj_class):
    392     if cls in self.type_pprinters:
    393         # printer registered in self.type_pprinters
--> 394         return self.type_pprinters[cls](obj, self, cycle)
    395     else:
    396         # deferred printer
    397         printer = self._in_deferred_types(cls)

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py:701](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py#line=700), in _dict_pprinter_factory.<locals>.inner(obj, p, cycle)
    699     p.pretty(key)
    700     p.text(': ')
--> 701     p.pretty(obj[key])
    702 p.end_group(step, end)

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py:394](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py#line=393), in RepresentationPrinter.pretty(self, obj)
    391 for cls in _get_mro(obj_class):
    392     if cls in self.type_pprinters:
    393         # printer registered in self.type_pprinters
--> 394         return self.type_pprinters[cls](obj, self, cycle)
    395     else:
    396         # deferred printer
    397         printer = self._in_deferred_types(cls)

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py:701](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py#line=700), in _dict_pprinter_factory.<locals>.inner(obj, p, cycle)
    699     p.pretty(key)
    700     p.text(': ')
--> 701     p.pretty(obj[key])
    702 p.end_group(step, end)

    [... skipping similar frames: RepresentationPrinter.pretty at line 394 (2 times), _dict_pprinter_factory.<locals>.inner at line 701 (1 times)]

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py:701](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py#line=700), in _dict_pprinter_factory.<locals>.inner(obj, p, cycle)
    699     p.pretty(key)
    700     p.text(': ')
--> 701     p.pretty(obj[key])
    702 p.end_group(step, end)

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py:394](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py#line=393), in RepresentationPrinter.pretty(self, obj)
    391 for cls in _get_mro(obj_class):
    392     if cls in self.type_pprinters:
    393         # printer registered in self.type_pprinters
--> 394         return self.type_pprinters[cls](obj, self, cycle)
    395     else:
    396         # deferred printer
    397         printer = self._in_deferred_types(cls)

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py:649](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py#line=648), in _seq_pprinter_factory.<locals>.inner(obj, p, cycle)
    647         p.text(',')
    648         p.breakable()
--> 649     p.pretty(x)
    650 if len(obj) == 1 and isinstance(obj, tuple):
    651     # Special case for 1-item tuples.
    652     p.text(',')

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py:419](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py#line=418), in RepresentationPrinter.pretty(self, obj)
    408                         return meth(obj, self, cycle)
    409                 if (
    410                     cls is not object
    411                     # check if cls defines __repr__
   (...)
    417                     and callable(_safe_getattr(cls, "__repr__", None))
    418                 ):
--> 419                     return _repr_pprint(obj, self, cycle)
    421     return _default_pprint(obj, self, cycle)
    422 finally:

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py:787](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/IPython/lib/pretty.py#line=786), in _repr_pprint(obj, p, cycle)
    785 """A pprint that just redirects to the normal repr function."""
    786 # Find newlines and replace them with p.break_()
--> 787 output = repr(obj)
    788 lines = output.splitlines()
    789 with p.group():

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py:121](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py#line=120), in ProxyMetaClass._no_proxy.<locals>.wrapper(self, *args, **kwargs)
    119 _osa(self, "__notproxied__", True)
    120 try:
--> 121     return method(self, *args, **kwargs)
    122 finally:
    123     _osa(self, "__notproxied__", notproxied)

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/jsonref.py:199](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/jsonref.py#line=198), in JsonRef.__repr__(self)
    197 def __repr__(self):
    198     if hasattr(self, "cache") or self.load_on_repr:
--> 199         return repr(self.__subject__)
    200     return "JsonRef(%r)" % self.__reference__

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py:163](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py#line=162), in Proxy.__getattribute__(self, attr)
    161 if Proxy._should_proxy(self, attr):
    162     return getattr(self.__subject__, attr)
--> 163 return _oga(self, attr)

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py:121](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py#line=120), in ProxyMetaClass._no_proxy.<locals>.wrapper(self, *args, **kwargs)
    119 _osa(self, "__notproxied__", True)
    120 try:
--> 121     return method(self, *args, **kwargs)
    122 finally:
    123     _osa(self, "__notproxied__", notproxied)

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py:243](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py#line=242), in LazyProxy.__subject__(self)
    240 except AttributeError:
    241     pass
--> 243 self.cache = super(LazyProxy, self).__subject__
    244 return self.cache

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py:121](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py#line=120), in ProxyMetaClass._no_proxy.<locals>.wrapper(self, *args, **kwargs)
    119 _osa(self, "__notproxied__", True)
    120 try:
--> 121     return method(self, *args, **kwargs)
    122 finally:
    123     _osa(self, "__notproxied__", notproxied)

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py:227](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py#line=226), in CallbackProxy.__subject__(self)
    225 @property
    226 def __subject__(self):
--> 227     return self.callback()

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py:121](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py#line=120), in ProxyMetaClass._no_proxy.<locals>.wrapper(self, *args, **kwargs)
    119 _osa(self, "__notproxied__", True)
    120 try:
--> 121     return method(self, *args, **kwargs)
    122 finally:
    123     _osa(self, "__notproxied__", notproxied)

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/jsonref.py:140](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/jsonref.py#line=139), in JsonRef.callback(self)
    138 else:
    139     base_doc = self.store[uri]
--> 140 result = self.resolve_pointer(base_doc, fragment)
    141 if result is self:
    142     raise self._error("Reference refers directly to itself.")

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py:121](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/proxytypes.py#line=120), in ProxyMetaClass._no_proxy.<locals>.wrapper(self, *args, **kwargs)
    119 _osa(self, "__notproxied__", True)
    120 try:
--> 121     return method(self, *args, **kwargs)
    122 finally:
    123     _osa(self, "__notproxied__", notproxied)

File [~/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/jsonref.py:181](http://localhost:8888/home/jovyan/miniforge3/envs/ipyautoui-dev/lib/python3.12/site-packages/jsonref.py#line=180), in JsonRef.resolve_pointer(self, document, pointer)
    179         document = document[part]
    180     except (TypeError, LookupError) as e:
--> 181         raise self._error(
    182             "Unresolvable JSON pointer: %r" % pointer, cause=e
    183         ) from e
    184 return document

JsonRefError: Error while resolving `#/$defs/Obj`: Unresolvable JSON pointer: '/$defs/Obj'

If I replace the reference section with:

"allOf": [{
        "$ref": "#/$defs/ObjSet"
    }]

the replace_refs works as expected.

Context

I noticed this issue through the use of pydantic when using their model_json_schema method. In pydantic==2.8 the JSON schema returned from a model returns with the allOf, whereas in pydantic==2.9 the model returns without the allOf.

I am no expert in JSON schema but from what they've said I think it's related to this: pydantic/pydantic#10029 (comment)

User dpeachey says that The JSON schema spec was updated to allow sibling keys alongside $ref keys so it is no longer required to wrap the $ref key in a allOf array with a single $ref element.

This is mentioned in the JSON schema spec here: https://json-schema.org/draft/2020-12/json-schema-core#appendix-G-2.6.1.10

Any help with this issue is much appreciated!

[gazpachoking]

The JSON schema spec was updated to allow sibling keys alongside $ref keys so it is no longer required to wrap the $ref key in a allOf array with a single $ref element.

This is indeed why it doesn't work here by default and has caused much confusion. The jsonref spec says All other properties of an object containing a $ref key are ignored. That means that the thing you are trying to reference is ignored in your first case by virtue of being another property in the object with the $ref. Jsonschema no longer uses the jsonref spec, and instead defines $refs themselves now, and apparently took that language about ignoring other sibling keys out of their spec.

The good thing is, we already added an option to this library to ignore that part of the jsonref spec. It should do what you want by using the merge_props=True argument to replace_refs.

[ollyhensby]

Thank you for the explanation @gazpachoking! Using merge_props=True has resolved this issue for us.

Is there any motivation to set merge_props=True as the default setting since the jsonschema now no longer uses the jsonref spec?