You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[Plugin EP] Allow EP to provide additional virtual devices (#26234)
### Description
Adds APIs to allow a plugin EP to create a virtual `OrtHardwareDevice`
that can be used for model cross-compilation. For example, this allows
an EP to create a compiled model for NPU on a device that does not have
an NPU.
#### Application code
An application must explicitly allow registered plugin EPs to create
virtual devices. This is currently done by using a registration name
that ends in the `".virtual"` suffix. Ex:
```c++
#include "onnxruntime_cxx_api.h"
#include "onnxruntime_ep_device_ep_metadata_keys.h"
const char* ep_registration_name = "my_ep_lib.virtual"; // IMPORTANT: ".virtual" suffix is a signal to EP library
ort_env->RegisterExecutionProviderLibrary(ep_registration_name, "my_ep.dll");
std::vector<Ort::ConstEpDevice> ep_devices = ort_env->GetEpDevices();
// ep_devices includes an OrtEpDevice from "my_ep.dll" that uses a virtual OrtHardwareDevice.
Ort::ConstEpDevice virtual_ep_device = std::find_if(ep_devices.begin(), ep_devices.end(),
[](Ort::ConstEpDevice& device) {
return device.EpName() == std::string("MyEpName");
});
// App can look in HW metadata to check if is virtual
Ort::ConstHardwareDevice virtual_hw_device = virtual_ep_device.Device();
std::unordered_map<std::string, std::string> metadata = virtual_hw_device.Metadata().GetKeyValuePairs();
assert(metadata[kOrtHardwareDevice_MetadataKey_IsVirtual] == "1");
// App can use the virtual OrtEpDevice in a session to, for example, compile a model
// ...
```
#### Plugin EP code
This PR introduces a new _optional_ C API function in the `OrtEpFactory`
struct called `SetEnvironmentOptions` that allows ORT to pass options
(as key/value pairs) to an EP factory. Currently, the only key supported
is `"allow_virtual_devices"`, which indicates to the EP factory that
creating virtual devices is allowed.
When the application registers a plugin EP library, ORT creates the
library's EP factories and checks if they implement the
`SetEnvironmentOptions` API function. If so, ORT calls
`ep_factory.SetEnvironmentOptions` with `"allow_virtual_devices"` set to
`"1"` if the EP registration name set by the application ends in the
`".virtual"` suffix (or `"0"` otherwise).
Here's an example implementation of
`OrtEpFactory::SetEnvironmentOptions` taken from a [test plugin EP that
supports a virtual
GPU](https://github.com/microsoft/onnxruntime/tree/adrianl/plugin-ep-specify-ort-hw-device/onnxruntime/test/autoep/library/example_plugin_ep_virt_gpu):
```c++
/*static*/
OrtStatus* ORT_API_CALL EpFactoryVirtualGpu::SetEnvironmentOptionsImpl(OrtEpFactory* this_ptr,
const OrtKeyValuePairs* options) noexcept {
auto* factory = static_cast<EpFactoryVirtualGpu*>(this_ptr);
const char* value = factory->ort_api_.GetKeyValue(options, "allow_virtual_devices");
if (value != nullptr) {
factory->allow_virtual_devices_ = strcmp(value, "1") == 0;
}
return nullptr;
}
```
An EP factory can create a virtual hardware device within
`OrtEpFactory::GetSupportedDevices` by using a new API function called
`CreateHardwareDevice`. The EP factory is expected to own the hardware
device instance, which should be released when the factory is destroyed
via `ReleaseHardwareDevice`.
The [test plugin EP shows an
implementation](https://github.com/microsoft/onnxruntime/blob/d87f8b86406525f5801a7a9933b1ced1eb40940c/onnxruntime/test/autoep/library/example_plugin_ep_virt_gpu/ep_factory.cc#L86)
of `OrtEpFactory::GetSupportedDevices` that creates a virtual GPU
device.
```c++
/*static*/
OrtStatus* ORT_API_CALL EpFactoryVirtualGpu::GetSupportedDevicesImpl(OrtEpFactory* this_ptr,
const OrtHardwareDevice* const* /*devices*/,
size_t /*num_devices*/,
OrtEpDevice** ep_devices,
size_t max_ep_devices,
size_t* p_num_ep_devices) noexcept {
size_t& num_ep_devices = *p_num_ep_devices;
auto* factory = static_cast<EpFactoryVirtualGpu*>(this_ptr);
num_ep_devices = 0;
// Create a virtual OrtHardwareDevice if application indicated it is allowed (e.g., for cross-compiling).
// This example EP creates a virtual GPU OrtHardwareDevice and adds a new OrtEpDevice that uses the virtual GPU.
if (factory->allow_virtual_devices_ && num_ep_devices < max_ep_devices) {
OrtKeyValuePairs* hw_metadata = nullptr;
factory->ort_api_.CreateKeyValuePairs(&hw_metadata);
factory->ort_api_.AddKeyValuePair(hw_metadata, kOrtHardwareDevice_MetadataKey_IsVirtual, "1");
auto* status = factory->ep_api_.CreateHardwareDevice(OrtHardwareDeviceType::OrtHardwareDeviceType_GPU,
factory->vendor_id_,
/*device_id*/ 0,
factory->vendor_.c_str(),
hw_metadata,
&factory->virtual_hw_device_);
// ...
OrtEpDevice* virtual_ep_device = nullptr;
status = factory->ort_api_.GetEpApi()->CreateEpDevice(factory, factory->virtual_hw_device_, ep_metadata,
ep_options, &virtual_ep_device);
// ...
ep_devices[num_ep_devices++] = virtual_ep_device;
```
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
0 commit comments