[QNN EP] Make QNN EP a shared library #23120

adrianlizarraga · 2024-12-16T08:57:04Z

Description

Makes QNN EP a shared library by default when building with --use_qnn or --use_qnn dynamic_lib. Generates the following build artifacts:
- Windows: onnxruntime_providers_qnn.dll and onnxruntime_providers_shared.dll
- Linux: libonnxruntime_providers_qnn.so and libonnxruntime_providers_shared.so
- Android: Not supported. Must build QNN EP as a static library.
Allows QNN EP to still be built as a static library with --use_qnn static_lib. This is primarily for the Android QNN AAR package.
Unit tests run for both the static and shared QNN EP builds.

Detailed changes

Updates Java bindings to support both shared and static QNN EP builds.
Provider bridge API:
- Adds logging sink ETW to the provider bridge. Allows EPs to register ETW callbacks for ORT logging.
- Adds a variety of methods for onnxruntime objects that are needed by QNN EP.
QNN EP:
- Adds ort_api.h and ort_api.cc that encapsulates the API provided by ORT in a manner that allows the EP to be built as either a shared or static library.
- Adds custom function to transpose weights for Conv and Gemm (instead of adding util to provider bridge API).
- Adds custom function to quantize data for LeakyRelu (instead of adding util to provider bridge API).
- Adds custom ETW tracing for QNN profiling events:
  - shared library: defines its own TraceLogging provider handle
  - static library: uses ORT's TraceLogging provider handle and existing telemetry provider.
ORT-QNN Packages:
- Python: Pipelines build QNN EP as a shared library by default. User can build a local python wheel with QNN EP as a static library by passing --use_qnn static_lib.
- NuGet: Pipelines build QNN EP as a shared library by default. build.py currently enforces QNN EP to be built as a shared library. Can add support for building a QNN NuGet package with static later if deemed necessary.
- Android: Pipelines build QNN EP as a static library. build.py enforces QNN EP to be built as a static library. Packaging multiple shared libraries into an Android AAR package is not currently supported due to the added need to also distribute a shared libcpp.so library.

Motivation and Context

…osed by the provider bridge.

…evert this in favor of doing the transpose manually in QNN EP

…entType(), DataTypeImpl::TensorTypeFromONNXEnum()

…tions not available in the provider bridge.

…+ iterators

…hat does not need to add new functionality to the provider bridge

…tructor

…d bug in qnn_configs_helper

… library

…EP static build.

…ted on platform

…static and dynamic QNN libs)

snnn · 2025-01-04T02:57:44Z

cmake/onnxruntime_providers_qnn.cmake

+    onnxruntime_add_include_to_target(onnxruntime_providers_qnn onnxruntime_common onnxruntime_framework onnx
+                                                                onnx_proto protobuf::libprotobuf-lite
+                                                                flatbuffers::flatbuffers Boost::mp11)
+    target_link_libraries(onnxruntime_providers_qnn)


This line is empty

cmake/onnxruntime_java.cmake

adrianlizarraga · 2025-01-04T01:28:28Z

tools/ci_build/build.py

+    parser.add_argument(
+        "--use_qnn",
+        nargs="?",
+        const="dynamic_lib",  # If provide --use_qnn without an arg, defaults to a dynamic library.


Perhaps change this to shared_lib for consistency?

adrianlizarraga · 2025-01-04T05:01:59Z

onnxruntime/core/providers/qnn/qnn_telemetry.h

+/// in core/platform/windows/telemetry.h/.cc. In this case, we forward method calls to the
+/// ORT Env's telemetry provider.
+/// </summary>
+class QnnTelemetry {


Hi @ivberg, If you're available, I'd would really appreciate a review of the ETW changes in this PR. Thanks.

adrianlizarraga added 30 commits December 7, 2024 00:09

Copy shared utils into qnn ep

d035fb4

Update QNN EP's initializer transpose logic to use only functions exp…

7e46a7d

…osed by the provider bridge.

Update comment

a155b33

Added TransposeBase::DoTranspose() to provider bridge. May elect to r…

e9c5f14

…evert this in favor of doing the transpose manually in QNN EP

Add TypeProto_Tensor_has_elem_type() to provider bridge

d0f64dc

Add to provider bridge: TensorTypeBase class, TensorTypeBase::GetElem…

f8bd2f6

…entType(), DataTypeImpl::TensorTypeFromONNXEnum()

Transpose initializers within QNN EP without using CPU EP utils

1f533a9

Rename transpose func

ccaefb3

Remove TransposeBase forward declaration from provider bridge

fb765c7

Rewrite SliceOpBuilder util GetInitializerInputData() to not use func…

0237bca

…tions not available in the provider bridge.

Revert addition of TensorTypeBase to provider bridge

e3705b2

Remove last use of GetTensorShapeFromTensorProto

6f0b3c6

Add DataTypeUtils::ToType(std::string&) to provider bridge

5939bf6

Add Logger::GetSeverity() to provider bridge

58dbf49

Add TensorShapeProto_Dimensions__size to provider bridge

f76b09a

Add utils::CreateSupportedPartitions() to provider bridege

d189fe6

Merge main and fix conflicts

a0b3c75

Use new namespace for NodeAttrHelper

48191ea

Add to provider bridge: GraphViewer::Nodes(), ConstGraphNodes struct …

e6afd72

…+ iterators

Replace usage of cbegin() and cend() in NodeAttrHelper with version t…

0b1e538

…hat does not need to add new functionality to the provider bridge

Add convenience function to get the default Env to provider bridge

fb3618d

Moving ORT includes to a separate header

6b581fd

Add Node::EdgeEnd wrapper class to provider bridge. Add NodeUnit cons…

a1129e5

…tructor

Move more header includes to ort_api.h

ba86c41

Add GraphViewer::NodeProducesGraphOutput() to provider bridge

2b1ea09

Replace use of InlinedVector with std::vector and fix newly discovere…

421cd78

…d bug in qnn_configs_helper

Eliminate use of qmath.h by introducing new quantization utils for QNN

d94e6f7

Move includes into qnn/ort_api.h

4eb1e80

Add TensorProto::has_data_type() to provider bridge

d86fb6c

Checkpoint: updating usage of provider bridge in ep

187e3b9

adrianlizarraga added 19 commits January 1, 2025 17:20

Merge branch 'main' into adrianl/qnn-ep-dynamic-lib

354203c

Fix java cmake conditions when building qnn as shared lib

d500cf8

Handle possibility of shared or static qnn lib in Java bindings

183ca05

properly dectect qnn build when building android+qnn AAR package

5e44199

Don't publish artifacts for QNN Win arm64 if QNN is built as a static…

29c6872

… library

python linter

02b71ed

don't include ntverp.h if not needed. set cmake source_group for QNN …

52199d7

…EP static build.

Add function to provider bridge to check if ETW for logging is suppor…

af16458

…ted on platform

python linting

1d61cae

Move NodeAttrHelper to ort_api.h/cc

dfc9baf

Make some ort_api.h functions inline

325d62f

cmake clean up

a72efb5

Clean up include paths

67abb6a

Edit pipeline job names

6335a88

Add QnnTelemetry for logging profiling events to etw (works for both …

f9e5c0c

…static and dynamic QNN libs)

Add comments to quantization utils

a4f6923

Merge branch 'main' into adrianl/qnn-ep-dynamic-lib

e3f6f5e

Fix QnnTelemetry for static lib case

4a0c7a1

Dont use QnnTelemetry on non-windows

0495a99

adrianlizarraga marked this pull request as ready for review January 4, 2025 00:52

adrianlizarraga changed the title ~~[QNN EP] [DRAFT] Make QNN EP a shared library~~ [QNN EP] Make QNN EP a shared library Jan 4, 2025

adrianlizarraga requested review from HectorSVC, snnn, jywu-msft, ivberg and edgchen1 January 4, 2025 00:59

snnn reviewed Jan 4, 2025

View reviewed changes

adrianlizarraga requested a review from RyanUnderhill January 4, 2025 04:26

adrianlizarraga commented Jan 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[QNN EP] Make QNN EP a shared library #23120

[QNN EP] Make QNN EP a shared library #23120

adrianlizarraga commented Dec 16, 2024 •

edited

Loading

snnn Jan 4, 2025

adrianlizarraga Jan 4, 2025

adrianlizarraga Jan 4, 2025 •

edited

Loading

[QNN EP] Make QNN EP a shared library #23120

Are you sure you want to change the base?

[QNN EP] Make QNN EP a shared library #23120

Conversation

adrianlizarraga commented Dec 16, 2024 • edited Loading

Description

Detailed changes

Motivation and Context

snnn Jan 4, 2025

Choose a reason for hiding this comment

adrianlizarraga Jan 4, 2025

Choose a reason for hiding this comment

adrianlizarraga Jan 4, 2025 • edited Loading

Choose a reason for hiding this comment

adrianlizarraga commented Dec 16, 2024 •

edited

Loading

adrianlizarraga Jan 4, 2025 •

edited

Loading