Marshmallow update has broken azure-ai-ml and azure cli ml extension

[friggog]

• Package Name: azure-ai-ml
• Package Version: 1.23.0
• Operating System: Any
• Python Version: Any

Describe the bug
Import of private name from marshallow has broken now they removed it in an update:

    from azure.ai.ml import MLClient
/.../site-packages/azure/ai/ml/__init__.py:11: in <module>
    from azure.ai.ml.entities._inputs_outputs import Input, Output
/.../site-packages/azure/ai/ml/entities/__init__.py:18: in <module>
    from ._assets._artifacts._package.base_environment_source import BaseEnvironment
/.../site-packages/azure/ai/ml/entities/_assets/__init__.py:9: in <module>
    from ._artifacts.code import Code
/.../site-packages/azure/ai/ml/entities/_assets/_artifacts/code.py:10: in <module>
    from azure.ai.ml._schema import CodeAssetSchema
/.../site-packages/azure/ai/ml/_schema/__init__.py:7: in <module>
    from ._data_import import DataImportSchema
/.../site-packages/azure/ai/ml/_schema/_data_import/__init__.py:7: in <module>
    from .data_import import DataImportSchema
/.../site-packages/azure/ai/ml/_schema/_data_import/data_import.py:9: in <module>
    from azure.ai.ml._schema.core.fields import NestedField
/.../site-packages/azure/ai/ml/_schema/core/fields.py:19: in <module>
    from marshmallow.fields import _T, Field, Nested
E   ImportError: cannot import name '_T' from 'marshmallow.fields' (/.../site-packages/marshmallow/fields.py)

Marshmallow version 3.24.0 introduces the issue

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/azure-ml-sdk @azureml-github.

[friggog]

This is the relevant PR: marshmallow-code/marshmallow#2723

[prise6]

Serious issue here. Monitoring in Azure ML failed also with:

cannot import name '_T' from 'marshmallow.fields

This happens in step Annotation - Compute Histogram with GenerationSafetyQualitySignal.

[DavidXuanxuan]

similar issue in my side when I run the code from azure.ai.ml import MLClient, automl.
The error message as below, cannot import name '_T' from 'marshmallow.fields' (/local_disk0/.ephemeral_nfs/envs/pythonEnv-47c589f2-1248-4f0d-a5b3-94e30d7df1a7/lib/python3.9/site-packages/marshmallow/fields.py)

[kingernupur]

We are working on a hotfix release for azure-ai-ml

[eugencutic]

Hi!
The same is happening for the CLI.
I am getting the following output on Windows Powershell:

PS C:\Program Files\Microsoft SDKs\Azure\CLI2> az extension add -n ml
PS C:\Program Files\Microsoft SDKs\Azure\CLI2> az ml component show --version <my-component-version> --name <my-component> --resource-group <my-rg> --workspace-name <my-workspace>
cannot import name '_T' from 'marshmallow.fields' (C:\Users\eugencutic\.azure\cliextensions\ml\marshmallow\fields.py)
'ml' is misspelled or not recognized by the system.

Will this hotfix solve the issue for the CLI as well?
If not, could someone also link this to the correct team/repo for the CLI?

Thanks

[friggog]

Yes I have also just hit the CLI issue exactly as you describe

[MetelStairs]

@eugencutic @friggog I manged to get it to work with running /opt/az/bin/python3 -m pip install marshmallow==3.23.2 after az extension add -n ml -y --allow-preview false, For CLI running in AzDo with Linux

[eugencutic]

PS C:\Program Files\Microsoft SDKs\Azure\CLI2> az extension remove -n ml
PS C:\Program Files\Microsoft SDKs\Azure\CLI2> .\python.exe -m pip show marshmallow
Name: marshmallow
Version: 3.23.2
Summary: A lightweight library for converting complex datatypes to and from native Python datatypes.
Home-page:
Author:
Author-email: Steven Loria [email protected]
License:
Location: C:\Users\eugencutic\AppData\Roaming\Python\Python312\site-packages
Requires: packaging
Required-by:
PS C:\Program Files\Microsoft SDKs\Azure\CLI2> az extension add -n ml -y --allow-preview false
PS C:\Program Files\Microsoft SDKs\Azure\CLI2> az ml component show --version <my-component-version> --name <my-component> --resource-group <my-rg> --workspace-name <my-workspace>
cannot import name '_T' from 'marshmallow.fields' (C:\Users\eugencutic\.azure\cliextensions\ml\marshmallow\fields.py)
'ml' is misspelled or not recognized by the system.

Examples from AI knowledge base:
https://aka.ms/cli_ref
Read more about the command in reference docs
PS C:\Program Files\Microsoft SDKs\Azure\CLI2>

Unfortunately, this workaround is not working for me.
I've also tried downgrading the ml extension to 2.31 and the azure cli to 2.65, which I know is a combination which was working before for me, but I am still seeing the same issue now.

[friggog]

@eugencutic

"C:/Program Files/Microsoft SDKs/Azure/CLI2/python.exe" -m pip install "marshmallow<3.24.0" --target "C:/Program Files/Common Files/AzureCliExtensionDirectory/ml" --upgrade

worked for me on windows - extension directory may vary, looks like yours is C:\Users\eugencutic\.azure\cliextensions\ml