metricflow_time_spine error on dbt 2024.8.221

[pestrov]

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

We're running dbt Cloud Versionless in our dev and prod environments, and yesterday we have started getting the metricflow_time_spine related error that does not allow us to run any run/compile/build commands.

We have fivetran ad_reporting package that seems to have a semantic model and generates metricflow_time_spine inside.

We don’t have a spine model in our own project and don’t specify it.

Relevant error log or model output

Running dbt...
22:13:53 Time spines without YAML configuration are in the process of deprecation. Please add YAML configuration for your 'metricflow_time_spine' model. See documentation to configure: https://docs.getdbt.com/docs/build/metricflow-time-spine
22:13:54 Encountered an error:
Object of type date is not JSON serializable

Expected behavior

The models should continue building.

dbt Project configurations

vars:
ad_reporting__amazon_ads_enabled: False # by default this is assumed to be True
ad_reporting__apple_search_ads_enabled: True # by default this is assumed to be True
ad_reporting__pinterest_ads_enabled: False # by default this is assumed to be True
ad_reporting__microsoft_ads_enabled: False # by default this is assumed to be True
ad_reporting__linkedin_ads_enabled: True # by default this is assumed to be True
ad_reporting__google_ads_enabled: True # by default this is assumed to be True
ad_reporting__twitter_ads_enabled: False # by default this is assumed to be True
ad_reporting__facebook_ads_enabled: False # by default this is assumed to be True
ad_reporting__snapchat_ads_enabled: False # by default this is assumed to be True
ad_reporting__tiktok_ads_enabled: False # by default this is assumed to be True
ad_reporting__reddit_ads_enabled: False
vars:
'dbt_date:time_zone': 'America/Chicago'

Package versions

packages:

• package: fivetran/app_reporting # https://github.com/fivetran/dbt_app_reporting
  version: 0.3.2

• package: fivetran/ad_reporting # https://github.com/fivetran/dbt_ad_reporting
  version: 1.9.0

What database are you using dbt with?

snowflake

dbt Version

2024.8.221

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

[fivetran-joemarkiewicz]

Hi @pestrov thanks for opening this issue and sorry you're seeing this error.

I'm not entirely in the loop on the nuances of dbt Cloud Versionless, but after some digging it seems that this is a mechanism for dbt Cloud users to always be using the latest versions of dbt Core without needing to manual upgrade. I'm not entirely sure why you are seeing this error, but I imagine this may be something dbt Labs should take a look at and provide some guidance. It looks like dbt Labs mentions in their considerations section of dbt Cloud Versionless that if you see any issues when using dbt Cloud Versionless that you should open a contact dbt Labs support and downgrade your dbt Core version in the meantime.

Have you been able to contact dbt Labs support and see what their thoughts are regarding this error you are seeing?

cc: @Jstein77 in case you have any understanding or visibility into this error message and the root cause.

[pestrov]

Thanks @fivetran-joemarkiewicz !

Yeah, I've reached out to dbt support and so far their response was "it looks like the package you are using is not yet supporting the latest dbt version, please reach out to them". I understand the complexity of this though. Do you guys have access to this dbt version? The debug logs say it is "2024.8.221".

The issue is that we have already migrated our whole project to 1.8 (tests -> data_tests), so we can't really downgrade to 1.7 (we'll lose all our tests).

I found a what looks like a related change from last week in dbt-core that might have caused this issue and have commented there as well —maybe this can help you in any way.

[fivetran-joemarkiewicz]

@pestrov thanks for sharing the additional details.

It seems the root of this issue has to do with compatibility with a new version of dbt core. I haven't dug to far into this new version, but after looking at the error message I imagine we can address this issue by providing yml documentation for the metric flow spine model.

I just created a branch which adds this yml documentation. Since I was unable to recreate the error myself, I am not 100% sure this will fix the issue, but it would be great if you can give it a test. If you would like to test this out, you can replace the official dbt hub Ad Reporting version in your packages.yml with the following:

packages:
  # - package: fivetran/ad_reporting
  #   version: [">=1.9.0", "<1.10.0"]
  - git: https://github.com/fivetran/dbt_ad_reporting.git
    revision: bugfix/metricflow-spine-yml
    warn-unpinned: false 

Let me know if you're able to test this out and if this fixes the issue you're seeing.

[pestrov]

Thanks for a quick test! This did remove the deprecation notice, but I'm still getting the JSON encoding error in manifest creation process. Will take a deeper look and report back.

[fivetran-joemarkiewicz]

Sounds good! Glad that resolved the deprecation notice, but sorry to hear that didn't sort out the JSON serializable error. I will continue to investigate as well.

Let me know if you're able to uncover any updates that need to be applied to resolve the error.

[fivetran-joemarkiewicz]

@pestrov one thought I had, do we know that the following error is related to the Ad Reporting package?

Object of type date is not JSON serializable

Are there any other models you think could result in this error.

Additionally, you state the version is 2024.8.221. I don't seem to see that as an available version for dbt Core. Do you know which listed dbt Core version you are using in your environment? I am unsure about how to find this in dbt Cloud, but I know you can obtain the version by running dbt --version in the CLI. Additionally, you can probably find the version at the top of the run log for your dbt job.

This will help our team troubleshoot and ideally recreate the issue and help provide a solution if it is related to the Ad Reporting package. Thanks!

[fivetran-joemarkiewicz]

Hi @pestrov, just wanted to post back here and see if you are still experiencing this issue or if you have found a possible solution?

In the meantime I will mark this issue as stale. Please let me know if you are still experiencing this issue or if you would like to continue this conversation.

[raphaelvarieras]

Hey @fivetran-joemarkiewicz - we're still seeing the issue. Wherever the ad_reporting package is used, we still get a warning about properly defining the time spine in YAML:

17:01:19  [WARNING]: Time spines without YAML configuration are in the process of deprecation. Please add YAML configuration for your 'metricflow_time_spine' model. See documentation on MetricFlow time spines:
https://docs.getdbt.com/docs/build/metricflow-time-spine and behavior change documentation: https://docs.getdbt.com/reference/global-configs/behavior-changes.

From what I can tell, it could be as simple as adding the following to one of the YAML files about this model:

models:
  - name: metricflow_time_spine
    time_spine:
      standard_granularity_column: date_day # column for the standard grain of your table
    columns:
      - name: date_day
        granularity: day # set granularity at column-level for standard_granularity_column

[fivetran-joemarkiewicz]

@raphaelvarieras thanks for chiming in here! Were you able to test this branch to see if this resolved the deprecation warning? If so, we could consider folding this into an upcoming release. Let me know if this branch fixes the warning. Thanks!

packages:
  # - package: fivetran/ad_reporting
  #   version: [">=1.9.0", "<1.10.0"]
  - git: https://github.com/fivetran/dbt_ad_reporting.git
    revision: bugfix/metricflow-spine-yml
    warn-unpinned: false 

[raphaelvarieras]

@pestrov - would you mind performing that test @fivetran-joemarkiewicz recommended while commenting out the code I added to see if it helps?