Skip to content

databricks_compute assignment is not respected for models executed in pre and post hooks #1268

@johnnyC9000

Description

@johnnyC9000

Describe the bug

databricks_compute assignment is not respected for models executed in pre and post hooks

Steps To Reproduce

The Elementary package is used in our dbt pipelines to capture performance data. Pre and post hooks are utilized to persist the data in multiple models. The Elementary models are assigned a specific compute using the databricks_compute attribute in dbt_project.yml. This assignment is visible when inspecting the manifest.json file. We would expect the model to use the compute assigned, however it falls back to using what is defined as the target default.

Expected behavior

It is expected that pre and post hook executions would acknowledge the usage of the databricks_compute assignment instead of using the default compute specified on a target.

Screenshots and log output

If applicable, add screenshots or log output to help explain your problem.

System information

The output of dbt --version:

using legacy validation callback
Core:
  - installed: 1.10.2 
  - latest:    1.10.15 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - databricks: 1.10.4 - Update available!
  - spark:      1.9.2  - Update available!
  - athena:     1.9.4  - Update available!

  At least one plugin is out of date with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation```

**The operating system you're using:**
Linux

**The output of `python --version`:**
Python 3.12.4

### Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions