dbt Core or dbt Cloud ?

Beyond the descriptions available online, which are often more marketing than operational reality, here’s what I’ve learned over the years working with these two versions of dbt.

dbt Cloud

• Fixed cost of $100 per user per month (let’s simplify and say €100)
• Integrated scheduling
• Lineage
• Documentation
• A long list of new features that dbt has recently developed (article to come on this topic)
• Some limits, however: 15,000 models built per month, 8 developers (a.k.a. Analytics Engineers), beyond that you need to contact them to define a budget

dbt Core

• No fixed monthly cost
• Scheduling to manage yourself (let’s say… Airflow!)
• Lineage
• Documentation

In my opinion, there are two factors that come into play when choosing:

• Money
• The complexity induced by the choice between dbt Core and dbt Cloud, and therefore… money!

I heard in a large company about the cost of dbt Cloud: “Do you realize, that means paying a monthly cost to dbt, it would cost us a fortune. No, it’s better to use dbt Core and not pay, we’ll save money!” As a freelancer who had just joined this client, I wasn’t going to change a company policy in a few days, with tight deadlines, so I used dbt Core.

This topic, in my opinion, is a real issue, and we can’t just think that on one side we have the subscription cost of dbt Cloud vs dbt Core being “free.”

Let’s break down the numbers. I think it’s not bad for an Analytics Engineer to start this way 😉 To simplify, I will reason annually.

dbt Cloud

• Subscription: €100 per month, so €1,200 annually
• Scheduling included in the subscription

dbt Core

• Subscription: €0
• Scheduling: €0

Did you say 0 for dbt Core’s scheduling? Hmm, well, let’s take Airflow, and… there you go!

Well, a few clarifications are needed. My experience has shown that several factors must be taken into account, and sometimes there are more or less “hidden” costs:

• You need to find a profile that masters Airflow. This adds time to search for the right profile, and thus costs.
• You’ll also need a bit more time to find a profile that’s comfortable with dbt Core as compared to dbt Cloud, because the person needs to be comfortable initializing dbt without everything being done automatically in dbt Cloud. So, more search time again.
• Once the profile is found, I’m convinced that on average, a profile that masters both dbt Core and Airflow will cost more than a profile that is comfortable with dbt Cloud.
• The initialization of dbt for the Core version vs. Cloud is much longer. That means time, and thus money.
• Scheduling via Airflow takes time, and compared to dbt Cloud, we know in advance who will take a few minutes and who will take… much longer with Airflow.
• When an Analytics Engineer leaves the project (yes, we have to take this into account), taking over an existing dbt Core + Airflow project seems more complicated than taking over a dbt Cloud project.
• The subscription per user. €0 annually for dbt Core, but €1,200 for dbt Cloud (as of December 27, 2024).

Summary:

It becomes clear very quickly that on one side, for dbt Core, we have an annual subscription cost of €0, but a significant list of factors to evaluate, which, trust me, will quickly drive up the bill. On the other side, for dbt Cloud, we have that fixed cost of €1,200 per user, but we get a simpler tool setup than with dbt Core.

I’ll mention just one number to help with your decision-making. A freelancer who masters Airflow + dbt Core will cost, I believe, €50 more per day than a profile who is comfortable with dbt Cloud. I’ve been kind, the reality is likely closer to €100 more per day. But let’s take €50 per day, 20 working days, that’s an additional €1,000 per month for this profile when using dbt Core + Airflow vs dbt Cloud. Do you remember the €1,200 per user annually for dbt Cloud? I’ll say no more…

Your move! Every company has its own skillset, its own costs, and each must, in all conscience, do the real cost estimation work to make an informed decision between dbt Core and dbt Cloud.

And beyond time and costs, there are other factors to consider. For example, with dbt Core, data doesn’t pass through yet another online platform, so you have complete control from A to Z. On the other hand, let’s not kid ourselves, dbt is a business, and they have decided to focus on what makes them money—dbt Cloud. So many features are only available there, and I’m not convinced they will develop them for dbt Core anytime soon. So, the choice isn’t as simple as it seems! Anyway, the additional features present in dbt Cloud vs dbt Core will be the subject of another article!

Dataly yours,
Martin