Production Database Changes You Should Fear the Most
As data engineers, we all know that production database changes can be a bit of a headache. Whether it's a schema change, a column type change, or a column name change, it can feel like we're always playing catch-up with the rest of the organization. But what if I told you that there are certain types of production database changes that you should be especially wary of?
Converting a Column to a New Table
One of the most dangerous production database changes is converting a column to a new table. For example, imagine that you have a boolean column that indicates whether a user is subscribed or not. If your organization decides to change that column to a table with subscriptions on it, you'll have to update all of your queries, data pipelines, and reporting tools to reflect the change. This can be a huge headache, especially if you're working with a large amount of data.
Column Type Changes
Another type of production database change that can cause problems is column type changes. For example, imagine that you have a column that is currently an integer, but your organization decides to add more detail to the data, so the column needs to change to a float. This can cause problems with your data pipelines, queries, and reporting tools, as they may not be able to handle the new data type.
Column Name Changes
A third type of production database change to be aware of is column name changes. For example, imagine that your organization discovers a typo in a column name, and decides to change it. This can cause problems with your data pipelines, queries, and reporting tools, as they may not be able to find the correct column.
Inverting a Boolean
A fourth type of production database change that can cause problems is inverting a boolean. For example, imagine that you have a column that is currently called "not_paid", but your organization decides to change it to "paid". This can cause problems with your data pipelines, queries, and reporting tools, as they may not be able to handle the change.So, what can you do to minimize the impact of these production database changes? Here are four strategies to consider:
Hold More Meetings
One way to minimize the impact of production database changes is to hold more meetings with the rest of your organization. By getting involved early in the process, you can learn about changes before they happen, and take steps to mitigate the impact.
Ask Your Software Engineers
Another strategy is to ask your software engineers for help. They may be able to provide you with information about changes or even offer solutions to problems that you're encountering.
Testing After Changes
A third strategy is to test your data pipelines, queries, and reporting tools after changes have been made. For example, you can use a tool like dbt tests to ensure that your data is still accurate and complete.
Active Testing Before Changes
Finally, you can use a tool like Grai to actively test your data pipelines, queries, and reporting tools before changes are made. This can help you identify and fix problems early on, minimizing the impact of production database changes.As data engineers, we all know that production database changes can be a bit of a headache. But by being aware of the types of changes that can cause problems, and taking steps to mitigate the impact, you can minimize the impact of production database changes on your organization.