Grai Matters: The No-Shave November Edition

For those not yet familiar, OpenLineage is an open metadata standard originally developed at WeWork as a component of the Marquez project. It provides a standard interchange format for lineage related metadata and is used heavily by companies like Astronomer. It's since been released as a Linux Foundation project seeing strong developer growth over the past year particularly amongst users who rely upon Airflow and as of October 2023 we are happy to announce we've rolled out alpha support for OpenLineage in Grai!

This is particularly exciting for the team here at Grai not just because it offers another opportunity to work with Open Source developers in the space but because it means users who have already instrumented their infrastructure for OpenLineage don't have to start over when adopting Grai. To that end, we can now provide support for a variety of new integrations including...

New Integration Support

• Airflow
• Dagster
• Spark
• Flink
• Great Expectations
• Keboola

All of these are officially live and available on the cloud. If you're self-hosting now is a great time to update! If you need help getting started with any of these integrations we've got up to-date documentation available for each or...

Support Chat

Relatedly, we've rolled real-time support capabilities into the web-app. If you're on the cloud you might have already noticed a chat window in the lower right hand corner of the page but if not it should look something like this.

Support is currently available most hours of the day as we have team members available in both the US and Europe.

GrAI (beta) release

The idea of an AI integrated component has been percolating amongst the team at Grai for a couple of months and not coincidentally, our name is actually a portmanteau of Great AI. We began exploring opportunities for AI enabled capabilities, lovingly nicknamed grAI, about a month and a half ago and are releasing to beta for self-hosted users with their own OpenAI API keys today.

You'll need to modify any existing deployments with API keys as outlined here, however, once configured you'll have access to a new "Chat" component on the sidebar.

In a very real sense the entire Grai ecosystem is a machine converting everything about an organizations data and data infrastructure into structured text perfect for large language models like ChatGPT. You can think of grAI as your organizations second-brain or as an on staff data steward. By systematically tracking metadata like the running code in each stage of an Airflow job we can uniquely identify potential logic errors, perform complex data lookup queries, and even write semantically correct SQL queries based on the metadata being automatically ingested in Grai. We will be rolling out more of this functionality over time including plugin integrations with various BI tools and the CLI providing active chat functions with the assistant wherever you're working.

For now, this remains a beta release but it really is already pretty cool. Much more to come on this in the future.

Odds N' Ends

• Various UI/UX updates
• expanded API reference documentation
• SOC2 Audit is nearing completion!
• Building custom Integrations just got easier than ever

Talk to you again soon,

~ The Grai Team.