Julian LaNeve is the Chief Technical Officer (CTO) at Astronomer, the driving pressure behind Apache Airflow and fashionable knowledge orchestration to energy every thing from AI to common analytics.
Julian does product and engineering at Astronomer the place he focuses on developer expertise, knowledge observability, and AI. He’s additionally the writer of Cosmos, an Airflow supplier for working dbt Core tasks as Airflow DAGs.
He’s captivated with all issues knowledge and open supply as he spends his spare time doing hackathons, prototyping new tasks, and exploring the newest in knowledge.
May you share your private story of the way you grew to become concerned with software program engineering, and labored your method as much as being CTO of Astronomer?
I’ve been coding since I used to be in center college. For me, engineering has at all times been an incredible inventive outlet: I can give you an thought and use no matter know-how’s crucial to construct in the direction of a imaginative and prescient. After spending a while in engineering, although, I wished to do extra. I wished to know how companies are run, how merchandise are offered and the way groups are constructed –– and I wished to study shortly.
I spent a couple of years working in administration consulting at BCG, the place I labored on all kinds of tasks in several industries. I realized a ton, however in the end missed constructing merchandise and dealing in the direction of a longer-term imaginative and prescient. I made a decision to hitch Astronomer’s product administration workforce, the place I might nonetheless work with prospects and construct methods (the issues I loved from consulting), however might additionally get very palms on constructing out the precise product and dealing with know-how.
For some time, I acted as a hybrid PM/engineer –– I’d work with prospects to know the challenges they had been dealing with and design merchandise and options as a PM. Then, I’d take the product necessities and work with the engineering workforce to truly construct out the product or characteristic. Over time, I did this with a bigger set of merchandise at Astronomer, which in the end led to the CTO position I’m now in.
For customers who’re unfamiliar with Airflow, are you able to clarify what makes it the perfect platform to programmatically writer, schedule and monitor workflows?
Apache Airflow is an open-source platform for creating, scheduling, and monitoring batch-oriented workflows. Airflow offers the workflow administration capabilities which can be integral to fashionable cloud-native knowledge platforms. It automates the execution of jobs, coordinates dependencies between duties, and offers organizations a central level of management for monitoring and managing workflows.
Information platform architects leverage Airflow to automate the motion and processing of information by means of and throughout numerous methods, managing advanced knowledge flows and offering versatile scheduling, monitoring, and alerting. All of those options are extraordinarily useful for contemporary knowledge groups, however what makes Airflow the perfect platform is that it’s an open-source undertaking –– which means there’s a group of Airflow customers and contributors who’re continuously working to additional develop the platform, resolve issues and share greatest practices.
Airflow additionally has many knowledge integrations with standard databases, functions, and instruments, in addition to dozens of cloud providers — and extra are added each month.
How does Astronomer use Airflow for inside processes?
We use Airflow a ton! Naturally, we now have our personal knowledge workforce that makes use of Airflow to ship knowledge to the enterprise and our prospects. They’ve some fairly subtle tooling they’ve constructed round Airflow that we’ve used as inspiration for characteristic growth on the broader platform.
We additionally use Airflow for some fairly untraditional use instances, nevertheless it performs very nicely. For instance, our CRE workforce makes use of Airflow to watch the lots of of Kubernetes clusters and hundreds of Airflow deployments we run on behalf of our prospects. Their pipelines run continuously to examine for points, and if we discover any, we’ll open proactive help tickets on behalf of our prospects.
I’ve even used Airflow for private use instances. My favourite (up to now) was after I was transferring to New York Metropolis. Should you’ve ever lived right here, you’ll know the rental market is loopy. Flats get rented out inside hours of them being listed. My roommates and I had a listing of standards all of us agreed upon (location, variety of bedrooms, bogs, and many others), and I constructed an Airflow DAG that ran each couple of minutes, pulled new listings from numerous condo itemizing websites, and texted me (thanks Twilio!) each time there was one thing new that matched our standards. The condo I’m now dwelling in was discovered because of Airflow!
Astronomer designed Astro, a contemporary knowledge orchestration platform, powered by Airflow. Are you able to share with us how this instrument allows firms to simply place Airflow on the core of their knowledge operations?
Astro allows organizations and extra particularly, knowledge engineers, knowledge scientists, and knowledge analysts, to construct, run, and develop their mission-critical knowledge pipelines on a single platform for all of their knowledge flows. It’s the solely managed Airflow service that gives excessive ranges of information safety and safety and helps firms scale their deployments and liberate assets to concentrate on their overarching enterprise objectives.
Certainly one of our prospects, Anastasia, a cutting-edge know-how firm, selected Astro to handle Airflow as a result of they didn’t have sufficient time or assets to take care of Airflow on their very own. Astro works on the again finish so groups can concentrate on core enterprise actions, somewhat than spending time on undifferentiated actions like managing Airflow.
One of many core parts of Astro is elastic scalability, might you outline what that is and why it’s necessary for cloud computing environments?
For us, this simply means our skill to satisfy the compute calls for of our prospects with out working a ton of infrastructure on a regular basis. Our prospects use our platform for all kinds of use instances, nearly all of which have excessive compute necessities (coaching machine studying fashions, processing huge knowledge, and many others). One of many core worth propositions of Astronomer is that, as a buyer, you don’t have to consider the machines working your pipelines. You deploy your pipelines to Astro, and may anticipate that they work. We’ve constructed a set of options and methods that assist scale our infrastructure to satisfy the altering calls for of our prospects, and it’s one thing we’re excited to maintain constructing upon sooner or later.
You had been liable for the Astronomer workforce constructing Ask-Astro, the LLM-powered chatbot for Apache Airflow. Are you able to share with us particulars on what’s Ask-Astro and the LLMs that energy it?
Our workforce at Astronomer has a few of the most educated Airflow group members and we wished to make it simpler to share their information. To try this, we created a reference implementation of Andreessen Horowitz’s Rising Architectures for LLM Purposes, which exhibits the commonest methods, instruments, and design patterns they’ve seen utilized by AI startups and complex tech firms. We began with some knowledgeable opinions about this reference implementation and Apache Airflow additionally performs a central position within the structure. Ask Astro is a real-life reference to indicate easy methods to glue all the varied items collectively.
Ask Astro is extra than simply one other chatbot. The Astronomer workforce selected to develop the appliance within the open and commonly submit about challenges, concepts, and options with the intention to develop institutional information on behalf of the group. What had been a few of the largest challenges that the workforce confronted?
The largest problem was the dearth of clear greatest practices locally. As a result of “state-of-the-art” was redefined each week, it was robust to know easy methods to method sure issues (doc ingestion, mannequin choice, output accuracy measurement, and many others). This was a key driver for us to construct Ask Astro within the open. We wished to ascertain a set of practices for LLM orchestration that work nicely for numerous use instances so our prospects and group might really feel well-prepared to undertake LLMs and generative AI applied sciences.
It’s confirmed to be an incredible alternative –– the instrument itself will get a ton of utilization, we’ve given a number of public talks on easy methods to construct LLM functions, and we’ve even began working with a choose group of shoppers to roll out inside variations of Ask Astro!
What’s your private imaginative and prescient for the way forward for Airflow and Astronomer?
I’m actually enthusiastic about the way forward for each Airflow and Astronomer. The Airflow group continues to develop and at Astronomer, we’re dedicated to fostering its growth, help and connection throughout groups and people.
With growing demand for data-driven insights and an inflow of information sources, knowledge engineers have a difficult job. We wish to lighten the load for these people and groups by empowering them to combine and handle advanced knowledge at scale. Right now, this additionally means supporting AI adoption and implementation. In 2023, like many different firms, we centered on how we are able to speed up AI use for our prospects. Our platform, Astro, accelerates AI deployment, streamlines ML growth, and offers the sturdy compute energy wanted for next-gen functions. AI will proceed to be a magnet for us this yr and we’ll help our prospects as new applied sciences and frameworks emerge.
As well as, Astronomer’s an incredible place to work and develop a profession. As the info panorama continues evolving, working right here will get increasingly thrilling. We’re constructing an incredible workforce right here and have numerous technical challenges to resolve. We additionally just lately moved our headquarters to New York Metropolis the place we are able to develop into an excellent larger a part of the tech group that exists there and we’ll be higher outfitted to draw the perfect, most expert expertise within the trade. Should you’re excited about becoming a member of the workforce to assist us ship the world’s knowledge on time, attain out!
Thanks for the nice interview, readers who want to study extra ought to go to Astronomer.
