Industrial Data Quality Podcast

Denis: Hi, my name is Denis, and welcome
to my new video, which will be a bit less

technical than my usual Databricks videos.

In this video, I really want to talk
a bit more about my career and more

specifically how I got into data
from a very unlikely, um, beginning

point, how I evolved my career over
the last six years, including having.

Both employed positions,
but also spending thirds.

So four years out of six working
as a freelancer and a contractor.

In fact, I am at the time, at
the time of making this video.

I've been self-employed for about two
years, and currently I see no reason

to go back to an employed position.

So let me tell you my
story of how I got there.

I graduated in 2000.

18, which was a while ago with a degree
in material science and materials

engineering with a minor in chemical
engineering from KU Leuven in Belgium.

I remember in my last year I did a, an
exchange here, which means I spent the

entire 12 months abroad in France at the
University of Chernobyl for five months.

Which landed me an internship position of
six months at the factory in the French

Alps that produced the liquid aluminum.

And this internship was hugely important
in what would eventually become my career

in data, because that's the first point.

I was really, really interacting
with data, and it all came

to pass because of a program.

Language you may be
familiar with called R.

We briefly had a couple
of courses at university.

Um, it was actually in economics where we
used R to analyze some economical data.

I didn't really think much of
it because I was never really

interested in programming.

I was always very interested in.

Physics, chemistry, and other sciences.

For me, programming in
Python was actually annoying.

I never really enjoyed that particular
course at uni, so I wasn't particularly

good at any, uh, at programming.

But at that internship, when I
started working, the main objective

I really did was analyze lots of data
coming from a SQL Server database.

And this really got me in touch
with the programming language,

well, I should say query language,
uh, SQL or SQL as we say in Europe.

And I spent, in essence the next six
months writing SQL queries, bumping that

data into r and r studio and making a
lots of, a lot of visual, nice graphs.

And my colleagues who hired
me at the internship saw that.

I was able to produce beautiful graphs in
a fraction of the time that they could.

Um, eventually I started to get
more and more work in data analysis.

I discovered that there was an entire
field that was coming up at that time.

It was data science, which I think was
in full swing around 2000, um, 18 and 19.

So I kind of wrote this height wave.

Until I completed some courses on Coursera
on machine learning, and that's basically

my first step into the data world.

So after that, I was
hired by the same company.

I did an internship at, the
company was called TRIMET.

They had their headquarters in
Essen, Germany, and they offered me

a job there as a process engineer.

I happily took this job because in
essence, um, the city of Essen was

only two hours away from my home
in Belgium, which mean that I could

prolong my, let's say European
travels by another year or two, right?

I would not have to go back
to Belgium to where I knew

everything and everyone already.

So that opportunity really excited me.

Um, my job itself as a process engineer
was more focused on process control.

I was developing a new control
system using, uh, MATLAB and

Simulink, which eventually got me
a bit away from the field of data.

I still did a lot of data analysis.

I discovered that Biden was actually
beating our, in terms of, um, importance.

Especially in the field of machine
learning, which is why I picked

up Python mostly on the sites.

Um, so on, on the job, I
was doing a lot of art.

I remember in the evenings I spent
a lot of time doing courses, Python,

um, seeing how all of that works.

Nevertheless, after about one year
and a half at this company in Sen.

I got a bit frustrated that most of
my day job was actually not revolving

around data or machine learning.

I really wanted to get deeper into
that field, which is why in 2020, right

before the start of Covid in March,
I remember that month quite well.

I found a new job.

It was at the data science consultancy
company called Stat Works, and they hired

me because they saw I had this intense
passion for data and machine learning.

They gave me a small case, which I managed
to solve quite well, and they were really

impressed by my, um, motivation that
showed itself in, for instance, competing

in Kaggle competitions and so on.

Kaggle was a website for.

Machine learning games
and competitions online.

I'm not sure if it's still very
popular, but at those days it was

like a way to prove your finesse and
your dexterity in machine learning.

So there I was hired as a, what they
call junior data science consultant.

And in essence, I would be
thought on various machine

learning projects and clients.

Unfortunately, because it was
covid, we all soon went into what

the Germans call could abide.

In essence, it means that
you're only employed at 50%,

so you only work half a week.

Nevertheless, that works was so nice to
pay us the full salary, which essentially

meant that I had half of the week
completed to myself, and I remember

really investing that time, getting
really deep into machine learning.

More specifically deep learning, which
was becoming very popular at that time,

especially for applications in computer
vision and natural language processing.

So as part of that journey, I in fact,
quickly discovered that do lots of

machine learning fast, but also cheap.

You need a workstation, and
that's how I spent the next

two months assembling hardware.

I fell into the rabbit hole of computer
building, which means choosing a good CPU,

choosing the correct graphical cart, which
was really the main component at that.

Uh, at that time, I purchased the 2080
TI by Nvidia, which was a very powerful

cart, also very expensive at that time,
and which allowed me to train some pretty

big intricate machine learning models
that I used for my gaggle competitions.

I remember particularly competing in the
Gaggle steel defect detection competition

and got there in the top 20%, which by
all means is not very impressive, but

it's something I was very proud of.

In essence, competing on Gaggle is
what eventually led me to switch from

R completely to Python and learning
the language completely by doing.

I never followed any courses on
Python, didn't really study any books.

But just by writing all of
this code and understanding

the machine learning frameworks
allowed me to master the language.

At the time.

I was also reached out by a former
colleague who worked with me at Premeds.

My first job was an engineer
in SEN, and he was still very

impressed by my ability in art.

And he knew that I was able to produce
nice graphs and applications quickly.

And given that he was a freelancer,
he worked at a, he had a

particular project for that same
company in a different division.

And he desperately needed a tool that
would allow him to visualize a lot

of signal coming from an industrial
controller, um, in one dashboard.

And.

He didn't really want to spend a
lot of time buying an expensive

tool or doing things himself.

So he reached out to me with an idea
that, Hey, de um, would you like to try

to build an app for me as a freelancer?

And I've always wanted to be
more independent of my employer,

be more entrepreneurial.

This first project really gave me the
opportunity to dip my toes into freelance.

I had additional time, I had already a
steady sort of income from my job, and

now I was able to earn, I think, 1.5

K Euros to build a nice
application in R, which was still

one of my favorite language.

So I took this opportunity, um, with
both hands built the application,

spent way too much time on it.

But it felt so satisfying delivering a
project from end start to end without

having someone looking over your shoulder.

No status meetings, no
agile meetings, dailies.

You just promised the work,
agree on the scope, and then

deliver it at the fixed price.

And this way of working appealed
to me so much that already just six

months into my new data consulting
job in Frankfurt, I quit the job

and decided to freelance full-time.

At that time, that was quite a bold
decision because as you remember,

COVID was still in full swing.

Companies were, uh, losing money.

Everyone was afraid.

Nevertheless, I built up, um, a nice nest
egg 'cause I always saved a lot of money.

Um, instead of spending it, and now
essentially I was completely unemployed, I

opened a freelancing business in Germany.

And was now looking for new clients to do.

Unfortunately, my colleague did not
need any additional work, so I was

in fact spending, I think it took
me at least two or three months

before I found my next contract.

My next contract was actually not
in R and not in machine learning.

It was in a Python framework
called Apache Airflow.

Now, Apache Airflow was a framework
that was putting a bit on the side.

During my first, during my second job
at Stats and also during the first month

of my freelancing when I still had my
clients, I was really fascinated by

the aspect of having a clear organized
way of building data pipelines,

essentially batch processing, scheduled
by a workflow orchestrator like Apache

Workflow, and by mid chance on some.

I think it's called a contractor
or contracting company.

I saw a project of a startup in
Cologne that needed someone who could

help them implement Apache airflow.

So I applied for that job, had I
think one or two interviews with

their team, and they liked me, and
so they hired me as freelancer, or

I should say more of a contractor.

And this was quite exciting
because this is what my first.

Real contracting gig with a company that
I couldn't get through via my own network.

This was really a cold, uh, contact.

I worked with them for
about six months, I think.

Delivered a project in a
startup, Apache Airflow.

I also worked quite a bit with
Apache Kafka at that time because

they were an IOT starter, but
really wanted to experiment with,

um, real time communication,
uh, via messaging frameworks.

So I did that for a while.

By now, we are reaching the end of 2020
and at the beginning of 2021, I actually

returned to Belgium and opened an LLC.

So I limited liability company because
my freelancing business was growing.

I realized that I want to do this further.

I want to contract more, and I
needed the company to do this to,

um, be able to secure bigger clients.

So I moved back to Belgium, continued
working for that client, um, for

that particular startup, but was
also looking for new additional work.

And the next project I actually
landed was for a very big company.

It was, uh, Johnson & Johnson in
Belgium, so a big pharma industry.

This was also, um, quite a big step for me
because I left the world of medium-sized

companies and small-sized companies.

I started working for enterprise clients
and I very soon became familiar with

the particular challenges of enterprise.

Being that you have, uh, very large
meetings, often not that much time coding,

you have a lot of different people, and
the scope of the work is so big that

you can essentially don't grasp it all.

So it's as if you're all working on.

Different parts of a big elephant.

Someone's working on the trunk,
someone is feeling the leg.

The only one is working on the
tail, but no one has the view on

the entire elephant very often.

This was a real contracting
position, which means that I got

there via a contracting company.

It was a data engineering company.

That found me.

I was looking for an additional
body or hands, hands on a

keyboard to put on this project.

So they were acting like an intermediary,
receiving a part of my daily rate.

Now, although the pay was good, the
framework that we were, that we were

working on was also very interesting.

It was called Kro.

Um, essentially a pi, uh, PySpark
framework for, uh, machine

learning on very big data.

I did that for a while, but very quickly
got frustrated by a, the many meetings

and the fact that I've lost the initial
freedom I gained by freelancing.

I suddenly had working hours, meetings,
scrum updates, so all the benefits of

being freelancer were no longer there.

I was essentially a full-time contractor.

The second thing I also didn't
really like was that I left.

The manufacturing world.

I was no longer in industry.

And coming from the aluminum sector, being
a materials engineer or metals engineer,

to be more specific, I really missed
the chemistry, the physics, the dust,

the heat from my previous factory jobs.

I wanted to work more
on industrial use cases.

So I did that contract for I
think about a year and a half.

Until I got a very interesting email.

We are now at the beginning of 2022.

In the middle, I was contracted
by, or I was contacted by a German

headhunting bureau that were
looking for a data engineer who

had some experience in aluminum.

At that time, a company called Novelis.

It was the.

I guess still is the world
leader in rolling and recycling.

They were looking for a data
engineer to work on their machine

learning projects in Germany.

And given that I then spoke German, I
had experience in manufacturing, even

in aluminum, and I had data engineering
experience from my last contracting

gig in Apache Airflow, which meant
I knew Spark, I knew Databricks.

I was hired.

As a data engineer, unfortunately
this meant that I had to

close my freelancing business.

At that point.

I was also ready to throw in the towel.

I think I got a bit impatient
and frustrated with the fact that

I wasn't really a freelancer.

I wasn't really free, and I had
this imposter syndrome where I felt

that you know what then is maybe you
need more experience in data before

you can actually be a freelancer.

So I closed my business in Belgium,
relocated to Germany, got a very

good, um, high, um, high paying job
as an employee for this company.

It was fully remote.

I was living in Berlin at that time,
while I did actually spend six months

at the factory in Copeland, which was
a deal with the company that I would

spend six months on site to learn a
business and then I could work remote

from anywhere based in Germany.

And I chose Berlin.

At that company I did
quite interesting projects.

I was part of a very smart team
of data scientists and I was there

talk to prepare the data pretty
much completely in Databricks.

So I wrote a lot of spark code,
did notebook orchestration created

of table medallion architecture.

Yeah, the bronze, silver,
and gold clean layers.

And generally passed on my knowledge
to a junior data engineer who

was also employed next to me.

And that was my first experience
really coaching junior people.

And I really enjoyed it.

Um, it was fun to not only consume
knowledge, but also be able

to pass it on to someone else.

I.

Even though I liked the job, I again,
very quickly noticed that I was

just not built to be an employee.

I really missed having the independence,
uh, that freelancing gave me.

I was also a bit frustrated by the
approach of the company when it came

to data ingestion of sensor data, which
was, um, this topic is, um, perhaps a bit

more too deep to explore in this video.

Let's say that particular challenge in
manufacturing, which actually forced me

to take my next step, which I will get
to a big challenge in manufacturing,

is that there's a very high volume
of time series data You have to know.

A factory is equipped
with many, many sensors.

All of these sensors produce data in
sometimes 20 millisecond frequency.

But let's assume even if it's one second
frequency, you're dealing with a lot of

time series generated every single moment.

And processing this data is not
as straightforward as processing

sale data or customer data that,
for example, Nike would be toast.

And at that time, I discovered.

Um, let's say the field of industrial iot.

So it's a bit similar to what I
did from my first startup gig when

I did a bunch of airflow work.

But there it was more focused on a
concept called the Unified Namespace.

And the unified namespace was a
concept developed by, uh, Walker

Walker Reynolds in the United States.

It was essentially an
event driven architecture.

Based on the MQTT protocol, it's a very
lightweight protocol, um, very commonly

used in IoT applications because allows
to transport data over insecure, um, or

unstable network in very large volumes.

It has a very small memory footprint,

but I got really upset by this technology
'cause I thought that, look, this

architecture is a solution to industrial.

Data infrastructure, in fact.

So what a lot of companies, including
Novelli was really struggling with, I felt

that the unified namespace was a solution.

So as an employee, I really tried
to push this concept in the company.

And yeah, of course I got
some interest, some traction.

I could do it as, um, let's
say as a side project.

But the whole strategy of
the company was set in stone.

Essentially what usually happens in a big
enterprise, some big consultants come in,

they develop a fancy 5-year-old, um, big
plan, and then the company executes that.

So I quickly realized that there was no
way the company would take a big risk

like that and explore this new technology.

So I felt that if I really wanted
to work on this thing called the

unified namespace, I would have to
go and work as a freelancer again.

Which meant that for, I think the
fourth time now, I quit my job.

I resigned, which troubled
the company very much.

They really tried to keep me there
'cause my work was generally good.

It was also very hard to hire
competent data engineers at that time.

Nevertheless, in July, 2023, I
resigned from this job and started

to work as a freelancer again.

Again, at this time I was in Berlin, so
in Germany, so I reopened a freelancing

business in Germany, and I even got
six months of funding because I managed

to, let's say, create a very coherent
business plan of serving the aluminum

industry in digital transformation.

That what would be my business venture.

I would, I would specialize.

On the aluminum industry and focus on data
integration using the unified namespace.

And I really hoped that my previous
background in aluminum or aluminum,

depending where you're from, and data will
allow me to build systems completely on my

own for this industry and now real life.

That's quite a naive venture,
even though it's quite niche,

it's a very conservative industry.

I tried to build a business out of
this for two years, so I focused

really hard on conferences.

I flew to two conferences in
aluminum industry, so once a

year, once it was in, um, Dubai.

I.

The second time was in Leo in France.

Shook a lot of hands,
talked to a lot of people.

I even flew to a conference in Istanbul
where I presented the unified native

space for the aluminum industry.

I wrote quite a lot of blog
posts and quite a lot of, um,

articles in industrial magazines.

I had a blog.

I had a daily, daily newsletter.

I also got in touch with a company called
the United Manufacturing Hub in Germany.

That was a company that open sourced
a unified namespace for the industry.

I worked with them briefly as a content
creator, but after two years, I just

noticed that you have to remember, my
background was always in data, machine

learning, data analysis, analytics,
data engineering, and I felt I was

drifting very far away into IIOT.

I was now dealing with integrating
data from controllers from industrial

SCADA systems, which stands for
Supervisory Control and Data Acquisition.

And I felt I was going too broad and
quite recently, actually, I would say

about a month ago, I secured my next gig,
which was for a big renewables company.

Again, a German company.

Um.

Oh, I have to mention in the fact
that I frankly also returned to

Belgium again, I, from Berlin.

That was in September, 2024 or last
year, I reopened the business in Belgium.

'cause I figured that I wanted to focus
my life in Belgium in the long run

that I would eventually return there.

I was also quite done
with, uh, with Germany and.

I found a very good contract through
this Belgian l lt, um, through a

large recruiting company called Hayes,
who offered me, who thought that I

had experience in data engineering,
especially in SCADA systems and

manufacturing, and they, someone that
could build a data quality framework

built on Databricks, and that's how
I actually got back into Databricks.

I saw that.

The old world, which I've left, namely
data and ventured into manufacturing,

suddenly showed itself back to me.

I was working once again with Spar in
Databricks and really enjoyed it, which

is why I now decided to focus my career
more on a specific platform instead

of a particular niche like aluminum or
manufacturing, or even unified namespace.

I felt this industry will need
a couple more years to really

implement this technology.

But for me, I wanted to work on
something that was already booming.

And the conference, the Databricks
Conference, which I'll be going to

in um, June in San Francisco has
60,000 attendees, which make me

realize that I think Databricks is
a technology that is going to stay.

It's a good product, it's a good product.

They make the whole data.

Let's say product lifecycle, very easy.

'cause they do the data management,
the processing, the model serving.

Databricks can do it all eventually,
which is why I decided to focus

my career, let's say, on my
freelancing business on Databricks.

I'm continually, I'm
continuously working on various.

Subfield such as data integration
into Databricks from SCADA systems,

data quality, bis, spark, and
processing data visualization.

So all the various experience that
I've built over the last five years,

I feel that they come nicely together
in this new focus, um, Databricks,

which I'm currently exploring.

So that's where I'm currently at.

Um, let's look at what the future
holds and what I learned from the past.

First of all, I thought that having
a degree in data was, well, it

was, first of all, impossible.

At that time, 2018, there weren't really
any data science university roles.

Sure there were some bootcamps.

I never did those.

Um, though I felt that you can get
quite far by still learning by yourself.

But whether that's still
through today, I don't know.

But it worked for me at that time.

I also realized that it's fine to look
for, um, for yourself and what you

want to do in the first, let's say,
five or even 10 years of your career.

I mean, I went from growth engineering,
from MATLAB and Simulink to SQL queries,

to Art Python, to playing with machine
learning on Gaggle to going more

into data engineering and bis, spark,
Databricks, going to manufacturing

into MQTT and Unified namespace.

And now back into Databricks.

And I felt that all the things
I've learned were all very

relevant, um, to my work now.

So I think at some point it's just about
sticking with something long enough,

even though I've jumped ships very long,
I've never had a job for two years.

I've always quit before Denmark
was never promoted, but still I did

really well because in the long term,
if you look at my career, data has

been the central point of it all.

I've approached it from a different
direction, sometimes more from analytics,

sometimes more from engineering.

But I've always worked a lot with data and
I've always focused on the fundamentals.

Things like SQL writing with
Python code, um, but also exploring

pattern performing frameworks.

So what do I hope you get from this video?

I think for this video it was
more, um, maybe even for me

just to say that it's fine.

To look for your way.

I think it's also fine to just follow your
intuition and as long as you're putting

the work and learn, you will be fine.

Even now, the thousand 25, when people say
that the market is crashing and tech jobs

are decreasing, I haven't really feel,
uh, felt, uh, lack of potential work.

Sure.

I've had like months before where
I was looking in between projects.

For work, but I've always felt that
if you know what you're doing and you

can create value for a company, you'll
never be out of, uh, out of business.

Maybe a final tip I would give you as a
viewer is that even now, even though I'm

back focused on Databricks, I'm always
very focused on one particular industry.

Like I try to avoid working
for, let's say Nike or Facebook

or anything, anything else.

Given that I've worked in manufacturing
for five years and industry in general,

I'm still very focused on serving
either manufacturing or utilities

or chemical companies, essentially
anything within the industry.

This really allows me to set me apart
from other freelancers and eventually

build experience in a particular niche.

So I think that's it for this video.

Uh, thank you for hearing
me ramble about my career.

I hope it was useful for you and for me.

It was nice to revisit the last six
years of my career and I'm quite

excited about what's to come on
the future and anything interesting

I will share with you on my chat.

Thank you and bye-bye.