By | Ilya Moshkov | www.kdnuggets.com
I switched to data engineering from development in 2017. Previously I had worked for ten years in desktop development, backend (mainly Java), and a little bit of frontend. Despite my strong IT experience, it wasn’t easy at first to figure out what data engineers do, how they differ from database administrators, how they’re connected to data analytics, and what they have to do with Big Data.
It was the magic of the phrase “Big Data” that determined what I’m doing now (this link provides a well-established definition of Big Data with the triad “Volume, Variety, Velocity” + an informative video from Amazon AWS). For me, the Big Data field looked like a challenge that I had to accept.
I became more interested in distributed systems, scalable technologies, and clouds, and went to conferences where relevant Big Data products like Hadoop, Kafka, Spark, etc. were analyzed. Colleagues who worked with Big Data appeared around me, and I flooded them with questions. Sometimes the answers I received weren’t entirely clear, which further fueled my curiosity.