You might have seen multiple posts around this subject as time keeps evolving and bringing changes into tech stack, however this includes recent discovery in data processing frameworks, visualization tools, ETL tools, Development notebooks, Data catalog..etc Over the time, we might have come across different terms like ETL, ELT, Reverse…

Throughout our data engineering journey, we’ve come across a myriad of database management systems (DBMS). But what sets DuckDB apart from the rest? And is it worth delving into? Let’s embark on a quest for answers. What’s DuckDB? The original purpose behind DuckDB’s creation was to empower analytical query workloads and facilitate…

In the realm of distributed database systems, the adoption of compaction strategies plays a pivotal role in the effective management of data storage. As the data landscape continues to evolve, a multitude of innovative compaction strategies have emerged, each catering to specific database technologies and their unique demands. …

Recently, few people might have heard about Apache Paimon. Undergoing incubation at the Apache Software Foundation (ASF), Apache Paimon is being sponsored by the Apache Incubator. Apache Paimon, with its key features built around datalake storage with ACID characteristics and support for DML operations has joined the space with other…

Introduction Ever since the introduction of Hadoop, computing power and storage are seen as two different entities (unlike OLTP databases where computing and storage are tightly coupled). For example, Map Reduce, Spark, Flink, Storm are frameworks to process big data in distributed environment, whereas HDFS, Cloud storage (S3, Azure Blob, Google…

Windows plays a major role and also defined as a core part in processing infinite streams. Windows split the incoming stream into buckets of finite size, over which required transformations can be applied. Though there are many components in windows that includes trigger, evictor, allowedLateness, sideOutputLateData…etc, this post focusses on…

Let’s begin with some issues faced in data engineering projects, followed by usage of Delta Lake clones and let’s take a final step by resolving issues. What’s clone in Delta Lake? It’s just a replica of a source table at a given point in time. In other database terminology, we…

Consider a common scenario of data engineering pipeline where raw data needs to be cleansed, transformed or aggregated before writing to a target system. For this case, usually we create 3-4 tables to store raw data, cleansed data, transformed data and aggregated data respectively. In order to implement, this needs…

So, what’s Bloom filter? Bloom filter index is a space-efficient probabilistic data structure that is used to test whether an element is a member of set. It skips the values of chosen columns, particularly for fields containing arbitrary text. A Bloom filter can tell you if a key is in a set and with…