🌸 💗 🌷

Hi Simran 🌸

Data Engineering, Modules 1 to 3. Easy notes, solved papers, mock tests and numericals you can practise.

0 / 0topics done

0answers written

0flashcards known

📝 How the paper looks

Question	What you do	Marks
Q.1	Six short questions (a to f). Answer any five.	5 × 2 = 10
Q.2	Answer (a) or (b). Long answer with a diagram.	5
Q.3	Answer (a) or (b). Long answer or case study.	5
Total	1 hour	20

Weightage: Module 1 is 20%, Module 2 is 40%, Module 3 is 40%.

🔎 Teacher's hints, decoded

Module 1: all theory. Q.2(b) likely comes from here. DE lifecycle diagram, DE vs DS vs DA.
Pandas and NumPy: the uses are asked.
Data scraping: Q.3(a) likely. Types, tools and why it is done. Open
NoSQL: "meaning" questions, Cypher numericals, drawing a graph from a description. Cypher
Lakehouse: features, a case study, schema enforcement and time travel on given data, warehouse vs lake vs lakehouse. Open
Flink: data streams, windowing, transformations. Open
Numericals from last year: Bloom filter, FM, HDFS blocks, Dijkstra. Practise

🗓️ Study plan (about 4 hours)

25 minModule 1: lifecycle diagram, DE vs DS vs DA, challenges, Pandas and NumPy uses.
40 minModule 2 part 1: scraping, NoSQL types, Neo4j and Cypher, Dijkstra.
30 minModule 2 part 2: S3, ADLS, warehouse vs lake vs lakehouse, schema enforcement, time travel.
40 minModule 3: real-time systems, DSMS, Bloom filter, FM, Flink, windows.
20 minNumericals: solve each tool's example by hand, then check.
60 minMock Paper 1 with the timer on. Mark yourself honestly.
15 minFlashcards the night before.

Tap a number to tick a step off.

Module 1 · Foundations of Data Engineering

What a data engineer does, why the job matters, the life of data from birth to report, and the two Python libraries (Pandas and NumPy) used every day. Almost everything here is theory, so learn the points and the two diagrams.

What is Data Engineering? 🌸

very likely2 marks

In simple words

Data is created everywhere: apps, websites, sensors, payments. Most of it is raw and messy. Data Engineering is the work of building the pipes that collect this data, clean it, store it and deliver it to the people who need it (analysts, data scientists, managers).

A data engineer sits in the middle:

Data ProducersData EngineersData Consumers

Producers are the apps and systems that create data. Consumers are the analysts, ML models and dashboards that use it. The engineer connects the two.

Think of it like

A city water system. Rain and rivers are the raw data. The data engineer builds the pipes, the filter plant and the tanks, so clean water reaches every tap. Nobody thanks the pipes, but without them nothing works.

Write this in the exam

Data Engineering is the development, implementation and maintenance of systems and processes that take in raw data and produce high-quality, consistent information that supports downstream use cases such as analysis and machine learning.
It focuses on building reliable data pipelines that collect, store, move and transform data.
It sits at the intersection of security, data management, DataOps, data architecture, orchestration and software engineering.
It connects data producers (source systems) to data consumers (analysts, data scientists, ML models).
It is the foundation on which Data Science and Data Analytics are built.

Keywords to drop

raw data, pipelines, high-quality consistent information, downstream use cases, producers and consumers, ETL

Role and responsibilities of a Data Engineer

very likely2 marks5 marks

In simple words

The data engineer is the person who makes sure the right data reaches the right place, on time, in a clean and trustworthy shape. They build the roads; analysts and data scientists drive on them.

Think of it like

Mumbai's dabbawalas. They collect lunch boxes from thousands of homes (sources), sort them at stations (processing), and deliver each box to the right office on time (serving). They do not cook the food and they do not eat it. Their job is reliable delivery.

Write this in the exam

Build and maintain data pipelines that move data from source systems to storage systems.
Design databases, data warehouses and data lakes so data is organised and easy to query.
Ensure data quality by cleaning, validating and transforming raw data through ETL (Extract, Transform, Load).
Integrate data from many sources and formats (structured, semi-structured, unstructured).
Manage data security, privacy and governance (access control, encryption, compliance).
Optimise performance and cost of storage and processing; monitor and fix failing pipelines.
Serve data to analysts, data scientists, BI dashboards and ML models, and work closely with these teams.

Exam tip

For 2 marks write 4 points. If asked "Roles and responsibilities" for 5 marks, write all 7 points and add the Producers → Engineers → Consumers flow as a small diagram.

Keywords to drop

pipelines, ETL, data quality, data warehouse, governance, serving

Importance of Data Engineering

2 marks

In simple words

Without data engineering, every analyst would spend most of their day fixing messy spreadsheets, and apps like UPI or Swiggy tracking could not show live updates. Good data engineering means fast, clean, trustworthy data.

Think of it like

A restaurant kitchen's prep team. They wash, cut and arrange everything before the chef starts. The chef (data scientist) cooks faster and the food (decisions) comes out right.

Write this in the exam

It lets huge volumes of data (Big Data) be handled efficiently without the system slowing down or crashing.
It improves data quality, so decisions taken from the data are accurate and reliable.
It saves time for data scientists and analysts, who no longer clean messy data themselves.
It enables real-time data flow, needed for UPI transactions, fraud detection and live food-delivery tracking.
It creates a single source of truth by integrating data from many systems.

Keywords to drop

Big Data, data quality, real-time, single source of truth, scalability

Challenges in Data Engineering

very likely2 marks5 marks

In simple words

Data comes from too many places, in too many shapes, too fast. It is often dirty, it must be kept safe, tools keep changing, and every extra machine costs money. These are the headaches a data engineer deals with daily.

Think of it like

Organising a big Indian wedding. Guests arrive from many cities (many sources), some RSVPs are wrong or duplicated (data quality), the guest list must stay private (security), the budget is fixed (cost), and the caterer changes every year (evolving tech).

Write this in the exam

Complexity of data sources: data comes from many sources in different formats (structured, semi-structured, unstructured) and must be integrated.
Volume and velocity: very large amounts of data arrive at high speed (Big Data).
Data quality: duplicate, missing or incorrect records must be removed or fixed.
Data security, privacy and governance: sensitive data must be protected while it is stored and transferred, and must follow laws.
Evolving technologies and hype cycles: tools change fast, so choosing the right stack is hard.
Performance optimisation and cost management: fast queries and pipelines without a huge cloud bill.
Data access and silos: data is locked inside separate departments and systems.
Compute capacity and query performance: enough processing power so queries stay fast as data grows.

Exam tip

For 2 marks, pick 4: sources, quality, security, volume. For 5 marks, write all 8 with one line each.

Keywords to drop

heterogeneous sources, data quality, governance, silos, scalability, cost

Data Engineering vs Data Science vs Data Analytics

very likely2 marks5 marks

In simple words

Three friends working on the same data. The engineer gets the data ready. The analyst looks back and says what happened. The scientist looks forward and predicts what will happen.

Think of it like

A cricket team's support staff. The groundsman prepares the pitch (engineer). The stats person tells you last season's averages (analyst). The strategist predicts how a bowler will perform tomorrow (scientist).

Write this in the exam

Basis	Data Engineering	Data Science	Data Analytics
Meaning	Builds systems and pipelines to collect and prepare data	Uses data to build predictive models	Studies past data to find trends and insights
Main goal	Make data available, clean and reliable	Predict future outcomes using Machine Learning	Explain what has already happened
Question answered	"Is the data ready and correct?"	"What will happen next?"	"What happened and why?"
Tools used	SQL, Apache Spark, Hadoop, Airflow, Kafka	Python, R, TensorFlow, Scikit-learn	Excel, Power BI, Tableau, SQL
Skills	Programming, databases, distributed systems	Statistics, ML, mathematics	Statistics, business understanding, visualisation
Output	Clean, structured data pipelines	Models and predictions	Reports, dashboards and charts

Bonus: Data Engineer vs Software Engineer

Both write code. A software engineer builds the app you use. A data engineer builds the plumbing that moves the data that app creates.

Basis	Data Engineer	Software Engineer
Builds	Data pipelines, warehouses, data lakes	Applications, websites, APIs
Focus	Flow, quality and storage of data	Features and user experience
Users	Analysts, data scientists, ML models	End customers
Tools	SQL, Spark, Kafka, Airflow	Java, JavaScript, frameworks, Git

Exam tip

"Differentiate" questions always go in a table. 2 marks = 4 rows. Always include Meaning, Goal, Tools and Output.

Keywords to drop

pipelines (DE), prediction / ML (DS), past trends / dashboards (DA)

Data Science Hierarchy of Needs

diagram2 marks

In simple words

Everyone wants the fancy top (AI, deep learning). But you cannot build the top floor of a building without the ground floor. This pyramid shows the order: first collect data, then move and store it, then clean it, then count and label it, then learn from it, and only then do AI.

The bottom two layers are the data engineer's job. That is why data engineering is called the foundation of data science.

Deep pink = the data engineer's layers. Each layer needs the one below it.

Write this in the exam

The Data Science Hierarchy of Needs is a pyramid showing that advanced data work depends on strong basic data work.
Level 1, Collect: instrumentation, logging, sensors, external data, user-generated content.
Level 2, Move/Store: reliable data flow, infrastructure, pipelines, ETL, structured and unstructured storage.
Level 3, Explore/Transform: cleaning, anomaly detection, preparation.
Level 4, Aggregate/Label: analytics, metrics, segments, aggregates, features, training data.
Level 5, Learn/Optimize: A/B testing, experimentation, simple ML algorithms.
Level 6, AI: deep learning.
Data engineers own the bottom layers (collect, move/store), which is why data engineering is the foundation of data science.

Keywords to drop

foundation, collect, move/store, pipelines, ETL, AI at the top

Data Lifecycle 🌸

very likelydiagram2 marks5 marks

In simple words

Data has a life, like a person. It is born (generated), gathered, kept somewhere, cleaned, looked after, studied, shown to people as charts, and finally deleted when nobody needs it. Then new data is born and the cycle repeats.

Think of it like

A Swiggy order. You place the order (generation), the app records it (collection), it is saved in a database (storage), duplicates and typos are fixed (processing), it is secured and backed up (management), Swiggy studies which dishes sell most (analysis), managers see it on a dashboard (visualisation), and after some years old order data is deleted (destruction).

The 8-stage data lifecycle from class notes. Arrows go clockwise.

Write this in the exam (8 stages, from class)

Generation: data is created by users, apps, sensors or transactions (a customer places an order).
Collection: data is gathered from the sources into the system.
Storage: data is saved in databases, data warehouses or data lakes.
Processing: raw data is cleaned, validated and transformed into a usable format.
Management: data is organised, secured, backed up and governed (access control, metadata).
Analysis: processed data is examined to find patterns, trends and insights.
Interpretation and Visualisation: insights are shown as charts and dashboards and turned into decisions.
Destruction: data that is no longer needed is securely deleted as per retention policy and privacy laws.

The 5-stage version (from the Q&A notes)

Some answers use a shorter 5-stage lifecycle. Know both. Each stage with its example:

IngestionStorageProcessingAnalysisVisualization

Stage	What happens	Example
Ingestion	Raw data is collected from sensors, apps, databases, websites. Batch (fixed intervals) or real-time/streaming (continuously).	A food delivery app collecting every order as soon as it is placed.
Storage	Ingested data is kept safely for later use. Structured data in relational databases, unstructured data in data lakes.	An e-commerce company storing orders in MySQL or Amazon S3.
Processing	Raw, messy data is cleaned: remove errors and duplicates, fill missing values, often with ETL tools.	Removing duplicate customers and fixing phone number formats.
Analysis	Processed data is examined for patterns and insights using statistics or ML.	Finding which product category sold most last month.
Visualization	Insights are shown as graphs, charts and dashboards for non-technical people.	Monthly sales as a bar chart on Power BI or Tableau.

Exam tip

Data Lifecycle = the life of the data itself (born to deleted). Data Engineering Lifecycle (next topic) = the engineer's work stages from source to serving. If the question says "Data Engineering Lifecycle", draw the next diagram, not this circle.

Keywords to drop

generation, ingestion, batch vs streaming, processing, governance, retention, destruction

Data Engineering Lifecycle 💗

very likelydiagram5 marks

In simple words

This is the most important question of Module 1. It came in the previous paper as a 5-mark question ("With a neat diagram, explain the elements of the Data Engineering Life cycle") and the teacher marked this area again.

The idea: data is generated in some source system. The engineer ingests it (brings it in), transforms it (cleans and shapes it), and serves it (hands it to users). Storage runs underneath all three. Below everything run the undercurrents: six things the engineer must take care of at every stage.

Think of it like

A restaurant. Farmers grow vegetables (generation). Trucks bring them to the kitchen (ingestion). Chefs wash, chop and cook (transformation). Waiters serve plates to customers (serving). The fridge and store room are used at every step (storage). Hygiene, the manager, the head chef's rules and the kitchen layout matter all the time (undercurrents).

Draw exactly this in the exam: Generation on the left, the three stages over a Storage bar, three outputs on the right, undercurrents at the bottom.

Each stage, explained

Stage 1, Generation (source systems and data origin): where data is born, for example an app database, IoT sensors, website clicks. The engineer does not control it but must understand it.
- Schema evolution: managing structural changes, for example the app team adds a new column.
- Generation velocity: the rate and volume at which data is generated.
- Team communication: keeping a strong connection with the application team so changes do not break pipelines.
Stage 2, Storage (the foundation): where data lives at every step. Chosen on the basis of access frequency, latency demands and cost.
- Hot data: used very often, kept on fast and costly storage (today's orders).
- Lukewarm data: used sometimes (last month's orders).
- Cold data: rarely used, kept on cheap archive storage (orders from 2019).
Stage 3, Ingestion: the process of moving data from the source system (generation) to the storage platform.
- Batch vs Streaming: batch collects data at fixed times (every night); streaming moves each record the moment it is created.
- Push vs Pull: in push, the source sends data to the pipeline; in pull, the pipeline goes and fetches data from the source.
- Managing bottlenecks: ingestion is often the slowest point, so it must handle spikes without breaking.
Stage 4, Transformation: turning raw data into useful data.
- Data structuring and cleaning: fix formats, remove duplicates, fill missing values.
- Data quality checks and validation: reject wrong records.
- Data modelling: design tables and relationships.
- Business logic application: apply company rules, for example "revenue = price × quantity − discount".
Stage 5, Serving: delivering data as per the requirement of each client.
- Analytics and BI: reports and dashboards.
- Machine Learning: features and training data for models.
- Reverse ETL: sending processed data back into business tools.

What is Reverse ETL?

Normal ETL takes data from apps into the warehouse. Reverse ETL does the opposite: it takes cleaned, useful results from the warehouse and pushes them back into the tools that teams use every day (CRM, marketing, support apps).

Example: the warehouse finds 5,000 customers who have not ordered in 30 days. Reverse ETL sends this list into the notification tool, which sends each of them a coupon.

The six undercurrents, one line each

Security: protect data with access control, encryption and least privilege.
Data Management: governance, data quality, metadata, lineage and privacy rules.
DataOps: DevOps ideas for data: automation, monitoring and quick recovery of pipelines.
Data Architecture: the overall design of systems so they meet business needs and can grow.
Orchestration: scheduling and coordinating jobs in the right order (for example Apache Airflow).
Software Engineering: writing clean, tested, version-controlled code for pipelines.

Write this in the exam (5 marks)

The Data Engineering Lifecycle describes the stages that turn raw data into useful data for analysts, data scientists and applications. (Draw the diagram above.)
Generation: source systems where data originates (app databases, IoT devices, logs). Key concerns: schema evolution, generation velocity, communication with the application team.
Storage: underlies every stage. Chosen by access frequency, latency and cost (hot, lukewarm and cold data).
Ingestion: moving data from source systems to storage. Batch or streaming, push or pull; bottlenecks must be managed.
Transformation: cleaning and structuring, data quality checks, data modelling and applying business logic.
Serving: delivering data for analytics and BI, machine learning and reverse ETL.
Undercurrents: security, data management, DataOps, data architecture, orchestration and software engineering support every stage.

Worked example: Swiggy orders

Stage	What happens at Swiggy
Generation	A customer places an order in the app; the order is saved in the app's database. Delivery partner phones send GPS points.
Ingestion	GPS points are streamed in real time; order records are also copied in batches every hour.
Storage	Raw data lands in a data lake (for example Amazon S3); today's orders are hot data, last year's orders are cold data.
Transformation	Duplicate orders are removed, cancelled orders are flagged, city names are standardised, revenue per restaurant is calculated.
Serving	Managers see a Power BI dashboard (analytics), the ETA model uses clean data (ML), and inactive customers get coupons (reverse ETL).
Undercurrents	Customer phone numbers are encrypted (security) and Airflow runs the jobs in order every night (orchestration).

Exam tip

Draw the diagram first, neatly, with a heading. Then write one short paragraph per stage. The worked example is optional, but adding 2 lines of it gets full marks easily.

Keywords to drop

generation, ingestion, transformation, serving, storage, undercurrents, reverse ETL, batch vs streaming, hot/cold data

ETL (Extract, Transform, Load)

2 marks

In simple words

ETL is the three-step recipe used everywhere in data engineering:

ExtractTransformLoad

Extract: take data out of the source (an app database, a CSV file, an API).
Transform: clean it and shape it (remove duplicates, fix formats, join tables, calculate totals).
Load: put the final clean data into the destination, usually a data warehouse.

Think of it like

Making chai. Extract: get tea leaves, milk and sugar from the shop. Transform: boil, mix and strain. Load: pour into cups for everyone.

Write this in the exam

ETL stands for Extract, Transform, Load: the process of moving data from source systems into a target system such as a data warehouse.
Extract reads data from sources like databases, files and APIs.
Transform cleans, validates, joins and aggregates the data and applies business rules.
Load writes the transformed data into the warehouse for reporting and analysis.
ELT (Extract, Load, Transform) loads raw data first into a data lake or cloud warehouse and transforms it later inside that system; common in modern cloud and lakehouse setups.

Keywords to drop

Extract, Transform, Load, data warehouse, ELT

Role of Python in Data Engineering

2 marks5 marks

In simple words

Python is the most common language data engineers use. It is easy to read and it has a ready-made library for almost every job: reading files, cleaning tables, scraping websites, talking to databases and cloud storage, and running big jobs on Spark.

Think of it like

A Swiss army knife. One tool, many blades: a blade for cleaning data (Pandas), one for maths (NumPy), one for websites (BeautifulSoup), one for big data (PySpark).

Write this in the exam

Simple, readable syntax makes pipelines quick to write and maintain.
Rich libraries: Pandas and NumPy for data manipulation, Requests and BeautifulSoup for scraping, SQLAlchemy for databases.
ETL scripting: Python scripts extract data from sources, transform it and load it into warehouses.
Big Data support: PySpark lets Python code run on Apache Spark clusters for huge datasets.
Orchestration and automation: Apache Airflow pipelines (DAGs) are written in Python.
Cloud integration: SDKs such as boto3 connect to Amazon S3 and other cloud services.
Visualisation: Matplotlib, Seaborn and Plotly for quick charts and data checks.

Keywords to drop

Pandas, NumPy, PySpark, Airflow, ETL scripts, libraries

Pandas: fundamentals and uses 🌸

likely2 marks

In simple words

Pandas is a Python library that gives you Excel-like tables inside code. You can load a CSV, remove bad rows, sort, filter, group and join tables with one line each.

Series: one single column of data (like one column in Excel).
DataFrame: a full table with rows and columns (many Series side by side).

Think of it like

Excel, but controlled by code, so the same cleaning steps can run automatically on a million rows every night.

import pandas as pd

s = pd.Series([10, 20, 30])            # a Series: one column

df = pd.read_csv("orders.csv")         # read a file into a DataFrame
df = df.dropna()                       # drop rows with missing values
df = df.drop_duplicates()              # remove repeated rows
sales = df.groupby("city")["amount"].sum()   # total sales per city
print(sales)

Write this in the exam: fundamentals

Pandas is a Python library for working with structured/tabular data, like rows and columns in Excel or a database table.
It has two main data structures: Series (a single labelled column, 1-D) and DataFrame (a table with rows and columns, a collection of Series).
It reads data directly from CSV, Excel, JSON and SQL into a DataFrame.
It has built-in functions for handling missing values, filtering, grouping and merging tables.

Write this in the exam: uses in Data Engineering

Ingestion: reading and writing data in different formats (CSV, Excel, JSON, SQL).
Cleaning: removing null and duplicate values and correcting data types.
Transformation: filter, sort, group-by and merge/join, the core of ETL.
Exploration: quick exploratory analysis before data goes to visualisation or ML tools.

Keywords to drop

Series, DataFrame, read_csv, dropna, groupby, merge, tabular data

NumPy: fundamentals and uses

likely2 marks

In simple words

NumPy (Numerical Python) is a Python library for fast maths on large lists of numbers. Its main object is the ndarray (n-dimensional array): a grid of numbers that can be 1-D (a list), 2-D (a matrix) or more.

Its superpower is vectorised operations: one line like prices * 1.18 applies GST to every price at once, with no loop, and much faster than a normal Python list.

Think of it like

A calculator that works on a whole column at once instead of one number at a time.

import numpy as np

prices = np.array([100, 250, 80, 400])
with_gst = prices * 1.18               # vectorised: no loop needed
print(with_gst.mean(), with_gst.max())  # statistics in one line

m = np.array([[1, 2], [3, 4]])          # a 2-D array (matrix)
print(m.shape)                          # (2, 2)

Write this in the exam: fundamentals

NumPy (Numerical Python) is a Python library for fast, efficient numerical and mathematical computation.
Its core structure is the ndarray (n-dimensional array), which stores large numeric data and is much faster than a Python list.
It supports vectorised operations: maths is applied to a whole array at once without loops.
It provides functions for linear algebra, statistics (mean, median, standard deviation) and random number generation.

Write this in the exam: uses in Data Engineering

Fast mathematical and statistical calculations on large numeric datasets.
Acts as the base on which Pandas and most ML libraries are built.
Handles multi-dimensional data such as matrices during processing and transformation.
Speeds up processing because NumPy operations run much faster than plain Python loops.

Basis	Pandas	NumPy
Main structure	Series and DataFrame	ndarray
Type of data	Tabular, labelled, mixed types (text, numbers, dates)	Numeric, same type in one array
Indexing	By labels (column names, index labels)	By integer positions
Best for	Cleaning and transforming tables (ETL)	Fast mathematical computation
Relation	Built on top of NumPy	Base library

Keywords to drop

ndarray, vectorised operations, linear algebra, statistics, faster than lists

Data Visualization: why it matters and the tools

2 marks5 marks

In simple words

A table with 10,000 numbers means nothing to a manager. One bar chart of the same numbers makes the story clear in five seconds. Visualisation turns data into pictures so people can understand it and decide quickly.

For a data engineer it is also a health check: a sudden drop in a "rows loaded per day" chart shows a broken pipeline immediately.

Think of it like

A cricket scoreboard with the worm graph. You do not read every ball; the graph shows who is winning at a glance.

Write this in the exam: importance in DE

It presents insights as graphs, charts and dashboards that are easy to understand.
It helps non-technical people (managers) grasp trends without reading raw numbers.
It exposes data quality problems quickly: outliers, missing values, sudden spikes or drops.
It helps monitor pipelines (data volume, failures, delays) through dashboards.
It reveals patterns and trends that support faster decision-making.

Write this in the exam: important tools

Matplotlib: the basic Python plotting library (line, bar, scatter, histogram).
Seaborn: built on Matplotlib; attractive statistical charts such as heatmaps and box plots.
Plotly: interactive web charts with zoom and hover.
Power BI: Microsoft's business intelligence tool for drag-and-drop dashboards.
Tableau: interactive dashboards for business users, connects to many data sources.

Fundamentals: pick the right chart

You want to show	Use	Example
Comparison between categories	Bar chart	Sales per city
Change over time	Line chart	Daily orders this month
Share of a whole	Pie chart	Payment modes: UPI vs card vs cash
Relation between two numbers	Scatter plot	Delivery distance vs delivery time
Spread of values	Histogram	Order values between ₹0 and ₹2,000

Keywords to drop

dashboards, trends, outliers, Matplotlib, Seaborn, Power BI, Tableau

Module 2 · Data Storage & Retrieval

How data is collected from the internet (scraping), where it is kept (object storage, NoSQL, graph databases, lakes and lakehouses) and how it is fetched back quickly. This is the biggest module and carries 40% of the paper.

Why Data Storage & Retrieval Matters

in her question list5 to 10 markscase study

In simple words

Every step of a data pipeline either writes data somewhere or reads it back. If the storage is slow, the whole pipeline is slow. If it is expensive, the company bleeds money. If it cannot grow, the system breaks on a big sale day. So choosing the right storage, and the right way to read from it, decides how fast, cheap and reliable the whole system is.

Think of it like

A kirana shop. Fast-selling items (Maggi, milk) sit on the front shelf, slow items go to the back godown, and old stock goes to the storeroom upstairs. The shopkeeper knows exactly where each thing is, so he can hand it over in seconds. Good storage and retrieval is that same arrangement, for data.

Write this in the exam

Storage and retrieval sit at the centre of the Data Engineering lifecycle: ingestion writes into storage, transformation reads and rewrites it, and serving reads it for analytics and ML.

Performance: the right storage (SSD, cache, columnar formats, indexing) gives low latency, so queries and dashboards respond quickly.
Pipeline reliability: every stage depends on reading the previous stage's output. Durable, consistent storage keeps the pipeline from breaking.
Cost: hot data on fast storage, cold data on cheap archive storage (tiering, lifecycle policies, compression) keeps the bill low.
Scalability: cloud object storage and distributed systems grow from GBs to petabytes without redesign.
Data quality and governance: schemas, versioning and access control on storage keep data correct and secure.
Supports every consumer: BI reports, ML training and real-time apps all retrieve from the same stored data.

Case study: an e-commerce company (Flipkart-style sale)

Ingestion: millions of orders, clicks and payment events arrive every minute during a sale.
Storage choice: raw clickstream and logs go to object storage (S3) in a data lake; orders go to an RDBMS; the shopping cart and sessions go to a key-value store like Redis for millisecond reads.
Retrieval for serving: the product page reads stock from cache (fast), the analytics team reads cleaned sales tables from a lakehouse (Delta Lake) for daily reports, and the ML team reads clickstream history to train the recommendation model.
Cost control: last year's logs move automatically to S3 Glacier through a lifecycle policy.
Result: fast checkout, correct reports, lower bill and a system that survives the sale-day load. A wrong storage choice (say, logs in the orders database) would slow down checkout for every customer.

Keywords to drop

performance, latency, scalability, cost, tiering, durability, pipeline, serving

What is Data Scraping and Why Do It? 🌸

teacher hint: Q3 a)2 marks

In simple words

Data scraping means writing a small program (a "scraper" or "bot") that visits a website, PDF or file and copies out the data you need, automatically. A human copy-pasting 10,000 product prices would take weeks. A scraper does it in minutes.

Think of it like

Sending a helper to the market with a notebook: "write down the price of every phone in every shop". Scraping is that helper, but a program, and it never gets tired.

Write this in the exam

Data scraping is the automated process of extracting data from websites, documents or other digital sources using scripts or tools.
Sources: websites (HTML pages), PDFs, databases, APIs and files (CSV, Excel, JSON).
Why it is done: organisations need large amounts of external data (prices, reviews, social media posts) that is not available in their own systems.
It saves manual effort and works at scale; a bot collects data quickly and repeatedly.
The scraped data is cleaned and stored (database, data lake) for analysis, Machine Learning or data products.

Keywords to drop

automated extraction, scraper / bot, external data, at scale, clean and store

Types of Data Scraping: Text, Image, Audio, Video

teacher hint: diff types2 to 5 marks

In simple words

Scraping is named after the kind of data being collected. Words, pictures, sound and video each need slightly different tools, and each is used for a different purpose.

Type	What is collected	Common tools	Used for	Example
Text	Articles, product descriptions, reviews, tweets, prices	BeautifulSoup, Scrapy, requests	Sentiment analysis, price tracking, NLP datasets	Customer reviews of a phone from Amazon
Image	Photos plus metadata (captions, tags, alt text)	requests + BeautifulSoup (find img src), Selenium, Scrapy image pipeline	Image classification, object detection training data	Thousands of product photos to train an image recognition model
Audio	Podcasts, songs, voice recordings, audio streams	requests, yt-dlp, APIs of audio platforms	Speech recognition, voice assistants, audio classification	Podcast episodes to build a speech-to-text dataset
Video	Video files or video metadata (title, views, comments, duration)	YouTube Data API, Selenium, yt-dlp	Recommendation systems, trend and engagement analysis	View counts and comments of trending YouTube videos

Write this in the exam

Text scraping: extracting written content (articles, reviews, tweets) from web pages using BeautifulSoup or Scrapy. Example: Amazon product reviews.
Image scraping: downloading images with metadata like captions and tags to build datasets for image classification. Example: product photos from a shopping site.
Audio scraping: collecting audio files or streams for speech recognition and voice assistants. Example: podcast episodes.
Video scraping: extracting video content or metadata (title, views, comments) for recommendation systems and trend tracking. Example: trending YouTube videos.

Exam tip

For a 5-mark version, write each type as a heading with definition, tool and example, then add one line on why scraping is done. The table above also works as the full answer.

Keywords to drop

text, image, audio, video, metadata, dataset for ML, BeautifulSoup, Scrapy

Web Crawling vs Web Scraping, and Web Scraping vs API Scraping

in her question list2 marks

In simple words

Crawling is wandering the internet following links to find pages (what Google does). Scraping is going to a known page and pulling out specific data. Scraping is a subset of the bigger crawling job.

API scraping is when the website itself offers a clean "data counter" (an API) that hands you data in JSON, with permission. Web scraping reads the messy HTML page meant for humans.

Think of it like

Crawling = walking through every lane of a market making a list of all shops. Scraping = going to one known shop and noting down its price list. API = the shop owner hands you a printed price list directly.

Basis	Web Crawling	Web Scraping
Goal	Discover and index pages by following links	Extract specific data from known pages
Scope	Broad, whole websites or the whole web	Narrow, targeted pages and fields
Output	List / index of URLs	Structured data (CSV, table, JSON)
Tool / bot	Crawler or spider (Googlebot, Scrapy spider)	Scraper (BeautifulSoup, Selenium)
Relation	The bigger process	A subset of crawling
Example	Google indexing every page	Collecting phone prices from Flipkart

Basis	Web Scraping	API Scraping
Data source	HTML page made for humans	API endpoint made for programs
Format	Unstructured HTML, needs parsing	Structured JSON / XML
Permission	May break Terms of Service	Officially allowed, uses an API key
Stability	Breaks when page layout changes	Stable, versioned
Limits	Can be blocked by the site	Rate limits and quotas
Example	Parsing tweets from the web page	Using the YouTube Data API for video stats

Write this in the exam

Web crawling is browsing the internet for a task, discovering relevant links and making an index of them.
Web scraping extracts the information from an identified or known web page; it is a subset of web crawling.
Crawling output is a list of URLs; scraping output is structured data.
API scraping gets structured JSON from an official endpoint with permission; web scraping parses raw HTML and breaks when the layout changes.

Keywords to drop

discover / index, extract, subset, spider, API key, JSON, rate limit

Applications of Data Scraping and Types of Web Scrapers

in her question list2 to 5 marks

In simple words

Anyone who needs data that lives on someone else's website uses scraping: investors watching prices, travel sites comparing hotels, resellers watching stock, ML teams building datasets and brands tracking what people say about them.

Write this in the exam

Applications of data scraping:

Financial and competitive market data: stock prices, news, competitor pricing for market analysis.
Travel and hospitality comparisons: flight and hotel prices collected from many sites (like MakeMyTrip, Trivago).
E-commerce reselling and auto purchasing: tracking price drops and stock availability, bots that buy when stock arrives.
Machine Learning and Data Science datasets: text, images and audio to train models.
Social media mining and analytics: posts, hashtags and comments for sentiment and brand monitoring.
Also: lead generation, research, news aggregation.

Types of web scrapers:

Browser extension scrapers: installed in Chrome, point and click, easy for small jobs, limited features.
Self-built scrapers: written in code (Python with requests, BeautifulSoup, Scrapy, Selenium), fully customisable, need programming skill.
Cloud-based scrapers: run on a provider's servers (for example Apify, Octoparse cloud), scale easily, run 24x7, handle IP rotation, cost money.

Keywords to drop

price monitoring, competitive analysis, sentiment, ML datasets, browser extension, self-built, cloud-based

Scraping Tools: urllib, requests, BeautifulSoup, Scrapy, Selenium

teacher hint: diff tools2 to 5 marksdiagram

In simple words

A scraper does two jobs: fetch the web page, then read the useful bits out of it. urllib and requests fetch. BeautifulSoup reads (parses) the HTML. Scrapy is a full factory that does fetching, reading and saving across thousands of pages. Selenium opens a real browser, for pages whose content only appears after JavaScript runs (like infinite scroll on Instagram).

Tool	What it does	Use it when
urllib	Built-in Python module to open URLs and download pages	No extra install allowed, basic downloads
requests	Easy library to send HTTP GET/POST requests with headers and cookies	Downloading static pages or calling APIs
BeautifulSoup	Parses HTML/XML into a tree, lets you find tags and pull text	Extracting data from a downloaded static page
Scrapy	Full framework: spiders, crawling, pipelines, export to CSV/JSON/DB	Large projects, many pages, crawling at scale
Selenium	Automates a real browser (click, scroll, type) through a WebDriver	Dynamic pages built by JavaScript, login forms, infinite scroll

Selenium: your script (framework) sends JSON commands to the WebDriver, which drives the real browser.

Every HTTP request and response = headers, one empty line, then the body (the HTML or JSON). Headers differ by request/response type (User-Agent, Content-Type, Cookie).

Write this in the exam

urllib / requests: send HTTP requests to the server and download the page. A request and response each have a header, an empty line and a body.
BeautifulSoup: parses the HTML into a tree-like structure so tags and text can be searched and extracted.
Scrapy: a complete framework for crawling many pages with spiders and saving the output through pipelines.
Selenium: when a page mixes static and dynamic (JavaScript) content, browser automation is needed before scraping. Selenium framework sends JSON commands to the WebDriver, which controls the browser.
Sitemap: a navigation map (sitemap.xml) listing all pages of a site, which helps a scraper find what to extract.

Keywords to drop

HTTP request/response, parser, spider, dynamic content, WebDriver, sitemap

BeautifulSoup: Steps and Parsers

in her question list2 marks

In simple words

BeautifulSoup takes the messy HTML text of a page and turns it into a neat family tree of tags (html → body → div → p). Then you can say "give me every price tag" and it hands them over. It cannot read HTML by itself, so it borrows a parser (the engine that understands HTML). You pick which parser.

Import the library: from bs4 import BeautifulSoup.
Send a request to the server and get the HTML (using requests).
Choose a parser and create the soup object. The response is converted into a tree-like structure of HTML elements.
Search the tree with find(), find_all() or CSS select().
Extract text or attributes and save them.

import requests
from bs4 import BeautifulSoup

html = requests.get("https://example.com").text
soup = BeautifulSoup(html, "html.parser")   # choose a parser

print(soup.title.text)                       # page title
for link in soup.find_all("a"):              # every link on the page
    print(link.get("href"))

Parser	How to call it	Speed	Notes
html.parser	`BeautifulSoup(html, "html.parser")`	Decent	Built into Python, no install needed
lxml (HTML)	`BeautifulSoup(html, "lxml")`	Very fast	Needs `pip install lxml`, lenient
html5lib	`BeautifulSoup(html, "html5lib")`	Slow	Parses exactly like a web browser, very lenient with broken HTML
lxml (XML)	`BeautifulSoup(xml, "xml")` or `"lxml-xml"`	Very fast	The only XML parser option

Write this in the exam

BeautifulSoup is a Python library for parsing HTML and XML documents into a parse tree for extracting data.
It does not have its own parsing mechanism, so an available parser is chosen when creating the soup object, e.g. soup_obj = BeautifulSoup(html_data, 'html.parser').
Parsers: html.parser (built-in), lxml (very fast, HTML), html5lib (browser-like, slow), lxml-xml (for XML).
Steps: import from bs4 → send request → choose parser and create soup → find elements → extract.

Keywords to drop

parse tree, soup object, html.parser, lxml, html5lib, find_all()

robots.txt and Legal & Ethical Considerations

in her question list2 to 5 marks

In simple words

robots.txt is a small text file at the root of every website (e.g. flipkart.com/robots.txt) that says which bots may visit which pages. It is the website's "house rules" board. A good scraper reads it first and obeys it.

Legal and ethical scraping means: follow the house rules, do not overload the server, do not grab people's personal data, and do not reuse copyrighted content as your own.

Think of it like

A housing society notice board: "Delivery boys allowed till Gate 2. No entry to the terrace. Visitors register at the gate." robots.txt is that board for bots.

# robots.txt of an example shop
User-agent: *              # rules for ALL bots
Disallow: /cart/           # do not crawl the cart
Disallow: /tv/*/pictures   # no TV picture pages
Allow: /products/

User-agent: BadBot         # one specific bot
Disallow: /                # blocked from everything

Crawl-delay: 10            # wait 10 seconds between requests
Sitemap: https://www.example.com/sitemap.xml

Write this in the exam

robots.txt gives crawler information and is useful for designing a web scraper. It is a guide consisting of rules:

User-agent: which bot the rule applies to; * means all web scrapers.
Disallow: paths the bot cannot crawl or scrape (e.g. /tv/*/pictures).
Allow: paths explicitly permitted.
Sitemap: a structured file listing all pages the bot should crawl.

Legal and ethical considerations:

Terms of Service (ToS): many sites forbid scraping in their ToS; breaking it can be a breach of contract.
robots.txt: respect the Disallow rules.
Rate limiting and server respect: add delays, avoid overloading the server (it can act like a DoS attack).
Data privacy (GDPR, CCPA, India's DPDP Act 2023): avoid collecting personal data (names, phone numbers, emails) without consent.
Intellectual property and copyright: scraped articles, images and videos may be copyrighted; do not republish them.
Legal precedents: court cases (e.g. hiQ Labs vs LinkedIn in the US) show public data and data behind a login are treated differently, so check the law before scraping.
Identify your bot honestly with a User-Agent and prefer an official API when one exists.

Keywords to drop

User-agent, Disallow, Sitemap, ToS, rate limiting, GDPR, copyright, consent

Workflow of Scraping Text Data and Storing it in a Database 💗

PYQ 5 marks5 marksdiagram

In simple words

This is the full journey: find the page, download it, pick out the text you want, clean it, check it, then save it neatly in a database table. Six steps. The example below collects book names and prices from a practice website (books.toscrape.com, which exists for scraping practice) and saves them in SQLite.

1. URL discovery2. Download (requests)3. Parse & extract (BeautifulSoup)4. Transform / clean5. Quality check6. Store in DB

Write this in the exam

URL discovery: identify the target pages (from the sitemap or by crawling links). Check robots.txt and ToS first. Example: books.toscrape.com/catalogue/page-1.html.
Downloading: send an HTTP GET request using the requests library with a User-Agent header; the server returns the HTML in the response body.
Parsing and extraction: create a BeautifulSoup object with a parser; locate the tags holding the data (each book is an <article class="product_pod">) and extract the title, price and rating text.
Transformation / cleaning: strip currency symbols and spaces, convert price to a number, remove duplicates, handle missing values.
Quality assurance: validate each record (title not empty, price greater than 0, count of rows as expected).
Storage: create a table and insert the records into a database (SQLite / MySQL for structured rows, MongoDB for flexible documents); the data is now ready for analysis.

Add delays between pages (rate limiting) and log errors so the scraper is polite and reliable.

Worked example: Python code

import re, sqlite3, requests
from bs4 import BeautifulSoup

# 1. URL discovery
URL = "https://books.toscrape.com/catalogue/page-1.html"

# 2. Download the page
resp = requests.get(URL, headers={"User-Agent": "StudentScraper/1.0"}, timeout=10)
resp.raise_for_status()

# 3. Parse and extract
soup = BeautifulSoup(resp.content, "html.parser")
rows = []
for book in soup.find_all("article", class_="product_pod"):
    title = book.h3.a["title"]
    price_text = book.find("p", class_="price_color").text      # e.g. "£51.77"
    rating = book.find("p", class_="star-rating")["class"][1]    # e.g. "Three"

    # 4. Transform / clean
    price = float(re.sub(r"[^0-9.]", "", price_text))
    rows.append((title.strip(), price, rating))

# 5. Quality check
rows = [r for r in rows if r[0] and r[1] > 0]
print(len(rows), "books scraped")      # 20 books per page

# 6. Store in the database
conn = sqlite3.connect("books.db")
conn.execute("""CREATE TABLE IF NOT EXISTS books (
                  id INTEGER PRIMARY KEY AUTOINCREMENT,
                  title TEXT, price REAL, rating TEXT)""")
conn.executemany("INSERT INTO books (title, price, rating) VALUES (?, ?, ?)", rows)
conn.commit()
conn.close()

Result: a table books with columns id, title, price, rating, for example (1, "A Light in the Attic", 51.77, "Three").

Exam tip

Draw the 6-box flow first, write one line per box, then give the example (even 8 to 10 lines of code or pseudo-code). Diagram + steps + example = full 5 marks.

Keywords to drop

URL discovery, HTTP GET, parse, extract, transform, quality assurance, store, SQLite / MongoDB

Storage Architecture: Raw Ingredients, Systems, Abstractions

2 to 5 marksdiagram

In simple words

Storage is built in three layers, like a cake. At the bottom is the hardware (disks, memory, network). On top of that sit storage systems, software that organises data on the hardware (a database, HDFS, S3). On top of those sit abstractions, the big ideas a company actually works with (a data lake, a warehouse, a lakehouse).

Think of it like

Bricks and cement (raw ingredients) → rooms and cupboards (storage systems) → "the house" your family lives in (the abstraction). Nobody says "I live in bricks".

Each outer layer is built on the one inside it.

Write this in the exam

Raw ingredients: the physical building blocks where data is actually stored and moved: HDD, SSD, RAM, networking, CPU, plus serialization and compression.
Storage systems: software that organises data on the raw ingredients: HDFS, object storage, RDBMS, cache, streaming storage.
Storage abstractions: higher-level combinations of storage systems that the data team uses: data lake, lakehouse, cloud data warehouse, data platform.
Serialization: converting in-memory objects into a byte/text format (JSON, CSV, Avro, Parquet) so they can be stored or sent over a network; deserialization converts them back into objects.
Compression: reducing storage size by encoding data in fewer bits. Lossless (gzip, Snappy, used for data files, exact original returns) and lossy (JPEG, MP3, some detail thrown away).
Why they matter in a pipeline: serialization lets different systems exchange data; compression cuts storage cost and network transfer time, making the pipeline faster and cheaper.

Keywords to drop

raw ingredients, storage systems, storage abstraction, serialization, deserialization, lossless / lossy, Parquet

Types of Databases, SQL vs NoSQL, Schema-on-Write vs Schema-on-Read

2 marks

In simple words

Databases can be grouped two ways. By where they live: one machine (centralized), many machines (distributed) or someone else's servers (cloud). By how they arrange data: tables (relational), non-table formats (NoSQL), a tree (hierarchical), a web of links (network) or objects like in programming (object oriented).

Schema-on-write = decide the shape of the data before saving (like a printed form you must fill correctly). Schema-on-read = save anything, decide the shape when reading (like a notebook where you scribble, and organise later).

Write this in the exam

By location: centralized, distributed, cloud databases.
By data organisation: relational (RDBMS), NoSQL, hierarchical, network, object oriented.

Basis	SQL (Relational)	NoSQL
Data model	Tables with rows and columns	Document, key-value, graph, wide-column
Schema	Fixed schema, defined first	Flexible / schema-less
Scaling	Vertical (bigger server)	Horizontal (more servers, sharding)
Consistency	Strong ACID transactions	Often BASE / eventual consistency
Query language	SQL	Varies: MongoDB queries, Cypher, Redis commands
Best for	Structured, transactional data (banking)	Big, fast-changing, semi/unstructured data
Examples	MySQL, PostgreSQL, Oracle	MongoDB, Redis, Neo4j, Cassandra

Basis	Schema-on-Write	Schema-on-Read
When structure is applied	Before data is stored	When data is read / queried
Load speed	Slower (validate first)	Fast (store as-is)
Query speed	Fast, data already clean	Slower, parse at read time
Flexibility	Low	High, any format
Used in	RDBMS, data warehouse	Data lake, NoSQL

Keywords to drop

relational, hierarchical, network, object oriented, horizontal scaling, schema-on-write, schema-on-read

NoSQL Databases: Meaning and Types 🌸

teacher: meaning will be asked, VIMP2 marks

In simple words

NoSQL = "Not Only SQL". Databases that do not force data into rows and columns. Each type stores data in the shape that fits it best:

Document: each record is a JSON "file card", and each card can have different fields (MongoDB).
Key-Value: a giant dictionary. Give the key, get the value instantly (Redis).
Graph: dots and lines, people and their connections (Neo4j).

Think of it like

Document DB = a file of Aadhaar forms where some people attached extra pages. Key-Value DB = a locker room: locker number (key) opens exactly one locker (value). Graph DB = a family tree chart on the wall.

Write this in the exam

NoSQL ("Not Only SQL") databases store data in formats other than the traditional rows-and-columns tables of relational databases.
They handle large volumes of unstructured or semi-structured data with high speed and flexibility.
They do not require a fixed schema, so new types of data can be added without redesigning the database.
They scale horizontally (sharding across many servers).

1. Document databases: data stored as documents in JSON/BSON format; each document can have a different structure. Example: MongoDB, which stores documents inside collections.

2. Key-Value databases: data stored as a unique key and its value, like a giant dictionary/hashmap; extremely fast lookups. Example: Redis, Amazon DynamoDB. Use: session ID (key) → login details (value).

3. Graph databases: data stored as nodes (entities) and edges (relationships) with properties. Example: Neo4j. Use: social networks, recommendations, fraud detection.

Bonus, wide-column: data stored column-family wise, read column by column (Cassandra, HBase).

RDBMS term	MongoDB (document) term
Database	Database
Table	Collection
Tuple / Row	Document
Attribute / Column	Key / Field
Column value	Field value

// one document in the "students" collection (MongoDB)
{
  "_id": 1,
  "name": "Simran",
  "age": 20,
  "skills": ["Python", "SQL"],
  "address": { "city": "Mumbai", "pin": "400071" }
}

// key-value (Redis)
SET session:9f2a "user_id=42;role=student"
GET session:9f2a

A document database is a special case of a key-value store where the value is a structured document (the key is a string such as _id).

Keywords to drop

Not Only SQL, schema-less, JSON/BSON, collection, key-value, nodes & edges, horizontal scaling, sharding

Graph Databases and Neo4j: Nodes, Relationships, Properties, Labels

teacher: description given, draw the graphdiagram2 to 5 marks

In simple words

A graph database stores things as circles (nodes) and how they are connected as arrows (relationships). Both can carry details (properties). A label is the "type" of a node, like Student or Course. Asking "who are my friends' friends" becomes following arrows, which is very fast.

Think of it like

An Instagram "mutual friends" map. People are circles, "follows" is an arrow. Neo4j is built to walk these arrows quickly.

Write this in the exam

A graph database stores data in the form of nodes and relationships, where nodes and relationships can have properties and labels.
Node: an entity, e.g. a Student or a Course.
Label: the category of a node, e.g. :Student, :Course.
Relationship (edge): a directed, named connection, e.g. ENROLLED_IN, FRIENDS_WITH.
Property: key-value details on a node or relationship, e.g. {name: "A", age: 18}.
Neo4j is the most widely used graph database; its query language is Cypher.
Used when relationships matter as much as the data: social networks, recommendation engines, fraud detection, route finding.

Worked example: her class notes (Student and Course tables)

RNo	Name	Age
1	A	18
2	B	19
3	C	20

Cid	C-title	Credit
C1	DBMS	4
C2	ML	4
C3	OS	4

Description: "A studies DBMS and ML, B studies ML and OS, C studies OS." Drawn as a graph:

Pink circles = Student nodes (properties rno, name, age). Lavender circles = Course nodes (properties cid, title, credit). Every arrow is an ENROLLED_IN relationship.

Method: "description is given, draw the graph"

Underline every noun that is a thing (student, teacher, city, product): these are nodes. Group them into labels.
Underline every verb that links two things (studies, teaches, follows, bought): these are relationships. Write them in CAPITALS, e.g. TEACHES.
Decide the direction of each arrow (who does what to whom).
Write the properties inside or beside each node, e.g. {name: "A", age: 18}, and on arrows if given (e.g. {since: 2025}).
Draw circles for nodes, labelled arrows for relationships, then (for full marks) write the Cypher CREATE query for it.

Keywords to drop

node, relationship / edge, property, label, directed, Neo4j, Cypher, highly connected data

Cypher Query Language: the "Numerical" Part 💗

PYQ + teacher: numericalnumerical2 marks

In simple words

Cypher is how you talk to Neo4j. It looks like ASCII art: ( ) is a node (a circle), -[ ]-> is an arrow. So (t:Teacher)-[:TEACHES]->(c:Course) literally draws "teacher → teaches → course".

CREATE = make new nodes/arrows.
MATCH = find existing ones (like SELECT/WHERE in SQL).
RETURN = show the result.
SET = update, DELETE = remove.

Cypher piece	Meaning	SQL cousin
`(s:Student {name:"A"})`	node, variable s, label Student, property name	a row in table Student
`-[:ENROLLED_IN]->`	directed relationship of type ENROLLED_IN	foreign key / join
`CREATE`	create nodes / relationships	INSERT
`MATCH ... WHERE`	find a pattern	SELECT ... WHERE
`RETURN`	output	SELECT list
`SET` / `DETACH DELETE`	update / delete node and its relationships	UPDATE / DELETE
`MERGE`	create only if it does not already exist	INSERT if not exists

Write this in the exam

Cypher is Neo4j's declarative query language (like SQL for graphs) used to create, read, update and delete nodes and relationships using ASCII-art patterns.
Nodes are written in round brackets (t:Teacher {name: "Meera"}): variable, label, properties.
Relationships are written in square brackets with an arrow -[:TEACHES]->: type in capitals, direction shown by the arrow.
CREATE adds data, MATCH finds a pattern, WHERE filters, RETURN outputs, SET updates, DETACH DELETE removes a node with its relationships, MERGE creates only if missing.

PYQ (2M): create a relationship TEACHES from Teacher "Meera" to Course "Data Engineering"

If both nodes already exist (the usual answer):

MATCH (t:Teacher {name: "Meera"}), (c:Course {name: "Data Engineering"})
CREATE (t)-[:TEACHES]->(c)
RETURN t, c;

If the nodes do not exist yet, one statement creates everything:

CREATE (t:Teacher {name: "Meera"})-[:TEACHES]->(c:Course {name: "Data Engineering"})
RETURN t, c;

Explain in one line: MATCH finds the Teacher node Meera and the Course node Data Engineering, CREATE draws a directed TEACHES arrow from the teacher to the course.

All the basic queries on the Student/Course graph

// CREATE nodes
CREATE (:Student {rno: 1, name: "A", age: 18}),
       (:Student {rno: 2, name: "B", age: 19}),
       (:Course {cid: "C1", title: "DBMS", credit: 4}),
       (:Course {cid: "C2", title: "ML", credit: 4});

// CREATE a relationship with a property
MATCH (s:Student {name: "A"}), (c:Course {title: "DBMS"})
CREATE (s)-[:ENROLLED_IN {year: 2026}]->(c);

// READ: which student is enrolled in which course
MATCH (s:Student)-[:ENROLLED_IN]->(c:Course)
RETURN s.name, c.title;

// WHERE: students older than 18
MATCH (s:Student) WHERE s.age > 18
RETURN s.name, s.age;

// UPDATE with SET
MATCH (s:Student {name: "A"}) SET s.age = 19;

// DELETE a node together with its relationships
MATCH (c:Course {title: "OS"}) DETACH DELETE c;

// Friends of friends (social network)
MATCH (me:Person {name: "Simran"})-[:FRIENDS_WITH]->(f)-[:FRIENDS_WITH]->(fof)
WHERE fof <> me AND NOT (me)-[:FRIENDS_WITH]->(fof)
RETURN DISTINCT fof.name;

Her Q15: illustrate a path in Neo4j and find the shortest path

Build the 4-city graph from the PYQ (distances as a property on each ROAD):

CREATE (a:City {name: "A"}), (b:City {name: "B"}),
       (c:City {name: "C"}), (d:City {name: "D"}),
       (a)-[:ROAD {distance: 10}]->(b), (a)-[:ROAD {distance: 15}]->(c),
       (a)-[:ROAD {distance: 20}]->(d), (b)-[:ROAD {distance: 35}]->(c),
       (b)-[:ROAD {distance: 25}]->(d), (c)-[:ROAD {distance: 30}]->(d);

// A path is a chain of nodes and relationships
MATCH p = (:City {name: "A"})-[:ROAD]->(:City {name: "B"})-[:ROAD]->(:City {name: "D"})
RETURN p;

// Shortest path by number of hops (built-in shortestPath)
MATCH (a:City {name: "A"}), (d:City {name: "D"}),
      p = shortestPath((a)-[:ROAD*]-(d))
RETURN [n IN nodes(p) | n.name] AS route, length(p) AS hops;
// route = ["A", "D"], hops = 1

// Shortest path by total distance (small graphs)
MATCH p = (:City {name: "A"})-[:ROAD*]-(:City {name: "D"})
RETURN [n IN nodes(p) | n.name] AS route,
       reduce(km = 0, r IN relationships(p) | km + r.distance) AS total
ORDER BY total ASC
LIMIT 1;
// route = ["A", "D"], total = 20

For big weighted graphs, Neo4j's Graph Data Science library runs Dijkstra directly: gds.shortestPath.dijkstra.stream(...) with relationshipWeightProperty: 'distance'.

Exam tip

Always write the label with a colon (:Teacher), properties in curly braces with quotes around text, the relationship type in CAPITALS inside [: ], and the arrow direction ->. Those four details are where marks go.

Keywords to drop

MATCH, CREATE, RETURN, WHERE, SET, DETACH DELETE, MERGE, shortestPath, pattern

Dijkstra's Shortest Path Algorithm (PYQ Numerical)

PYQ 5 marksnumericaldraw graph

In simple words

Dijkstra finds the cheapest route from one city to all others. Start at the source with distance 0 and everyone else at "infinity". Repeatedly pick the nearest city not yet finalised, lock it in, and check if going through it makes any neighbour cheaper. Repeat until every city is locked.

Think of it like

Google Maps checking "is it shorter to go straight, or via that flyover?" for every junction, always expanding from the closest junction first.

Method

Draw the graph from the distance matrix (each non-zero entry = an edge with that weight).
Set dist(source) = 0, all others = ∞. Visited set = empty.
Pick the unvisited node with the smallest distance and mark it visited.
Relax its neighbours: if dist(u) + weight(u, v) < dist(v), update dist(v) and note "via u".
Repeat steps 3 and 4 until all nodes are visited. Read the path backwards using "via".

Write this in the exam

Dijkstra's algorithm finds the shortest path from a source node to all other nodes in a graph with non-negative edge weights.
It is greedy: at each step it permanently selects the unvisited node with the smallest tentative distance.
It then relaxes the edges of that node: dist(v) = min(dist(v), dist(u) + w(u, v)).
It stops when all nodes are visited; the path is traced back using the predecessor ("via") of each node.

Worked example: the PYQ (source A, destination D)

	A	B	C	D
A	0	10	15	20
B	10	0	35	25
C	15	35	0	30
D	20	25	30	0

Step	Node picked (visited)	dist B	dist C	dist D	What happened
0	none	∞	∞	∞	Start: source A = 0
1	A (0)	10 (via A)	15 (via A)	20 (via A)	Relax A's neighbours
2	B (10)	10	15 (10+35 = 45, no change)	20 (10+25 = 35, no change)	B finalised
3	C (15)	10	15	20 (15+30 = 45, no change)	C finalised
4	D (20)	10	15	20	D finalised, done

Answer: shortest path from A to D is A → D with cost 20.

Check against the other routes: A → B → D = 10 + 25 = 35, A → C → D = 15 + 30 = 45, A → B → C → D = 75, A → C → B → D = 75. All are longer.

Shortest paths from A to every city: A → B = 10, A → C = 15, A → D = 20 (all direct).

Exam tip

Marks come from: the drawn graph with weights (1), the step table showing each relaxation (2 to 3), the final path and cost (1). Write the "no change" comparisons (45 > 15 and so on) so the examiner sees you applied the algorithm, since here the direct edge wins.

Keywords to drop

source node, relaxation, visited set, greedy, non-negative weights, distance table

Evolution of Storage Types

2 marks

In simple words

As data grew from a few files to petabytes, storage kept changing to keep up. Each new type fixed a problem of the older one.

File system (early days)RDBMS (databases)Block storage (enterprise era)Object storage (cloud era)Data lake (big data era)Lakehouse (modern era)

Write this in the exam

File system: data stored as files in folders; simple but hard to search and share at scale.
RDBMS: structured tables with SQL and ACID transactions; rigid schema, hard to scale for huge data.
Block storage: data split into fixed-size blocks on SAN disks; very fast, used for databases and virtual machines, expensive.
Object storage: each object = data + metadata + unique ID in a flat namespace, accessed with REST APIs; cheap and almost unlimited (S3, ADLS, GCS).
Data lake: object storage used to hold all raw data (structured, semi, unstructured) in one place; lacks ACID and governance.
Lakehouse: data lake + warehouse features (ACID, schema enforcement, time travel) using Delta Lake, Iceberg or Hudi.

Keywords to drop

file, block, object, flat namespace, REST API, data lake, lakehouse

Cloud Object Storage 🌸

syllabus 2.12 to 5 marks

In simple words

Object storage keeps each piece of data (a photo, a CSV, a video) as one sealed parcel called an object. Each parcel has the content, a label describing it (metadata) and a unique number (ID / key). All parcels sit in big containers called buckets. There are no folders inside folders; everything lies flat and is found by its unique key, over the internet, using simple API calls.

Think of it like

An Amazon delivery warehouse. The warehouse = bucket. Each parcel = object. The sticker on the parcel (address, weight) = metadata. The tracking number, like an Aadhaar number for the parcel = unique ID. Workers find any parcel instantly by tracking number, not by walking through shelves.

Write this in the exam

Cloud object storage stores data as independent units called objects, each made of data + metadata + a unique identifier, instead of files in folders or blocks on a disk.
Objects are kept in containers called buckets in a flat namespace (no directory hierarchy) and accessed through REST APIs over the internet.
Durability: 99.999999999% (11 nines), data is copied across multiple locations.
Scalability: stores exabytes without pre-provisioning capacity.
Lifecycle management: automatically moves or deletes data based on age or access pattern.
Storage classes: tiered pricing based on cost and access frequency (hot, cool, archive).
Versioning: keeps multiple versions of the same object, so nothing is overwritten by mistake.
Pay-as-you-use: pay only for what is stored and transferred.
Ideal for unstructured data: images, videos, backups, logs, data lakes. Examples: Amazon S3, Azure Blob / ADLS Gen2, Google Cloud Storage.

Basis	File storage	Block storage	Object storage
Unit	Files in folders	Fixed-size blocks	Objects (data + metadata + ID)
Structure	Hierarchical directories	Raw volume, no structure	Flat namespace in buckets
Metadata	Limited (name, size, date)	None	Rich, custom metadata
Access	File path (NFS / SMB)	Attached disk (SAN, iSCSI)	HTTP REST API
Scalability	Limited	Moderate	Virtually unlimited
Speed	Good for shared files	Fastest, lowest latency	Higher latency, high throughput
Best for	Shared documents	Databases, VM disks	Backups, media, data lakes
Example	NAS, Amazon EFS	Amazon EBS	Amazon S3, ADLS, GCS

Keywords to drop

object = data + metadata + ID, bucket, flat namespace, REST API, 11 nines durability, lifecycle, versioning, pay-as-you-go

Amazon S3 (Simple Storage Service) and its Storage Classes

syllabus 2.12 to 5 marks

In simple words

S3 is Amazon's object storage. You make a bucket (its name must be unique in the whole world, like a username), and drop objects into it. Every object has an address like s3://bucket-name/object-key. Amazon silently keeps 3 copies in 3 different data centres (Availability Zones), so data almost never gets lost.

Storage classes are price plans. Data used every day sits in Standard (costly to store, free to read). Old data nobody opens goes to Glacier (very cheap to store, but you pay and wait to take it out).

Think of it like

Clothes at home: daily wear in the almirah (Standard), festive wear in the upper shelf (Standard-IA), winter blankets packed in the loft (Glacier Flexible), grandmother's wedding saree in a sealed trunk at the village house (Deep Archive). Cheaper to keep as you go down, slower to take out.

Write this in the exam

Amazon S3 is AWS's cloud object storage service (an IaaS offering): durable (11 nines), highly available and scalable.
It stores any file type: images, videos, audio, database backups, documents, CSVs. It stores data; it is not a server that runs application code.
Data is stored as objects placed in containers called buckets. Bucket names are globally unique.
Object address: s3://bucket-name/object-key, e.g. s3://college-data/student-data/2026/students.csv (protocol / bucket name / object key).
S3 keeps copies of each object across at least 3 Availability Zones (except One Zone-IA).
Features: tiered storage classes, lifecycle policies (auto-move old data to cheaper classes), versioning, encryption, access control.
Ways to access a bucket: AWS Management Console (UI), AWS CLI, AWS SDKs, REST API, VPC Endpoint, S3 Access Points.

Storage class	Use it when
S3 Standard	General purpose, frequently accessed data, low latency
S3 Intelligent-Tiering	Access pattern is unknown or changing; S3 moves data between tiers automatically, no retrieval fee
S3 Standard-IA	Infrequently accessed but needs rapid access when needed; lower storage price, per-GB retrieval fee
S3 One Zone-IA	Infrequent data that can be recreated if lost (secondary backups); 1 AZ, about 20% cheaper than Standard-IA
S3 Glacier Instant Retrieval	Archive data read about once a quarter, still needs milliseconds access
S3 Glacier Flexible Retrieval	Archives and backups read once or twice a year; minutes to 12 hours
S3 Glacier Deep Archive	Cheapest, long-term compliance data kept for years; 12 to 48 hours

Feature	Standard	Intelligent-Tiering	Standard-IA	One Zone-IA	Glacier Instant	Glacier Flexible	Glacier Deep Archive
Durability	11 nines	11 nines	11 nines	11 nines	11 nines	11 nines	11 nines
Availability	99.99%	99.9%	99.9%	99.5%	99.9%	99.99%	99.99%
Availability Zones	≥ 3	≥ 3	≥ 3	1	≥ 3	≥ 3	≥ 3
Retrieval time	milliseconds	milliseconds	milliseconds	milliseconds	milliseconds	minutes to 12 hours	12 to 48 hours
Retrieval cost	none	none	per GB	per GB	per GB	per GB	per GB
Min storage duration	none	none	30 days	30 days	90 days	90 days	180 days
Min billable object size	none	none	128 KB	128 KB	128 KB	40 KB	40 KB

Exam tip

For a 2-mark S3 question: define, bucket + object + key with the s3:// example, 11 nines durability, and name 3 features. For storage classes, the 7 names in order (Standard → Deep Archive) plus "cheaper to store, costlier and slower to retrieve as you go down" is the core idea.

Keywords to drop

bucket, globally unique, object key, Availability Zones, storage classes, lifecycle policy, versioning, IaaS

Azure Data Lake Storage Gen2 and Google Cloud Storage

syllabus 2.12 marks

In simple words

ADLS Gen2 is Microsoft's version of S3, built for big data. Its special power: it allows real folders inside folders (hierarchical namespace), which S3 does not have. That makes renaming or deleting a whole folder of big data fast, and lets you set permissions per folder.

Google Cloud Storage (GCS) is Google's version: buckets and objects, with 4 price plans.

Write this in the exam

Azure Data Lake Storage Gen2:

Microsoft Azure's storage service for huge volumes of structured and unstructured data, i.e. a data lake.
Built on Azure Blob Storage (low cost object storage) with a Hierarchical Namespace (HNS) that supports folders and directories. S3 uses a flat namespace.
Security through ACLs (Access Control Lists) on files and folders, and identity through Azure Active Directory (now called Microsoft Entra ID).
Integrates with big data tools: Azure Databricks, Apache Spark, Azure Synapse, HDInsight.
Access tiers: Hot, Cool, Cold, Archive.

Google Cloud Storage:

Google Cloud's object storage; objects stored in buckets, addressed as gs://bucket/object.
Storage classes: Standard (frequent), Nearline (about once a month, 30-day minimum), Coldline (about once a quarter, 90-day minimum), Archive (about once a year, 365-day minimum).
Integrates with BigQuery, Dataproc (Spark/Hadoop) and Vertex AI.

Basis	Amazon S3	Azure ADLS Gen2	Google Cloud Storage
Provider	AWS	Microsoft Azure	Google Cloud
Structure	Flat object storage (bucket + key)	Hierarchical namespace (real folders)	Flat object storage (bucket + object)
Address	`s3://bucket/key`	`abfss://container@account.dfs.core.windows.net/path`	`gs://bucket/object`
Access control	IAM policies, bucket policies	Azure AD (Entra ID) + ACLs	Cloud IAM
Best suited for	General purpose object storage	Big Data analytics workloads	Analytics with BigQuery, ML
Cheap tiers	IA, Glacier classes	Cool, Cold, Archive	Nearline, Coldline, Archive

Keywords to drop

hierarchical namespace, flat namespace, Blob storage, ACL, Azure AD, Databricks, Nearline / Coldline

Data Warehouse

teacher: diff warehouse / lake / lakehouse2 to 5 marks

In simple words

A data warehouse is a clean, organised store of structured business data, built for answering questions and making reports. Data from many systems (billing, sales, insurance) is extracted, cleaned into one format and loaded (ETL). After that, a manager's question like "which store sold more this month" becomes one quick query.

Think of it like

A well-arranged library: every book catalogued and shelved by subject before it is allowed in. Finding anything is fast, but a book with no proper catalogue entry is not accepted.

Structured data (POS, CRM, billing)ETL: Extract → Transform → LoadData WarehouseReports / BI dashboards

Write this in the exam

A data warehouse is a central repository of structured, analytics-ready data integrated from multiple sources, designed for reporting and business analysis (BI).
Data enters through ETL: Extract from sources → Transform (clean, standardise, integrate) → Load into the warehouse.
Uses schema-on-write: data must fit the schema before loading.
Organised in star / snowflake schemas (fact tables with dimension tables), optimised for fast SQL aggregation.
Stores historical data, is subject-oriented and non-volatile.
Examples: Amazon Redshift, Google BigQuery, Snowflake, Azure Synapse.
Limitation: only structured data, expensive storage, not suited to ML on raw data.

Worked example: her class notes (POS + Insurance schemas)

POS table: (store_nm, cust_name, device, device_price, date_of_sale)
Insurance table: (store_nm, cust_name, period_of_insurance, plan, price)

-- Q1: total device sales for store "3, Jay St, New York"  (1 table)
SELECT SUM(device_price) AS device_sales
FROM pos
WHERE store_nm = '3, Jay St, New York';

-- Q2: total sales (device OR insurance) for that store  (2 tables)
SELECT (SELECT SUM(device_price) FROM pos       WHERE store_nm = '3, Jay St, New York')
     + (SELECT SUM(price)        FROM insurance WHERE store_nm = '3, Jay St, New York')
       AS total_sales;

-- Q3: compare store 1 and store 2 (needs data normalised into one model)
SELECT s.store_name, SUM(f.amount) AS total_sales
FROM sales_fact f JOIN store_dim s ON f.store_id = s.store_id
GROUP BY s.store_name;

Point of the example: Q1 needs one table, Q2 needs two systems combined, Q3 needs both systems cleaned and normalised (same store written the same way, one sales fact table with a store dimension). Doing that integration once, through ETL, is exactly the job of a data warehouse.

Keywords to drop

structured, ETL, schema-on-write, BI / reporting, star schema, fact & dimension, Redshift / BigQuery / Snowflake

Data Lake, and Data Lake vs Data Warehouse 💗

PYQ 2M (any four points)2 to 5 marksdiagram

In simple words

A data lake is a huge, cheap storage pool (built on S3 / ADLS / GCS) where you dump all data as-is: tables, JSON, logs, images, videos. No rules at entry; structure is decided later when someone reads it (schema-on-read). Great for flexibility and ML, but if nobody manages it, it becomes a messy "data swamp".

Think of it like

Warehouse = a neatly arranged almirah, only folded, ironed clothes go in. Lake = a big godown where everything is dumped in bags; cheap and holds anything, but finding one specific shirt takes effort.

Data lake medallion architecture: Bronze → Silver → Gold.

Write this in the exam

A data lake is a centralised repository that stores structured, semi-structured and unstructured data in its raw form at any scale.
Uses schema-on-read: data is stored without enforcing a schema.
Built on object storage: S3, ADLS Gen2, GCS.
Architecture (medallion): Bronze (raw) → Silver (cleaned, validated) → Gold (business-level, aggregated).
Advantages: low cost storage, any data format, separation of storage and compute, supports ML and data science.
Problems / limitations: (1) data congestion and inconsistency, (2) duplicate or missing records, (3) schema drift, (4) data quality issues, (5) concurrency problems (no ACID: atomicity and consistency are not governed), (6) governance and lineage gaps. Unmanaged lakes turn into data swamps.

Basis	Data Lake	Data Warehouse
Type of data	Structured, semi-structured, unstructured	Only structured
Schema	Schema-on-read	Schema-on-write
Data state	Raw, as ingested	Cleaned, processed, analytics-ready
Processing	ELT (load first, transform later)	ETL (transform before load)
Storage cost	Low (cheap object storage)	High (specialised storage)
Users	Data scientists, ML engineers	Business analysts, managers
Use cases	ML, exploration, big data	BI reports, dashboards, SQL analytics
Reliability	No ACID, risk of data swamp	ACID, high data quality
Examples	S3, ADLS Gen2, GCS	Redshift, BigQuery, Snowflake

Exam tip

"Differentiate using any four points" = pick 4 rows from the table (type of data, schema, cost, users are the safest). Always write it as a table.

Keywords to drop

raw data, schema-on-read, Bronze / Silver / Gold, schema drift, data swamp, ELT vs ETL, no ACID

Lakehouse Architecture: Features, Layers, and Warehouse vs Lake vs Lakehouse 🌸

teacher: features, properties, VIMPcase studydiagram5 marks

In simple words

A lakehouse = the cheap, hold-anything storage of a data lake + the rules and reliability of a warehouse. The data still sits as files on S3/ADLS, but a smart "table layer" (Delta Lake, Iceberg or Hudi) sits on top and keeps a register of every change. That register gives ACID transactions, blocks bad data (schema enforcement) and lets you see old versions (time travel). One system now serves both BI reports and ML.

Think of it like

The same big godown (lake), but now with a strict store manager and a register at the gate (the table layer). Every bag coming in is checked and written in the register, so nothing gets lost, mixed up or overwritten, and you can always look up what the godown held last Tuesday.

Data sourcesData ingestionCloud object storageLakehouse table (Delta / Iceberg / Hudi)Compute (Spark, SQL) → BI + ML

Lakehouse = object storage + a table/metadata layer + compute, serving BI and ML from the same data.

Write this in the exam

A lakehouse combines the low-cost, flexible storage of a data lake with the data management and reliability features of a data warehouse (Lakehouse = Data Warehouse + Data Lake).

Features / properties:

Object storage foundation: data stays on cheap cloud storage (S3, ADLS, GCS) in open formats like Parquet.
Table management layer with ACID transactions: reliable concurrent reads and writes.
Schema enforcement and schema evolution: bad data is rejected, planned changes are allowed.
Time travel / versioning: query or restore older versions of a table.
Metadata and governance layer above the data lake: auditing, lineage, access control.
Unified analytics platform: BI reporting and ML/AI on the same data; structured and unstructured data together.
Removes the need for two separate systems (lake + warehouse), reducing cost and duplication.

Layers: Object storage (S3) → Table layer (Delta Lake, Apache Iceberg, Apache Hudi) → Compute (Spark, SQL).

Basis	Data Warehouse	Data Lake	Lakehouse (e.g. Delta Lake)
Type of data	Only structured	Structured, semi, unstructured	Structured, semi, unstructured
Schema	Schema-on-write	Schema-on-read	Both, with schema enforcement and evolution
Storage cost	High, specialised storage	Low, object storage	Low, object storage
ACID transactions	Yes	No	Yes (via Delta / Iceberg / Hudi)
Data quality	High	Low, risk of data swamp	High, governed
Versioning / history	Limited, latest data	None built in	Time travel through transaction log
Use cases	BI and SQL reporting	ML, data science, raw storage	BI + ML/AI on the same data
Examples	Redshift, Snowflake, BigQuery	S3, ADLS Gen2, GCS	Databricks Delta Lake, Iceberg tables

Exam tip

If the question says "Delta Lake vs warehouse vs lakehouse", note in one line that Delta Lake is the open-source storage layer that turns a data lake into a lakehouse, then use this 3-column table.

Keywords to drop

lake + warehouse, ACID, table layer, open format (Parquet), schema enforcement, time travel, unified analytics, metadata & governance

Open-Source Storage Layers: Delta Lake, Apache Iceberg, Apache Hudi

syllabus 2.34 to 5 marks

In simple words

Plain files on S3 cannot guarantee correctness when many people read and write at once. These three tools are a software layer placed on top of the files that keeps a log of every change, which gives database-like safety. Think of three brands of the same idea:

Delta Lake (by Databricks): best friend of Spark, famous for ACID and time travel.
Apache Iceberg (by Netflix): built so many engines (Spark, Flink, Trino) can share huge tables safely.
Apache Hudi (by Uber): built for data that keeps changing (updates and deletes all the time, like live ride data).

ACID in simple words, using a UPI payment: Atomicity = money leaves your account and reaches your friend, or nothing happens at all. Consistency = balances always follow the rules (never negative). Isolation = two payments at the same moment do not mix up. Durability = once "success" shows, it stays even if the server crashes.

Write this in the exam

An open-source storage layer is software on top of cloud object storage (S3 / ADLS) that adds database-like features (ACID transactions, schema checks) to plain files. It is needed because object storage alone cannot guarantee consistency when many users read and write at the same time.

Delta Lake:

Open-source storage layer created by Databricks that adds ACID transactions on top of data lake storage.
Maintains a transaction log (_delta_log) recording every change, enabling reliable reads and writes.
Supports schema enforcement, schema evolution and time travel.

Apache Iceberg:

Open table format created at Netflix for very large analytic tables.
Hidden partitioning (users do not need to know the partition layout) and safe schema evolution.
Multi-engine support: Spark, Flink, Trino can work on the same table at the same time.

Apache Hudi (Hadoop Upserts Deletes and Incrementals):

Created at Uber for fast-changing datasets with frequent updates and deletes.
Efficient upserts (update + insert), useful for streaming / CDC data.
Incremental processing: only changed data is reprocessed, not the whole dataset.

Basis	Delta Lake	Apache Iceberg	Apache Hudi
Created by	Databricks	Netflix	Uber
Main strength	ACID transactions + time travel	Schema evolution + multi-engine support	Fast upserts on streaming / changing data
Metadata	Transaction log (`_delta_log` JSON files)	Metadata files + snapshots	Timeline of commits
Special feature	Schema enforcement, RESTORE	Hidden partitioning	Incremental queries, record-level updates
Best suited for	Spark-based lakehouses	Large analytic tables across engines	Frequently updated / real-time datasets

Keywords to drop

ACID, transaction log, time travel, hidden partitioning, multi-engine, upsert, incremental processing, Databricks / Netflix / Uber

Schema Enforcement, Schema Evolution and Time Travel (with given data) 💗

teacher: data will be givencase study2 to 5 marks

In simple words

Schema enforcement = a strict gatekeeper. The table says "amount must be a number". If a row arrives with amount = "five hundred" (text), the whole write is rejected, so bad data never gets in.

Schema evolution = a planned, allowed change. You tell the table "a new column city is coming, accept it", and it adds the column.

Time travel = see or bring back an older version of the table, like restoring a WhatsApp chat backup from last week after deleting messages by mistake.

Worked example: an `orders` Delta table

Schema: order_id INT, customer STRING, amount DOUBLE

Version 0: table created. Version 1: two rows inserted.

order_id	customer	amount
101	Riya	500.0
102	Aman	250.0

1. Schema enforcement (bad data arrives)

order_id	customer	amount
103	Kabir	"five hundred" (a string)

bad_df.write.format("delta").mode("append").save("/lake/orders")
# AnalysisException: A schema mismatch detected when writing to the Delta table.
# Failed to merge incompatible data types DoubleType and StringType (column: amount)

Result: the write is rejected, the table stays at version 1 and stays correct. The same happens if the batch has an unexpected extra column.

2. Schema evolution (a planned new column)

new_df.write.format("delta").mode("append") \
      .option("mergeSchema", "true").save("/lake/orders")   # creates version 2

order_id	customer	amount	city (new)
101	Riya	500.0	NULL
102	Aman	250.0	NULL
104	Neha	800.0	Mumbai

Old rows get NULL in the new column; nothing breaks.

3. A mistake, then time travel

-- version 3: someone forgot the WHERE clause
UPDATE orders SET amount = 0;          -- every amount is now 0

DESCRIBE HISTORY orders;               -- lists versions 0, 1, 2, 3 with time and operation

SELECT * FROM orders VERSION AS OF 2;                        -- see the correct data
SELECT * FROM orders TIMESTAMP AS OF '2026-09-20 10:00:00';  -- or by date and time

RESTORE TABLE orders TO VERSION AS OF 2;   -- bring it back (recorded as version 4)

Version	Operation	State
0	CREATE TABLE	empty
1	WRITE (insert 101, 102)	correct
2	WRITE with mergeSchema (104 + city)	correct
3	UPDATE amount = 0	wrong
4	RESTORE to version 2	correct again

Write this in the exam

Schema enforcement: checks incoming data against the table's defined structure (column names, data types) and rejects data that does not match, preventing bad records from corrupting the table. Offered by Delta Lake and Iceberg.
Schema evolution: allows intentional schema changes (e.g. adding a column) using options like mergeSchema, without rewriting the table.
Time travel: lets users query or restore an older version of the table using a version number or timestamp (VERSION AS OF, TIMESTAMP AS OF, RESTORE).
How it works: Delta Lake keeps a transaction log in the _delta_log folder. Every commit is a numbered JSON file (000...000.json, 000...001.json) listing which data files were added or removed. Old data files are kept, so reading version 2 means reading exactly the files listed at version 2. (Old files are deleted only by VACUUM, default retention 7 days.)
Iceberg provides the same using snapshots in its metadata; Hudi using its commit timeline.
Uses: recovering from accidental updates/deletes, audits and compliance, reproducing ML experiments on an old snapshot.

Case study tip (from her notes)

If the scenario says a table was "accidentally updated with wrong data" or "an old snapshot is needed for audit", it is a Time Travel use case. If it says "a new column with a wrong data type is breaking the pipeline" or "prevent bad records from entering the table", it is a Schema Enforcement use case. Always name the storage layer (Delta Lake / Iceberg / Hudi) that offers the feature and state in one line how the transaction log / metadata makes it possible.

Keywords to drop

schema enforcement, schema evolution, mergeSchema, time travel, VERSION AS OF, RESTORE, transaction log, _delta_log, audit

How to Answer a Lakehouse Case Study

teacher: case study aayegacase study4 to 5 marks

In simple words

Every case study hides a few keywords. Each keyword points to one feature or one tool. Spot them, match them with the table below, then justify your choice using the same words the question used.

If the question says...	Answer with
raw + structured data together, one system, lower cost	Lakehouse architecture
multiple teams / tools / engines (Spark, Flink, Trino) on the same tables	Apache Iceberg (multi-engine, schema evolution)
history of changes, audit, compliance, rollback, accidental update	Time travel + transaction log (Delta Lake)
bad records, wrong data type, new column breaking the pipeline	Schema enforcement / evolution
frequent updates and deletes, streaming, CDC, only changed data	Apache Hudi (upserts, incremental processing)
Spark-based company, strong ACID	Delta Lake
only structured BI reporting	Data warehouse
cheap raw storage for ML, no strict rules needed	Data lake

Answer template (5 marks)

Name the architecture (usually Lakehouse) with a one-line definition.
Name the storage layer (Delta / Iceberg / Hudi) and why.
Map each requirement in the question to a feature (one bullet each).
Draw the layered diagram: sources → object storage → table layer → compute → BI/ML.
One-line conclusion on benefits (cost, reliability, one copy of data).

Write this in the exam

Identify the requirements (keywords) in the scenario.
Propose the architecture (Lakehouse / warehouse / lake) with a one-line definition.
Choose the storage layer (Delta Lake / Iceberg / Hudi) and justify it with the scenario's own keywords.
Map each requirement to a feature: ACID, schema enforcement, schema evolution, time travel, upserts, multi-engine support.
Draw the layered architecture diagram and conclude with the benefits.

Case 1 (from her notes, 4M): raw + structured data together, many teams on different tools, history of all changes for compliance

Suggest a Lakehouse architecture, since raw and structured data can live together in one system instead of a separate lake and warehouse.
Suggest Apache Iceberg if multiple engines (Spark, Flink, Trino) must query the same tables, as Iceberg is built for multi-engine compatibility and safe schema evolution.
Suggest Delta Lake if the company is Spark-based and specifically needs strong ACID and time travel for compliance and audit history.
Justify with the keywords: "multiple teams/tools" → schema evolution + multi-engine support; "history of changes" → time travel + transaction log; "raw and structured together" → lakehouse.

Case 2 (practice): a bank's card-transaction data

Question: A bank receives millions of card transactions every hour as a stream. Old transactions are often corrected (chargebacks, refunds), so records must be updated after they land. Analysts run daily fraud reports, the ML team trains fraud models on the same data, and regulators ask "what did the data look like on the 1st of last month". Suggest an architecture and storage layer. Justify.

Model answer:

Architecture: a Lakehouse on cloud object storage (S3 / ADLS), because the same data must serve both BI fraud reports and ML model training at low cost.
Storage layer: Apache Hudi, because transactions arrive as a stream and keep getting corrected; Hudi supports fast upserts (update + insert) and incremental processing, so only changed records are reprocessed.
ACID transactions ensure a refund update and a new transaction written at the same time do not corrupt the table.
Time travel (Hudi's commit timeline) answers the regulator's "as of the 1st of last month" question for audit.
Schema enforcement stops malformed transactions (e.g. text in the amount field) from entering.
If the bank were fully Spark-based with fewer updates, Delta Lake would be an equally valid choice for ACID + time travel.
Diagram: card network stream → ingestion (Kafka) → S3 → Hudi tables → Spark / SQL → fraud dashboard + ML model.

Keywords to drop

justify with keywords, lakehouse, multi-engine, upsert, time travel, audit, ACID, schema enforcement

Module 3 · Stream Data Processing

Data that never stops arriving: live locations, card swipes, clicks. This module is about handling it the moment it comes in, plus the tricks (Bloom filter, FM, DGIM) and the tool (Apache Flink) used for it.

Stream data processing: bounded vs unbounded data 🌸

very likely2 marks

In simple words

Data comes in two shapes. Bounded data has a clear start and end, like last month's sales file. You collect it all, then process it in one go. That is batch processing.

Unbounded data never ends, like the live location of every Uber car. It keeps flowing, so you process each piece as it arrives. That is stream processing. The data is "always available" and "arrives in a stream", one record after another.

Think of it like

A newspaper is batch: yesterday's news, printed once, complete. A live cricket score on Cricbuzz is a stream: every ball updates the score immediately and the match is not over yet.

Basis	Bounded data (Batch)	Unbounded data (Stream)
Start and end	Fixed start and end	Has a start, no end
When processed	After all data is collected	Continuously, as each record arrives
Latency	High (minutes to hours)	Low (milliseconds to seconds)
Data size	Known and finite	Unknown, keeps growing
Example	Monthly salary report, daily backup	UPI fraud check, live GPS tracking
Tools	Hadoop MapReduce, Spark batch	Apache Flink, Kafka Streams, Spark Streaming

Write this in the exam

Stream data processing is the continuous processing of data as soon as it is generated, instead of storing it first and processing it later in batches.
Data is of 2 categories: bounded (fixed start and end, used in batch processing) and unbounded (continuous, no end, used in stream processing).
In streaming, data is always available and arrives as a stream of records; speed (velocity) and storage (volume) needs are very different from batch.
Example: fraud detection on live card transactions, live cab tracking.

Keywords to drop

bounded, unbounded, batch, continuous, low latency, real-time

Key characteristics of the flow of data

2 marks

In simple words

Streaming data behaves differently from a normal file. It keeps coming, comes fast, comes in huge amounts, sometimes out of order, and people expect answers instantly. These 8 points describe that behaviour.

Think of it like

Orders pouring into Zomato on a Friday night: non-stop, very fast, sometimes a delayed order shows up after a newer one, and every customer wants the rider assigned right now.

Write this in the exam

Real-time and continuous: data is generated and flows without stopping.
Ordered or unordered arrival: records may not arrive in the sequence they were created (network delays).
Constantly changing patterns: the nature of data changes over time (traffic at 9 am vs 3 am).
High velocity: data arrives very fast, often thousands of records per second.
High volume: the total amount becomes huge over time.
Real-time response requirement: results are needed within milliseconds or seconds.
Sequential access: data is read once, in the order it arrives; it cannot be revisited freely.
Continuous queries: queries keep running on the stream and keep producing results.

Keywords to drop

velocity, volume, out-of-order, sequential access, continuous query

Real-time systems and their categories

very likely5 marks

In simple words

A real-time system is one that must respond within a fixed time limit. The question is: how bad is it if the answer comes late? Based on that, there are 3 types.

Hard: late means disaster (a pacemaker). Soft: late is annoying but the system still works (a stock price that updates a second late). Near real-time: a few seconds or minutes of delay is fine (a Skype video call that buffers a little).

Think of it like

An airbag must open in milliseconds (hard). A WhatsApp "typing..." status can lag a bit (soft). Your Swiggy order status updating every few seconds is fine (near real-time).

Classification	Latency	Tolerance for delay	Example
Hard real-time	Microseconds to milliseconds	None: a delay means total system failure	Pacemaker, airbag, aircraft control
Soft real-time	Milliseconds to seconds	Low: a delay lowers quality but no system failure	Online stock quotes, online gaming
Near real-time	Seconds to minutes	High: delays are acceptable, no system failure	Skype video, dashboards, social media feeds

Write this in the exam

A real-time system is a system that must process input and produce a response within a strict, specified time limit (deadline).
Correctness depends on both the result and the time at which the result is produced.
Based on tolerance for delay, real-time systems are classified as hard, soft and near real-time (draw the table above).
Hard: latency in micro/milliseconds, zero tolerance, missing a deadline causes total failure. Example: pacemaker.
Soft: latency in milliseconds to seconds, low tolerance, a missed deadline reduces quality only. Example: online stock quotes.
Near real-time: latency in seconds to minutes, high tolerance. Example: Skype video.
A streaming data system is a non-hard real-time system (soft or near real-time): it responds quickly, but a small delay does not crash the system.

Exam tip

For 5 marks, the table plus one line each and the line "streaming data systems are non-hard real-time" gets full marks.

Keywords to drop

deadline, latency, tolerance for delay, hard / soft / near, non-hard real-time

Types of queries on stream data: standard vs ad-hoc

very likely2 marks

In simple words

You can ask a stream two kinds of questions. A standard (standing) query is set up once and keeps running forever, giving a fresh answer every time new data comes. An ad-hoc query is a one-time question asked at some moment about what is happening right now.

Think of it like

A standing query is a Google alert: "tell me every time my name appears". An ad-hoc query is typing a question into Google once: "what is the weather right now?".

Basis	Standard / Standing query	Ad-hoc query
When created	Registered in advance, stored in the query repository	Asked on the spot by a user
How long it runs	Runs continuously, forever	Runs once, then ends
Answer	Output stream that keeps updating	A single answer about the current state
Storage need	System knows it in advance, so it keeps only the summary it needs	Not known in advance, so the system must keep summaries or a window of recent data
Example	"Alert when temperature of any sensor crosses 50°C"	"How many cars crossed junction 4 in the last 10 minutes?"

Write this in the exam

Queries on stream data are of two types: standing (standard / continuous) queries and ad-hoc queries.
Standing query: permanently stored in the system and executed continuously on the incoming stream; produces output whenever new data arrives. Example: report the maximum temperature every hour.
Ad-hoc query: asked once about the current state of the stream; not known in advance, so the DSMS keeps a sliding window or summary of recent data to answer it. Example: number of unique users in the last 5 minutes.
Both are handled by the query processor of the DSMS.

Keywords to drop

standing query, ad-hoc query, continuous, query repository, sliding window

DSMS architecture (Data Stream Management System) 💗

very likelydraw diagram2 / 5 marks

In simple words

A normal database (DBMS) stores data first and answers questions later. A DSMS is built for streams: data keeps flowing in, the system cannot store everything, so it keeps small summaries and keeps answering questions on the fly.

Data enters through a gatekeeper (the stream input regulator), gets kept in three kinds of storage, and a query processor keeps running the saved questions and sends answers out as output streams.

Think of it like

A traffic control room. Camera feeds pour in (input streams). An operator filters out useless footage (regulator). A whiteboard shows live counts (working storage), a register keeps hourly totals (summary storage), a rulebook says which camera is where (metadata). A set of standing orders like "alert if jam longer than 1 km" (query repository) is checked non-stop (query processor), and alerts go out (output stream).

Abstract architecture of a DSMS (same as the class notes). Draw this first in the exam.

Write this in the exam (2 marks: sketch + define elements)

Input streams: continuous, unbounded sequences of data records entering the system from many sources.
Stream input regulator: controls and filters incoming data (e.g. using a Bloom filter), drops irrelevant records and manages the arrival rate.
Working storage: fast, temporary memory holding the current window of recent data used by running queries.
Summary storage: keeps compact summaries (counts, averages, sketches) of old data, since the full stream cannot be stored.
Metadata storage: stores information about the streams and queries: schema, sources, windows, timestamps.
Query repository: stores the registered standing queries (and accepts ad-hoc user queries).
Query processor: executes the queries continuously on the storage and produces results.
Output streams: the continuous results sent to users, dashboards or alert systems.

5-mark version: DSMS for a Smart City Traffic Management case study

Step 1: Identify the sources of data. Traffic cameras, GPS devices in cars, roadside sensors (speed and vehicle count), public transport system (bus and metro locations), mobile app user reports (accident, pothole, jam).

Step 2: Identify data velocity and processing needs.

Source	Velocity	Processing need
Traffic cameras	Very high (video frames every second)	Near real-time vehicle counting, accident detection
GPS devices in cars	High (location every few seconds per car)	Average speed per road, jam detection
Roadside sensors	High, steady	Vehicle count per junction per minute
Public transport system	Medium	Bus and metro arrival estimates
Mobile app reports	Low, irregular (bursty)	Verify and broadcast incident alerts

Step 3: Map each DSMS element to the city.

Input streams: the 5 sources above flow in continuously.
Stream input regulator: removes duplicate GPS pings and blurry frames, drops fake app reports, controls the arrival rate at rush hour.
Working storage: the last 5 minutes of speeds and counts for every junction.
Summary storage: hourly and daily average traffic per road, used for comparison and planning.
Metadata storage: location of each camera and sensor, road map, junction IDs, schema of each stream.
Query repository and query processor: standing query "average speed per junction every 1 minute; alert if below 10 km/h"; ad-hoc query from the control room "how many buses are near station X right now?".
Output streams: signal timing changes, jam alerts on Google Maps style apps, live dashboard for traffic police.

Exam tip

For the case-study version: draw the diagram, write the two steps (sources, velocity), then map every box to the case. That mapping is where the marks are.

Keywords to drop

stream input regulator, working / summary / metadata storage, query repository, query processor, output stream, standing query

Stream data processing algorithms: the big 3

2 marks

In simple words

A stream is too big to store fully, so these algorithms answer questions using very little memory. Each one answers a different question.

Algorithm	Question it answers	Used for	Example
Bloom filter	"Have I seen this item before? Is it in my set?"	Membership test, input regulation (filtering)	Is this email address on the spam blacklist?
DGIM method	"How many 1s are in the last N bits?"	Counting in a sliding window	How many failed logins in the last 1000 attempts?
FM (Flajolet-Martin)	"How many different items have I seen?"	Counting distinct elements	How many unique visitors came to the website today?

Write this in the exam

Bloom filter: a probabilistic data structure that uses a bit array and k hash functions to test whether an element belongs to a set; answers "probably yes" or "definitely no". Used for input regulation.
DGIM (Datar-Gionis-Indyk-Motwani) method: estimates the number of 1s in the last N bits of a binary stream using buckets whose sizes are powers of 2, with O(log² N) memory.
FM (Flajolet-Martin) algorithm: estimates the number of distinct elements in a stream using hashing and the maximum number of trailing zeros R; estimate = 2^R.

Keywords to drop

membership, sliding window count, distinct count, probabilistic, low memory

Bloom filter (with solved numericals) 🌸

very likelynumerical2 / 5 marks

In simple words

A Bloom filter answers one question very fast using very little memory: "Is this item in my list?"

Instead of storing the whole list, it keeps a row of light switches (a bit array, all OFF = 0 at the start). For every item in the list, a few formulas (hash functions) each point to one switch, and those switches are turned ON (1).

To check a new item, run the same formulas. If any switch it points to is OFF, the item is definitely not in the list. If all are ON, it is probably in the list (maybe other items turned those switches on, so it can be wrong: a false positive). It never says "no" to an item that really is in the list: no false negatives.

Think of it like

A wedding bouncer who did not memorise the full guest list. He only remembers a few clues per guest ("wears glasses, from Pune, surname starts with S"). If you miss even one clue, you are surely not on the list. If you match all clues, you are probably a guest, though a stranger could match by luck.

Steps to solve any Bloom filter question

Draw the bit array of size m and set all bits to 0.
Write the given hash functions (e.g. H1(x) = (3x + 3) mod 6).
Apply every hash function to every element of the set; make a table.
Set the bit at each calculated index to 1. Write the final array.
For each test element, calculate the hash values again.
If all those bits are 1 → "may be present (probably yes)". If any bit is 0 → "definitely not present".

Worked example 1 (class notes example)

Question: A Bloom filter uses a bit array of size 8 (index 0 to 7) and four hash functions: H1(x) = (3x + 3) mod 6, H2(x) = (3x + 7) mod 8, H3(x) = (2x + 9) mod 2, H4(x) = (2x + 3) mod 5. Insert the set {8, 10}. Check whether 50 and 62 are present.

Step 1: Initial array: all zeros.

Index	0	1	2	3	4	5	6	7
Bit	0	0	0	0	0	0	0	0

Step 2: Apply the hash functions to the set elements.

Hash function	x = 8	x = 10
H1 = (3x + 3) mod 6	27 mod 6 = 3	33 mod 6 = 3
H2 = (3x + 7) mod 8	31 mod 8 = 7	37 mod 8 = 5
H3 = (2x + 9) mod 2	25 mod 2 = 1	29 mod 2 = 1
H4 = (2x + 3) mod 5	19 mod 5 = 4	23 mod 5 = 3

Step 3: Set bits 3, 7, 1, 4 (from 8) and 3, 5, 1, 3 (from 10) to 1. Final array:

Index	0	1	2	3	4	5	6	7
Bit	0	1	0	1	1	1	0	1

Step 4: Test the elements.

Hash function	x = 50	Bit at index	x = 62	Bit at index
H1 = (3x + 3) mod 6	153 mod 6 = 3	1	189 mod 6 = 3	1
H2 = (3x + 7) mod 8	157 mod 8 = 5	1	193 mod 8 = 1	1
H3 = (2x + 9) mod 2	109 mod 2 = 1	1	133 mod 2 = 1	1
H4 = (2x + 3) mod 5	103 mod 5 = 3	1	127 mod 5 = 2	0

Result:

50: bits 3, 5, 1, 3 are all 1 → 50 may be present in the set. (50 was never inserted, so this is a false positive: the known weakness of a Bloom filter.)
62: bit 2 is 0 → 62 is definitely not present in the set.

Note: the notebook shows H2(50) = 1. The correct value is 3 × 50 + 7 = 157, and 157 mod 8 = 5. The final answer for 50 stays the same.

Worked example 2 (practice)

Question: Bit array of size 10 (index 0 to 9). h1(x) = (x + 1) mod 10, h2(x) = (3x + 2) mod 10. Insert {5, 12, 20}. Check 12, 7 and 31.

Element	h1 = (x + 1) mod 10	h2 = (3x + 2) mod 10	Bits set
5	6 mod 10 = 6	17 mod 10 = 7	6, 7
12	13 mod 10 = 3	38 mod 10 = 8	3, 8
20	21 mod 10 = 1	62 mod 10 = 2	1, 2

Index	0	1	2	3	4	5	6	7	8	9
Bit	0	1	1	1	0	0	1	1	1	0

Test	h1	h2	Bits	Verdict
12	3	8	1, 1	May be present (true: 12 is in the set)
7	8 mod 10 = 8	23 mod 10 = 3	1, 1	May be present (false positive: 7 was never inserted)
31	32 mod 10 = 2	95 mod 10 = 5	1, 0	Definitely not present

PYQ (2 marks): Spam filter with a blacklist of 10 million addresses. Bloom filter or FM?

Answer: Bloom filter.

The task is a membership test ("is this email address in the blacklist?"). A Bloom filter is built exactly for membership testing. FM only counts distinct elements; it cannot say whether a particular address is in a set.
A Bloom filter stores only a bit array, not the addresses. For 10 million addresses with about 1% false positives it needs roughly 96 million bits ≈ 12 MB and about 7 hash functions, far less than storing every address.
Lookup is very fast: k hash calculations, O(k), independent of list size.
No false negatives: a blacklisted address is never let through. A rare false positive can be double-checked against the full list kept on disk.

Write this in the exam (theory)

A Bloom filter is a space-efficient probabilistic data structure used to test whether an element is a member of a set.
It consists of a bit array of m bits (initially 0) and k independent hash functions.
Insert: hash the element with all k functions and set those k bits to 1.
Query: hash the element; if all k bits are 1 → "probably present"; if any bit is 0 → "definitely not present".
False positives are possible, false negatives are not. Elements cannot be deleted in a basic Bloom filter.
In a DSMS it is used for input regulation / filtering. Applications: spam filtering, malicious URL check, username availability, cache lookups.

Keywords to drop

probabilistic, bit array, k hash functions, false positive, no false negative, membership test, space-efficient

FM (Flajolet-Martin) algorithm (with solved PYQ) 💗

very likelynumerical2 / 5 marks

In simple words

FM answers: "How many different items have come in the stream?" without remembering all of them.

Each item is turned into a number by a hash function, and that number is written in binary. Count the zeros at the right end (the tail length, or trailing zeros). Seeing a long tail of zeros is rare: a number ending in "00" appears about 1 in 4 times, "000" about 1 in 8 times. So if the longest tail seen is R, there were roughly 2^R different items.

Think of it like

Counting unique visitors at a mall without a register. If someone tells you "today I met a person whose phone number ended in 0000", you guess a lot of different people must have come, because that is rare. The rarer the pattern seen, the bigger the crowd.

Steps to solve

For each element x in the stream, calculate h(x).
Write h(x) in binary (use enough bits for the mod value, e.g. mod 5 → 3 bits, mod 32 → 5 bits).
Find the tail length r(x) = number of trailing zeros (zeros at the right end).
R = maximum tail length over all elements.
Estimated number of distinct elements = 2^R.

PYQ solved: stream 3, 1, 4, 1, 5, 2, 6, 5 with h(x) = (2x + 1) mod 5

x	2x + 1	h(x) = (2x + 1) mod 5	Binary (3 bits)	Tail length r
3	7	2	010	1
1	3	3	011	0
4	9	4	100	2
1	3	3	011	0
5	11	1	001	0
2	5	0	000	0 (by convention)
6	13	3	011	0
5	11	1	001	0

Maximum tail length R = 2 (from x = 4, h = 100).
Estimated distinct elements = 2^R = 2² = 4.
Actual distinct elements = {3, 1, 4, 5, 2, 6} = 6. FM gives an estimate, not the exact count.

Exam tip

h(x) = 0 has no 1 bit, so the tail length is undefined. The usual class convention takes it as 0 (answer: R = 2, estimate 4). Some teachers count 000 as 3 trailing zeros, which gives R = 3 and estimate 8. Write one line saying which convention you used, then solve.

Practice example

Question: Stream 4, 5, 6, 4, 7, 5, 8 with h(x) = (3x + 2) mod 16 (4-bit binary). Estimate the distinct count.

x	3x + 2	h(x)	Binary (4 bits)	Tail length r
4	14	14	1110	1
5	17	1	0001	0
6	20	4	0100	2
4	14	14	1110	1
7	23	7	0111	0
5	17	1	0001	0
8	26	10	1010	1

R = 2 → estimate = 2² = 4 (actual distinct = {4, 5, 6, 7, 8} = 5).

Write this in the exam (theory)

The Flajolet-Martin (FM) algorithm estimates the number of distinct elements in a data stream in a single pass using very little memory.
Each element is hashed to a bit string; r(x) is the number of trailing zeros (tail length) of h(x).
R = max r(x) over the stream; estimated distinct count = 2^R.
Reason: the probability that a hash value ends in r zeros is 1/2^r, so seeing a tail of R zeros suggests about 2^R distinct elements.
Duplicates hash to the same value, so repeats do not change the estimate.
Accuracy is improved by using many hash functions, grouping them, taking the average within groups and the median across groups.

Keywords to drop

distinct elements, hash function, trailing zeros / tail length, R = max r, 2^R, single pass

DGIM method: counting 1s in a sliding window

numerical2 marks

In simple words

The stream is a long line of 0s and 1s (1 = "an event happened", like a failed login). Question: how many 1s in the last N bits? Storing all N bits is too costly, so DGIM groups the 1s into buckets. Each bucket holds 1, 2, 4, 8... ones (powers of 2) and only remembers its size and the time of its newest 1.

To estimate: add all bucket sizes, but take only half of the oldest bucket, because part of it may already be outside the window.

Think of it like

Counting coins by stacking them into piles of 1, 2, 4, 8. You never keep more than two piles of the same size: when a third appears, the two oldest are merged into one bigger pile.

DGIM rules

The right end of every bucket is a 1; a bucket of size s contains exactly s ones.
Every bucket size is a power of 2 (1, 2, 4, 8, ...).
There are one or two buckets of each size, never three.
Sizes never decrease as you go back in time (older buckets are bigger or equal).
A bucket is dropped when its end time falls outside the window of N bits.
When a new 1 arrives: create a bucket of size 1; if there are now three buckets of the same size, merge the two oldest into one of double size (repeat upward).

Worked example

Window N = 20, current time t = 100. Buckets (size, end time), oldest to newest: (4, 83), (2, 88), (2, 92), (1, 96), (1, 99). All end times are after 100 − 20 = 80, so all are inside the window.

Estimate of 1s = 1 + 1 + 2 + 2 + (4 ÷ 2) = 8. (The true count lies between 7 and 10.)

A new 1 arrives at t = 101:

Step	Buckets (oldest → newest)
Add (1, 101)	(4,83) (2,88) (2,92) (1,96) (1,99) (1,101): three buckets of size 1
Merge (1,96) + (1,99) → (2,99)	(4,83) (2,88) (2,92) (2,99) (1,101): three buckets of size 2
Merge (2,88) + (2,92) → (4,92)	(4,83) (4,92) (2,99) (1,101): valid

New estimate = 1 + 2 + 4 + (4 ÷ 2) = 9.

Write this in the exam

DGIM (Datar-Gionis-Indyk-Motwani) estimates the number of 1s in the last N bits of a binary stream.
The window is divided into buckets; each bucket stores its size (a power of 2) and the timestamp of its most recent 1.
At most two buckets of each size are kept; when three exist, the two oldest are merged.
Estimate = sum of sizes of all buckets except the oldest + half the size of the oldest bucket.
Memory needed is only O(log² N) bits, and the error is at most 50%.

Keywords to drop

sliding window, buckets, power of 2, timestamp, merge, half of the oldest bucket

What is Apache Flink? 🌸

very likely2 marks

In simple words

Apache Flink is free, open-source software that processes streams of data the moment they arrive. It can also process normal files (batch), but live streams are its speciality.

In Flink, every piece of data is an event: a small record of something that happened ("Riya paid ₹250 at 7:02 pm", "car 42 is at this location"). Flink keeps reading events, calculates results in milliseconds, and remembers what it has seen so far (state), such as a running total.

Think of it like

A cashier at a busy counter who updates the day's total after every single bill, instead of adding up all the bills at night. If the power goes off, the cashier's notebook (checkpoint) lets the counting restart from the right place, so no bill is counted twice.

Write this in the exam

Apache Flink is an open-source, distributed stream processing framework for stateful computations over unbounded and bounded data streams.
An event is a piece of data that represents something that happened at a point in time.
Features: very low latency and high throughput; stateful processing; exactly-once state consistency using checkpoints; fault tolerance; runs on YARN, Kubernetes or standalone.
Supports event time (when the event actually happened) as well as processing time (when Flink sees it), so out-of-order data is handled correctly.
Watermarks tell Flink "no older events are expected now", so a time window can be closed and its result produced.
Use cases: fraud detection, real-time dashboards, live recommendations, IoT sensor monitoring.

Keywords to drop

stateful, event, unbounded / bounded, low latency, exactly-once, checkpoint, event time, watermark

Flink architecture 💗

very likelydraw diagram5 marks

In simple words

Flink works like a small company. The Client is the customer who hands over the job. The JobManager is the manager (master) who plans the work and hands it out. The TaskManagers are the workers who actually do the processing. Each worker has a few task slots: think of them as desks, one piece of work per desk.

Inside the JobManager are 3 helpers: the Dispatcher (front desk that accepts jobs), the ResourceManager (hands out desks, i.e. task slots) and the JobMaster (supervises one particular job).

Flink runtime: Client → JobManager (Dispatcher, ResourceManager, JobMaster) → TaskManagers with task slots.

Write this in the exam

Flink follows a master-worker architecture with three main parts: Client, JobManager and TaskManagers.
Client: not part of the runtime; it prepares the program, converts it into a dataflow graph (JobGraph) and submits it to the JobManager. It initialises the process.
JobManager (master): coordinates the distributed execution: schedules tasks, reacts to failures, coordinates checkpoints. Its components:
- ResourceManager: allocates and de-allocates task slots; integrates with YARN / Kubernetes / standalone.
- Dispatcher: provides a REST interface to submit jobs and starts a new JobMaster for each job; runs the Web UI.
- JobMaster: manages the execution of a single JobGraph.
TaskManagers (workers): execute the tasks of the dataflow; they buffer and exchange data streams with each other. Each TaskManager has task slots, the smallest unit of resource scheduling, which decide the parallelism.
A task ≈ an operation ≈ a transformation (map, filter, window...) running in parallel.

Keywords to drop

master-worker, JobGraph, JobManager, ResourceManager, Dispatcher, JobMaster, TaskManager, task slot, parallelism

Flink layered API (levels of abstraction)

very likelydraw diagram5 marks

In simple words

Flink offers 4 ways to write a program, stacked like floors of a building. The bottom floor gives total control but needs the most code. Each floor up is easier and shorter to write, but gives less fine control. At the top you write plain SQL.

Think of it like

Getting food: cooking from raw ingredients (ProcessFunction), using a recipe kit (DataStream API), ordering from a menu with options (Table API), or saying "one thali please" (SQL).

Flink's layered APIs: higher layers are easier, lower layers give more control.

Write this in the exam

Flink provides 4 levels of abstraction for developing streaming and batch applications.
1. Stateful Stream Processing (ProcessFunction), lowest level: gives full control over events, state and time (timers). Most expressive but most code.
2. DataStream / DataSet API (core APIs): common building blocks such as map, filter, keyBy, window, join, aggregate. Most Flink programs are written here.
3. Table API: declarative, works on tables with a schema; operations like select, groupBy, join; programs are optimised automatically.
4. SQL, highest level: same meaning as the Table API but written as SQL queries; most concise.
Moving up: more concise and easier; moving down: more expressive and flexible. Layers can be mixed in one program.

A tiny code idea for each layer

ProcessFunction (full control, e.g. alert on a big payment):

public class BigTxnAlert extends KeyedProcessFunction<String, Txn, String> {
  public void processElement(Txn t, Context ctx, Collector<String> out) {
    if (t.amount > 50000) out.collect("Check card " + t.card);
  }
}

DataStream API (total amount per city every minute):

payments
  .filter(p -> p.amount >= 10)
  .keyBy(p -> p.city)
  .window(TumblingEventTimeWindows.of(Time.minutes(1)))
  .sum("amount");

Table API (clicks per user):

clicks.groupBy($("user"))
      .select($("user"), $("url").count().as("cnt"));

SQL (clicks per user in 1-minute tumbling windows):

SELECT user_id, COUNT(*) AS clicks
FROM clicks
GROUP BY TUMBLE(ts, INTERVAL '1' MINUTE), user_id;

Keywords to drop

levels of abstraction, ProcessFunction, DataStream API, Table API, SQL, expressive vs concise

What is a data stream?

theory2 marks

In simple words

A data stream is a never-ending line of records arriving one after another over time. Each record usually carries a timestamp: the moment the event really happened.

Think of it like

The ID-card swipes at the college gate: every swipe creates one new record, all day, every day, as long as the gate is open.

Swipe at 9:01Swipe at 9:01Swipe at 9:03...

Write this in the exam

A data stream is a continuous, never-ending sequence of data records (events) generated over time, one after another, instead of arriving all at once.
Each record usually has a timestamp showing when the event actually happened.
Data streams are unbounded: they have no fixed end; new data keeps arriving as long as the source is active.
Records may arrive out of order and must usually be processed in a single pass.
Example: every click on a website, live GPS updates from a delivery rider, UPI transactions.

Keywords to drop

continuous, unbounded, timestamp, event, single pass

Windowing: tumbling, sliding, session 🌸

very likelydraw diagram2 / 5 marks

In simple words

A stream never ends, so a question like "total orders" never gets a final answer. Windowing cuts the endless stream into small finite pieces (windows), and the calculation (sum, average, count) is done per piece. Then "orders in the last 1 minute" has a real answer.

Tumbling window: fixed size, no overlap, back to back (every 1 minute: 10:00 to 10:01, 10:01 to 10:02...).
Sliding window: fixed size, but starts every "slide" interval, so windows overlap (last 10 minutes, updated every 5 minutes).
Session window: no fixed size; a window stays open while activity continues and closes after a gap of inactivity.
Global window: one window for everything, closed by a custom trigger. Count windows close after N elements (e.g. every 100 events) instead of after a time.

Think of it like

Tumbling: hourly news bulletins, each covering its own hour. Sliding: "top songs of the last 7 days", refreshed daily, so days overlap. Session: a chat conversation that ends when nobody types for 10 minutes.

Same events (dots), three ways to group them into windows.

Basis	Tumbling	Sliding	Session
Size	Fixed	Fixed	Not fixed
Overlap	No	Yes (when slide < size)	No
Defined by	Window size	Window size + slide	Inactivity gap
One event belongs to	Exactly 1 window	1 or more windows	Exactly 1 session
Example	Orders per minute	Average speed over last 10 min, every 1 min	One user's browsing visit

Write this in the exam

Since a data stream is unbounded, windowing divides the stream into finite chunks called windows so that aggregations (sum, count, average) can be computed.
Tumbling window: fixed-size, non-overlapping windows; each event belongs to exactly one window. Example: total orders every 1 minute.
Sliding window: fixed size with a slide interval; windows overlap, so one event can belong to many windows. Example: last 10 minutes, updated every 5 minutes.
Session window: groups events by periods of activity; a window closes after an inactivity gap. Example: a user's website session.
Global / count windows: one window for all data with a custom trigger, or windows of N elements.
Windows can be based on event time or processing time; watermarks decide when an event-time window is complete.

Keywords to drop

finite chunks, tumbling, sliding, session gap, count window, aggregation, watermark

Transformations in stream processing

very likely2 / 5 marks

In simple words

Transformations are the actions applied to the stream to turn raw data into the answer: change each record (map), keep only some records (filter), group them (keyBy), and combine them (reduce / aggregate). Chained together, they form the pipeline.

Think of it like

Sorting mangoes at a market: throw away the rotten ones (filter), peel each one (map), group by variety (keyBy), weigh each group (reduce / sum).

Transformation	What it does	Example on a payments stream
map	One record in → one changed record out	Convert amount from ₹ to $
flatMap	One record in → zero, one or many out	Split a sentence into words
filter	Keeps only records that meet a condition	Keep payments ≥ ₹10
keyBy	Groups records by a key	Group payments by city
reduce / aggregate	Combines records into one value (sum, count, min, max, avg)	Total amount per city
window	Applies the calculation to a window of records	Total per city every 1 minute
union / join	Merges two streams / matches records from two streams	Join payments with a fraud-list stream

Payments streamfilter (≥ ₹10)map (amount)keyBy (city)window (1 min)sumOutput

Write this in the exam

Transformations are operations applied on a data stream to convert, filter or combine records into the required output.
Common transformations: map (change each record), flatMap (one to many), filter (keep records meeting a condition), keyBy (partition by key), reduce / aggregate (sum, count), window, union / join.
They are applied directly on streams or inside windows, and are chained to form a dataflow pipeline.
Example: filter out transactions below ₹10 from a live payment stream, then map the rest to their amount and sum per city every minute.

5 marks: Differentiate Data Streams, Windowing and Transformations

Basis	Data Streams	Windowing	Transformations
Meaning	Continuous, unbounded flow of data records arriving over time	Technique to split an unbounded stream into finite, manageable chunks	Operations applied to data to convert, filter or combine it
Purpose	To represent real-time data as it is generated	To allow calculations (sum, avg, count) on an endless stream	To produce the actual useful output from raw data
Nature	Raw, continuous input (the source)	A grouping mechanism applied on the stream	A processing step applied on the stream or window
Example	Live GPS location updates from a delivery rider	Grouping the GPS updates every 30 seconds into one window	Calculating the average speed from the GPS points in that window
Dependency	Exists independently; the base of the whole pipeline	Needs a data stream to operate on	Can be applied directly on a stream or after windowing

Keywords to drop

map, flatMap, filter, keyBy, reduce, aggregate, window, join, dataflow

Distributed data processing: Hadoop ecosystem and MapReduce

bonus: came in last year's paperdraw diagram5 marks

In simple words

When data is too big for one computer, it is split across many computers that work together. Hadoop is the classic toolkit for this. HDFS stores the big file in pieces (blocks) across machines. MapReduce processes those pieces in parallel: every machine works on its own piece (Map), then the partial results are grouped and combined (Reduce).

Think of it like

Counting votes in an election. Each booth counts its own votes (map), all counts for the same candidate are collected together (shuffle), and a final counter adds them up per candidate (reduce).

Component	Job in one line
HDFS	Distributed file system: stores huge files as blocks (128 MB) with 3 copies each
YARN	Resource manager: decides which job gets how much CPU and memory
MapReduce	Batch processing model: Map (process pieces in parallel) then Reduce (combine)
Hive	SQL-like queries (HiveQL) on data in HDFS; for reports and analysis
Pig	Scripting language (Pig Latin) for data transformation and cleaning
HBase	NoSQL column-oriented database on HDFS for fast, random read/write
Sqoop	Imports / exports structured data between RDBMS (MySQL, Oracle) and HDFS
Flume	Collects and moves streaming log / event data (e.g. social media feeds) into HDFS
Kafka	Distributed messaging system for real-time data streams
Spark	Fast in-memory processing engine (batch, streaming, ML, SQL)
ZooKeeper	Coordination service: configuration, naming, synchronisation of the cluster
Oozie	Workflow scheduler: runs Hadoop jobs in order, on a schedule
Mahout	Machine learning library (clustering, classification, recommendations)

MapReduce word count: the standard diagram to draw.

Write this in the exam

Distributed data processing splits a large dataset across many machines (nodes) of a cluster and processes the parts in parallel.
MapReduce is Hadoop's programming model with two phases: Map converts each input split into key-value pairs; Reduce aggregates all values of the same key.
Phases: Input → Splitting → Mapping → Shuffling & Sorting (group by key) → Reducing → Output.
Data locality: the computation is sent to the node where the data block is stored, reducing network traffic.
Fault tolerance: HDFS keeps 3 replicas of each block; failed tasks are re-run on another node.

Keywords to drop

HDFS, YARN, Map, Reduce, shuffle and sort, key-value pairs, data locality, replication factor 3

HDFS blocks and mappers numerical (PYQ)

numerical2 marks

In simple words

HDFS cuts every big file into fixed-size pieces called blocks (default 128 MB). MapReduce starts one mapper per block (one input split per block). So: number of blocks = file size ÷ block size, rounded up, and minimum mappers = number of blocks.

Rule

Convert both sizes to the same unit (1 GB = 1024 MB, 1 TB = 1024 GB).
Blocks = ⌈file size ÷ block size⌉ (ceiling: any leftover makes one extra, smaller block).
Minimum mappers = number of blocks (one mapper per input split).
If replication is asked: stored block copies = blocks × replication factor (default 3).

PYQ solved: 50 GB file, block size 128 MB

File size = 50 GB = 50 × 1024 MB = 51,200 MB
(i) Total blocks = 51,200 ÷ 128 = 400 blocks
(ii) Minimum mappers = one per block = 400 mappers

Practice variants

Question	Working	Answer
1 TB file, 128 MB blocks	1 TB = 1024 × 1024 MB = 1,048,576 MB; ÷ 128	8,192 blocks, 8,192 mappers
500 MB file, 128 MB blocks, replication 3	500 ÷ 128 = 3.9 → ceil = 4 (3 full blocks of 128 MB + 1 block of 500 − 384 = 116 MB)	4 blocks, 4 mappers, 4 × 3 = 12 stored copies

Exam tip

Show the unit conversion line, it carries marks. The last smaller block uses only its real size on disk (116 MB, not 128 MB). If a teacher uses 1 GB = 1000 MB, 50 GB gives 50,000 ÷ 128 = 390.6 → 391 blocks; write the conversion you used.

Write this in the exam

Number of blocks = ⌈File size ÷ Block size⌉.
50 GB = 50 × 1024 = 51,200 MB; 51,200 ÷ 128 = 400 blocks.
Each block is one input split and each split is processed by one mapper, so minimum mappers = 400.

Keywords to drop

block size 128 MB, input split, one mapper per split, ceiling, replication factor

Kafka in simple words

2 marks

In simple words

Kafka is a super-fast post office for live data. Apps that create data (producers) drop messages into named boxes (topics). Apps that need the data (consumers) read from those boxes at their own speed. Messages are kept for a while, so a slow reader does not lose anything.

Think of it like

A WhatsApp group per subject: senders post (producers), members read whenever they open it (consumers), and the chat history stays for a while.

ProducersKafka brokers (topics → partitions)Consumers

Write this in the exam

Apache Kafka is a distributed, fault-tolerant publish-subscribe messaging / event streaming platform for high-throughput real-time data.
Producer: publishes records to a topic. Consumer: subscribes to topics and reads records; consumers in a consumer group share the work.
Topic: a named category of messages. Partition: a topic is split into ordered partitions spread across brokers for parallelism; each record has an offset.
Broker: a Kafka server storing partitions; a cluster has many brokers, and partitions are replicated for fault tolerance.
Cluster metadata is managed by ZooKeeper (older versions) or KRaft (newer versions). Kafka often feeds Flink or Spark for processing.

Keywords to drop

publish-subscribe, producer, consumer group, topic, partition, offset, broker, replication

Case study (PYQ 5M): Hadoop ecosystem for a retail company 💗

case studydraw diagram5 marks

In simple words

The company has two kinds of data: neat tables of purchases (structured, sitting in a normal database) and messy social media comments (unstructured text). The answer picks one Hadoop tool for each step: bring the data in, store it, process it, analyse it, and says why each tool fits.

Think of it like

A kitchen: Sqoop and Flume are the delivery vans, HDFS is the big storeroom, Spark and MapReduce are the cooks, Hive and Mahout are the waiters who serve the final dish (reports and insights), YARN is the kitchen manager.

Draw this flow first, then write the table.

Stage	Component	Role	Why (justification)
Ingestion	Sqoop	Imports purchase history from the RDBMS into HDFS	Built for bulk transfer of structured tables between RDBMS and Hadoop
Ingestion	Flume / Kafka	Collects social media comments continuously	Designed for high-volume streaming event and log data
Storage	HDFS	Stores both structured and unstructured data	Cheap, scalable, fault tolerant (3 replicas), any format
Storage	HBase	Fast lookup of a customer's latest purchases / profile	NoSQL, random real-time read/write on top of HDFS
Processing	MapReduce / Spark	Cleans data, joins purchases with comments, runs sentiment scoring	Parallel processing of huge data; Spark is faster (in-memory)
Analysis	Hive	SQL queries: sales by region, top products, repeat customers	Analysts already know SQL; HiveQL runs on HDFS data
Analysis	Pig	Scripts to transform and clean the text comments	Less code than raw MapReduce for data flows
Analysis	Mahout / Spark MLlib	Sentiment classification, product recommendations, customer segments	Ready machine learning algorithms that scale on the cluster
Management	YARN, Oozie, ZooKeeper	Resource allocation, daily job scheduling, cluster coordination	Keep the whole pipeline running reliably

Write this in the exam

The company has structured data (purchase history in an RDBMS) and unstructured data (social media comments), so the Hadoop ecosystem is suitable because it handles both at scale.
Ingestion: Sqoop imports the structured purchase tables; Flume (or Kafka) streams the social media comments into Hadoop.
Storage: HDFS stores both types cheaply with replication; HBase gives fast real-time access to customer records.
Processing: MapReduce or Spark cleans and joins the data and computes sentiment of comments in parallel.
Analysis: Hive for SQL-style sales reports; Pig for transformation scripts; Mahout / Spark MLlib for sentiment analysis and recommendations.
Management: YARN allocates resources, Oozie schedules the daily workflow, ZooKeeper coordinates the cluster.
Outcome: link what customers buy with what they say, to improve products, stock and marketing.

Exam tip

Group the answer under the three words in the question: storage, processing, analysis. Each tool gets one line of role and one line of "why". The diagram earns easy marks.

Keywords to drop

structured vs unstructured, Sqoop, Flume, HDFS, HBase, Spark, Hive, Mahout, YARN, justification

60:00

Vivekanand Education Society's Institute of Technology, Chembur

Data Engineering · Mid Term Test 2025-26

12/09/2025Time: 1 hourMarks: 20CO1 20% · CO2 40% · CO3 40%

Q.1 Attempt any FIVE of the following 10 marks

a) Create a Cypher query to create a relationship TEACHES from a Teacher node named "Meera" to a Course node named "Data Engineering". 2MCO2

If both nodes already exist, find them with MATCH and join them with CREATE:

MATCH (t:Teacher {name: "Meera"}),
      (c:Course  {name: "Data Engineering"})
CREATE (t)-[:TEACHES]->(c);

If the nodes do not exist yet, one statement creates both nodes and the relationship:

CREATE (t:Teacher {name: "Meera"})-[:TEACHES]->(c:Course {name: "Data Engineering"});

(t:Teacher {name:"Meera"}): a node with label Teacher and property name.
-[:TEACHES]->: a relationship of type TEACHES; the arrow shows the direction, teacher to course.
To check: MATCH (t:Teacher)-[:TEACHES]->(c:Course) RETURN t.name, c.name;

Marks for: MATCH with the right labels and properties (1), CREATE with the TEACHES type and the correct arrow direction (1).

b) Differentiate using any four points between Data Lake and Data Warehouse. 2MCO1

Basis	Data Lake	Data Warehouse
Type of data	Structured, semi-structured and unstructured (logs, images, JSON)	Only structured data in rows and columns
Schema	Schema-on-read: stored raw, schema applied when reading	Schema-on-write: data must fit the schema before loading
Processing	ELT: load first, transform later	ETL: transform first, then load
Cost	Low cost, built on object storage (S3, ADLS)	Higher cost, specialised storage and compute
Users	Data scientists and ML engineers	Business analysts, BI reporting
Examples	Amazon S3, Azure Data Lake Storage	Snowflake, Amazon Redshift, Google BigQuery

Marks for: any four correct rows, half a mark each. A table scores better than paragraphs.

c) A dataset of 50 GB needs to be processed using Hadoop MapReduce. If each block size is 128 MB, calculate: (i) Total number of blocks (ii) Minimum number of mappers required. 2MCO2

Convert to MB	50 GB = 50 × 1024 = 51,200 MB
(i) Number of blocks	51,200 ÷ 128 = 400 blocks
(ii) Minimum mappers	One mapper per block (input split), so 400 mappers

Rule: blocks = file size ÷ block size, rounded up. If it does not divide exactly, the last block is smaller but still counts as one block.

Marks for: GB to MB conversion with 1024 (0.5), 400 blocks (0.5), 400 mappers with the reason "one mapper per input split" (1).

d) Stream consists of the integers 3, 1, 4, 1, 5, 2, 6, 5. Our hash functions will all be of the form h(x) = 2x+1 mod 5. Determine the tail length for each stream element and the resulting estimate using FM algorithm. 2MCO3

h(x) gives values 0 to 4, so each value is written in 3 bits. Tail length r = number of zeros at the right end.

x	2x + 1	h(x) = (2x+1) mod 5	Binary	Tail length r
3	7	2	`010`	1
1	3	3	`011`	0
4	9	4	`100`	2
1	3	3	`011`	0
5	11	1	`001`	0
2	5	0	`000`	0 (h = 0 taken as 0)
6	13	3	`011`	0
5	11	1	`001`	0

R = maximum tail length = 2. Estimated number of distinct elements = 2^R = 2² = 4.

Exam tip

Write one line saying how you treat h(x) = 0. Some teachers count 000 as 3 trailing zeros, which makes R = 3 and the estimate 8. The actual number of distinct elements is 6, and FM only gives an estimate.

Marks for: the table with binary and tail lengths (1), R = 2 and estimate 2^R = 4 (1).

e) Sketch the abstract architecture of the Data Stream Management System (DSMS) and define the elements of DSMS. 2MCO3

Abstract architecture of a DSMS

Input streams: continuous data arriving from sources such as sensors or apps.
Stream input regulator: controls the rate of incoming data and filters or samples it (Bloom filter works here).
Working storage: holds the recent data or window being processed right now.
Summary storage: keeps compact summaries (counts, averages, sketches) instead of all raw data.
Metadata storage: information about the streams, schemas and queries.
Query processor: runs the queries on the streams and produces results.
Query repository: stores registered standing (continuous) queries.
User queries: standing or ad-hoc questions sent by users.
Output streams: results sent out continuously to users or dashboards.

Marks for: a labelled diagram (1), short definitions of the elements (1).

f) A spam filter needs to quickly decide whether an incoming email address belongs to a blacklist of 10 million addresses. Storing the entire list is costly. Which algorithm would you use: Bloom Filter or FM Algorithm? Justify your answer. 2MCO3

Answer: Bloom Filter.

The question is a membership test: "is this email in the blacklist?". A Bloom filter is built for exactly this.
It stores only a bit array and a few hash functions, not the 10 million addresses, so it needs very little memory.
A lookup takes a few hash computations, so it is very fast (constant time).
It never gives a false negative: if it says "not in the list", the email is surely clean. A rare false positive can be double-checked against the full list.
FM algorithm only estimates how many distinct elements a stream has. It cannot tell whether one particular email is present, so it does not fit.

Marks for: choosing Bloom filter (0.5), membership + memory + speed reasons (1), why FM does not fit (0.5).

Q.2 Answer (a) OR (b) 5 marks

a) Consider the following distance matrix of 4 cities. Construct a graph with 4 nodes. Apply Dijkstra's graph algorithm to find shortest paths from a source node 'A' to a destination node 'D'. 5MCO2

	A	B	C	D
A	0	10	15	20
B	10	0	35	25
C	15	35	0	30
D	20	25	30	0

Step 1: Draw the graph. Every pair of cities has a road, so each node connects to the other three.

A is the source, D is the destination

Step 2: Iterations. Start with A = 0 and every other node = ∞. Each step, pick the unvisited node with the smallest distance and update its neighbours.

Step	Node picked	B	C	D
Start	none	∞	∞	∞
1	A (0)	10 (A)	15 (A)	20 (A)
2	B (10)	10	15 (A), since 10+35 = 45 > 15	20 (A), since 10+25 = 35 > 20
3	C (15)	10	15	20 (A), since 15+30 = 45 > 20
4	D (20)	10	15	20

Step 3: Check every route from A to D.

Route	Distance
A → D	20 (shortest)
A → B → D	10 + 25 = 35
A → C → D	15 + 30 = 45
A → B → C → D	10 + 35 + 30 = 75
A → C → B → D	15 + 35 + 25 = 75

Answer: the shortest path from A to D is the direct road A → D with distance 20. Shortest distances from A: B = 10, C = 15, D = 20.

Marks for: graph drawn with weights (1), initialisation (0.5), iteration table with correct updates (2.5), final path and distance (1).

b) With a neat diagram, explain the elements of the Data Engineering Life cycle. 5MCO1

Data Engineering lifecycle

Generation: data is created in source systems such as apps, databases, IoT sensors. The engineer tracks schema evolution, generation velocity, and talks with the app team.
Storage: runs under every stage. The choice depends on access frequency, latency and cost: hot, lukewarm or cold data.
Ingestion: moving data from source systems into the storage platform. Batch or streaming; push or pull; handling bottlenecks.
Transformation: converting raw data into useful data: structuring and cleaning, quality checks and validation, data modelling, applying business logic.
Serving: giving the final data to users: Analytics and BI dashboards, Machine Learning models, and Reverse ETL (sending data back into business tools).
Undercurrents: practices that support every stage: security, data management, DataOps, data architecture, orchestration and software engineering.

Example: in a food delivery app, orders are generated in the app, ingested in real time, stored in a data lake, cleaned and aggregated, then served to a sales dashboard and a delivery-time prediction model.

Marks for: a neat labelled diagram (2), each stage explained (2), undercurrents and an example (1).

Q.3 Answer (a) OR (b) 5 marks

a) Illustrate the workflow of scraping text data from a website and storing it in a database with an example. 5MCO2

URL discoverySend HTTP requestParse HTMLExtract textClean and transformQuality checkStore in database

URL discovery: find the pages to scrape (sitemap, category pages). Check robots.txt and Terms of Service first.
Send request: download the page with the requests library; check status code 200.
Parse: give the HTML to BeautifulSoup with a parser (html.parser or lxml) to get a tree of HTML elements.
Extract: pick the needed text with find(), find_all() or CSS selectors.
Transform: clean the text: strip spaces, remove currency symbols, convert prices to numbers, remove duplicates.
Quality check: skip rows with missing fields; validate types.
Store: insert rows into a database (SQLite, MySQL or MongoDB) so they can be queried later.

Example: scrape book titles and prices from books.toscrape.com (a practice site) into SQLite.

import requests, sqlite3
from bs4 import BeautifulSoup

url = "https://books.toscrape.com/"
res = requests.get(url, headers={"User-Agent": "student-project"})
soup = BeautifulSoup(res.text, "html.parser")

rows = []
for book in soup.find_all("article", class_="product_pod"):
    title = book.h3.a["title"].strip()
    price = float(book.find("p", class_="price_color").text.replace("£", ""))
    rows.append((title, price))

con = sqlite3.connect("books.db")
con.execute("CREATE TABLE IF NOT EXISTS books (title TEXT, price REAL)")
con.executemany("INSERT INTO books VALUES (?, ?)", rows)
con.commit()
print(con.execute("SELECT COUNT(*) FROM books").fetchone())

Marks for: workflow diagram (1.5), each step explained (2), a real example with code or clear steps (1.5).

b) A retail company wants to analyze customer purchase history (structured data) and social media comments (unstructured data). Which Hadoop ecosystem components would you use for storage, processing, and analysis? Justify. 5MCO3

Sales DBSqoopHDFSSpark / MapReduceHiveReports

Social mediaFlume / KafkaHDFSSpark (sentiment)Mahout / MLlib

Need	Component	Why
Bring in purchase history	Sqoop	Imports structured tables from the company's RDBMS into Hadoop
Bring in social comments	Flume or Kafka	Collects continuous, unstructured text streams from social platforms
Storage	HDFS	Stores both structured and unstructured data cheaply across many machines, with replication
Fast lookups	HBase	NoSQL store on HDFS for quick reads of a single customer's history
Processing	MapReduce or Spark	Cleans and joins large data in parallel; Spark is faster because it works in memory
Analysis (structured)	Hive	SQL-like queries (HiveQL) on purchase data: top products, sales per region
Analysis (text)	Spark MLlib or Mahout, Pig	Sentiment analysis and recommendations; Pig scripts for text cleaning
Resource management	YARN	Allocates cluster resources to all these jobs
Scheduling	Oozie	Runs the daily import, process and report pipeline in order

Justification: the data is large and mixed (structured + unstructured). HDFS stores both in one place, Sqoop and Flume bring each kind in, Spark or MapReduce process it at scale, and Hive plus MLlib turn it into insights such as "customers who complained on social media bought less next month".

Marks for: correct component for storage, processing and analysis (3), justification tied to structured vs unstructured data (2).

Vivekanand Education Society's Institute of Technology, Chembur

Data Engineering · Mock Paper 1

Time: 1 hourMarks: 20CO1 20% · CO2 40% · CO3 40%

Q.1 Attempt any FIVE of the following 10 marks

a) Write Cypher queries to (i) create a Student node named "Riya" and a Course node titled "Data Engineering", and (ii) create an ENROLLED_IN relationship from Riya to the course with the property since: 2026. 2MCO2

(i) Create the two nodes

CREATE (s:Student {name: "Riya"}),
       (c:Course {title: "Data Engineering"});

(ii) Create the relationship with a property

MATCH (s:Student {name: "Riya"}),
      (c:Course {title: "Data Engineering"})
CREATE (s)-[:ENROLLED_IN {since: 2026}]->(c);

Check the result

MATCH (s:Student)-[r:ENROLLED_IN]->(c:Course)
RETURN s.name, r.since, c.title;

( ) is a node, :Student is its label and {name: "Riya"} is its property.
-[:ENROLLED_IN]-> is a directed relationship; {since: 2026} is a property stored on the relationship.
MATCH finds the existing nodes first so duplicates are not created.

Marks for: correct CREATE for both nodes with labels (1M), MATCH + CREATE of the relationship with arrow direction and the since property (1M).

b) Differentiate between Data Warehouse and Lakehouse (any four points). 2MCO2

Basis	Data Warehouse	Lakehouse
Type of data	Only structured data (rows and columns)	Structured, semi-structured and unstructured data together
Storage cost	Expensive, uses specialised storage	Cheap, built on cloud object storage (S3, ADLS)
Schema	Schema-on-write (data must fit before loading)	Supports schema-on-read plus schema enforcement and evolution
Workloads	Mainly BI reports and SQL analytics	BI, SQL, Machine Learning and AI on the same data
History	Usually keeps only the latest clean data	Time travel to older versions using the transaction log (Delta Lake)
Example	Amazon Redshift, Snowflake, Teradata	Delta Lake, Apache Iceberg, Apache Hudi on S3

Marks for: any four correct points of difference in table form (0.5M each).

c) What are the different types of data scraping? Name one tool or use for each. 2MCO2

Data scraping is the automated extraction of data from websites, documents or other digital sources using scripts or tools. Its four types are:

Text scraping: extracts articles, reviews, product descriptions or tweets. Tools: BeautifulSoup, Scrapy. Example: collecting Amazon product reviews.
Image scraping: downloads images with captions or tags to build ML datasets. Tools: Scrapy image pipeline, Selenium. Example: product photos for image classification.
Audio scraping: collects podcasts, songs or voice clips for speech recognition. Tools: requests for file download, yt-dlp. Example: podcast episodes for speech-to-text training.
Video scraping: extracts videos or metadata (title, views, comments, duration). Tools: YouTube Data API, Selenium. Example: trending video views for engagement analysis.

Marks for: definition (0.5M), four types with one tool or example each (1.5M).

d) A Bloom filter uses a bit array of size 8 (index 0 to 7) and two hash functions h1(x) = (x + 1) mod 8 and h2(x) = (3x + 2) mod 8. Insert the elements 5 and 10. Then check whether 7 and 13 are present in the set. 2MCO3

Step 1: Initialise all 8 bits to 0.

Index	0	1	2	3	4	5	6	7
Bit	0	0	0	0	0	0	0	0

Step 2: Hash the elements to insert and set those bits to 1.

Element	h1 = (x + 1) mod 8	h2 = (3x + 2) mod 8	Bits set
5	6 mod 8 = 6	17 mod 8 = 1	1, 6
10	11 mod 8 = 3	32 mod 8 = 0	0, 3

Index	0	1	2	3	4	5	6	7
Bit	1	1	0	1	0	0	1	0

Step 3: Test the query elements.

Test	h1	h2	Bits checked	Result
7	8 mod 8 = 0	23 mod 8 = 7	bit 0 = 1, bit 7 = 0	Definitely NOT present
13	14 mod 8 = 6	41 mod 8 = 1	bit 6 = 1, bit 1 = 1	Probably present

13 was never inserted, yet both its bits are 1 (set by 5). This is a false positive. A Bloom filter can give false positives but never false negatives.

Marks for: hash values of inserted elements and final bit array (1M), correct verdict for both test values with the false positive noted (1M).

e) What is a real-time system? Explain its categories with suitable examples. 2MCO3

A real-time system is a system that must respond to input within a fixed time limit (deadline). The correctness depends on the result and on how fast the result arrives.

Category	Latency	Tolerance for delay	Example
Hard real-time	Microseconds to milliseconds	None: a missed deadline means total system failure	Pacemaker, airbag system
Soft real-time	Milliseconds to seconds	Low: a delay reduces quality but the system does not fail	Online stock quotes
Near real-time	Seconds to minutes	High: no system failure	Skype video, live dashboards

Streaming data systems are non-hard (soft or near) real-time systems.

Marks for: definition (0.5M), three categories with latency and one example each (1.5M).

f) State the uses of Pandas in Data Engineering. 2MCO1

Pandas is a Python library for tabular data. Its two main structures are the Series (one column) and the DataFrame (a full table).

Ingestion: reads and writes CSV, Excel, JSON and SQL tables with read_csv(), read_sql(), to_csv().
Cleaning: handles missing values and duplicates with dropna(), fillna(), drop_duplicates(), and fixes data types with astype().
Transformation (the T of ETL): filter, sort, groupby() and merge() tables.
Exploration: quick checks with head(), describe(), info() before data goes to visualisation or ML tools.

Marks for: one line on what Pandas is (0.5M), three or four uses with a function name each (1.5M).

Q.2 Answer (a) OR (b) 5 marks

a) With a neat diagram, explain the Data Engineering lifecycle and its undercurrents. 5MCO1

The Data Engineering lifecycle is the set of stages that turn raw data from source systems into useful data for analytics and machine learning.

Data Engineering lifecycle with its six undercurrents

Stages

Generation: data is produced by source systems such as app databases, IoT sensors, logs and APIs. The engineer tracks schema evolution, generation velocity and keeps communication with the application team.
Storage: the foundation under every stage. Chosen by access frequency, latency and cost: hot, lukewarm and cold data.
Ingestion: moving data from source systems to the storage platform, in batch or streaming, using push or pull modes, while managing bottlenecks.
Transformation: structuring and cleaning, data quality checks and validation, data modelling and applying business logic.
Serving: delivering data to consumers for Analytics and BI, Machine Learning and Reverse ETL (sending processed data back into business tools).

Undercurrents (practices that support every stage): Security, Data Management, DataOps, Data Architecture, Orchestration and Software Engineering.

Example: a food delivery app generates orders (generation), Kafka moves them into S3 (ingestion, storage), Spark removes duplicates and joins with restaurant data (transformation), and a Power BI dashboard shows daily sales (serving).

Marks for: neat labelled diagram (1.5M), five stages explained (2.5M), undercurrents listed (0.5M), example (0.5M).

b) Explain the role of a Data Engineer and the challenges faced in Data Engineering, with a suitable example. 5MCO1

Data Engineering is the development, implementation and maintenance of systems that take in raw data and produce high-quality, consistent information for analysis and machine learning. The Data Engineer sits between data producers and data consumers.

Data ProducersData EngineersData Consumers

Role of a Data Engineer

Builds and maintains data pipelines that move data from source systems to storage.
Designs databases, data warehouses and data lakes so data is organised and easy to query.
Performs ETL (Extract, Transform, Load): cleans, validates and transforms raw data.
Ensures data quality, security and governance across the pipeline.
Supplies ready, trustworthy data to Data Scientists and Analysts.

Challenges

Complexity of data sources: many sources in structured, semi-structured and unstructured formats.
Volume and velocity: Big Data arriving at high speed.
Data quality: duplicate, missing or incorrect records.
Security, privacy and governance: protecting sensitive data (GDPR and similar laws).
Evolving technologies and hype cycles: tools change fast.
Performance and cost: query performance, compute capacity and cloud bills; data silos block access.

Example: in a UPI payments company, the Data Engineer builds a streaming pipeline from millions of transactions per minute into a data lake, removes duplicate transaction IDs, masks account numbers for privacy, and serves clean data to the fraud-detection ML team. The challenges here are high velocity, data quality and privacy.

Marks for: definition (0.5M), role with four or five points (2M), challenges with four or five points (2M), example (0.5M).

Q.3 Answer (a) OR (b) 5 marks

a) Case study: ShopKart stores its orders in a Delta Lake table orders (order_id INT, customer STRING, amount DOUBLE). Version 1 of the table is shown below. An analyst then ran a wrong UPDATE that set every amount to 0 (version 2). Later, a new batch arrived with amount = "five hundred" (a text value). Explain schema enforcement and time travel on this data, with queries. 5MCO2

order_id	customer	amount
101	Asha	1200.0
102	Rahul	450.0
103	Neha	800.0

Delta Lake is an open-source storage layer that adds ACID transactions on top of cloud object storage. Every change is recorded in a transaction log (the _delta_log folder), and this log makes both features possible.

Table history (from the transaction log)

Version	Operation	State of amount
0	CREATE TABLE	empty table
1	INSERT 3 rows	1200.0, 450.0, 800.0 (correct)
2	UPDATE amount = 0	0.0, 0.0, 0.0 (wrong)

1. Schema enforcement (the "five hundred" batch)

Schema enforcement checks every incoming write against the table's schema (column names and data types).
The column amount is DOUBLE, and "five hundred" is a STRING, so Delta Lake rejects the whole write with a schema mismatch error. No bad row enters the table.
This protects the pipeline: reports and ML models reading amount never break on text values.

INSERT INTO orders VALUES (104, 'Vikram', 'five hundred');
-- Error: schema mismatch / cannot cast 'five hundred' to DOUBLE. Write rejected.

If a genuine new column is needed later, schema evolution allows it on purpose with mergeSchema = true.

2. Time travel (the wrong UPDATE)

Time travel lets users query or restore an older version of the table using the transaction log.
Version 1 still has the correct amounts, so the data can be read and restored without any backup file.

DESCRIBE HISTORY orders;                       -- see versions 0, 1, 2
SELECT * FROM orders VERSION AS OF 1;          -- view correct data
SELECT * FROM orders TIMESTAMP AS OF '2026-09-20 10:00:00';
RESTORE TABLE orders TO VERSION AS OF 1;       -- undo the wrong update

Conclusion: schema enforcement prevents bad data from entering, and time travel recovers from bad data that already entered. Both work because of Delta Lake's transaction log. Time travel also helps audits and reproducing old reports.

Marks for: Delta Lake and transaction log (0.5M), version history table (1M), schema enforcement applied to the given batch with error (1.5M), time travel queries and restore (1.5M), conclusion (0.5M).

b) A bank wants to monitor UPI payments in real time and flag fraud (for example, the same account paying from two cities within 5 minutes). Explain the DSMS architecture with respect to this case study. 5MCO3

A Data Stream Management System (DSMS) manages continuous, unbounded data streams and runs queries on them as data arrives, unlike a DBMS which stores data first and queries it later.

Step 1: Sources of data: UPI app transactions, the bank's payment switch, device and location (GPS/IP) data, and merchant terminals.

Step 2: Velocity and processing need: thousands of payments per second; a fraud decision is needed within a second (soft real-time).

DSMS architecture for real-time UPI fraud monitoring

Elements of the DSMS in this case

Input streams: UPI transactions, switch logs and device/location events.
Stream input regulator: controls the incoming rate, filters and samples data. A Bloom filter here can drop payments to known-safe merchants quickly.
Working storage: holds the current window of recent transactions (for example, the last 5 minutes per account).
Summary storage: keeps compact synopses such as count of payments per account per hour and average amount.
Metadata storage: stores stream schemas, window sizes and fraud rules.
Query repository: stores registered queries.
Query processor: runs the queries on arriving data and emits results.
Output streams: fraud alerts to the bank app, SMS to the user, and a live dashboard.

Types of queries:

Standard (standing / continuous) query: registered once and runs forever, for example "alert if one account pays from two cities within 5 minutes".
Ad-hoc query: asked once when needed, for example "how many payments above ₹50,000 happened in the last hour?".

Marks for: DSMS definition (0.5M), labelled diagram (1.5M), elements mapped to the case (2M), standard vs ad-hoc queries with case examples (1M).

Vivekanand Education Society's Institute of Technology, Chembur

Data Engineering · Mock Paper 2

Time: 1 hourMarks: 20CO1 20% · CO2 40% · CO3 40%

Q.1 Attempt any FIVE of the following 10 marks

a) What is a NoSQL database? Explain its types with one example each. 2MCO2

NoSQL means "Not Only SQL". It is a database that stores data in formats other than the rows-and-columns tables of a relational database. It needs no fixed schema, handles large volumes of unstructured and semi-structured data, and scales horizontally.

Document database: stores data as JSON/BSON documents inside collections; each document can have different fields. Example: MongoDB.
Key-value database: stores data as a unique key and its value, like a giant dictionary; very fast lookups. Example: Redis, Amazon DynamoDB (session ID to login details).
Graph database: stores data as nodes (entities) and relationships (edges), both with properties. Example: Neo4j, queried with Cypher.
Column-family (wide column) database: stores data by columns for fast analytics on huge data. Example: Apache Cassandra, HBase.

Marks for: meaning of NoSQL with two features (0.5M), types with an example each (1.5M).

b) Differentiate between Amazon S3 and Azure Data Lake Storage Gen2. 2MCO2

Basis	Amazon S3	Azure Data Lake Storage Gen2
Provider	Amazon Web Services (AWS)	Microsoft Azure (built on Azure Blob Storage)
Namespace	Flat namespace: bucket + object key, no real folders	Hierarchical namespace (HNS): real folders and directories
Address	s3://bucket-name/object-key	abfss://container@account.dfs.core.windows.net/path
Security	IAM policies and bucket policies	Microsoft Entra ID (Azure Active Directory) and ACLs on files and folders
Best for	General-purpose object storage: backups, media, data lakes	Big Data analytics with Azure Databricks, Synapse and Spark

Marks for: any four correct differences (0.5M each).

c) A stream consists of the integers 1, 3, 2, 1, 2, 3, 4, 3, 1, 2, 3, 1. The hash function is h(x) = (6x + 1) mod 5. Determine the tail length for each element and estimate the number of distinct elements using the FM algorithm. 2MCO3

FM (Flajolet-Martin) method: hash each element, write the hash in binary, count the trailing zeros (tail length r). Take R = maximum r. Estimated distinct count = 2^R.

Since h(x) is between 0 and 4, 3-bit binary is used.

x	6x + 1	h(x) = (6x + 1) mod 5	Binary	Tail length r
1	7	2	010	1
3	19	4	100	2
2	13	3	011	0
1	7	2	010	1
2	13	3	011	0
3	19	4	100	2
4	25	0	000	0 *
3	19	4	100	2
1	7	2	010	1
2	13	3	011	0
3	19	4	100	2
1	7	2	010	1

* Convention: a hash value of 0 (000) has no 1 bit, so its tail length is taken as 0.

R = max(1, 2, 0, 1, 0, 2, 0, 2, 1, 0, 2, 1) = 2

Estimated distinct elements = 2^R = 2² = 4

Check: the actual distinct elements are {1, 2, 3, 4} = 4, so the estimate is exact here.

Marks for: hash and binary table with tail lengths (1M), R and 2^R estimate (1M).

d) What is robots.txt? Explain its role in designing a web scraper. 2MCO2

robots.txt is a plain text file kept at the root of a website (for example https://www.flipkart.com/robots.txt) that tells crawlers and scrapers which pages they may and may not access.

User-agent: *
Disallow: /account/
Disallow: /checkout/
Allow: /products/
Sitemap: https://example.com/sitemap.xml

User-agent: which bot the rule is for; * means all bots.
Disallow: paths that must not be crawled or scraped.
Allow: paths that are permitted.
Sitemap: a structured file listing all pages the bot should crawl.

Role: the scraper reads robots.txt first and follows its rules, so it scrapes only permitted pages, respects the server, avoids legal and ethical problems, and uses the sitemap to discover URLs.

Marks for: definition and location (0.5M), directives explained (1M), role in scraper design (0.5M).

e) Differentiate between a tumbling window and a sliding window in stream processing. 2MCO3

Windowing splits an unbounded stream into finite chunks (windows) so that calculations like sum, count and average can be done.

Basis	Tumbling window	Sliding window
Size	Fixed size	Fixed size plus a slide interval
Overlap	Non-overlapping: windows sit back to back	Overlapping: a new window starts every slide
Each event belongs to	Exactly one window	One or more windows
Example	Orders per minute: 10:00-10:01, 10:01-10:02	Orders in the last 5 minutes, updated every 1 minute
Use	Periodic reports (per hour, per day)	Moving averages, trend and fraud detection

Tumbling: [0-5) [5-10) [10-15)Sliding (size 10, slide 5): [0-10) [5-15) [10-20)

Marks for: any four correct differences (0.5M each).

f) A 10 GB file is stored in HDFS with a block size of 128 MB and a replication factor of 3. Calculate (i) the number of blocks, (ii) the minimum number of mappers required, and (iii) the total block copies stored in the cluster. 2MCO3

Step	Working	Result
Convert to MB	10 GB × 1024	10,240 MB
(i) Number of blocks	10,240 ÷ 128	80 blocks
(ii) Minimum mappers	One mapper per block (input split)	80 mappers
(iii) Block copies	80 × 3 (replication)	240 copies = 30 GB of raw disk

If the size does not divide exactly, round the block count up (the last block is smaller but still counts as one block).

Marks for: correct unit conversion and blocks (1M), mappers with reason and replicated copies (1M).

Q.2 Answer (a) OR (b) 5 marks

a) A social app has the following data: "Aman follows Priya. Priya follows Kabir. Kabir likes post P1. Aman likes post P1." Draw the graph data model and write Cypher queries to (i) create this graph, (ii) find the friends-of-friends of Aman, and (iii) find the shortest FOLLOWS path from Aman to Kabir. 5MCO2

In a graph database, entities are nodes with labels (User, Post) and connections are relationships (FOLLOWS, LIKES). Both can hold properties.

Graph model: 3 User nodes, 1 Post node, FOLLOWS and LIKES relationships

(i) Create the graph

CREATE (a:User {name: "Aman"}),
       (p:User {name: "Priya"}),
       (k:User {name: "Kabir"}),
       (post:Post {id: "P1"}),
       (a)-[:FOLLOWS]->(p),
       (p)-[:FOLLOWS]->(k),
       (k)-[:LIKES]->(post),
       (a)-[:LIKES]->(post);

(ii) Friends-of-friends of Aman (people followed by the people Aman follows)

MATCH (a:User {name: "Aman"})-[:FOLLOWS]->(friend)-[:FOLLOWS]->(fof)
WHERE fof <> a
RETURN fof.name;

Result: Kabir (Aman → Priya → Kabir).

(iii) Shortest FOLLOWS path from Aman to Kabir

MATCH p = shortestPath(
  (a:User {name: "Aman"})-[:FOLLOWS*]->(k:User {name: "Kabir"})
)
RETURN p, length(p);

Result: path Aman → Priya → Kabir, length 2.

Bonus: posts liked by both Aman and Kabir (common interest):

MATCH (:User {name: "Aman"})-[:LIKES]->(post)<-[:LIKES]-(:User {name: "Kabir"})
RETURN post.id;   // P1

Such multi-hop queries are fast in Neo4j because relationships are stored directly, while an RDBMS would need several self-joins.

Marks for: correct graph diagram with labels and arrow directions (1.5M), CREATE query (1M), friends-of-friends query with result (1M), shortestPath query with result (1M), one line on why graph DB suits this (0.5M).

b) Consider the following distance matrix of 5 cities (∞ means no direct road). Construct the graph and apply Dijkstra's algorithm to find the shortest paths from source A to all other nodes, and the shortest path from A to E. 5MCO2

	A	B	C	D	E
A	0	4	2	∞	∞
B	4	0	1	5	∞
C	2	1	0	8	10
D	∞	5	8	0	2
E	∞	∞	10	2	0

Graph (undirected, weights = distances):

Source node A is highlighted

Rule: start with distance 0 at A and ∞ elsewhere. Each step, visit the unvisited node with the smallest distance and update its neighbours: new distance = min(old, distance of visited node + edge weight).

Step	Visited node	B	C	D	E
Initial	-	∞	∞	∞	∞
1	A (0)	4 (A)	2 (A)	∞	∞
2	C (2)	min(4, 2+1) = 3 (C)	2	2+8 = 10 (C)	2+10 = 12 (C)
3	B (3)	3	2	min(10, 3+5) = 8 (B)	12 (C)
4	D (8)	3	2	8	min(12, 8+2) = 10 (D)
5	E (10)	3	2	8	10

(The letter in brackets is the previous node on the best path so far.)

Destination	Shortest distance	Path
B	3	A → C → B
C	2	A → C
D	8	A → C → B → D
E	10	A → C → B → D → E

Answer: the shortest path from A to E is A → C → B → D → E with total distance 2 + 1 + 5 + 2 = 10. The direct-looking route A → C → E costs 12, so it is not the shortest.

Marks for: graph drawn from the matrix (1M), iteration table with correct updates (2.5M), final distances and A to E path with total (1.5M).

Q.3 Answer (a) OR (b) 5 marks

a) Illustrate the workflow of scraping product data (name, price, rating) from an e-commerce website and storing it in MongoDB, with code. 5MCO2

Web scraping is the automated extraction of information from a known web page. Here the scraper collects product name, price and rating, cleans them, and stores each product as a document in MongoDB.

Check robots.txtURL discoveryDownload (HTTP GET)Parse + extractTransform / cleanQuality checkStore in MongoDB

Check robots.txt and Terms of Service: confirm the product pages are allowed; plan rate limiting.
URL discovery: collect category and product page URLs, using the sitemap.
Download: send an HTTP GET request with the requests library and a User-Agent header.
Parse and extract: BeautifulSoup converts the HTML into a tree; select the name, price and rating tags. (Use Selenium if the page loads prices with JavaScript.)
Transform and clean: remove "₹" and commas, convert price to a number and rating to a float, strip spaces.
Quality assurance: skip rows with a missing name or price, remove duplicates.
Store: insert the documents into a MongoDB collection using pymongo.

import time
import requests
from bs4 import BeautifulSoup
from pymongo import MongoClient

url = "https://example-shop.com/mobiles"
headers = {"User-Agent": "Mozilla/5.0 (student project)"}

# 1. Download the page
html = requests.get(url, headers=headers, timeout=10).text

# 2. Parse into a tree using the lxml parser
soup = BeautifulSoup(html, "lxml")

# 3. Extract + clean each product
products = []
for card in soup.find_all("div", class_="product-card"):
    name = card.find("h2", class_="title").get_text(strip=True)
    price = card.find("span", class_="price").get_text(strip=True)
    rating = card.find("span", class_="rating").get_text(strip=True)
    products.append({
        "name": name,
        "price": float(price.replace("₹", "").replace(",", "")),
        "rating": float(rating),
        "source": url,
    })
    time.sleep(1)   # polite rate limiting

# 4. Store in MongoDB (database: shop, collection: products)
client = MongoClient("mongodb://localhost:27017/")
collection = client["shop"]["products"]
if products:
    collection.insert_many(products)
print(collection.count_documents({}), "products stored")

Sample stored document:

{ "_id": ObjectId("..."), "name": "Redmi Note 14", "price": 17999.0,
  "rating": 4.3, "source": "https://example-shop.com/mobiles" }

MongoDB suits scraped data because it is schema-flexible: a product with extra fields (for example "discount") can be stored without changing any table design.

Marks for: workflow diagram or steps (1.5M), code for request + BeautifulSoup extraction (1.5M), cleaning and MongoDB insert (1.5M), robots.txt / ethics or why MongoDB (0.5M).

b) Case study: a food delivery app wants to keep raw clickstream events (JSON), structured order records and food images in one place. Several teams will run analytics on the same tables using Spark and Trino, and the company must keep a history of all changes for audits. Suggest a suitable architecture and storage layer, and justify your choice with a diagram. 5MCO2

Suggested architecture: Lakehouse on cloud object storage (Amazon S3), with Apache Iceberg as the open table format (Delta Lake is the alternative if the company is fully Spark-based).

Lakehouse for the food delivery app

Justification (matching the keywords in the case)

"Raw and structured data in one place" → Lakehouse: it combines the cheap, flexible storage of a data lake (JSON, images) with warehouse features (tables, ACID) for orders. No separate lake plus warehouse, so less cost and duplication.
"Several teams using Spark and Trino" → Apache Iceberg: Iceberg is an open table format built for multi-engine access, so Spark, Trino and Flink can read and write the same tables safely at the same time. Hidden partitioning means teams query correctly without knowing the partition layout.
"History of all changes for audits" → time travel: every commit creates a snapshot in Iceberg metadata (the transaction log in Delta Lake), so auditors can query the table as it was on any past date.
Schema enforcement and evolution: bad records with wrong types are rejected; new columns (for example "delivery_rating") can be added safely.
Medallion layers: Bronze keeps raw clickstream and images, Silver holds cleaned and joined data, Gold holds business-ready tables such as daily sales per city.

Alternative: if all teams used only Spark, Delta Lake would also fit, with ACID transactions and time travel through its transaction log. Apache Hudi fits best when data has very frequent upserts.

Marks for: naming Lakehouse (1M), choosing Iceberg or Delta with reason (1M), labelled diagram (1.5M), justification of time travel, schema handling and multi-engine support (1.5M).

🧮 Numericals lab

Each tool solves the numerical step by step in the same table you write in the exam. Solve it by hand first, then press the button to check.

Bloom filter

Is an element in the set? Answer is "maybe yes" or "definitely no".

Bit array size

Elements in the set

Elements to test

Hash functions (one per line)

Flajolet-Martin (FM): count distinct elements

Stream

Hash function

When h(x) = 0

HDFS blocks and mappers

File size

Unit

Block size (MB)

Replication factor

Dijkstra shortest path

Type the distance matrix. Leave a cell empty or type - when there is no direct road.

Number of nodes

Source

Destination

💗 Flashcards

tap to flip

answer

x	6x + 1	h(x) = (6x + 1) mod 5	Binary	Tail length r
1	7	2	010	1
3	19	4	100	2
2	13	3	011	0
1	7	2	010	1
2	13	3	011	0
3	19	4	100	2
4	25	0	000	0 *
3	19	4	100	2
1	7	2	010	1
2	13	3	011	0
3	19	4	100	2
1	7	2	010	1

x	6x + 1	h(x) = (6x + 1) mod 5	Binary	Tail length r
1	7	2	010	1
3	19	4	100	2
2	13	3	011	0
1	7	2	010	1
2	13	3	011	0
3	19	4	100	2
4	25	0	000	0 *
3	19	4	100	2
1	7	2	010	1
2	13	3	011	0
3	19	4	100	2
1	7	2	010	1

x	6x + 1	h(x) = (6x + 1) mod 5	Binary	Tail length r
1	7	2	010	1
3	19	4	100	2
2	13	3	011	0
1	7	2	010	1
2	13	3	011	0
3	19	4	100	2
4	25	0	000	0 *
3	19	4	100	2
1	7	2	010	1
2	13	3	011	0
3	19	4	100	2
1	7	2	010	1