<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[my-blogs]]></title><description><![CDATA[Mongodb aggregation pipeline from basic to internal workings.]]></description><link>https://lavkush-maurya-blogs.hashnode.dev</link><image><url>https://cdn.hashnode.com/res/hashnode/image/upload/v1593680282896/kNC7E8IR4.png</url><title>my-blogs</title><link>https://lavkush-maurya-blogs.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Sat, 20 Jun 2026 03:22:49 GMT</lastBuildDate><atom:link href="https://lavkush-maurya-blogs.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Understanding YouTube’s Image Processing Pipeline]]></title><description><![CDATA[When I had the idea to build my own movie review platform, I started wondering how I would handle images for different screen sizes. That question led me to learn about YouTube’s image processing pipe]]></description><link>https://lavkush-maurya-blogs.hashnode.dev/understanding-youtube-s-image-processing-pipeline</link><guid isPermaLink="true">https://lavkush-maurya-blogs.hashnode.dev/understanding-youtube-s-image-processing-pipeline</guid><category><![CDATA[youtube]]></category><category><![CDATA[inner working]]></category><category><![CDATA[Pipeline]]></category><category><![CDATA[Processing]]></category><category><![CDATA[compression]]></category><category><![CDATA[Image Compression]]></category><category><![CDATA[image processing]]></category><category><![CDATA[image]]></category><dc:creator><![CDATA[Lavkush Maurya]]></dc:creator><pubDate>Tue, 21 Apr 2026 06:09:22 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/b2e76f12-4f53-4afe-bf1b-f597e1cd5f25.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>When I had the idea to build my own movie review platform, I started wondering how I would handle images for different screen sizes. That question led me to learn about YouTube’s image processing pipeline, so I could build something similar for my project.</p>
<p>So, here’s what happens behind the scenes:</p>
<h3><strong>Step 1: Upload (Input Stage)</strong></h3>
<p>When a creator uploads a thumbnail with high resolution (e.g., 1280×720) in JPEG or PNG format, the image may be converted into more efficient formats such as WebP or AVIF to improve loading speed and optimize performance. The processed image is then stored as the master (original) image.</p>
<p>At this stage, YouTube also uses an NSFW detection system to filter out any images that are inappropriate.</p>
<h3><strong>Step 2: Derivative Generation (Multiple Resolutions)</strong></h3>
<p>After the first step, multiple versions of the master image are created. The system resizes the master image into different resolutions, such as:</p>
<ul>
<li><p><strong>default</strong> → 120×90</p>
</li>
<li><p><strong>mqdefault</strong> → 320×180</p>
</li>
<li><p><strong>hqdefault</strong> → 480×360</p>
</li>
<li><p><strong>maxresdefault</strong> → up to 1280×720+</p>
</li>
</ul>
<p>Each version is generated directly from the master image to maintain quality and avoid repeated compression.</p>
<h3><strong>Step 3: Compression(Perceptual Optimization)</strong></h3>
<p>Human vision is more sensitive to brightness (luminance), edges, faces, and text, and less sensitive to fine color details (chrominance). Modern image compression techniques take advantage of this:</p>
<p><strong>1. Chroma Subsampling</strong> Instead of storing full color detail, the image keeps full brightness information while reducing color resolution.</p>
<p><strong>2. Frequency-Based Compression (DCT)</strong> The image is divided into small blocks (usually 8×8). Each block is transformed into frequency components:</p>
<ul>
<li><p>Low frequency → smooth areas</p>
</li>
<li><p>High frequency → sharp edges and details</p>
</li>
</ul>
<p><strong>3. Quantization</strong> This is the “lossy” stage, where small details are rounded off and less important frequency data is reduced to save space.</p>
<p><strong>4. Perceptual Weighting</strong> Different parts of the image are given different levels of importance during compression. Human eyes are:</p>
<ul>
<li><p>Highly sensitive to edges (text, outlines)</p>
</li>
<li><p>Very sensitive to faces</p>
</li>
<li><p>Less sensitive to smooth backgrounds, noise, and minor color variations</p>
</li>
</ul>
<p>Compression algorithms take advantage of these characteristics to preserve what matters most.</p>
<p><strong>5. Adaptive Compression</strong> Compression is applied unevenly across the image:</p>
<ul>
<li><p>Complex areas → higher quality</p>
</li>
<li><p>Simple areas → more compression</p>
</li>
</ul>
<p><strong>6. Post-Processing</strong> After resizing, slight sharpening is often applied to restore edge clarity and improve perceived image quality.</p>
<h3><strong>Step 4: Storage</strong></h3>
<p>All generated image variants are stored within YouTube’s infrastructure, which relies on Google’s distributed storage systems.</p>
<h3><strong>Step 5: CDN Distribution</strong></h3>
<p>Images are delivered through Google’s CDN, which uses edge servers to serve content from locations closest to the user for faster loading.</p>
<h3><strong>Step 6: Device Aware Delivery</strong></h3>
<p>YouTube selects the appropriate image based on factors such as screen size, network speed, and the UI context (e.g., whether the image is displayed in a grid or full view).</p>
<img src="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/4e76ae5b-2829-4e1a-8187-fcba8b7d5b34.png" alt="" style="display:block;margin:0 auto" />

<p>In the end, YouTube’s thumbnail system isn’t built on magic. it’s built on smart engineering decisions. By combining efficient compression techniques, multiple image variants, perceptual optimization, and fast CDN delivery, it ensures that thumbnails look sharp while loading quickly across all devices.</p>
<p>If you’re building your own platform, the key takeaway is simple: store a high-quality master image, generate optimized versions, and deliver them intelligently based on user context. With these principles, you can create a system that feels fast, scalable, and professional, just like the best platforms on the web.</p>
<h3>Sources</h3>
<ul>
<li><p>YouTube Data API – Thumbnails</p>
</li>
<li><p>Google Developers – WebP Image Format</p>
</li>
<li><p>JPEG Compression (DCT &amp; Quantization) – Technical Overview</p>
</li>
<li><p>Cloudflare – How CDNs Work</p>
</li>
<li><p>Google Cloud Vision API – SafeSearch Detection</p>
</li>
</ul>
]]></content:encoded></item><item><title><![CDATA[How Does react actually works?]]></title><description><![CDATA[In recent days, I’ve been diving deeper into React and exploring not just how to use it, but how it actually works under the hood. While most tutorials do a great job explaining concepts step by step,]]></description><link>https://lavkush-maurya-blogs.hashnode.dev/how-does-react-actually-works</link><guid isPermaLink="true">https://lavkush-maurya-blogs.hashnode.dev/how-does-react-actually-works</guid><category><![CDATA[React]]></category><category><![CDATA[fiber]]></category><category><![CDATA[React Fiber]]></category><category><![CDATA[reconcilliation]]></category><category><![CDATA[vite]]></category><category><![CDATA[vite-react-app]]></category><dc:creator><![CDATA[Lavkush Maurya]]></dc:creator><pubDate>Mon, 13 Apr 2026 07:55:53 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/fc0bf9ef-51e8-43a3-aa63-bb246f2c9b73.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>In recent days, I’ve been diving deeper into React and exploring not just how to use it, but how it actually works under the hood. While most tutorials do a great job explaining concepts step by step, it’s easy to feel overwhelmed when trying to connect everything—components, hooks, reconciliation, and rendering—all at once.</p>
<p>This article is my attempt to simplify those internal concepts and build a clear mental model of how React works behind the scenes, especially focusing on Fiber, rendering phases, and how updates flow through the system.</p>
<p>In this blog, I assume that you already know the basics of React, such as:</p>
<ul>
<li><p>How to create functional components.</p>
</li>
<li><p>How to use <code>Vite</code> to create a React project.</p>
</li>
<li><p>What props are.</p>
</li>
<li><p>The basic usage of React Hooks.</p>
</li>
</ul>
<p>These are the prerequisites for understanding this blog.</p>
<p>If we take a look at the project structure of React, we see a single <code>index.html</code> file. That is one of the reasons React applications are called Single Page Applications (SPAs).</p>
<p>This HTML file contains a <code>div</code> element with an <code>id="root"</code>. This is the element where all React components are rendered.</p>
<h2>React element Objects</h2>
<p>These are like normal javascript objects that holdZ properties of an element.</p>
<pre><code class="language-javascript">const App = () =&gt;{
    return(
        &lt;div&gt;App Component&lt;/div&gt;
    )
}
</code></pre>
<p>When we use <code>console.log(App())</code>, it outputs an object:</p>
<pre><code class="language-json">{
    "$$typeof": Symbol(react.element),
    key: null,
    props: {children: "App Component"},
    ref: null,
    type: "div",
}
</code></pre>
<p>Each of those function calls gets converted into <code>React.createElement()</code>. That is the reason we import the React module whenever we create a <code>JSX</code> component. Each function call returns an element object, as shown above.</p>
<p>Whenever react parses the component function to the <code>React.createElement()</code> it is transpiled by babel.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/c12a0b3a-f0dd-46df-befa-b3e1c018b271.png" alt="" style="display:block;margin:0 auto" />

<p>These elements are:</p>
<ul>
<li><p>Immutable in nature</p>
</li>
<li><p>Lightweight</p>
</li>
</ul>
<p>They are just a description of the UI, not the actual DOM. Now, using this object, React creates an element tree, also known as the Virtual DOM.</p>
<p>In React we do not call the component as a function but instead we passes it as <code>&lt;App/&gt;</code> html tag. React first creates an object using <code>React.createElement()</code>. If the <code>type</code> property is a function, React calls it. This returns more JSX, which is again converted into <code>React.createElement()</code> calls. Eventually, we get an object that can be used to create the Virtual DOM.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/092b4680-6a21-4dbc-829a-86224732a060.png" alt="" style="display:block;margin:0 auto" />

<p><strong>Flow:</strong></p>
<ol>
<li><p>JSX</p>
</li>
<li><p><code>React.createElement(App)</code></p>
</li>
<li><p>Creates element object {type: App}</p>
</li>
<li><p>React sees type is a function</p>
</li>
<li><p>calls App()</p>
</li>
<li><p>Gets more JSX</p>
</li>
<li><p>Converts again in objects</p>
</li>
<li><p>Builds Virtual DOM Tree</p>
</li>
</ol>
<h2>What is Virtual DOM?</h2>
<p>You have heard the term Virtual DOM many times in this blog. The Virtual DOM is created by React. When React renders a component, it creates a tree of React elements (plain JavaScript objects), which are used to construct and update the Virtual DOM internally. After the creation of the Virtual DOM, it is synced with the browser DOM.</p>
<p>When the page loads for the first time, the Virtual DOM is fully rendered and inserted into the browser DOM.</p>
<p>If some part of the tree changes, updating the entire tree is not an efficient approach. Instead, React compares the previous and the updated trees and determines the minimum number of changes required to update the DOM. This process is done using a <strong>diffing</strong> algorithm.</p>
<h2>The Diffing Algorithm</h2>
<p>This is the main algorithm that compares changes between two trees and updates the DOM accordingly. It helps maintain the performance of the application because it reduces the time required to render elements.</p>
<p>Assumptions:</p>
<ul>
<li><p>Different trees are created when there are elements of two different types.</p>
</li>
<li><p>In a list of child elements that often changes, we should provide a unique <code>key</code> as a prop. We can see that in the element object, <code>key</code> is a separate property. This is because it plays a crucial role in the diffing algorithm.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/f18bcbde-134b-4c95-a4d0-edfcec3bfda2.png" alt="" style="display:block;margin:0 auto" /></li>
</ul>
<p>React will rebuild the tree from the start if the elements inside <code>div</code> changes.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/60d407ef-570e-4bfb-9134-b6b81ba88b3a.png" alt="" style="display:block;margin:0 auto" />

<p>In this case, the attributes are updated while the elements remain the same, so the state is preserved. The tree is not fully re-rendered.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/086c516b-0efd-41fd-9bb1-abe6991e8473.png" alt="" style="display:block;margin:0 auto" />

<p>First, React compares the initial items in the list. If they are the same, it then detects any changes in the Virtual DOM and appends the new element to the end of the <code>&lt;ul&gt;</code> in the browser DOM.</p>
<p>If we add an element at the beginning, React will regenerate the tree, which is less efficient. Instead, we can use the <code>key</code> property to tell React which element is new.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/9c57688a-948d-4a62-b7c2-43f49d0055fb.png" alt="" style="display:block;margin:0 auto" />

<h2>Rendering</h2>
<p>React itself doesn’t directly update the UI. Instead, it relies on a renderer like React DOM to apply changes to the browser.</p>
<p>After React determines what needs to change during the render phase, it passes those updates to the renderer. The renderer then performs the actual DOM operations, such as creating, updating, or removing elements.</p>
<p>In simple terms:</p>
<ul>
<li><p>React decides <strong>what should change</strong></p>
</li>
<li><p>The renderer decides <strong>how to update the UI</strong></p>
</li>
</ul>
<p>Understanding how React works internally can feel overwhelming at first, but breaking it down into concepts like Fiber, rendering phases, and the role of the renderer makes it much easier to grasp.</p>
<p>At a high level, React is all about efficiently calculating changes and updating the UI in a predictable way. Once you understand how it separates <em>what to update</em> from <em>how to update</em>, many advanced concepts start to make sense naturally.</p>
<p>This article reflects my learning journey, and there’s still more to explore. If you’re learning React too, take it step by step—these concepts become clearer with time and practice.</p>
<h3>Resources</h3>
<ul>
<li><p><strong>React internals</strong> - <a href="https://youtu.be/7YhdqIR2Yzo?si=FCVd0b9WwqZhqlh3">https://youtu.be/7YhdqIR2Yzo?si=FCVd0b9WwqZhqlh3</a></p>
</li>
<li><p><strong>React Fiber</strong> - <a href="https://youtu.be/0ympFIwQFJw?si=WRmbB78F53atzRPu">https://youtu.be/0ympFIwQFJw?si=WRmbB78F53atzRPu</a></p>
</li>
<li><p><strong>Full Stack React Course</strong> - <a href="https://youtu.be/Bvwq%5C_S0n2pk?si=Q8vaDNw4n%5C_TivpIx">https://youtu.be/Bvwq\_S0n2pk?si=Q8vaDNw4n\_TivpIx</a></p>
</li>
</ul>
]]></content:encoded></item><item><title><![CDATA[MongoDB Aggregation Pipeline: From Basics to Behind the Scenes]]></title><description><![CDATA[In this blog, you are going to understand the internals of the MongoDB aggregation pipeline. Most of us, when we first face this aggregation pipeline, get confused about why they use the $ sign inside]]></description><link>https://lavkush-maurya-blogs.hashnode.dev/mongodb-aggregation-pipeline-from-basics-to-behind-the-scenes</link><guid isPermaLink="true">https://lavkush-maurya-blogs.hashnode.dev/mongodb-aggregation-pipeline-from-basics-to-behind-the-scenes</guid><category><![CDATA[software architecture]]></category><category><![CDATA[MongoDB]]></category><category><![CDATA[Aggregation Pipeline]]></category><category><![CDATA[Mongodb Aggregation Pipeline]]></category><category><![CDATA[Query]]></category><category><![CDATA[query engine]]></category><dc:creator><![CDATA[Lavkush Maurya]]></dc:creator><pubDate>Tue, 07 Apr 2026 02:50:41 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/14a0f90f-3ea5-4be6-96ce-f8ac4d92a80f.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>In this blog, you are going to understand the internals of the MongoDB aggregation pipeline. Most of us, when we first face this aggregation pipeline, get confused about why they use the <code>$</code> sign inside [] brackets. Inside the [] brackets are the stages. These stages are created using {} braces. Each Stage is using operations to transform the data. e.g. group, project, addField, lookup etc..</p>
<p>The aggregation pipeline looks like:</p>
<img src="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/ece79d7a-1399-4aa8-b3e3-3a5654ba8f1a.png" alt="" style="display:block;margin:0 auto" />

<p>Now we are going to understand that behind this aggregation pipeline, there is a mini query engine that handles each step of the aggregation process. I also provided the technical jargons for each step.</p>
<hr />
<h2>Step 1: Parsing the Data</h2>
<h3>Parsing, Logical Plan, AST(Abstract Syntax Tree)</h3>
<p>At first MongoDB reads this pipeline and turn it into a structure of steps. This helps to analyze and execute the query more efficiently.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/71ae6caf-ab81-40f1-85e4-ae1df2f95171.png" alt="" style="display:block;margin:0 auto" />

<h2>Step 2: Query Optimizer</h2>
<h3>Predicate Pushdown, Stage Re-ordering</h3>
<p>It reorder the query to make it faster to execute and improves performance without changing the result.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/5cab4709-2619-4799-99bc-c02f15cea7f2.png" alt="" style="display:block;margin:0 auto" />

<h2>Step 3: Query Planner</h2>
<h3>IXSCAN, COLLSCAN, B-Tree Index</h3>
<p>It decides the proper way to fetch data. IXSCAN refers to index-based scanning, whereas COLLSCAN means scanning the entire collection. The index used in IXSCAN can be on any field, not just <code>_id</code>, but also combinations like <code>{ _id, name }</code>.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/a4b69296-01b1-4721-8716-2508a3baa5b2.png" alt="" style="display:block;margin:0 auto" />

<h2>Step 4: Physical Plan</h2>
<h3>Execution Plan, Operators</h3>
<p>It converts the optimized plan into a physical execution plan, where MongoDB decides how each stage will actually run using specific operators and algorithms.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/27b3518b-9ea2-48f6-ab9d-64ce4ab70867.png" alt="" style="display:block;margin:0 auto" />

<h2>Step 5: Execution Engine</h2>
<h3>Slot Based Execution Engine(SBE)</h3>
<p>It uses small memory boxes (slots) instead of moving full documents, which makes execution faster and more efficient.</p>
<pre><code class="language-javascript">{
    "customerId": 01,
    "amount": 10000,
    "total": 1000,
}
</code></pre>
<img src="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/8c32c3a4-d5d8-4145-968a-f34129973b86.png" alt="" style="display:block;margin:0 auto" />

<h2>Step 6: Iterator Model</h2>
<h3>Volcano Model, next()</h3>
<p>Each stage processes data by pulling one document at a time from the previous stage using the <code>next()</code> function.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/79ba9144-ae6c-457e-8223-02f069e9298c.png" alt="" style="display:block;margin:0 auto" />

<h2>Step 7: Streaming VS Blocking</h2>
<p>Streaming refers to processing one document at a time. Blocking refers to processing all the data only after it is fully available. Aggregation operations like <code>match</code> and <code>project</code> are considered streaming, while <code>group</code> and <code>sort</code> are considered blocking.</p>
<h2>Step 8: Stage Algorithms</h2>
<h3>Hash Aggregation, External Merge Sort, Nested Loop Join</h3>
<p>It defines which algorithm is used in each stage to perform the operation.</p>
<p>e.g. group - <code>Hash Table</code>, sort - <code>chunks-&gt;sort-&gt;merge-&gt;Result</code></p>
<img src="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/f8bccd14-69fa-4fd0-8f3d-7b15985fcdf0.png" alt="" style="display:block;margin:0 auto" />

<h2>Step 9: Expression Evaluation</h2>
<h3>Expression Tree, Evaluation Engine</h3>
<p>It uses an expression evaluation engine to process expressions (like <code>sum</code>, <code>multiply</code>) by converting them into expression trees and evaluating them for each document.</p>
<img src="https://cdn.hashnode.com/uploads/covers/69d3e64d40c9cabf445715f5/9382d1a9-1ffe-40d5-9c3f-c2115d1e4733.png" alt="" style="display:block;margin:0 auto" />

<p>What happens internally:</p>
<pre><code class="language-plaintext">Doc1 → price = 100, quantity = 2 → totalPrice = 200  
Doc2 → price = 50, quantity = 3 → totalPrice = 150  
</code></pre>
<p>For each document, MongoDB evaluates expressions like <code>multiply</code> by reading field values, applying the operation, and storing the result.</p>
<h2>Step 10: Memory Management</h2>
<h3>Memory Threshold, Disk Spill, External Processing</h3>
<p>It processes small data in RAM, but when the data size exceeds the limit (around 100MB), it uses disk storage temporarily.</p>
<h2>Step 11: Final output (Cursor)</h2>
<h3>Cursor, Batching, Lazy Fetching</h3>
<p>Instead of sending all results at once, it sends results gradually using a cursor. This saves memory, provides a faster response, and is scalable for large datasets.</p>
<hr />
<p>The aggregation pipeline is not just a sequence of operations. it is a mini query engine that processes data efficiently using different strategies and algorithms.</p>
<p>Understanding this makes you a better developer when working with MongoDB.</p>
]]></content:encoded></item></channel></rss>