But when I looked at outcomes, the pattern was obvious: I had become excellent at preparing and inconsistent at executing.

That’s the trap.

Preparation is useful, but it can also become a socially acceptable way to delay exposure. Instead of shipping one rough thing, we polish our inputs. We save another article, queue another video, revise the plan one more time, and tell ourselves we’re “still working on it.” Technically true. Practically stagnant.

The hardest part is that this behavior doesn’t feel like procrastination. It feels smart. It feels thoughtful. It feels like you’re reducing risk.

In reality, you’re often postponing feedback.

And feedback is the only thing that makes you better.

I like thinking about this in ML terms. A book, a course, or a framework is like a high-quality model trained on someone else’s data. Useful, absolutely. But your life is the production environment. Your constraints, your timing, your pressure, your edge cases none of that is in the training set. Until you deploy in your own environment, you don’t know what survives contact with reality.

No deployment means no signal. No signal means no improvement.

What changed things for me was a small but strict rule: every preparation session has to end with one concrete output. Not better notes. Not a cleaner plan. Output.

If I studied system design, I had to publish one architecture sketch. If I watched an LLM tutorial, I had to run one tiny experiment. If I planned a side project, I had to ship one narrow slice that same day.

The output could be small and imperfect. It just couldn’t be imaginary.

That one constraint exposes everything. It shows whether I’m learning to apply or learning to avoid. It also makes momentum real, because execution compounds in a way preparation never does.

I still plan. I still read. I still research deeply. But now I treat preparation as support work, not the main work.

If you feel stuck, try this for a month: prepare less than you execute. Keep learning, but force conversion. The goal is not to know more in private. The goal is to become effective in public.

Because in the end, unread books, saved threads, and polished plans don’t change your trajectory.

Shipped work does.

The GitHub repo is here: https://github.com/abhinav-TB/VaultClip

Live demo: https://vaultclip.gamezdev.workers.dev

VaultClip is an open-source, browser-only video Q&A app. You select a video or audio file, the app extracts the audio locally, transcribes it locally, builds searchable context locally, and lets you chat with the media using local AI. No uploads. No accounts. No backend. No analytics pipeline quietly collecting your files.

That constraint, everything happens in the browser shaped almost every technical decision in the project.

Why build this?

A lot of useful information is trapped inside videos: lectures, meeting recordings, interviews, demos, tutorials, walkthroughs, research calls, debugging sessions, and personal notes. The usual way to find something is still manual scrubbing: drag the timeline, listen for a few seconds, miss it, go back, try again.

Cloud AI tools can solve that problem, but they often require uploading the file. That is fine for public content, but it is a bad fit for private recordings. A meeting recording, an interview, a classroom lecture, or an unreleased demo may contain sensitive information. Even when a provider has good policies, the user still has to trust an external system with the media.

VaultClip takes the opposite approach:

• the file stays on your machine

• inference runs in your browser

• temporary data stays in browser memory

• after the initial model download, the app can work offline

• the code is open source and auditable

The goal was not just to make video Q&A work. The goal was to make it work while preserving the user's control over the media.

The product idea

The user flow is intentionally small:

1. Pick a video or audio file.

2. Preview it in the browser.

3. Let VaultClip extract and transcribe the audio.

4. Ask natural-language questions.

5. Get answers grounded in timestamped transcript context.

Examples of questions:

• "What were the main decisions in this meeting?"

• "When does the speaker explain the architecture?"

• "Summarize the action items."

• "Find the part where they talk about latency."

• "What are the objections raised in this interview?"

The important part is that the app does not need to upload the file to answer these questions.

Architecture at a high level

VaultClip is a React + TypeScript single-page app built with Vite. The heavier work is moved out of the UI thread into Web Workers, because media processing and model inference can easily freeze the browser if handled directly in React components.

The pipeline looks like this:

User selects media
        ↓
Validate file size, duration, and format
        ↓
Preview the file locally
        ↓
Extract audio with ffmpeg.wasm in a Web Worker
        ↓
Transcribe audio chunks with Gemma via Transformers.js + WebGPU
        ↓
Create timestamped transcript segments
        ↓
Build local retrieval context
        ↓
Answer questions in a local chat interface

The current repository is here:

https://github.com/abhinav-TB/VaultClip

The app is deployed as a static app on Cloudflare Workers, but the deployed app is still not a backend for user files. The Worker serves the frontend assets and handles app routing. The actual media processing happens in the user's browser.

Why browser-only AI is interesting

For a long time, "AI app" almost automatically meant "send data to an API." That is still the right architecture for many products, especially when models are large or collaboration matters. But browsers have become much more capable.

With WebGPU, Web Workers, WebAssembly, and libraries like Transformers.js, it is now realistic to run useful local AI workflows inside a normal web app. The browser can:

• decode and preview media

• run ffmpeg.wasm for audio extraction

• use WebGPU for accelerated inference

• cache model files locally

• manage temporary data in memory

• provide a familiar UI without installation

This makes a new class of apps possible: local-first AI tools that feel like web apps but behave more like private desktop utilities.

VaultClip is an experiment in that direction.

Keeping media private

Privacy in VaultClip is not just a sentence in the README. It is a design constraint.

The selected media file is stored through the browser's File API. It is not serialized into Redux, not uploaded to a server, and not persisted to a remote database. The extracted audio is also kept locally, and object URLs are cleaned up when they are no longer needed.

The app has no backend API for media. The only unavoidable network step is the initial download of model assets. Once the model is cached, the workflow can run locally.

This matters because video files are often high-context. They can include faces, voices, screenshares, credentials accidentally visible in a demo, internal project names, customer details, or personal conversations. A privacy-first architecture should not ask for that data unless it truly needs it.

VaultClip's answer is: for this use case, the app does not need it.

The local processing pipeline

The first technical challenge is turning an arbitrary media file into something the model can use.

Videos are large, varied, and browser support depends on codecs. VaultClip starts with guardrails: file size limits, duration limits, supported formats, and browser preview checks. These constraints are not glamorous, but they make the app much more reliable. Browser memory is finite, and local inference gets expensive quickly.

Once the user confirms the file, VaultClip extracts audio using ffmpeg.wasm inside a Web Worker. That keeps the UI responsive while the app does CPU-heavy work. The extracted audio is converted into a transcription-friendly format, such as WAV at 16 kHz.

The transcription step processes audio in chunks. Chunking matters because long media files can exceed practical generation limits, and smaller chunks make progress easier to track. VaultClip keeps timestamp ranges for transcript segments so answers can point back to where the information came from.

After transcription, the app prepares local context for question answering. Instead of sending the whole transcript to a server, VaultClip builds a local retrieval layer and feeds relevant context into the chat flow.

Running Gemma in the browser

VaultClip uses Gemma through Transformers.js with WebGPU acceleration. The model lifecycle has a few stages:

1. load or fetch model assets

2. initialize the runtime

3. warm up inference

4. process transcription and chat tasks

5. surface progress and failure states in the UI

This is very different from calling a hosted API. With an API, the hard work is hidden behind an HTTP request. In the browser, the app has to care about model download size, browser support, GPU availability, memory pressure, initialization time, and failure recovery.

That tradeoff is worth it for VaultClip because the privacy model becomes much stronger. The user's media does not leave the device just because they want an AI summary.

State management and workers

One lesson from the project: do not put large browser objects directly into global app state.

VaultClip uses Redux Toolkit for serializable UI and workflow state, but the actual File objects and raw audio bytes live in separate registries. Redux stores metadata like file name, size, duration, object URL, processing status, transcript segments, and model state.

That separation keeps the app easier to debug and avoids pushing non-serializable heavy objects through the normal React state flow.

The Web Worker boundary is also important. The UI should not know too much about how ffmpeg or the model runtime works. Instead, a worker client provides a promise-based interface for long-running tasks. This keeps the app structure cleaner:

• React handles the user experience

• Redux tracks state transitions

• workers handle expensive compute

• small registry modules own large in-memory objects

• shared types define the contracts between pieces

That architecture made the app easier to grow without turning every component into an orchestration layer.

The hard parts

The hardest parts were not the UI screens. The hard parts were browser limitations and edge cases.

Browser memory is not straightforward. Web APIs do not give a reliable cross-browser way to know total system memory or GPU memory. That means the app needs conservative limits and clear failure states instead of pretending it can process every file.

WebGPU support is also still uneven. Chrome and Edge are the best targets right now. Firefox and Safari are improving, but for this kind of app, WebGPU support is a real requirement.

Another subtle issue is transcription coverage. Longer chunks can produce incomplete coverage or weaker timestamp behavior, so VaultClip defaults to smaller chunk sizes. The app is designed around practical reliability instead of theoretical maximum throughput.

There is also the first-run experience: downloading a local model can be large. That is not as seamless as an API call, but the upside is that the model can be cached and reused locally.

What VaultClip is good for

VaultClip is useful when privacy matters more than cloud-scale convenience.

Some examples:

• students reviewing long lectures

• engineers searching through demo recordings

• researchers analyzing interviews

• creators reviewing raw footage

• professionals summarizing meeting recordings

• anyone who wants Q&A over media without uploading it

It is not trying to replace every cloud transcription or video intelligence product. It is designed for the local-first case: give me useful AI over my file, but keep the file with me.

Current limitations

The current version is still an MVP. The main limitations are:

• WebGPU is required for the best experience

• first-time model download is large

• browser memory limits constrain file size and duration

• transcript timestamps are segment-level, not word-level

• the app focuses on one active media session at a time

• very long recordings still need better chunking and persistence strategies

These constraints are acceptable for the first version because they keep the privacy model simple and the architecture understandable.

Why open source matters here

For privacy-focused tools, open source is not just a nice extra. It is part of the trust model.

If an app claims "your files never leave your device," users should be able to inspect how that is implemented. VaultClip's code is available here:

https://github.com/abhinav-TB/VaultClip

The repo includes the architecture, worker pipeline, model runtime integration, media guardrails, deployment setup, and local development instructions.

If you want to run it locally:

git clone https://github.com/abhinav-TB/VaultClip.git
cd VaultClip
npm install
npm run dev -- --host 127.0.0.1

Then open http://127.0.0.1:5173 in a WebGPU-capable browser like Chrome or Edge.

What I learned

The biggest lesson from building VaultClip is that browser AI is no longer just a toy category. It still has rough edges, but it is useful enough for real workflows when the scope is chosen carefully.

The second lesson is that privacy-first design needs to happen at the architecture level. You cannot bolt it on at the end. If the core workflow assumes uploads, then privacy becomes policy language. If the core workflow keeps data local, privacy becomes a property of the system.

VaultClip is my attempt to build the second kind of app.

Closing thoughts

We are used to thinking of the browser as a thin client for server-side intelligence. But for many personal AI tasks, especially ones involving private files, the browser can be the compute environment.

VaultClip shows that a modern web app can process media, run local models, build retrieval context, and provide useful Q&A — all without asking the user to hand over their video.

If you are interested in local-first AI, browser ML, WebGPU, or privacy-preserving media tools, check out the project here:

https://github.com/abhinav-TB/VaultClip

And try the live demo here:

https://vaultclip.gamezdev.workers.dev

If you find the idea useful or want to support more local-first browser AI experiments, please consider giving the GitHub repo a star. It helps the project reach more people who care about private AI tools.

It sounds obvious at first: if the assistant remembers everything, it should become more useful over time. It should know your preferences, your projects, your writing style, your tools, your schedule, and all the tiny details that usually get lost between conversations.

But in practice, an assistant that remembers everything becomes noisy.

It starts carrying around stale assumptions. It overfits to one-off conversations. It treats temporary preferences like permanent truth. It may remember the exact wrong thing with too much confidence.

The real question is not: how do we make the assistant remember more?

The better question is: what should an assistant actually remember?

And just as importantly: when should something become memory, when should it become a skill, and when should it simply be forgotten?

Memory is not a transcript

A useful assistant should not treat memory as a giant chat log.

A transcript is what happened. Memory is what remains useful later.

Those are different things.

If I tell an assistant, “I’m debugging a Next.js issue right now,” that may be useful for the current session. But it probably does not need to become long-term memory.

If I tell it, “When I write technical notes, I prefer concept-first explanations with concrete examples,” that is durable. It should remember that.

If I say, “Use this folder for my Obsidian notes,” that is also durable. Forgetting it causes repeated friction.

The assistant should not store every message. It should extract stable facts, preferences, workflows, and decisions that reduce future back-and-forth.

Good memory feels invisible. You notice it because the assistant stops asking the same question twice.

Bad memory feels creepy or annoying. You notice it because the assistant brings up irrelevant details or acts like yesterday’s temporary context is now your identity.

The three buckets: context, memory, and skills

I think assistant knowledge should be split into three buckets.

1. Temporary context

This is the working set for the current task.

Examples:

• the bug we are debugging right now

• the article draft currently open

• the command output from five minutes ago

• the user’s latest instruction in this thread

• a design choice that only matters for this feature branch

Temporary context should be easy to use and easy to discard. Not everything needs ceremony. Most things belong here.

2. Long-term memory

This is durable information that will probably be useful across many future sessions.

Examples:

• the user’s preferred name and timezone

• canonical folders, repos, and tools

• writing style preferences

• recurring project goals

• important constraints like “do not send emails without asking”

• personal workflows that are stable but not complex enough to become a skill

Memory should answer the question: will this help the assistant behave better next week?

If the answer is no, it probably does not belong there.

3. Skills

Skills are different from memory.

Memory says: remember this fact or preference.

A skill says: when this kind of task appears, follow this repeatable procedure.

That distinction matters.

For example, “Abhi prefers detailed beginner-friendly notes” is memory.

But “when Abhi asks to create a tutorial note, save it to the canonical Obsidian folder, use this structure, include examples, and run a Nextcloud scan afterward” is a skill.

The skill is not just remembering a preference. It is packaging a workflow.

When repeatable work becomes a skill

Some agent systems make this workflow explicit. In OpenClaw, for example, a skill is a small instruction package the agent can load when a certain kind of task appears. Hermes-style systems use a similar idea: repeated work should not live only in the model’s memory or in scattered chat history. It should become a reusable workflow.

The idea is simple: if an agent repeatedly performs the same kind of task, capture the procedure as a skill. The next time that task appears, the agent does not have to rediscover the process from scratch.

That is extremely useful.

But it also creates a temptation: turning everything into a skill.

I do not think that is the right move.

If every preference, tiny habit, and one-off instruction becomes a skill, the system becomes heavy. The assistant spends more time deciding which skill applies than actually helping. Skills start overlapping. Some become stale. Others encode workflows that were only useful once.

A skill should exist when there is a repeatable pattern with enough structure to justify it.

Good candidates for skills:

• writing a Hashnode-ready blog draft

• generating a tailored resume from a job description

• creating a tutorial-style Obsidian note

• triaging GitHub issues

• running a health check on a server

• using a specific browser automation flow

• preparing a release checklist

Bad candidates for skills:

• “I like concise replies”

• “Use this nickname for me”

• “This project is important this week”

• “Remember that I prefer Python examples”

• “For this one article, use a slightly more opinionated tone”

Those belong in memory or temporary context.

A good rule of thumb:

If it is a stable preference, store it in memory. If it is a repeatable procedure, make it a skill. If it only matters right now, keep it in context.

OpenClaw’s approach: tell the agent what to store or create

In OpenClaw, this distinction becomes practical.

You can tell the agent to remember things. You can also tell it to create or update skills for repeatable workflows.

That gives the user more control over the assistant’s evolution.

Instead of hoping the model magically guesses what matters, the system can support explicit instructions like:

Remember that I prefer technical notes with detailed examples.

or:

Create a skill for writing my blog posts so future drafts follow this structure.

or:

Update the resume tailoring skill with this new rule.

This is important because users should not have to re-explain stable preferences forever, but they also should not have every random message fossilized into permanent memory.

The assistant should be proactive, but not presumptuous.

A strong agent should sometimes say:

This looks like a repeatable workflow. Want me to turn it into a skill?

Or:

This seems like a durable preference. I’ll remember it for next time.

That small distinction makes the assistant feel much more trustworthy.

What should actually go into memory?

Here is my personal filter.

An assistant should remember things that are:

Durable

They remain true beyond the current session.

“Use my Nextcloud Obsidian vault for notes” is durable. “I am currently comparing two blog titles” is not.

Actionable

They change how the assistant behaves.

“User likes practical examples” is actionable. “User once mentioned React in passing” is probably not.

User-approved or clearly implied

Some things should only be stored when the user asks. Personal details, sensitive context, and preferences that could be wrong should not be silently collected.

The assistant can infer lightweight working context, but durable memory should be handled carefully.

Small enough to stay useful

Memory should be compressed.

Do not store the whole conversation. Store the lesson.

Instead of:

On Tuesday, the user asked me to write a note about web workers, then corrected the examples, then said they like this style...

Store:

User prefers technical notes that are concept-first, sectioned, and include detailed examples in the style of the web-workers note.

That is useful.

What should become a skill?

A workflow should become a skill when it has triggers, steps, and quality checks.

For example, “write a blog post” is not just a preference. It has a shape:

• detect the topic

• create a strong title and slug

• include frontmatter

• write in the user’s style

• save to the right folder

• run any required indexing step

• respond with the saved path

That deserves a skill.

The same applies to job applications, note generation, browser automation, server health checks, inbox triage, and other repeatable workflows.

The key is that a skill should reduce future cognitive load. It should make the agent more reliable, not more bureaucratic.

If a skill does not improve reliability, it is probably just documentation wearing a costume.

My take: memory should be boring, skills should be opinionated

This is the part I care about most.

Memory should be boring.

It should store stable facts, preferences, constraints, and pointers. It should not be dramatic. It should not try to model the user’s soul from three messages. It should not turn every interaction into permanent identity.

Skills, on the other hand, should be opinionated.

A good skill should encode a clear way of doing a task. It should have defaults. It should know the destination folder. It should know the expected format. It should know what “done” means.

Memory helps the assistant know you.

Skills help the assistant do things for you.

Those are related, but they are not the same.

The danger of too much automatic memory

A lot of AI products talk about memory like it is magic. But automatic memory has real failure modes.

It can store stale information. It can create privacy problems. It can make the assistant overconfident. It can also make users feel watched instead of helped.

There is a subtle difference between:

“Nice, it remembered how I like my notes structured.”

and:

“Why did it bring up that random thing I said two months ago?”

The first feels useful. The second feels invasive.

This is why I prefer memory systems where the assistant can write things down, but the user can also explicitly say what should be remembered, forgotten, or promoted into a skill.

The assistant should not be a hoarder. It should be a careful librarian.

A practical framework

When deciding what to do with new information, the agent can ask three questions.

First:

Will this matter after the current task?

If no, keep it in temporary context.

Second:

Is this a stable fact, preference, or constraint?

If yes, store it in memory.

Third:

Is this a repeatable workflow with steps and quality checks?

If yes, create or update a skill.

That is the whole system.

Context is for now.

Memory is for later.

Skills are for doing it better next time.

Closing thoughts

The best AI assistant is not the one that remembers the most.

It is the one that remembers the right things, forgets the noise, and turns repeated work into reliable workflows.

That is the difference between a chatbot with a long transcript and an actual assistant that compounds over time.

For OpenClaw, I think the sweet spot is clear: let the user tell the agent what to remember, let the agent suggest when something should become a skill, and keep the boundary between memory and skills sharp.

Not every detail deserves permanence.

Not every workflow deserves a skill.

But when the assistant gets the distinction right, it starts to feel less like software you operate and more like a teammate who actually learns how to work with you.

A Brief History: From RNNs to “Attention Is All You Need”

Recurrent Neural Networks (RNNs) and their gated variants (LSTMs) once dominated sequence modeling, but they struggled with long-range relationships due to sequential processing and vanishing gradients. In 2014, Bahdanau et al. introduced the attention mechanism to help RNN-based translation models focus on relevant parts of the input sequence at each decoding step. Instead of encoding an entire sentence into a single vector (a “context vector”) as in vanilla sequence-to-sequence models, attention allowed the decoder to dynamically look at (“attend to”) specific encoder states (words) when producing each output word. This alignment mechanism significantly improved translation quality by bypassing the fixed-length bottleneck of RNNs.

Fast-forward to 2017: Vaswani et al.’s paper “Attention Is All You Need” dispensed with recurrence altogether and proposed the Transformer – an architecture built entirely on attention layers and feed-forward networks. The Transformer’s breakthrough was showing that you can achieve state-of-the-art results in translation (and soon many other tasks) using only attention mechanisms in a highly parallelizable model, making training faster than RNNs. The Transformer introduced several key ideas we’ll unpack below: self-attention, multi-head attention, positional encoding, and residual connections. These innovations enabled models to capture global context and long-term dependencies more effectively than RNNs

Transformer Fundamentals: Attention and Friends

Before building our mini GPT, let’s understand the core concepts of the Transformer architecture in an approachable way.

Self-Attention: Paying Attention to Context

At the heart of Transformers is the self-attention mechanism. In simple terms, self-attention lets every token (word or subword) in a sequence look at every other token and decide how much attention to give to each. This means each word can gather useful information from the whole sequence when computing its representation, rather than only from nearby words or previous words. As a result, the model can capture relationships like long-distance dependencies (e.g. pronoun-reference links, subject-verb agreements) much better than an RNN, which processes words one by one

How does self-attention work? Imagine we have a sentence, and we want to compute new representations for each word that include contextual information. The Transformer does this by creating three vectors for each token: a Query vector (Q), a Key vector (K), and a Value vector (V). You can think of this like an information retrieval system: each token’s Query is like a question, and each other token has a Key (an identifying summary) and a Value (the content info). To figure out how much one word should pay attention to another, we compute a similarity score between the Query of the first word and the Key of the second word (for all pairs). These scores (often scaled dot-products) form an attention matrix of size n×n (for n tokens). We then apply a softmax to turn these scores into attention weights that sum to 1 for each query word. Finally, each word’s output representation is computed as a weighted sum of all Value vectors, using these attention weights.

In essence, self-attention produces a context-aware representation for each token. If word A’s Query aligns strongly with word B’s Key, then word A’s representation will include a lot of word B’s information (Value). For example, in the sentence “The animal didn’t cross the road because it was tired,” the pronoun “it” might have a high attention weight on “animal”, effectively letting “it” incorporate context from “animal” into its representation to clarify the reference

This mechanism is powerful because it’s order-agnostic (any token can attend to any other, regardless of position) and parallelizable (all token relationships can be computed simultaneously). It gives Transformers the flexibility to capture both local and long-distance relationships in a sequence. Notably, Vaswani et al. used a specific form called scaled dot-product attention: they compute scores as dot products of Q and K, scale by √d_k (the vector dimension) to stabilize gradients, and softmax

Positional Encoding: Adding Order to Sequences

One catch: since self-attention doesn’t inherently know the positions of tokens (unlike RNNs, which process in order), Transformers need a way to inject sequence order information. This is done via positional encoding. In the original Transformer, they added a deterministic positional embedding to token embeddings using sine and cosine functions of varying frequencies for each position Intuitively, these positional encodings produce a unique pattern for each position; the model can learn to infer relative positions (e.g. one position’s encoding dot-product another’s indicates how far apart they are). In practice, some implementations use learned positional embeddings (trainable vectors for each position index) instead of sinusoidal formulas, both work to give the model a sense of word order.

The result is that the input to the Transformer is enriched with positional information: TokenEmbedding + PositionEmbedding for each token. This allows the self-attention layer to be aware of sequence ordering (so it doesn’t treat a sentence as a bag of words).

Multi-Head Attention: Many Perspectives at Once

A single self-attention head (one set of Q, K, V projections) is great, but the Transformer improves on this with multi-head attention. Instead of computing one attention distribution, the Transformer uses multiple attention “heads” in parallel. Each head operates on the same input but with its own learned projection matrices, so it might focus on different aspects of the sequence. For example, one head could attend to syntactic relationships (e.g., noun-adjective), another to long-range dependencies, and yet another to anaphora resolution. The idea is that multiple heads allow the model to capture diverse types of relationships simultaneously.

In practice, if the model’s hidden size is \(d_{model}\), and we choose \(h\) heads, we project the input into \(h\) different subspaces of dimension \(d_k = d_{model}/h\). Each head yields its own attention output (size \(n \times d_v\); often \(d_v\) is chosen equal to \(d_k\)). We then concatenate the outputs of all heads (getting back \(n \times d_{model}\)), and pass it through a final linear layer so that multi-head attention still produces an output of the same dimension \(d_{model}\).

Formally, if \(head_i(Q,K,V)\) is the output of the \(i\)-th attention head, then:

\[MultiHead(Q,K,V) = Concat(head_1, ..., head_h) W^O\]

where \(W^O\) is a learned projection matrix.

Don’t worry if the details sound complex; the key takeaway is that each head can attend to different patterns or parts of the sequence, and the model merges this information. Empirically, this improves the model’s capacity and performance.

Feed-Forward Networks and Residual Connections

After the attention mechanism, each Transformer layer includes a simple position-wise feed-forward network (FFN). This is just an MLP applied to each token’s vector independently: typically a linear layer expanding the dimension (e.g., from \(d_{model}\) to some larger \(d_{ff}\)), a non-linearity (ReLU), then a linear layer back to \(d_{model}\). For example, in Vaswani et al., \(d_{model}=512\) and \(d_{ff}=2048\) – so a token’s 512-dim representation is converted to 2048-dim, non-linearly transformed, then back to 512-dim. This FFN gives the model extra transformation power at each position (learning higher-level features or mixing the information that attention collected). Importantly, because the feed-forward is applied to each position separately (no interactions across tokens), it does not destroy the contextual mixing done by attention; instead, it processes each token’s enriched representation in isolation. This also means it can be parallelized for all tokens.

Critically, Transformers employ residual connections (skip connections) and layer normalization around both the attention sub-layer and the FFN sub-layer. In each layer, the input to the sub-layer is added to its output (“adding the skip”) and then normalized. These residual connections help gradients flow and allow the model to retain the original input features while the sub-layer learns a refinement. Layer normalization stabilizes training by normalizing the output at each layer. In simple terms, the computation in one Transformer layer looks like:

Self-Attention layer:

\[X_{\text{attn}} = \text{SelfAttn}(\text{LayerNorm}(X_{\text{in}}))\]

Then:

\[X_{\text{mid}} = X_{\text{in}} + X_{\text{attn}}\]

(This applies attention to normalized input, then adds it back to the input).

Feed-Forward layer:

\[X_{\text{ffn}} = \text{FFN}(\text{LayerNorm}(X_{\text{mid}}))\]

Then:

\[X_{\text{out}} = X_{\text{mid}} + X_{\text{ffn}}\]

This "add & normalize" structure is repeated in every layer. It allows deep stacking of layers without losing the original signal or causing training to diverge.

Transformer Architecture Recap

Putting it all together, the original Transformer architecture consists of an Encoder (stack of N identical layers, each with self-attention + FFN) and a Decoder (stack of N layers, each with self-attention, encoder-decoder attention, aka cross-attention, and FFN). The encoder-decoder attention in each decoder layer allows the decoder to attend to the encoder’s output (source sentence) in addition to attending to previous decoder tokens. This is crucial for tasks like translation, where the decoder needs to focus on the source sequence. In purely generative settings (like GPT language models), we don’t use an encoder at all; we only have decoder blocks with self-attention.

One important detail for decoder self-attention (used in language generation) is masking. The decoder operates autoregressively: when predicting the next token, it should not peek at later positions. Thus, decoder self-attention uses a causal mask to zero out attention weights to future tokens. This mask is a triangular matrix that prevents information flow from position j to i if j > i. In training, this masking ensures the model can’t trivially see the ground-truth next word; at inference, it naturally only attends to already generated tokens.

Now that we’ve covered the theory, let’s get our hands dirty with a practical implementation of these ideas in a mini GPT model.

Building GPTMini: An End-to-End Example

Our project, GPTMini, is a scaled-down GPT-like language model. We will outline each component: a custom Byte-Pair Encoding tokenizer, preparing the WikiText dataset, defining the Transformer (decoder-only) architecture in code, training the model, and finally generating text. The goal is to demystify how these pieces fit together in code, step by step.

Byte-Pair Encoding (BPE) Tokenizer

Real-world Transformer models don’t typically operate on characters or whole words directly – they use subword tokenization. Byte-Pair Encoding (BPE) is a popular subword tokenization algorithm used by GPT-2, GPT-3, BERT, and others. BPE was originally a data compression technique (Gage, 1994) where the most frequent pair of bytes in a text is merged into a single symbol, iteratively. In NLP tokenization, we start with all characters (or bytes) as initial tokens and repeatedly merge the most frequent adjacent token pair to form a new token until we reach a desired vocabulary size. This allows common multi-character sequences (like “ing” or “tion”) to become one token, while rare sequences remain split into smaller pieces. The result is a vocabulary of subword units that balance vocabulary size and representation of word parts, handling unknown words gracefully (any word can be composed from subword pieces).

In GPTMini, we implemented a custom BPE tokenizer from scratch. First, we gather the corpus text and initialize the vocabulary with all unique characters. Then we count character pair frequencies and merge the most frequent pair into a new token, update the corpus by replacing that pair everywhere, and repeat. For example, suppose the corpus has “low” and “lowest” frequently – BPE might merge “l” + “o” -> “lo”, then perhaps “lo” + “w” -> “low”, and eventually “low” + “est” -> “lowest”. After enough merges, common words or chunks become single tokens.

Here’s a simplified snippet (for illustration) of how one might learn BPE merges in Python:

from collections import Counter

def learn_bpe_merges(corpus_tokens, num_merges):
    # corpus_tokens is a list of lists of tokens (start with char tokens per word)
    for _ in range(num_merges):
        # Count all adjacent token pairs in the corpus
        pair_counts = Counter()
        for tokens in corpus_tokens:
            for i in range(len(tokens)-1):
                pair = (tokens[i], tokens[i+1])
                pair_counts[pair] += 1
        if not pair_counts: 
            break
        # Find the most frequent pair
        best_pair = max(pair_counts, key=pair_counts.get)
        # Merge the best pair in all token sequences
        new_token = best_pair[0] + best_pair[1]
        for i, tokens in enumerate(corpus_tokens):
            corpus_tokens[i] = merge_tokens(tokens, best_pair, new_token)
        # Add new_token to vocabulary (and remove old tokens if needed)
        vocab.add(new_token)
    return vocab

# Helper to merge a specific pair in one token list
def merge_tokens(token_list, pair, merged_token):
    i = 0
    merged_list = []
    while i < len(token_list):
        if i < len(token_list)-1 and (token_list[i], token_list[i+1]) == pair:
            merged_list.append(merged_token)
            i += 2  # skip the merged pair
        else:
            merged_list.append(token_list[i])
            i += 1
    return merged_list

In practice, one would also handle spaces or use byte-level BPE (operating on raw bytes to include all characters, which GPT-2 did for robustness. After training BPE on our WikiText corpus, we save the merge rules and vocabulary. Then we can use it to encode any text into a sequence of token IDs and decode IDs back to text.

For example, after training, encoding a word like “transformers” might yield tokens ["trans", "form", "ers"] if those subwords are in the vocab. The tokenizer also appends special tokens as needed (like an end-of-sequence or padding token, though GPT usually doesn’t use start/end tokens explicitly, just an end-of-text token).

Why BPE? It provides a balanced vocabulary where common words are one token and uncommon words are split into meaningful pieces. This drastically reduces out-of-vocabulary issues compared to word-level tokenization and is more efficient than character-level (which would make sequences very long). BPE is used by GPT-2 and GPT-3, for instance, to achieve a vocabulary of around 50,000 tokens that can encode text from a variety of languages

Dataset Preparation with WikiText

For training GPTMini, we need a large text corpus. We chose the WikiText dataset, a popular benchmark for language modeling. WikiText-2 (the smaller version) contains about 2 million tokens of Wikipedia articles (extracted from “Good” and “Featured” quality articles), and WikiText-103 (the full version) has over 100 million tokens. These are clean, curated Wikipedia texts that are great for training a language model from scratch.

We downloaded the WikiText corpus and applied our BPE tokenizer to it, converting the text into a stream of token IDs. Since language models are usually trained to predict the next token in a sequence, we formatted the data as sequences of a fixed length (say, 128 tokens long). Each training sample is a snippet of the tokenized text (128 tokens), and the target is the same sequence shifted one position to the left (so at each position, the model learns to predict the next token). We slide this window through the entire corpus to generate many training examples. Alternatively, one can randomly sample chunks of text of length 128 each epoch.

Here’s how we can create training sequences in code:

token_ids = tokenizer.encode(full_text)  # convert entire text to list of token IDs
seq_length = 128
train_data = []
for i in range(0, len(token_ids) - seq_length):
    input_seq = token_ids[i : i+seq_length]
    target_seq = token_ids[i+1 : i+seq_length+1]
    train_data.append((input_seq, target_seq))

In practice, we wouldn’t explicitly materialize train_data as a list of all sequences (which could be huge); instead, we use a Dataset object that reads batches on the fly. For example, in PyTorch, you might create a torch.utils.data.Dataset that wraps the token list and yields slices of length 128. We also shuffle the data or randomize starting positions for each epoch to alleviate fixed segmentation.

After this preparation, we have a dataset of input-target pairs of token sequences. Now we’re ready to build the Transformer model that will learn to predict tokens.

The GPTMini Model: Transformer Decoder Architecture

GPTMini’s architecture is a Transformer decoder, essentially the same building blocks as the original Transformer’s decoder part, but possibly with fewer layers or smaller dimensions for simplicity. Conceptually, our model will do the following:

Take a sequence of input tokens, embed them, add positional encodings, pass them through a stack of Transformer decoder blocks (with self-attention and feed-forward sublayers), and output a probability distribution for the next token at each position. Because we train it with next-token prediction, when we feed a sequence, the model’s output at position t is trying to predict the token at t+1.

Let’s break down the components in code. We’ll use PyTorch-like pseudo-code for clarity:

Embeddings: We need an embedding layer for token IDs, and we’ll add positional encodings. If we use learnable position embeddings, we can just have a second embedding table for positions.

import torch.nn as nn

class GPTMini(nn.Module):
    def __init__(self, vocab_size, d_model, max_seq_len, n_layers, n_heads, d_ff):
        super().__init__()
        self.token_emb = nn.Embedding(vocab_size, d_model)
        self.pos_emb   = nn.Embedding(max_seq_len, d_model)
        self.layers    = nn.ModuleList([
            DecoderBlock(d_model, n_heads, d_ff) for _ in range(n_layers)
        ])
        self.norm      = nn.LayerNorm(d_model)
        self.out_head  = nn.Linear(d_model, vocab_size)

Here DecoderBlock will be a class implementing one Transformer decoder layer (self-attention + feed-forward). We pass in n_heads and d_ff (hidden size of feed-forward) as well. We also have a final linear out_head layer that maps the final hidden state of each position to vocabulary logits for prediction.

Self-Attention Sub-layer: We implement this as part of the DecoderBlock. We need to compute multi-head masked self-attention. Let’s define a single attention head first, then generalize.

A simplified single-head attention (without batching for brevity) could be:

def single_head_attention(Q, K, V, mask=None):
    scores = Q @ K.T              # (T_q × T_k) raw attention scores
    scores = scores / math.sqrt(Q.size(1))  # scale by √d_k
    if mask is not None:
        scores.masked_fill_(mask == 0, float('-inf'))  # mask out future positions
    weights = scores.softmax(dim=-1)   # (T_q × T_k) attention weights
    return weights @ V                # weighted sum of values (T_q × d_v)

In a multi-head setting, we have n_heads different projection matrices for Q, K, V. In code, we often combine them for efficiency. For example, we project the input X (of shape [batch_size, T, d_model]) into Q, K, V of shapes [batch_size, T, d_model] each, then split those into [batch_size, n_heads, T, d_k]. Each head then performs the above operation in parallel. After that, we concatenate the heads and project back to d_model. Here’s how a multi-head self-attention forward might look:

class MultiheadSelfAttention(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        assert d_model % n_heads == 0
        self.d_k = d_model // n_heads
        self.n_heads = n_heads
        # Projection layers for Q, K, V (we use one big weight for all heads for each)
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        # Output projection
        self.W_o = nn.Linear(d_model, d_model)

    def forward(self, X, mask=None):
        B, T, C = X.shape  # batch, seq_length, d_model
        # Project inputs to multi-head Q, K, V
        Q = self.W_q(X)  # shape (B, T, C)
        K = self.W_k(X)
        V = self.W_v(X)
        # Split into heads: reshape to (B, T, n_heads, d_k) then transpose
        Q = Q.view(B, T, self.n_heads, self.d_k).transpose(1, 2)  # (B, n_heads, T, d_k)
        K = K.view(B, T, self.n_heads, self.d_k).transpose(1, 2)  # (B, n_heads, T, d_k)
        V = V.view(B, T, self.n_heads, self.d_k).transpose(1, 2)  # (B, n_heads, T, d_k)
        # Compute scaled dot-product attention for each head
        # (We transpose K to (B, n_heads, d_k, T) for matmul)
        scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_k)  # (B, n_heads, T, T)
        if mask is not None:
            scores = scores.masked_fill(mask[:, None, None, :] == 0, -1e9)
        weights = torch.softmax(scores, dim=-1)  # (B, n_heads, T, T)
        attn_output = torch.matmul(weights, V)   # (B, n_heads, T, d_k)
        # Concatenate heads back together
        attn_output = attn_output.transpose(1, 2).contiguous().view(B, T, C)  # (B, T, d_model)
        return self.W_o(attn_output)  # final linear proj

Don’t worry if the tensor manipulations are a bit much, the idea is we computed attention for each head and recombined. The mask we pass will be a tensor that is 0 where we want to mask out future tokens (for positions beyond the current timestep in the decoder) and 1 elsewhere. This ensures the model only attends to positions ≤ current position.

Decoder Block: Now we create the DecoderBlock using the attention and a feed-forward sub-layer:

class DecoderBlock(nn.Module):
    def __init__(self, d_model, n_heads, d_ff):
        super().__init__()
        self.self_attn = MultiheadSelfAttention(d_model, n_heads)
        self.feed_forward = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.ReLU(),
            nn.Linear(d_ff, d_model),
        )
        self.ln1 = nn.LayerNorm(d_model)
        self.ln2 = nn.LayerNorm(d_model)

    def forward(self, x, mask=None):
        # Self-attention sub-layer with residual connection
        attn_out = self.self_attn(self.ln1(x), mask=mask)
        x = x + attn_out
        # Feed-forward sub-layer with residual connection
        ff_out = self.feed_forward(self.ln2(x))
        x = x + ff_out
        return x

Each DecoderBlock applies layer norm to the input (ln1(x)), performs masked self-attention, adds it back, then layer norm again and feed-forward, and adds it back. We don’t show an encoder-decoder attention here because GPTMini is decoder-only (no encoder to attend to).

Now we can complete the GPTMini.forward method:

    def forward(self, idx):
        B, T = idx.shape  # idx is input token indices (batch, seq_length)
        # Create causal mask for attention (shape T×T, lower triangular ones)
        device = idx.device
        mask = torch.tril(torch.ones(T, T, device=device)).bool()  # True where allowed
        # Embed tokens and positions
        token_embeddings = self.token_emb(idx)            # (B, T, d_model)
        positions = torch.arange(0, T, device=device).unsqueeze(0)  # (1, T)
        pos_embeddings = self.pos_emb(positions)          # (1, T, d_model)
        x = token_embeddings + pos_embeddings             # combine token + position
        # Apply each Transformer decoder layer
        for layer in self.layers:
            x = layer(x, mask=mask)
        # Final layer norm (often used at end of Transformer decoder stack, e.g. GPT-2)
        x = self.norm(x)  # (B, T, d_model)
        # Output logits for each position and token in vocab
        logits = self.out_head(x)  # (B, T, vocab_size)
        return logits

A few things to note: we constructed a mask using torch.tril (lower-triangular matrix of ones) which will be broadcast to shape (B, n_heads, T, T) inside attention to mask out future positions. We add token and positional embeddings to get our initial x. After the Transformer layers, logits contains scores for each vocabulary token at each sequence position.

During training, we will call model(input_ids) to get logits, which has shape [batch_size, seq_length, vocab_size]. We then compare it to the target_ids (the ground-truth next tokens) to compute loss. Typically we use CrossEntropy loss applied to each position. In PyTorch, we’d flatten the predictions and targets to shape [batch_size*seq_length] when computing F.cross_entropy(logits.view(-1, vocab_size), targets.view(-1)).

To recap, GPTMini’s architecture includes everything we discussed: token + positional embeddings, multi-head masked self-attention in each layer, residual connections with layer norm, feed-forward networks, and an output projection. It’s essentially a small GPT model.

Training Loop

With the model and data ready, we train GPTMini using a next-token prediction objective (language modeling). We iterate over the dataset of input-target sequences, and for each batch, do the following:

1. Forward pass: Compute logits = model(input_ids).

2. Loss computation: Compare logits to target_ids. We use cross-entropy loss, which is perfect for classification over the vocabulary for each position. In code: loss = F.cross_entropy(logits.view(-1, vocab_size), target_ids.view(-1)). This calculates the average negative log-likelihood of the correct tokens.

3. Backward pass: optimizer.zero_grad(); loss.backward(); optimizer.step(). We typically use the Adam optimizer (often with weight decay and a learning rate schedule) as used in the original Transformer paper.

A simplified training loop:

import torch
import torch.nn.functional as F

model = GPTMini(vocab_size, d_model=384, max_seq_len=128, n_layers=6, n_heads=6, d_ff=1024)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)

for epoch in range(num_epochs):
    for input_ids, target_ids in train_loader:
        logits = model(input_ids)              # (batch, T, vocab_size)
        loss = F.cross_entropy(logits.view(-1, vocab_size), target_ids.view(-1))
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print(f"Epoch {epoch} loss: {loss.item():.4f}")

We also validate the model on a held-out validation set (WikiText provides train/validation/test splits) to ensure it’s not overfitting and to track its performance (often measured in perplexity, which is \(\exp(\text{loss})\)). Perplexity tells us how well the model predicts the sample on average the lower, the better.

Text Generation with GPTMini

Once trained, we can use GPTMini to generate text by sampling from the model’s output distribution iteratively. Here’s the general procedure for text generation (also known as autoregressive decoding):

1. Start with a prompt (some initial text). Tokenize it to get input IDs.

2. Feed the input through the model to get output logits for the next token.

3. Convert logits to probabilities (e.g., apply softmax, possibly with a temperature to control randomness).

4. Sample a token from the probability distribution (or pick the highest probability token for greedy decoding).

5. Append the sampled token to the input sequence.

6. Repeat steps 2–5 until you reach a desired length or an end-of-sequence token.

For example, suppose we want GPTMini to generate text after the prompt "Once upon a time":

prompt = "Once upon a time"
input_ids = tokenizer.encode(prompt)  # list of token IDs
for _ in range(50):  # generate 50 tokens
    idx_tensor = torch.tensor(input_ids)[None, :]  # shape (1, seq_len)
    logits = model(idx_tensor)  # forward pass
    next_token_logits = logits[0, -1, :]          # logits for the last token
    # Optionally apply temperature scaling and/or top-k filtering:
    probs = torch.softmax(next_token_logits / temperature, dim=-1)
    next_id = torch.multinomial(probs, num_samples=1).item()  # sample next token
    input_ids.append(next_id)

After this loop, input_ids will contain the original prompt tokens plus 50 newly generated tokens. We can then decode input_ids back to text with tokenizer.decode(input_ids).

To improve quality, one often uses strategies like top-k or top-p (nucleus) sampling to avoid low-probability tokens and encourage more plausible outputs. For instance, top-k sampling picks from the k most probable tokens (e.g., k=50) instead of the entire distribution, which reduces the chance of odd words. Top-p sampling chooses the smallest set of tokens whose cumulative probability exceeds a threshold p (e.g,. 0.9), and samples from that set.

GPTMini, being small, won’t produce Shakespearean prose, but it will learn basic English structure and generate somewhat coherent sentences related to Wikipedia topics in the training data. For example, if prompted with “Once upon a time”, it might continue with something like “in a small village, a young girl named Alice was reading about kings and queens.” (The output will vary each time due to randomness in sampling.)

Putting It All Together

We’ve now gone from theoretical foundations to a working miniature GPT model:

• Attention mechanisms enabled the model to handle dependencies without RNNs, examining all words at once

• The Transformer architecture uses stacked layers of multi-head self-attention and feed-forward networks with residual connections, allowing deep, parallel computation.

• We built a BPE tokenizer to preprocess text into subword units, which is exactly how real GPT models handle open-vocabulary input

• We prepared the WikiText dataset as a training corpus, a standard benchmark with millions of tokens from Wikipedia

• We defined the GPTMini model in code, including masked multi-head self-attention and all the trimmings of a Transformer decoder.

• We ran a training loop optimizing next-word prediction, and then used the trained model to generate text with a simple sampling strategy.

Key Takeaways and Future Directions

Transformers and attention mechanisms have transformed (pun intended) the field of NLP. By letting models attend to all parts of the input, we overcome the bottlenecks of older architectures and achieve greater parallelism and context integration. The self-attention mechanism is the backbone of modern large language models, enabling them to capture meaning across entire documents. In our GPTMini, we saw how these ideas materialize in code, from tokenization to model forward pass.

Going further, one could experiment with improvements and extensions: for example, adding Dropout in the model (the original Transformer used dropout in various places) for regularization, using adaptive softmax or tied input-output embeddings for efficiency, scaling up the model size, or pretraining on a larger corpus. Modern large language models also introduce techniques like transformer-decoder block scaling, learning rate warmups, gradient clipping, and more – but the essence remains the architecture we’ve built.

Transformers aren’t limited to text; they’re used in vision (ViT – Vision Transformer), audio, and multi-modal models. The versatility of attention mechanisms means they can be applied wherever we have sequential or set data that benefits from global interactions.

We hope this guided tour – from the intuition of attention to coding a mini GPT – has demystified how these powerful models work. With these fundamentals, you can better understand breakthroughs like GPT-3 and beyond, or even build your own Transformer models for fun and learning. Happy experimenting with attention and Transformers, sometimes, a little attention goes a long way!

1. Import the necessary modules and libraries. This will typically include the hashlib library for generating hashes and the datetime library for managing timestamps.

Copy codeimport hashlib
import datetime

1. Define a Block the class that will represent each block in the chain. This class should have the following attributes:

• timestamp: a timestamp representing the time at which the block was created

• data: the data that the block will store (e.g. a transaction record)

• previous_hash: a hash of the previous block in the chain

• hash: a hash of the current block

Copy codeclass Block:
  def __init__(self, timestamp, data, previous_hash):
    self.timestamp = timestamp
    self.data = data
    self.previous_hash = previous_hash
    self.hash = self.calculate_hash()

  def calculate_hash(self):
    # Generate the hash of the block using the SHA-256 algorithm
    sha = hashlib.sha256()
    sha.update(self.timestamp.encode('utf-8') + self.data.encode('utf-8') + self.previous_hash.encode('utf-8'))
    return sha.hexdigest()

1. Define a Blockchain class that will manage the blocks in the chain. This class should have the following attributes and methods:

• chain: a list of blocks in the chain

• create_genesis_block(): a method that creates the first block in the chain (also called the "genesis block")

• latest_block(): a method that returns the latest block in the chain

• add_block(data): a method that adds a new block to the chain

Copy codeclass Blockchain:
  def __init__(self):
    self.chain = [self.create_genesis_block()]

  def create_genesis_block(self):
    # Create the first block in the chain
    timestamp = datetime.datetime.now()
    data = "Genesis Block"
    previous_hash = "0"
    return Block(timestamp, data, previous_hash)

  def latest_block(self):
    # Return the latest block in the chain
    return self.chain[-1]

  def add_block(self, data):
    # Create a new block and add it to the chain
    timestamp = datetime.datetime.now()
    previous_hash = self.latest_block().hash
    new_block = Block(timestamp, data, previous_hash)
    self.chain.append(new_block)

1. Create an instance of the Blockchain class, add some data to it, and print the resulting chain to verify that it is working as expected.

Copy codeblockchain = Blockchain()
blockchain.add_block("Transaction Data 1")
blockchain.add_block("Transaction Data 2")

for block in blockchain.chain:
  print(f"Timestamp: {block.timestamp}")
  print(f"Data: {block.data}")

To complete the implementation of the blockchain, you will need to add some additional functionality to the Block and Blockchain classes.

For the Block class, you will need to add a method that verifies the integrity of the block by checking that the calculated hash of the block matches the hash stored in the block itself. You can do this by adding the following method to the Block class:

def is_valid(self):
    # Check if the calculated hash of the block matches the stored hash
    return self.calculate_hash() == self.hash

For the Blockchain class, you will need to add a method that verifies the integrity of the entire chain by checking that the hash of each block matches the previous block's hash, and that each block is valid according to the is_valid() method of the Block class. You can do this by adding the following method to the Blockchain class:

def is_chain_valid(self):
    # Check if each block in the chain is valid
    for i in range(1, len(self.chain)):
        current_block = self.chain[i]
        previous_block = self.chain[i-1]
        if not current_block.is_valid():
            return False
        if current_block.previous_hash != previous_block.hash:
            return False
    return True

Once you have added these methods, your blockchain should be complete and ready for use. You can test it by adding some data to the chain and verifying that it is valid according to the is_chain_valid() method.

# Create a new blockchain
blockchain = Blockchain()

# Add some data to the blockchain
blockchain.add_block("Transaction Data 1")
blockchain.add_block("Transaction Data 2")

# Check if the chain is valid
if blockchain.is_chain_valid():
  print("The blockchain is valid.")
else:
  print("The blockchain is not valid.")

This is just a basic implementation of a blockchain using Python. In a real-world scenario, you would need to add additional functionality to make it more robust and secure, such as the ability to handle multiple transactions in each block and the ability to handle multiple chains (i.e. a "fork" in the blockchain).

One of the key challenges in NLP is the fact that human language is incredibly complex and varied. There are thousands of languages, dialects, and accents, and even within a single language there can be a wide range of vocabulary, syntax, and grammar. This makes it difficult for computers to accurately process and understand natural language inputs.

To overcome this challenge, NLP researchers and practitioners use a variety of techniques and algorithms. These include:

• Part-of-speech tagging: This is the process of identifying the part of speech (e.g. noun, verb, adjective) for each word in a sentence. Part-of-speech tagging is important for understanding the structure and meaning of a sentence.

• Named entity recognition: This is the process of identifying and classifying named entities (e.g. people, organizations, locations) in a text. Named entity recognition is important for extracting useful information from a text.

• Sentiment analysis: This is the process of determining the sentiment (e.g. positive, negative, neutral) of a piece of text. Sentiment analysis is often used to gauge the public opinion of a product, service, or event.

• Machine translation: This is the process of automatically translating text from one language to another. Machine translation has improved dramatically in recent years, but it still faces challenges in dealing with complex syntax and cultural differences.

One of the most promising areas of NLP research is the development of deep learning algorithms. These algorithms use neural networks to learn the underlying patterns and structures of language and can achieve impressive results in tasks such as language translation and text summarization.

In conclusion, NLP is a fascinating and rapidly-evolving field of artificial intelligence that has a wide range of applications. By using sophisticated algorithms and techniques, NLP researchers and practitioners are making impressive progress in enabling computers to understand and process human language.

Let's first understand how a typical web application works without blockchain. A website, after coding, is uploaded to a server, which serves the user's request. This server also connects to the database to fetch and dynamically deliver data to the user.

Now let us see how blockchain is different from this former setup. A web app built using blockchain is called a web3 application, and most of the other web applications we use today without blockchain are called web3 applications. As a web2 application has a front end and a back end, a blockchain application has two parts. The major components of a web3 app are the front end and smart contracts. So as you might have guessed, the front-end parts of a web2 and web3 application are similar.

Smart Contracts are code written a particular programming language which varies depending on the blockchain we choose. For Ethereum, the programming language used is solidity. These smart contracts will have functions to add and remove data from the blockchain. These smart contracts are then published to the blockchain, a chain of nodes(computers) that serves the function of a backend server but in a decentralized manner. When published, smart contracts will generate an ABI(Application Binary Interface) and return an address.

The front end uses the ABI and the address to interact with the blockchain. The connection to the blockchain is made using a wallet. Any user can execute different functions in the smart contract using a UI interface in the front end if the function being executed changes the data in the blockchain, some fees need to be paid. These fees are called gas fees, which are paid using a wallet connected to the application.

Code Walk Through

Library imports

# Necessary Library imports
import torch
from torch.utils.data import Dataset , DataLoader , random_split
import torch.nn.functional as F
import torch.nn as nn

The following are the major import statements we need to build our demo app,we can discuss the use of each of these libraries when we use them in the upcoming sections.

Creating a dataset

Tensors

Nos = 1000
no_of_features = 5
x = torch.randn(Nos,no_of_features) #taking a random sample from a normal distribution
y1 = torch.ones(Nos//2)        # creating first half of the binary output
y2 = torch.zeros(Nos//2)      # creating second half of the binary output
y = torch.cat((y1,y2))        #final output array
shuffled_y = [y[i] for i in torch.randperm(len(y))] #shuffling output tensor
y = torch.tensor(shuffled_y)

We are not using any premade/complicated dataset for this particular tutorial instead we can make a dataset of our own using some PyTorch utilities. The basic building blocks of PyTorch are tensors, they are n-dimensional arrays that are similar to NumPy arrays but have the capability to perform operations in a GPU. If you are familiar with NumPy most of the methods in tensors will be similar.

Our independent variable x has 1000 items defined with 5 features, torch.randn() takes in the shape of the tensor to be created a returns a tensor of the required dimension, actually, there are two functions to create a random tensor rand() and randn() , the only difference is radn() creates a tensor whose elements follow a normal distribution. To create the dependent variable y which act as the prediction we create two nos/2 dimensional vectors y1 and y2 , which contain only ones and zeros respectively and the corresponding functions used to generate them are self-explanatory. Then we use torch.cat to concatenate these two tensors and as a final step we shuffle these ones and zeros and the final output is stored in y.

Dataset class

# creating a custom dataset class inheriting from torch.utils.data.Dataset class
class MyDataset(Dataset):
  def __init__(self,x,y) -> None:
    self.x = x
    self.y = y

  def __len__(self):
    return len(self.y)

  def __getitem__(self,idx):
    return self.x[idx] , self.y[idx]

dataset = MyDataset(x,y)

# spliting the dataset for training and testing
train_set , test_set = random_split(dataset, [800, 200], generator=torch.Generator().manual_seed(42))

Now we have our data as a tensor, but to use our data with a PyTorch model we need to convert it to a child class of superclass called Dataset, which can be imported from torch.utils.data.Dataset.The child class should implement init() , len() , getitem() functions to make the class valid for future uses . The init() function is the constructor to our class this is where we do all the preprocessing required for our dataset, here it takes two arguments that are x and y tensors which we created earlier. The len() function returns the length of the dataset and the getitem() function takes an index as an argument and returns the feature and output tensor at that particular index. Finally, we split the dataset into two,train_set and test_test which can be used for training and testing respectively.

Dataloaders

# creating dataloaders for train and test set
train_loader = DataLoader(train_set,batch_size = 16,shuffle = True)
test_loader = DataLoader(test_set,batch_size = 16,shuffle = True)
sample_x, sample_y = next(iter(train_loader))
sample_x.shape , sample_y.shape

A Dataloader class in PyTorch helps to load the data in batches and shuffle them during each epoch of training for better randomization. The above code creates two data loaders for both train and test sets. Pytorch dataloaders are basically python iterators and the next elements can be extracted using the next() in python

Defining a Model

class MyModel(nn.Module):
  def __init__(self):
    super(MyModel,self).__init__()
    self.layer1 = nn.Linear(5,300)
    self.layer2 = nn.Linear(300,1)
    self.relu = nn.ReLU()
    self.sigmoid = nn.Sigmoid()

  def forward(self,x):
    x = self.layer1(x)
    x = self.relu(x)
    x = self.layer2(x)
    x = self.sigmoid(x)
    return x

model = MyModel()
print(model)

for parameter in model.parameters(): 
  print(parameter.data.shape)

Now we have our dataset and data loaders ready, next let us define our model architecture. In PyTorch, we represent a Model through a class that inherits from the base class nn.Module. The class should implement two functions init() and the forward . The init() function defines the layers for our model with help of a neural network sub-module called nn . In the above code snippet, we define a linear layer with the function nn.Linear() which takes in two parameters which are the input and output layer dimensions. The forward function is the function that gets executed when we call the model in the training loop it is called inside a call() function inside the model class , this is the place where the forward pass takes place

Notice that the input to the first layer is 5 this is the number of features we have in our dataset and the output of our 2nd/final layer is 1 since we are doing a binary classification problem and the output is either 0/1 based on the probability in this final layer. We can also see a non-linear activation function like ReLu which does not change the dimension of our layers

You might be wondering where are the weights and bias of our model stored, this is automatically implemented by PyTorch as we use the base class nn.Module. We can use the parameters() in the model class to get all the parameters in our model with the initialiased values. In our case our model will have 4 matrices of parameters with the following dimensions torch.Size([300, 5]), torch.Size([300]), torch.Size([1, 300]), torch.Size([1]), We can see that the first and third ones are weights and the 2nd and last ones are biases of our model.

Training and Testing Loops

# Optimising the model parameters

loss_fn = F.binary_cross_entropy
optimizer = torch.optim.SGD(model.parameters(),lr = 1e-4)

# training loop for train set
def train(train_loader,model , loss_fn,optimizer):

  for batch,(x,y) in enumerate(train_loader):

    pred = model(x)
    loss = loss_fn(pred,y.unsqueeze(1))

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if batch % 50 == 0:
      loss , current = loss.item(),batch * len(x)
      print(f"loos : {loss}, record no {current} ")

#testing loop

def test(test_loader,loss_fn):
  size = len(test_loader.dataset)

  model.eval()
  test_loss , accuracy = 0 , 0
  for batch,(x,y) in enumerate(train_loader):
    pred = model(x)
    loss = loss_fn(pred,y.unsqueeze(1))
    test_loss += loss.item()
    accuracy += (pred.round()==sample_y.unsqueeze(1)).sum()

  print(f"test_loss = {loss/16}")
  print(f"test accuracy = {(accuracy/Nos)}")


epochs = 10

for t in range(epochs):
  print(f"starting epoch {t}")
  train(train_loader,model,loss_fn,optimizer)
  test(test_loader, loss_fn)

Let's train the model with the dataset we created. Training is a fancy term for updating the weights and biases of our model such that the output predicted by our model is close to the original output, a loss function is a metric that tells us how different are our prediction and output, here we use a loss function called binary_cross_entropy which we import from the functional package. An optimizer updates the weights to minimize the loss function here we use SGD from the torch.optim package with a learning rate of 1e-4.

After we define our optimizer and loss function we define our training loop which loops the data loader and in each iteration it passes the x(i) value to the model to generate predictions, this prediction along with the actual y label is passed to loss function to generate loss , here we unsqueeze y to add a single dimension to it so that it matches the predicted value's dimension, An important thing to note is that a PyTorch model class will only accept values in batches and it returns the prediction of each element in the batch, here the dimension of the tensor pred is [16,1] as our batch size was 16.

Pytorch uses dynamic computation graphs which will help us to calculate gradients easily without an extra step, this makes PyTorch more pythonic than other frameworks. optimizer.zero_grad() clears the calculated gradients from the previous step and loss.backwords() calculate the gradients of the loss function with respect to the X .Optimiser.step() updates the weights and biases accordingly.

Now let's talk about the testing loop. It is very much similar to the training loop but we are not updating the weights but simply validating the training hence we add model.eval() to tell the Pytorch the same, the default mode is model.train() so we don't have to add it explicitly in the training loop. For each epoch, we find the accuracy and loss with the training set. Now lets us closely examine how loss and accuracy are calculated in each epoch. For every batch prediction in an epoch, we obtain an avg loss for the batch from the loss function we sum it up and divide it by the batch size to find the avg loss for the entire dataset for that epoch. For each batch prediction, we round it off, compare it with its label and add the total number of correct predictions to the accuracy variable at the end of the epoch we divide the number of correct predictions by the total size of our dataset to find out the accuracy after that epoch.

Conclution

Phew!, congrats if you could make it to the end, all these might seem a bit overwhelming in the beginning but you will get used to it soon. Please note that the dataset we used is just for educational purposes and we did not try to make any best predictions here. That's all folks, if you want me to explain any other topic do comment it out.

Before understanding what is docker let's try to find the answer to the question "why Docker". Any software developed is for solving a particular problem statement. Let's consider a situation in which you developed an application and send your source files to your manager for production or a friend to test it out. Even though your application works fine on your machine when your friend or manager runs it on their machine or in a production server it just didn't work. There can be many reasons for this, lets say your application was build using nodejs 14 but the person who ran that has node 12 in their system or you developed the app in a Linux environment but the person who ran it might have used windows and it crashed the app. This problem of "it works on my machine" is what docker tries to solve. So what if there is a way to ship your machine to production rather than shipping your source code, this will ensure that the program works where ever it is made to run. Dockerising an application helps us to do this, in the upcoming sections we will try to see how to do it.

Installation

Installing docker is pretty straightforward, you can follow these instructions according to your operating system https://docs.docker.com/get-docker/. It's a good thing to note that in all these installation setups docker is installed inside a Linux machine or a Linux virtual machine(Docker Desktop), that is because docker only runs operating systems with the same kernel. If you don't understand this do not worry, you will get this once you use are familiarised with docker.

Dockerfile, Docker Images, Docker Containers

Now let's understand some important terms in Docker. Let's start with Docker Images, as the name suggests it is a snapshot of the machine you need to ship to production and a docker file is a blueprint to configure how your image should be built. Once we build a docker file and create a docker image, we can run the image to create a container which is a process that runs our applications in the specified environment

Containerizing a Node Js Application

Let's try to use all the things we learnt till now and containerise a simple Nodejs Application. You can find the starter code here. Once you are ready with the starter nodejs application let's begin by creating a DockerFile in the root directory of the application.

FROM node:12

WORKDIR /app

COPY package.json ./

RUN npm install

COPY . .

ENV PORT=8080

EXPOSE 8080

CMD [ "npm", "start" ]

Now copy and paste the following contents into the Dockerfile file. Let's break down each of these statements. The FROM command specifies the base image or the starting image that we are using , it can be ubuntu, centos or something particular like node which already have node js installed inside that. next, we specified the working directory, this is where all our files should be stored. the copy command copies the package.json to the working directory then the RUN command runs 'npm install' which installs the necessary packages needed. then we copy the rest of the app to the working directory. The ENV sets an Environment variable to 'port' then we Expose the port and finally starts the application using the CMD command

Now let's build an image using the docker file we made. Use the following command to run the docker file.

docker build -t node-docker-demo:v1 .

This command builds our docker image.

docker images ps

the above command can be used to show all the docker images in our system

Now lets the run the image in a container

 docker run -p 8080:8080 -d node-docker-demo:v1

the above command starts and the container runs our application, then expose the port 8080 to the host machine. Now you can visit http://localhost:8080/ in your browser to view our app.

Congrats on running your first containerized application, now we can stop the container
Run the following command to get the list of containers currently running in our machine

docker ps

copy the id of the container that we started and run the following command to stop it.

docker stop <container_id>

Conclusions

That was a small introduction to docker, if you are more curious checkout more commands and options in the official documentation Follow me for more content like this

Before getting into these let us understand what exactly is meant by a network request. In simple terms, a network request is a request made by a client-side application to a server for exchanging different types of resources like images, webpages, JSON etc. We are going to see three different types of methods to make a network request which includes XMLHttp, fetch and Axios. Let's try to understand each of these methods one by one.

XMLHttp

const getdata = () =>{
    var xhr = new XMLHttpRequest();
    xhr.open("GET", "https://reqres.in/api/products/1");
    xhr.onload = function(){
        console.log(JSON.parse(xhr.response));
    };
    xhr.send();
}
getdata()

The code consists of a simple js function making using of XMLHttp for getting data from an API endpoint. Now let's break down the code one by one. The XMLHttpRequest() object is a built-in method in the browser it initializes an XMLHttp object . The open method in the xhr object primarily takes in two arguments. An HTTP method and the URL to the resource, this method initializes a request. The onload method can be assigned a function that can be fired when the request receives some data. The data we receive will not be in JSON format, but we can parse it to JSON using the inbuild JSON object in the browser and finally the send() method sends the request to the server

This method of doing this is fine but since it's asynchronous, we can wrap the function with a promise.

const getdata = () =>{
    return new Promise((resolve, reject) => {
        var xhr = new XMLHttpRequest();
        xhr.open("GET", "https://reqres.in/api/products/1");
        xhr.onload = function(){
            resolve(JSON.parse(xhr.response));
        };
        xhr.send();
    })

}
getdata()
.then(data => {
    console.log(data);
})

This is a quite vague implementation, it doesn't handle the rejected case but I just wanted to show you that promises could be used alongside with XMLHttp requests. A common doubt you must have now is XMLHttp requests have XML in its name but we are using JSON data, this is a very genuine question, the answer to this question is we could use both JSON and XML data with XMLHttp, it was named so because XML was a common standard in earlier days. So that's all we have for XMLHttp you could read more about it in the MDN docs.

Fetch API

XMLHttp requests work fine but, but too much code for a single request, fetch simplifies this process by implementing promises under the hood so we don't have to do it manually.

fetch('https://reqres.in/api/products/1')
.then(res =>console.log(res))

Wow that was short, but there is a small problem, if you check the response body it will be a readable stream and not any form of usable data. To convert it to any other format there are methods provided by the fetch API, one of these is res.json() which will convert the response to JSON and returns another promise which can be resolved to get the data. The following code does what we have mentioned bellow

fetch('https://reqres.in/api/products/1')
.then((res)=>{
    return res.json();
})
.then(data=>{
    console.log(data);
})

you could read more about it in the MDN docs

Axios

Unlike the other two methods, we discussed above Axios is not a native browser API it's a javascript package that can either be installed using npm or used with a CDN. Let's see a sample GET request using Axios that fetches the same data as we did before

axios.get('https://reqres.in/api/products/1')
.then(res=>{
    console.log(res)
})

you can notice it does not return a readable stream instead returns the entire data, unlike the fetch API. So is that the only difference, the answer is no. Axios provide a lot of things under the hood like XSRF protection, the ability to intercept HTTP requests etc.

Conclusion

That's all for this post, hope you got a foundation on the different methods to make a network request in js so that you could practise and learn more about these soon. If you loved my content consider following me for more of these and if you need a blog post any related content do mention it in the comments

ECMAScript 2015 or ES6 was the second major revision to JavaScript. Many of you probably might be using it with popular front-end libraries such as react. Even though ES6 syntax is much better and as developers we loved to use it quite often, there are few versions of browsers that won't support ES6. Due to this reason we use babel a javascript transpiler to compile the ES6 code to lower versions. This is done in tools such as create react app.

Now let's come to Nodejs a javascript runtime that runs javascript outside the browser. Let's see how a simple express app would look while using ES6 and ES6

// ES5 version
var express = require('express');
var app = express();

app.get('/', function (req, res) {
   res.send('Hello World');
})

var server = app.listen(8081, function () {
   var host = server.address().address
   var port = server.address().port

   console.log("Example app listening at http://%s:%s", host, port)
})

// ES6 version
import express from 'express';
var app = express();

app.get('/', (req, res)=> {
    res.send('Hello World');
})

app.listen(3000, () => {
    console.log("Example app listening at port 3000")
})

Some of the changes we can observe here are,the use of ES6 imports and arrow functions but it's not limited to that, you can check here for more differences and additions.

METHODS TO WRITE ES6 CODE

METHOD 1

From nodejs 13+ ES6 has been added as an experimental feature. You can enable this by adding type = "module" to your package.json.

{
  "name": "demo",
  "version": "1.0.0",
  "type":"module",  <--
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^4.17.1"
  }
}

METHOD 2

Here if use babel to transpile our ES6 code to ES5 code. We need to install the following packages to do that.

 npm install --save-dev @babel/cli @babel/core @babel/node @babel/preset-env

We can also install an additional dependency called nodemon to restart our server automatically when we make changes

npm install --save-dev nodemon

Now lets us configure our package.json to use babel to transpile our ES5 code to ES6 Create a folder called src at the root of the project and move the index.js(the file with your express app) file into it

Let's add the following scripts to our package.json

 "scripts": {
    "build": "babel ./src --out-dir dist",
    "start": "nodemon  --exec babel-node src/index.js "
  },

Run the following code to start the development server and to transpile your code respectively.

npm start  // starts a development server

npm run build  //transpile the code to ES6 and store it in a directory called dist

The final project structure would look like the one bellow

The final package.json will look the like the one bellow

{
  "name": "demo",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "build": "babel ./src --out-dir dist",
    "start": "nodemon  --exec babel-node src/index.js "
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^4.17.1"
  },
  "devDependencies": {
    "@babel/cli": "^7.14.8",
    "@babel/core": "^7.15.0",
    "@babel/node": "^7.14.9",
    "@babel/preset-env": "^7.15.0"
  }
}

That's it , you are now ready to get up and running with ES6 🎉. If you have any queries reach out to me via Twitter .

Most of us have heard the term CORS from an error message, popping up while doing a network request. Before getting into solving this error let's understand what does CORS exactly mean and check whether it's really bad as you think.

In the olden days of the web, we were only allowed to share resources between the same origin, here origin can be considered as a single domain or a server. This was fine until microservice architecture became widely adopted and there was a need for the browser to communicate to different domains. This is where CORS come into the picture.

CORS

It is an HTTP-header based mechanism that allows a server to indicate any other origins (domain, scheme, or port) than its own from which a browser should permit loading of resources. CORS also relies on a mechanism by which browsers make a “preflight” request to the server hosting the cross-origin resource, in order to check that the server will permit the actual request. In that preflight, the browser sends headers that indicate the HTTP method and headers that will be used in the actual request.

There are two types of cross-origin requests:

1. Safe requests.
2. All the others.

Safe Requests are simpler to make, so let’s start with them.

A request is safe if it satisfies two conditions:

1. Safe method: GET, POST or HEAD

2. Safe headers – the only allowed custom headers are:

   • Accept,

   • Accept-Language,

   • Content-Language,

   • Content-Type with the value application/x-www-form-urlencoded, multipart/form-data or text/plain.

Any other request is considered “unsafe”. For instance, a request with a PUT method or with an API-Key HTTP header does not fit the limitations

Handling CORS Errors

The reason for you to get a CORS error is that your request does not come under the safe request category, one way of fixing this error would be adding a proxy server, this makes use of the property that CORS restrictions are not there in server to server communications. If you are working on a react app build using create-react-app, the react-scripts provide you with a proxy server, but this is only for development purposes so it's not a proper solution.

If you have control over the API then the best solution would be to add the CORS headers before sending the response to your request.

Access-Control-Allow-Origin: *
Access-Control-Allow-Origin: <origin>
Access-Control-Allow-Origin: null

The above headers can be used to allow some or all certain origins to access the server resources. Similarly, there are more headers that you can add to allow different content-Types, methods etc. You can learn more about the different headers here.

Hopefully, you are now able to understand what caused this error and a possible fix, now let's try to implement this solution using code, I am going to use express js for this particular demo.

The following code shows a simple express app

const express = require('express');

const app = express();

app.use(express.json());

app.listen(3000, () => {
  console.log(`listening on port 3000`)
})

// routes
app.get('/', (req, res) => res.render('home'));
app.put('/books',  (req, res) => {
  books = updatebook() // some function to update books
  res.send("books updated")
});

Then the browser is requesting for the /books resource from another origin there will be a CORS error as we are not attaching any CORS header to the response hence the preflight request will fail to raise the CORS Error. Now let's see how to add a simple express middleware that can add this necessary header to the responses.

const express = require('express');

const app = express();

app.use(express.json());

app.use((req, res, next) => {
  res.header('Access-Control-Allow-Origin', '*');
  res.header('Access-Control-Allow-Headers','*');
  next();
});

app.listen(3000, () => {
  console.log(`listening on port 3000`)
})

// routes
app.get('/', (req, res) => res.render('home'));
app.put('/books',  (req, res) => {
  books = updatebook() // some function to update books
  res.send("books updated")
});

app. use applied this middleware to all requests in our server, you can read more about express middleware here.

So from the next request, our server adds 'Access-Control-Allow-Origin' and 'Access-Control-Allow-Headers' to our requests so this will allow the browser to accept the resources from our server and CORS error will not be there. Note that the headers you need to include are the ones needed according to your question, do refer to the link I shared before for reading more about the other headers available

Before wrapping up I would like to add one more thing, that is about the different modes available in browsers fetch API, it is just a way to control how the browser should attempt to cross-origin requests and will not help in accessing cors restricted resources.

The different mode options available are the following:

• "cors" – the default, cross-origin requests are allowed, as described in Fetch: Cross-Origin Requests,
• "same-origin" – cross-origin requests are forbidden,
• "no-cors" – only safe cross-origin requests are allowed. This option may be useful when the URL for fetch comes from a 3rd-party, and we want a “power off switch” to limit cross-origin capabilities.