I've been integrating AI into my development workflow, but I quickly realized that costs can spiral out of control if not monitored carefully. Premium AI coding assistants with their subscription models and "pro" tiers can easily become expensive-especially for personal and open-source projects where budgets are tight. I found myself needing to track every API call and watch my usage like a hawk.

The cost barrier to AI-assisted development has finally crumbled. After months of experimenting with various setups, I've found a combination that lets me build production-quality software while spending almost nothing: Claude Code paired with OpenRouter's free tier models.

The Evolution of My AI Coding Setup​

When I first started using AI coding assistants, I was quickly confronted with the reality that quality comes at a price. Premium models from Anthropic, OpenAI, and others can easily run up bills of $50-100+ per month for regular usage. As an indie developer, that's not sustainable.

My journey went something like this:

1. Early experiments: Direct API calls to premium models (expensive) - Claude Opus/Sonnet were excellent but prohibitively expensive
2. Intermediate phase: Mix of premium and open-source models (moderate cost) - Tried cheaper Mistral models on paid APIs but quality was significantly worse
3. Local experiments: Attempted running models locally with Ollama - MacBook Air M2 (16GB) lacks VRAM for large models, small models (Gemma) have poor quality, and even with swap, medium models (quantized Qwen) have terrible performance
4. Current setup: Claude Code + OpenRouter free tier (near-zero cost)

The breakthrough came when I realized that for many coding tasks, I don't need the most expensive, cutting-edge models. I need models that are:

• Good enough to understand code context
• Capable of generating functional code
• Fast enough to not break my flow
• Free or extremely cheap

Why Claude Code?​

Claude Code has become my primary interface for AI-assisted coding because it offers something unique: a terminal-native experience that feels like pair programming with a knowledgeable colleague.

Key advantages:

• Seamless terminal integration: No context switching between IDE and chat interface
• File-aware operations: Understands your project structure and can read/write files directly
• Command-based workflow: Natural integration with existing development practices
• Session memory: Remembers context across interactions within a session
• Agentic loop: Claude Code continuously analyses context and iteratively proposes and executes actions (edits, reads, exec) for reactive and proactive assistance.
• Tools and skills: Built‑in capabilities like ultrareview, planning, dependency management and other micro‑tools streamline the workflow.

What really sets Claude Code apart is how it handles complex, multi-file operations. Rather than just suggesting code snippets, it can actually implement features across multiple files, run tests, and even debug issues.

I use a hybrid approach: a large model for the planning phase (Claude Code plan mode), then a cheap/free model for the actual development. This lets me structure the task with the power of a big model while keeping costs low during coding.

The OpenRouter Free Tier Advantage​

OpenRouter's free tier has been a game-changer. Through their platform, I get access to several high-quality models at zero cost:

Models I regularly use:

• Nemotron 3 Super (my current primary, listed as free on OpenRouter but incurs a small per‑request fee or uses your credit balance, so it may cost dollars depending on your OpenRouter usage settings): Excellent for code generation and understanding - I chose this specifically for its thinking capabilities, excellent time to first token, and token throughput
• Mistral models: Strong performance on coding tasks
• Google's Gemma models: Surprisingly capable for their size
• OpenAI GPT OSS (120b): Large open‑source model with deep understanding of text and code, ideal for detailed prompts and tasks requiring extensive context.

Important note about free tier limits: While the models are free to use, OpenRouter implements rate limits to prevent abuse. Free tier users are limited to 20 requests per minute. Additionally, there are daily limits based on purchased credits:

• Less than 10 credits: 50 free model requests per day
• At least 10 credits: 1000 free model requests per day

I've found that purchasing just $10 in credits OpenRouter (which lasts a long time given how inexpensive these models are) gives me plenty of headroom for daily development work. The cost is negligible compared to traditional AI coding assistants.

The key insight is that different tasks benefit from different models:

• Architecture/design discussions: Use the most capable free model available
• Straightforward code generation: Mid-tier models work perfectly
• Debugging/log analysis: Even smaller models excel at pattern matching
• Learning/exploration: Any model can help explain concepts

Cost Analysis: Nearly Free Development​

Let's break down the actual costs:

Traditional approach:

• Claude Pro: $20/month
• GitHub Copilot: $10/month
• OpenAI API usage: $20-50/month (depending on usage)
• Total: $50-80/month

My current approach:

• Claude Code: Free (open-source CLI tool)
• OpenRouter free tier: $0
• Occasional premium model usage for complex tasks: <$2/month
• Total: <$2/month

That's a 95-97% cost reduction while maintaining excellent development velocity.

Quality Considerations​

You might wonder: does using free models compromise quality? My experience says no, with some caveats:

Where free models excel:

• Boilerplate code generation
• Standard algorithms and data structures
• API integration code
• Documentation and comments
• Bug fixes and refactoring

Where you might want premium models:

• Complex architectural decisions
• Novel algorithm development
• Edge case handling in security-critical code
• When maximum reliability is required

The trick is knowing when to use which. For 80% of my coding tasks, the free models on OpenRouter are more than sufficient.

Tips for Maximizing Free Tier Usage​

1. Be specific in your prompts: Vague requests waste tokens and give poorer results
2. Break problems into smaller chunks: Easier for models to handle, less context needed
3. Use the right model for the task: Match model capability to task complexity
4. Leverage Claude Code's file operations: Let it read your codebase to provide better context
5. Cache good prompts: Save effective prompt patterns for reuse
6. Monitor usage: Keep an eye on which models you're using most effectively

The Bigger Picture: AI for All​

What excites me most about this setup isn't just the cost savings-it's the democratizing effect. When AI-assisted development becomes accessible to virtually anyone with an internet connection, we unlock tremendous creative potential.

Imagine:

• Students in developing countries learning to code with AI assistance
• Hobbyists building projects without worrying about API costs
• Entrepreneurs validating ideas quickly and cheaply
• Open-source contributors able to contribute more effectively

This isn't just about saving money-it's about lowering the barrier to entry for software creation.

Getting Started Yourself​

If you want to try this approach:

1. Install Claude Code: brew install claude-code (via Homebrew)

2. Sign up for OpenRouter: OpenRouter (free tier available immediately)

3. Configure your OpenRouter API key in Claude Code settings

   Claude Code talks to Anthropic by default. To route through OpenRouter you must set the following environment variables before running Claude Code:

   export ANTHROPIC_BASE_URL="https://openrouter.ai/api"
   export ANTHROPIC_AUTH_TOKEN="<your‑OpenRouter‑API‑key>"
   export ANTHROPIC_API_KEY=""
   

   The empty ANTHROPIC_API_KEY tells Claude Code to skip the built‑in Anthropic key and use the custom base URL and token above. This lets you alias any model name (e.g., gpt-oss-120b:free) to the OpenRouter endpoint.

4. Start with simple tasks to get a feel for how the models respond

5. Experiment with different models to find what works best for your workflow

Model configuration: I'm using openai/gpt-oss-120b:free with maxContextTokens set to 135000 in ~/.claude/settings.json to stay within the model’s context window.

The Future of Affordable AI Coding​

As open-source models continue to improve and platforms like OpenRouter expand their free tiers, I believe we're heading toward a future where high-quality AI-assisted coding is accessible to everyone, regardless of budget.

The tools are already here. The cost barrier is falling. All that's left is to build.

I don't trust apps with my crypto. Every wallet, exchange, and dapp is a black box. You hand over your private key, click "send", and hope the code does what it claims. So I built my own scripts instead: small, auditable Python programs that I can read and understand completely.

The differences between Stellar and Ethereum force you to confront each blockchain's design philosophy. Here's what I learned building scripts for both.

Getting Started: Keypair Generation​

The first difference hit me immediately: how do you fund a testnet account?

With Stellar, it's almost too easy. Generate a keypair, then call Friendbot once:

keypair = Keypair.random()
public_key = keypair.public_key

url = f"https://friendbot.stellar.org?addr={public_key}"
response = urllib.request.urlopen(url)
body = json.loads(response.read())
funding_tx = body.get("hash")

Boom. 10,000 XLM, instantly. Stellar was designed for developers.

Ethereum? Different story entirely.

account = Account.create()
address = account.address
private_key = account.key.hex()

Then you have to go find a faucet to fund the account on the testnet. I used the Ethereum Sepolia Faucet from Google, which let me get started quickly with 0.05 Sepolia ETH.

Ethereum's testnet requires you to prove you're human. This isn't a flaw—it's a design choice. Sepolia is scarce by design, and that scarcity forces you to be intentional about testing.

The Decimal Problem​

I discovered the decimal limits problem by accident. I tried sending 10.12345678 XLM and Stellar rejected it silently at the RPC level. Turns out Stellar stops at 7 decimals:

value = Decimal(amount)
decimals = abs(value.as_tuple().exponent)  # Stellar: max 7

Ethereum goes much further. 18 decimals, because wei (10^-18 ETH) is the smallest unit:

value = Decimal(amount)
amount_wei = Web3.to_wei(value, "ether")  # Ethereum: max 18

This taught me something: blockchains aren't interchangeable. They have hard constraints baked into their protocol. You can't just move code from one to the other—you have to understand and respect each system's limits.

Memos: Different Tradeoffs​

I wanted to include descriptive memos in transactions—something like "invoice #12345" to track payments. Stellar made this simple:

memo = "payment"
builder.add_text_memo(memo)  # 28 UTF-8 bytes, free

28 bytes, included in the base fee. Done.

Ethereum doesn't have native memos. Instead, you encode data into the transaction itself:

memo = "payment"
data = "0x" + memo.encode("utf-8").hex()
tx["data"] = data  # Each byte ≈ 16 gas

This is expensive. A 256-byte memo costs ~4,000 gas, which at current rates is about $0.10. So on Ethereum, you think twice before adding metadata to a transaction. On Stellar, you just throw it in. Different design, different tradeoffs.

The Destination Problem​

When I tested sending to a non-existent account, I got completely different behaviors.

On Stellar, it fails:

server.load_account(destination)  # Check if account exists

Stellar makes you prove the destination account is real. If it doesn't exist, you can't send. You need a create_account operation first, then send. Two steps. This forces you to think about what you're doing.

On Ethereum, any address is valid:

destination = Web3.to_checksum_address(destination)  # Normalize with checksum

Ethereum will happily send funds to an address that doesn't exist yet. The address is valid, the transaction succeeds, and now those funds are locked in a contract address nobody owns. Forever. The checksum helps you avoid typos, but it can't save you from a wrong address.

Fee Models: The Biggest Difference​

This is where Stellar and Ethereum revealed their fundamentally different philosophies.

Stellar's fee model is dead simple:

base_fee = server.fetch_base_fee()
ops_count = len(transaction.transaction.operations)
fee = base_fee * ops_count / 10_000_000

print(f"Fee: {fee} XLM ({ops_count} operations)")

Fetch the base fee, multiply by operation count, done. Fees are predictable. They scale gently with network load. You can estimate gas costs accurately before submitting.

Ethereum's model is... more complex:

gas_price = w3.eth.gas_price
tx = {"from": source, "to": dest, "value": amount_wei}
gas_limit = w3.eth.estimate_gas(tx)

fee = gas_limit * gas_price
print(f"Fee: ~{Web3.from_wei(fee, 'ether'):.8f} ETH")

You have to estimate. And the estimate can change. The ~ in my output isn't cosmetic—it's a warning that this fee is approximate. Gas prices spike. Estimates are wrong. You have to check again right before submitting. This forces a different workflow: always dry-run first.

Sequences and Nonces: The Sync Problem​

Both blockchains use counters to ensure transactions don't get duplicated or reordered. But they surface the problem differently.

Stellar's sequence numbers auto-increment:

source_account = server.load_account(public_key)
builder = TransactionBuilder(source_account=source_account, ...)
transaction = builder.build()

Convenient, but also dangerous. If you submit two transactions concurrently, they'll have the same sequence number and one will fail. For scripts, I document it as a limitation. For production systems, you'd need a queue or mutex.

Ethereum's nonce is something you fetch fresh:

nonce = w3.eth.get_transaction_count(source_address)
tx = {"nonce": nonce, ...}

Same problem, same solution needed, but now it's explicit. You're responsible for fetching the nonce. You see the problem immediately. In a way, Ethereum forces you to think about concurrency.

Key Management: Two Philosophies​

For testing, both blockchains let you use plain text private keys. But Ethereum also shipped with a production-ready key format: the encrypted keystore.

Stellar: plain text with restricted permissions:

with open(f"keys/{public_key}.secret", "w") as f:
    os.chmod(f.name, 0o600)  # chmod 600
    f.write(secret_key)

For testnet, this is fine. For mainnet, you're on your own.

Ethereum: built-in encryption:

password = getpass.getpass("Choose password:")
keystore = Account.encrypt(private_key, password)

# Use it later
python send_eth.py --keystore keystores/0x123....json
# Prompts for password at runtime

So I built both workflows. For Ethereum mainnet, I always use encrypted keystores. The password is never stored—I type it on each use. It's an extra step, but it matters.

RPC Reliability: Stellar vs. Ethereum​

Stellar has one official endpoint: Horizon. Rock solid.

Ethereum has hundreds of public RPC endpoints, and they're all flaky:

rpcs = ["https://ethereum-sepolia-rpc.publicnode.com", "https://rpc.sepolia.org", ...]

for rpc_url in rpcs:
    w3 = Web3(Web3.HTTPProvider(rpc_url))
    if w3.is_connected():
        break

Public RPCs go down. They rate-limit. They're unreliable. So I retry multiple endpoints. It's annoying, but necessary.

Error Messages: Structured vs. Parsed​

When a transaction fails, Stellar is explicit:

# Horizon returns structured codes like:
# "tx_insufficient_balance", "tx_bad_seq", "op_no_destination"

Read the code, understand what went wrong, fix it.

Ethereum returns error messages:

# RPC errors arrive as strings like:
# "insufficient funds", "nonce too low", "gas too low"

I had to write string parsing logic to extract the actual error. Not elegant, but it works.

Audit Logging: Non-Negotiable​

After a successful transaction on either blockchain, I log it immediately:

timestamp = datetime.now(timezone.utc).isoformat()
log_line = f"{timestamp} | {network} | from={source} | to={dest} | amount={amount} | tx={tx_hash}\n"

with open(f"transactions.{network}.log", "a") as f:
    f.write(log_line)

Never log private keys—just source, destination, amount, and tx hash. This is your receipt. Months later, you'll need proof of what happened. The log is that proof.

Dry-Run: Always Test First​

Both scripts support a --dry-run flag that simulates the transaction without submitting:

if dry_run:
    print(f"From   : {source}")
    print(f"To     : {destination}")
    print(f"Amount : {amount}")
    print(f"Fee    : {fee}")
    print(f"Balance: {balance}")
    return

On Ethereum especially, this is critical. You see the gas cost before committing. You verify the balance. You check the destination address. The workflow becomes: dry-run testnet, dry-run mainnet, submit mainnet.

The Real Lesson​

I built these scripts because I don't trust black boxes with my crypto. Every exchange, every wallet, every dapp is a centralized point of failure. Any one of them could have a bug that drains your account. Any one could get hacked. Any one could disappear.

But when I read my own code—when I wrote every line—I know exactly what happens when I hit enter. No hidden endpoints. No intermediaries. Just Python + standard libraries + direct blockchain calls.

Stellar and Ethereum are solid platforms. But they can't protect you from:

• A buggy wallet that miscalculates gas
• An exchange that locks your account
• A dapp that requests too many permissions
• A private key stolen from a browser cache

Self-custody through auditable code is the answer.

If You're Building This Too​

1. Trust code, not UIs, You should understand every line that moves your assets.
2. Start on testnet, Free funds. Zero risk. Perfect for learning.
3. Log everything, You need proof. The log is your receipt.
4. Dry-run first, See the fee. See the structure. Before you commit.
5. Keep it simple, No abstractions. No magic. Just straightforward steps.

If you're not comfortable reading the code that moves your assets, don't use it. And if you're using a service you can't read the code for, you're trusting someone else's judgment with your money.

Read the code. Audit it. Modify it. Own it.

Source Code​

Both scripts are open source:

• Stellar XLM: https://github.com/Lunik/stellar_xlm
• Ethereum: https://github.com/Lunik/ethereum

A while ago I wrote a blog post about how I created this blog. Back then it was Pelican, a Python static site generator. Then I migrated it to MkDocs — also Python, a bit more modern, better support for my bilingual setup. Now here we are again. Third time.

This is the story of why I did it and how it went.

The old setup​

Let me describe what I had before so you understand why I wanted to change it.

The portfolio​

A raw HTML/CSS/JavaScript website. No build step. No bundler. Just files on a disk that I uploaded to S3.

It worked. But it was painful to maintain. All the CSS was in one 600-line file mixing colors, layout and components. There was no clear separation between data and presentation. The JavaScript was procedural, with eval() calls to map string property names to DOM attributes. Dependencies were pulled from CDNs with hardcoded version numbers in <script> tags.

Every time I needed to touch something I had to rediscover how it was put together.

The blog​

MkDocs is a perfectly fine static site generator. I have nothing bad to say about it. But it was a separate project, with its own Python environment, its own GitLab CI pipeline, and its own S3 deployment. Two projects, two pipelines, two deployment processes.

And MkDocs is not really designed for a blog. I was using the MkDocs Material Blog plugin which is great but adds a layer on top of something that wasn't built for that use case from the start.

The CI situation​

Two .gitlab-ci.yml files. Two separate cache strategies. Two separate deploy jobs. When I wanted to make a change that touched both the portfolio and the blog — like updating the footer links — I had to coordinate two separate pipelines and two separate deployments. It was annoying.

Why Docusaurus and Vite​

Docusaurus​

Docusaurus is a React-based static site generator made by Meta, primarily designed for documentation. But it has a blog mode that is quite good. It supports:

• Bilingual content out of the box with its i18n system
• Reading time estimation
• Author profiles
• RSS/Atom feeds
• Tag pages

It's built on top of Node.js and npm, which means I'm back in the ecosystem I hate but at least it's the same ecosystem as the portfolio build tooling. Consistency has a value.

I considered Astro and 11ty. Astro is interesting but it felt like more configuration for the same result. 11ty is elegant in its simplicity but the Docusaurus i18n support is hard to beat without writing it yourself.

Vite​

Vite is the obvious choice for the portfolio. It's fast, it handles multi-page apps well, and it has first-class support for ES modules. The migration from raw HTML + procedural JavaScript to a proper Vite multi-page app gave me the structure I needed without forcing me to adopt a component framework.

The monorepo structure​

The decision to merge the two projects into one repository was the most important one. Everything else follows from it.

my_website_v2/
├── package.json          # npm workspaces root
├── packages/
│   ├── website/          # Vite portfolio
│   └── blog/             # Docusaurus blog
└── dist/                 # assembled output

The root package.json declares both packages as npm workspaces and orchestrates the build:

{
  "scripts": {
    "build": "npm run build:website && npm run build:blog && npm run assemble",
    "assemble": "mkdir -p dist && cp -r packages/website/dist/. dist/ && mkdir -p dist/blog && cp -r packages/blog/build/. dist/blog/"
  }
}

The portfolio builds into dist/, the blog builds into dist/blog/. The URL structure is preserved: / for the portfolio, /blog/ for the blog. One rclone sync command on the CI to push everything to S3.

Migrating the portfolio​

CSS refactor​

The first thing I did was extract the Gruvbox color palette into CSS custom properties:

:root {
  --gb-bg:      #1d2021;
  --gb-bg1:     #282828;
  --gb-fg:      #ebdbb2;
  --gb-cyan:    #689d6a;
  --gb-yellow:  #d79921;
  /* ... */
}

Then replaced every hardcoded color literal in the component stylesheets with these variables. No visual change, but now the theme is in one place.

JavaScript modules​

The main challenge was the eval() pattern. The original code used string property names to set dynamic CSS variables:

// Before
config.forEach(function(item) {
  Object.keys(item).forEach(function(key) {
    eval("element." + key + " = item[key]");
  });
});

I replaced this with an explicit property map:

// After
const PROPERTY_MAP = {
  textContent: (el, v) => { el.textContent = v },
  href: (el, v) => { el.href = v },
  style: (el, v) => { el.setAttribute('style', v) },
};

Not as clever, but it works without eval() and it's readable.

UAParser.js​

The portfolio uses UAParser.js to display the visitor's browser and OS on the terminal homepage. Before it was loaded from a CDN:

<script src="https://cdn.jsdelivr.net/npm/ua-parser-js@1/src/ua-parser.min.js"></script>

Now it's a proper npm dependency imported as an ES module:

import { UAParser } from 'ua-parser-js'

Migrating the blog content​

The migration script​

I wrote a one-shot Node.js script to migrate all the MkDocs posts to Docusaurus format. The main things it had to handle:

Frontmatter: MkDocs Material uses categories and tags as separate fields, Docusaurus only has tags. The script merges them and deduplicates. It also removes the readtime field since Docusaurus computes reading time automatically.

Admonitions: MkDocs uses !!! warning "Title" syntax, Docusaurus uses :::warning[Title]. A regex handles the conversion:

content = content.replace(
  /^!!!\s+(\w+)(?:\s+"([^"]*)")?\n\n?((?:(?:    |\t)[^\n]*\n?)+)/gm,
  (_, type, title, body) => {
    const dedented = body.replace(/^(    |\t)/gm, '')
    const header = title ? `:::${type}[${title}]` : `:::${type}`
    return `${header}\n${dedented.trim()}\n:::\n`
  }
)

Directory structure: MkDocs posts are in docs/en/posts/slug/index.md with no date in the path. Docusaurus wants blog/YYYY-MM-DD-slug/index.md. The script reads the date: field from frontmatter and prefixes the directory name.

Truncation markers: MkDocs uses <!-- more --> for the post excerpt cutoff. Docusaurus uses <!-- truncate -->. A simple find and replace.

Bilingual content​

The EN posts go in packages/blog/blog/. The FR posts go in packages/blog/i18n/fr/docusaurus-plugin-content-blog/. Images are stored with the EN post and FR posts reference them with the same relative path — Docusaurus resolves them correctly at build time.

The fun problems​

Node 25 and the localStorage error​

The Docusaurus SSG build crashed on Node.js 25 with:

Cannot initialize local storage without a --localstorage-file path.

Node.js 25 ships the Web Storage API natively, but unlike browsers, it requires an explicit file path to persist data. Docusaurus uses localStorage internally during SSG for theme persistence and had no idea this API now needed initialization.

The fix is a Node.js CLI flag:

NODE_OPTIONS='--localstorage-file=/tmp/docusaurus-ls' docusaurus build

It's in the package.json build script and in the GitLab CI variables block.

Non-breaking spaces​

HTML collapses consecutive regular spaces into one. Everyone knows that. Fewer people remember it when writing strings that will be injected as innerHTML.

The terminal homepage has two commands whose output relies on column alignment: locate (links padded to a common width before the # comment) and ls -l /bin (file sizes and dates right-aligned). The original source used U+00A0 non-breaking spaces for that padding, not regular spaces. In HTML,   or its Unicode equivalent is the only whitespace character that does not collapse.

During the migration, every text editor, diff tool, and copy-paste involved in moving those strings treated U+00A0 as an ordinary space. They all came out as U+0020. The columns disappeared silently — no build error, no test failure, just a cosmetic regression that you only notice when you look at the page.

The fix is surgical: find the original source, extract the raw bytes, restore them. A one-liner in Python to grep the original file and count \xa0 occurrences confirmed it, then a direct byte-level substitution put them back.

Lesson: if column alignment in HTML matters, use a <pre> or white-space: pre and keep the string in a separate file where a linter cannot touch it.

webpack 5.106 and webpackbar​

Docusaurus 3 uses webpack 5 internally. webpackbar — the progress bar plugin — was updated to 6.0.1, which passed options (name, color, reporters) to webpack's ProgressPlugin. But webpack 5.100+ removed those options from ProgressPlugin.

The result: the blog build crashed at startup with an obscure plugin options error.

The fix was to pin webpack to 5.95.0 as a direct devDependency in packages/blog/package.json. npm workspace resolution picks it up and uses it instead of resolving the latest 5.x version.

{
  "devDependencies": {
    "webpack": "5.95.0"
  }
}

Not great, but it works until Docusaurus ships a fix upstream.

The CI pipeline​

The new .gitlab-ci.yml is three stages: install, build, deploy.

install → build → pages (develop branch)
                → publish (master branch)

The pages job copies dist/ to public/ for GitLab Pages on the develop branch. The publish job runs rclone sync to push dist/ to S3 on the master branch.

Before this, I had two pipelines with two separate S3 deploy jobs, one of which used an exclusion pattern to avoid overwriting the blog that was deployed from the other pipeline. Now it's a single rclone sync with no exclusions:

rclone sync --checksum --delete-during --quiet \
  --s3-provider AWS \
  --s3-access-key-id "$AWS_ACCESS_KEY_ID" \
  --s3-secret-access-key "$AWS_SECRET_ACCESS_KEY" \
  --s3-region eu-west-3 \
  ./dist :s3:website-lunik.tiwabbit.fr

--delete-during removes files from the bucket that no longer exist in dist/ as the sync runs. --checksum uses checksums rather than modification times to detect changes — more reliable for files that get reassembled from the same source. Clean.

Conclusion​

Three iterations of this blog. Pelican, then MkDocs, now Docusaurus. Each time the tooling gets better and the maintenance overhead gets lower.

The monorepo was the right call. Having the portfolio and the blog in the same repository, with the same build pipeline, is genuinely less annoying than maintaining two separate projects. I should have done it earlier.

The source is at gitlab.com/Lunik/my_website_v2.

Right click on the image below and save it as an SVG file.

Don't forget to credit Apple for the original design. All right reserved. And me for the approximation.

In this article, I will explain why Python Virtual Environments are so cool when you know how to use them properly. I will also show you how to use them the right way.

What are we talking about ?​

Python Virtual Environments are a way to isolate your Python environment from the system's Python environment. This is useful when you are working on multiple projects that require different versions of Python or different versions of the same package.

I have heard many times that people are afraid of using Python Virtual Environments because they think it is complicated and unnecessary. But at the same time, there are more than hundreds of python packages that tries to solve the same problem : pyenv, virtualenv, pipenv, conda, poetry, pew, and many more.

A little bit of history​

Let's go back to the basics. Wow there is a built-in module for that : venv ! Who knew ?! It was first introduced in Python 3.3 with the PEP 405 – Python Virtual Environments.

This PEP proposes to add to Python a mechanism for lightweight “virtual environments” with their own site directories, optionally isolated from system site directories. Each virtual environment has its own Python binary (allowing creation of environments with various Python versions) and can have its own independent set of installed Python packages in its site directories, but shares the standard library with the base installed Python.

So what about all these people trying to solve the same problem ? Well, they are trying to make it easier to use and to manage. But in the end, they all provide the same thing : a way to isolate your Python environment.

How to use them the right way ?​

You only need to know 3 commands to use Python Virtual Environments the right way.

Create a new virtual environment​

python -m venv myenv

This command will create a new directory called myenv that contains a Python environment. You can replace myenv with any name you want. The most common name is venv of .venv. (Don't forget to add it to your .gitignore file if you are using git).

Activate the virtual environment​

source myenv/bin/activate

This command will activate the virtual environment. You can see that the prompt has changed. It now contains the name of the virtual environment in the beginning of the line.

(myenv) user@host:~$

You can check the Python binaries that are used by the virtual environment with the following command.

which python

Deactivate the virtual environment​

deactivate

This command will deactivate the virtual environment. You are now back to the system's Python environment.

Now what ?​

Now that you know how to use Python Virtual Environments, you can use them in your projects.

In my routine, when I start a new project or when I work on an existing project, the first thing I do is to create a new virtual environment. Then I activate it and I install the required packages.

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

This allows me to work on multiple projects that require different versions of Python or different versions of the same package without any conflicts. Without this technique, I would have to deal with a lot of conflicts and I would have to uninstall and reinstall packages all the time.

Conclusion​

Python Virtual Environments are a great way to isolate your Python environment from the system's Python environment. They are easy to use and they are very useful when you are working on multiple projects that require different versions of Python or different versions of the same package. You should use them in your projects. It will save you a lot of time and headaches.

And remember, Python already provides a built-in module to create virtual environments. You don't need to use any third-party package to do that. Trust Python, not some random package you found on the internet.

Today I have upgraded my Fedora server from version 33 to 38 but all did not go as planned.

I have OpenZFS installed to handle my storage and the current released version of OpenZFS (2.2.2) is not compatible with the latest kernel (6.7). I was stuck with my new kernel and unable to access my data.

For reference here is the error message I was getting:

zfs list

Output:

The ZFS modules cannot be auto-loaded.
Try running 'modprobe zfs' as root to manually load them.

And:

modprobe zfs

Output:

modprobe: FATAL: Module zfs not found in directory /lib/modules/6.7.3-100.fc38.x86_64

Troubleshooting​

The first thing I did was to check what kernel version is supported by the latest ZFS release (2.2.2) and I found out that the latest supported kernel is 6.6. I missed it by one minor version.

Well that's not a big deal, I can just install the previous kernel and boot on it, right?

Let's see what kernels are available:

dnf --releasever=38 --showduplicates list kernel

Output:

Paquets disponibles
kernel.x86_64          6.2.9-300.fc38          fedora  
kernel.x86_64          6.7.3-100.fc38          pdates 

Install previous kernel​

Not really what I was expecting. I was hoping to see the previous kernel version 6.6 but it's not there. Let's try to install the 6.2.9 version and see what happens:

dnf install kernel-6.2.9-300.fc38

Let's reboot and see if it works.

uname -r

Output:

6.7.3-100.fc38.x86_64

After rebooting I was still booting on the latest kernel 6.7.

Let's plug a monitor and see what's happening during the boot process.

Grub2​

During the boot process, I got the Grub2 menu and I was able to select the previous kernel version 6.2.9 but it was not the default boot image. I had to manually select it every time I rebooted the server.

Troubleshooting again​

Let's have a look at the Grub2 configuration to see what's the default configuration:

cat /etc/default/grub

Output:

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="resume=/dev/mapper/fedora_nubs-swap rd.lvm.lv=fedora_nubs/root rd.lvm.lv=fedora_nubs/swap rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

Hum. GRUB_DEFAULT=saved. Loooks like the default boot image is saved somewhere. Let's have a look at the Grub2 configuration file:

grep saved /etc/grub2.cfg

Output:

   set default="${saved_entry}"
if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
function savedefault {
    saved_entry="${chosen}"
    save_env saved_entry

The grub2 configuration file is quite complex and I don't want to mess with it but it seeams that it sets the default boot image to the saved_entry variable.

I found out there is a binary to work with grub2 environment variables: grub2-editenv.

grub2-editenv list

Output:

saved_entry=8bfb6e63623d4ee6aab1634ebafdd5d4-5.6.15-200.fc31.x86_64
kernelopts=root=/dev/mapper/fedora_nubs-root ro resume=/dev/mapper/fedora_nubs-swap rd.lvm.lv=fedora_nubs/root rd.lvm.lv=fedora_nubs/swap rhgb quiet systemd.unified_cgroup_hierarchy=0
boot_success=0
boot_indeterminate=1

5.6 ! That's an old kernel version. I guess my previous upgrade from Fedora 31 was not as clean as I thought.

How can I change the default boot image ? And what is this long hexadecimal string at the beginning of the saved_entry variable ?

Changing the default boot image​

We first need to find the list of available boot images. I found another handy binary to do that grubby.

grubby --info ALL

Output:

index=0
kernel="/boot/vmlinuz-6.7.3-100.fc38.x86_64"
args="ro resume=/dev/mapper/fedora_nubs-swap rd.lvm.lv=fedora_nubs/root rd.lvm.lv=fedora_nubs/swap rhgb quiet systemd.unified_cgroup_hierarchy=0"
root="/dev/mapper/fedora_nubs-root"
initrd="/boot/initramfs-6.7.3-100.fc38.x86_64.img"
title="Fedora Linux (6.7.3-100.fc38.x86_64) 38 (Server Edition)"
id="8bfb6e63623d4ee6aab1634ebafdd5d4-6.7.3-100.fc38.x86_64"

index=1
kernel="/boot/vmlinuz-6.5.12-100.fc37.x86_64"
args="ro resume=/dev/mapper/fedora_nubs-swap rd.lvm.lv=fedora_nubs/root rd.lvm.lv=fedora_nubs/swap rhgb quiet systemd.unified_cgroup_hierarchy=0"
root="/dev/mapper/fedora_nubs-root"
initrd="/boot/initramfs-6.5.12-100.fc37.x86_64.img"
title="Fedora Linux (6.5.12-100.fc37.x86_64) 37 (Server Edition)"
id="8bfb6e63623d4ee6aab1634ebafdd5d4-6.5.12-100.fc37.x86_64"

index=2
kernel="/boot/vmlinuz-6.2.9-300.fc38.x86_64"
args="ro resume=/dev/mapper/fedora_nubs-swap rd.lvm.lv=fedora_nubs/root rd.lvm.lv=fedora_nubs/swap rhgb quiet systemd.unified_cgroup_hierarchy=0"
root="/dev/mapper/fedora_nubs-root"
initrd="/boot/initramfs-6.2.9-300.fc38.x86_64.img"
title="Fedora Linux (6.2.9-300.fc38.x86_64) 38 (Server Edition)"
id="8bfb6e63623d4ee6aab1634ebafdd5d4-6.2.9-300.fc38.x86_64"

index=3
kernel="/boot/vmlinuz-0-rescue-8bfb6e63623d4ee6aab1634ebafdd5d4"
args="ro resume=/dev/mapper/fedora_nubs-swap rd.lvm.lv=fedora_nubs/root rd.lvm.lv=fedora_nubs/swap rhgb quiet systemd.unified_cgroup_hierarchy=0"
root="/dev/mapper/fedora_nubs-root"
initrd="/boot/initramfs-0-rescue-8bfb6e63623d4ee6aab1634ebafdd5d4.img"
title="Fedora (0-rescue-8bfb6e63623d4ee6aab1634ebafdd5d4) 30 (Thirty)"
id="8bfb6e63623d4ee6aab1634ebafdd5d4-0-rescue"

Wait a minute. The output list is really similar to what I have on my monitor during the boot process. And the id field as the same format as the saved_entry variable.

I guess if I change the saved_entry variable to the id of the boot image I want to boot on, it should work.

grub2-editenv - set saved_entry='8bfb6e63623d4ee6aab1634ebafdd5d4-6.2.9-300.fc38.x86_64'

Checking the saved_entry variable:

grub2-editenv list

Output:

saved_entry=8bfb6e63623d4ee6aab1634ebafdd5d4-6.2.9-300.fc38.x86_64
kernelopts=root=/dev/mapper/fedora_nubs-root ro resume=/dev/mapper/fedora_nubs-swap rd.lvm.lv=fedora_nubs/root rd.lvm.lv=fedora_nubs/swap rhgb quiet systemd.unified_cgroup_hierarchy=0
boot_success=1
boot_indeterminate=1

Looks good. Time for the final test. Rebooting the server...

I see a the Grub2 menu and the default selected image is the one I set. Don't move ... 5 ... 4 ... 3 ... 2 ... 1 ... 0.

It boots !

Last check:

uname -r

Output:

6.2.9-300.fc38.x86_64

And my ZFS ?

zfs list

Output:

The ZFS modules cannot be auto-loaded.
Try running 'modprobe zfs' as root to manually load them.

Argh. I forgot to reinstall the ZFS modules for the new kernel. Let's do that.

dnf install zfs

zfs list

Output:

NAME                           USED  AVAIL  REFER  MOUNTPOINT
poule                         3.64T  3.47T   141K  /poule
...

It's alive !

Conclusion​

I hope this guide will help you to configure the default boot image in Grub2. It's a simple task but it can be tricky if you don't know where to look. Also be very careful when playing with Grub2, you can easily break your system. Always have a backup plan (I didn't ... YOLO).

Today I was confronted with a quite annoying problem. I have a server with a 1Gbps network card connected to a 1Gbps switch. I was trying to backup my data as usual but the transfer was slower than usual. Using ethtool I saw that the link speed was only 100Mbps !

Debug the current link speed​

First of all, I need to retreive the name of the network interface. I tougth it was eth0 but it was enp2s0. Thank's Fedora for the new naming convention !

So using the ip command I can get list of all network interfaces and their current status.

ip link show

Output :

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 12:34:56:78:9a:bc brd ff:ff:ff:ff:ff:ff

The enp2s0 interface is the one I'm looking for. I can now use ethtool to get the current link speed.

ethtool enp2s0

Output :

Settings for enp2s0:
	Supported ports: [ TP	 MII ]
	Supported link modes:   10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  100baseT/Full
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Link partner advertised link modes:  10baseT/Half 10baseT/Full
	                                     100baseT/Half 100baseT/Full
	                                     1000baseT/Half 1000baseT/Full
	Link partner advertised pause frame use: Symmetric Receive-only
	Link partner advertised auto-negotiation: Yes
	Link partner advertised FEC modes: Not reported
	Speed: 100Mb/s
	Duplex: Full
	Auto-negotiation: on
	master-slave cfg: preferred slave
	master-slave status: slave
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: external
	MDI-X: Unknown
	Supports Wake-on: pumbg
	Wake-on: g
	Link detected: yes

The Speed line is the one I'm looking for. It says 1000b/s which is not what I'm expecting. Auto-negotiation is on so the network card should be able to negociate the link speed with the switch.

My card supports 1000baseT/Full as seen in the Supported link modes line. The switch also supports 1000baseT/Full as seen in the Link partner advertised link modes line. So why is the link speed only 100Mbps ?

Force the link speed​

I can force the link speed using the ethtool command.

ethtool -s enp2s0 speed 1000

Let's check the link speed again.

ethtool enp2s0

Output :

Settings for enp2s0:
	Supported ports: [ TP	 MII ]
	Supported link modes:   10baseT/Half 10baseT/Full
	                        100baseT/Half 100baseT/Full
	                        1000baseT/Full
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  1000baseT/Full
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Link partner advertised link modes:  10baseT/Half 10baseT/Full
	                                     100baseT/Half 100baseT/Full
	                                     1000baseT/Half 1000baseT/Full
	Link partner advertised pause frame use: Symmetric Receive-only
	Link partner advertised auto-negotiation: Yes
	Link partner advertised FEC modes: Not reported
	Speed: 1000Mb/s
	Duplex: Full
	Auto-negotiation: on
	master-slave cfg: preferred slave
	master-slave status: slave
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: external
	MDI-X: Unknown
	Supports Wake-on: pumbg
	Wake-on: g
	Link detected: yes

The Speed line now says 1000Mb/s which is what I'm expecting.

Conclusion​

I don't really know why the link speed was only 100Mbps. Maybe there was a problem with the switch or the cable. I'll investigate later. For now, I'm happy with my 1Gbps link speed.

This article describes how to use a SSH CA to authenticate to SSH servers. This is particularly useful when you have a lot of servers to manage and you want to avoid the hassle of managing a lot of SSH keys.

The basic idea is to have a CA (Certificate Authority) that will sign the public keys of the users. Then, the users will be able to authenticate to the servers using their signed public key. This way, you don't have to manage the public keys of the users on the servers, you only have to manage the public keys of the CA. You can also limit the validity of the signed public keys to a certain amount of time or to a certain set of servers and users.

Prerequisites​

This article assumes that you have a basic knowledge of SSH and that you have a SSH server running on a Linux machine.

The theory​

What is a SSH CA ?​

A SSH CA is simply an basic SSH key pair (public and private) that is used to sign other SSH public keys. The public key of the CA is then distributed to the servers and the private key is kept secret.

How does it work ?​

When a user wants to authenticate to a server, the server will ask the user to provide a public key. The user will then provide a public key that has been signed by the CA. The server will then check the signature of the public key using the public key of the CA. If the signature is valid, the user will be authenticated.

The practice​

Generate the CA key pair​

The first step is to generate the CA key pair. To do so, you can use the ssh-keygen command:

ssh-keygen \
  -t rsa \
  -b 4096 \
  -f my_ssh_ca

This will generate two files: my_ssh_ca and my_ssh_ca.pub. The first one is the private key of the CA and the second one is the public key of the CA.

Generate a personal SSH key pair (optional)​

This step assumes that you already have a personal SSH key pair. If you don't, you can generate one using the ssh-keygen command:

ssh-keygen \
  -t ed25519 \
  -b 256 \
  -f my_ssh_key

Distribute the CA public key​

Configure the SSH server​

First, you need to configure the SSH server to accept the SSH CA public key. To do so, you need to authenticate to the server and become root. Then, you need to edit the /etc/ssh/sshd_config file and add the following line at the end of the file:

Match Address 192.168.42.*
  TrustedUserCAKeys /etc/ssh/ssh_user_ca
  AuthorizedPrincipalsFile /etc/ssh/authorized_principals/%u
  RevokedKeys /etc/ssh/revoked_keys

Then, you need to create the /etc/ssh/ssh_user_ca file and add the public key of the SSH CA to it:

cat my_ssh_ca.pub > /etc/ssh/ssh_user_ca

For now we don't have any SSH CA public key to revoke, so we only need to create an empty file for the RevokedKeys parameter:

touch /etc/ssh/revoked_keys

Finally, we need to create the /etc/ssh/authorized_principals directory to store the list of SSH principals allowed to authenticate to users:

mkdir /etc/ssh/authorized_principals

Configure principal-based authentication​

As an example, we will configure the SSH server to allow me to authenticate to the wabbit user. To do so, we need to create the /etc/ssh/authorized_principals/wabbit file and add the following line to it:

wabbit_ops

This will allow me to authenticate to the wabbit user using the wabbit_ops principal. Attention, a principal is not a username, it's just a name that will be used to identify the user or a role. You can use whatever name you want, but it's better to use a name that is easy to remember.

Apply the configuration​

Once the configuration is done, you need to restart the SSH server to apply the changes:

systemctl restart sshd

Configure the SSH client​

Now that the SSH server is configured, we need to configure the SSH client to use the SSH CA to authenticate to the SSH server.

Sign the personal SSH public key​

First, we need to sign the personal SSH public key using the CA private key. To do so, we need to use the ssh-keygen command:

ssh-keygen \
  -s my_ssh_ca \
  -I lunik@my_server \
  -n wabbit_ops \
  -V +1d \
  my_ssh_key.pub

This will generate a my_ssh_key-cert.pub file that contains the signed public key.

To check the signature of the signed public key, you can use the ssh-keygen command:

ssh-keygen \
  -L \
  -f my_ssh_key-cert.pub

This will print the following output:

my_ssh_key-cert.pub:
        Type: ssh-ed25519-cert-v01@openssh.com user certificate
        Public key: ED25519-CERT SHA256:Nbe7SXv788c0t3g8D8GQ/5dwHXX4zXtoaEC7r97h5oM
        Signing CA: RSA SHA256:jitBsmIydxjgdW7DttD9piKIWmb4RPQQ7i7wfQ+M4Vg (using rsa-sha2-512)
        Key ID: "lunik@my_server"
        Serial: 0
        Valid: from 2023-09-19T19:13:00 to 2023-09-20T19:14:07
        Principals: 
                wabbit_ops
        Critical Options: (none)
        Extensions: 
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

Authenticate to the SSH server​

Now that the SSH client is configured, we can authenticate to the SSH server using the signed public key. To do so, we need to use the ssh command:

ssh \
  -i my_ssh_key \
  wabbit@my_server

I have a lot of media to watch and I don't want to spend time searching for it, downloading it, renaming it, moving it, etc. I want to be able to request a movie or a TV show and watch it in less than 5 minutes. I also want to be able to watch it on any device, anywhere, anytime.

I do not use this setup to download and watch pirated content. No pirated content was downloaded during the writing of this article. Placeolder files were used instead.

Requirements​

We are going to use the following tools :

• Plex : Media server and player
• Sonarr : TV shows manager
• Radarr : Movies manager
• Prowlarr : Torrents indexer
• Overseerr : Requests manager
• qBittorrent : Torrents client

We are also going to need a linux based server with for hosting all of this.

Deployment​

For the deployment, we are going to use Docker and Docker Compose.

Docker Compose​

The full docker-compose.yml file is available here.

Then we can start the stack with the following command :

docker compose up -d

Then we can check that everything is running with :

docker compose ps

The output should look like this :

NAME                              IMAGE                                    COMMAND             SERVICE             CREATED             STATUS              PORTS
auto-media-center-prowlarr-1      lscr.io/linuxserver/prowlarr:latest      "/init"             prowlarr            20 minutes ago      Up 20 minutes       0.0.0.0:9696->9696/tcp
auto-media-center-overseerr-1     lscr.io/linuxserver/overseerr:latest     "/init"             overseerr           9 minutes ago       Up 9 minutes        0.0.0.0:5055->5055/tcp
auto-media-center-plex-1          lscr.io/linuxserver/plex:latest          "/init"             plex                9 minutes ago       Up 9 minutes        0.0.0.0:3005->3005/tcp, 0.0.0.0:8324->8324/tcp, 0.0.0.0:1900->1900/udp, 0.0.0.0:32410->32410/udp, 0.0.0.0:32400->32400/tcp, 0.0.0.0:32412-32414->32412-32414/udp, 0.0.0.0:32469->32469/tcp, 5353/udp
auto-media-center-qbittorrent-1   lscr.io/linuxserver/qbittorrent:latest   "/init"             qbittorrent         20 minutes ago      Up 20 minutes       0.0.0.0:6881->6881/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:6881->6881/udp
auto-media-center-radarr-1        lscr.io/linuxserver/radarr:latest        "/init"             radarr              20 minutes ago      Up 19 minutes       0.0.0.0:7878->7878/tcp
auto-media-center-sonarr-1        lscr.io/linuxserver/sonarr:latest        "/init"             sonarr              20 minutes ago      Up 19 minutes       0.0.0.0:8989->8989/tcp

Setup​

Now that everything is running, we can start the setup.

Plex​

This is the first thing we are going to setup. Plex is the media server and player. It is the core of our media center.

First we need to access the service through the web interface. The default port is 32400. So we can access it at http://<server-ip>:32400/web.

If you don't have a Plex account, you can create one for free. Then you will be able to access the web interface.

After going through the wizard, you should see something like this :

We can now add our media libraries. For this example, I will add a TV shows library and a movies library.

First the movies library configuration :

Then the TV shows library configuration :

Finally Plex server should look like this :

QBittorrent​

qBittorrent is a torrents client. It will allow us to download torrents.

First we need to access the service through the web interface. The default port is 8080. So we can access it at http://<server-ip>:8080. The default username is admin and the default password is adminadmin.

You should see something like this :

Then we need to configure the download folder in (Options > Downloads). For this example, I will use /downloads as the download folder and /incomplete-downloads as the incomplete downloads folder. Don't forget to click Save at the bottom of the page.

Sonarr​

Sonarr is a TV shows manager. It will allow us to search for TV shows and download them automatically.

First we need to access the service through the web interface. The default port is 8989. So we can access it at http://<server-ip>:8989.

You should see something like this :

Indexers​

We are going to see the configuration of the indexers in (Settings > Indexers). This will be automatically configured by Prowlarr later.

Download client​

We are going to configure the download client in (Settings > Download Client). For this example, I will use the following configuration :

We can test the connection by clicking on Test at the bottom of the page.

Note : You can set the Initial State to Paused if you don't want Sonarr to download all the episodes of the TV shows you add.

Media management​

We are going to configure the media management in (Settings > Media Management). This will allow Sonarr to rename and move the downloaded episodes from qBittorret in the right folder for Plex to find them.

For this example, I will use the following configuration :

Test​

Now we should be able to add a TV show. In (Series > Add Series), we can search for Game of Thrones and add it :

Then we click on the Game of Thrones card and we should see something like this :

Then we can click on the Interractive Search button (small person icon) to search the first season torrents :

We can click on the Download button to trigger the download of the first season.

On the (Activity > Queue) we can see the download progress :

And on qBittorrent we can see the download progress :

And finally when the download is finished, Sonarr will rename and move the episode in the right folder. And Plex will detect the new episode and add it to the library.

Radarr​

Radarr is a movies manager. It will allow us to search for movies and download them automatically.

First we need to access the service through the web interface. The default port is 7878. So we can access it at http://<server-ip>:7878.

You should see something like this :

Indexers​

We are going to see the configuration of the indexers in (Settings > Indexers). This will be automatically configured by Prowlarr later.

Download client​

We are going to configure the download client in (Settings > Download Client). For this example, I will use the following configuration :

We can test the connection by clicking on Test at the bottom of the page.

Note : You can set the Initial State to Paused if you don't want Radarr to download all the Films you add.

Media management​

We are going to configure the media management in (Settings > Media Management). This will allow Radarr to rename and move the downloaded films from qBittorret in the right folder for Plex to find them.

For this example, I will use the following configuration :

Test​

Now we should be able to add a TV show. In (Series > Add Series), we can search for Avengers and add it :

Then we click on the The Avengers card and we should see something like this :

Then we can click on the Search tab to search the film torrents :

We can click on the Download button to trigger the download of the first season.

On the (Activity > Queue) we can see the download progress :

And on qBittorrent we can see the download progress :

And finally when the download is finished, Radarr will rename and move the episode in the right folder. And Plex will detect the new episode and add it to the library.

Prowlarr​

Prowlarr is a torrents indexer. It will allow us to search for torrents on multiple trackers at once.

First we need to access the service through the web interface. The default port is 9696. So we can access it at http://<server-ip>:9696.

You should see something like this :

Then we need to add some indexers. For this example, I will add the following indexers :

Apps​

This section in Settings will allow us to link Prowlarr to Radarr and Sonarr. This way, all the indexers we add in Prowlarr will be automatically added in Radarr and Sonarr.

Add two applications :

By checking in Radarr and Sonarr, we can see that the indexers have been successfully added in the Indexers section.

Overseerr​

As you can see, Radarr and Sonarr are great tools. But they are not very user friendly. Overseerr will allow us to request TV shows and movies in a nice web interface.

First we need to access the service through the web interface. The default port is 5055. So we can access it at http://<server-ip>:5055.

During the first setup, we need to configure the following fields :

Plex Settings

This will allow Overseerr to connect to Plex and look if media are already available in the library.

Radarr Settings

This will allow Overseerr to connect to Radarr and trigger the download of requested movies.

Sonarr Settings

This will allow Overseerr to connect to Sonarr and trigger the download of requested TV shows.

If everything is configured correctly, you should see something like this in the Discovery tab :

We can see that the movie Avengers and TV show Game of Thrones are already available in Plex.

Notes : User that have access to your Plex server will be able to request movies and TV shows. They can login with their Plex account.

You can configure the users in the Users tab. You can for example create a local users that will be able to acces Overseerr without a Plex account.

Requesting media​

We will now try to request a movie from Overseerr. We will use a local user for this example.

Let's try to request the movie Deerskin. From the Discover tab, we can search for Deerskin and we should see something like this :

Then we can click on the Request button to request the movie.

Going back to the Requests tab with our admin account, we can see the request :

We can click on the Approve button to approve the request. This will trigger the download of the movie in Radarr and download it like we saw earlier in the Radarr section.

And finally when all the process is finished, we can see the movie is available in Overseerr and Plex.

Now that I have generated by own custom certification chain in the previous article "x509 certificat managment" I need to install the Certificate Authority (CA) on my iPhone.

Install the certificate​

First I need to install the CA certificate on the device.

The file was on the device and I opened it from the file explorer.

Then I need to go in IOS settings. In the "Général" section.

Then in the "VPN et gestion de l'appareil" section.

Click on the CA in the "Profil téléchargé" section.

I have followed the install process.

Then verified the installation.

Finally I need to approve the CA as a root authority on the device. By going back to the "Général" section. Now in the "Informations" section.

Then in the "Réglages des certificats" section.

Finally toggled the certificate and completed the installation.

Assuming you are using the GitLab Terraform state feature in your self managed instance and you are using the embded backup utility provided by GitLab.

The Terraform state files are encrypted before they are stored. This means that you cannot retreiv the content at rest. For this purpose, GitLab use application secrets (and derive new secrets from thoses keys when needed) to encrypt sensitive content.

You want to retreiv the content of a state file from a GitLab backup. Like explained in this issue, it's not possible to easily retreiv a decrypted content is the instance is offline.

Recover procedure​

Retreiving GitLab instance secrets​

Using the GitLab documentation about application secrets you need to retreiv the value of the db_key_base.

Retreiving project informations​

Terraform state storage is linked to a GitLab project. If the project still exists in your instance, save the project_id for later.

If you have deleted the project we are going to need the database backup (also stored in the GitLab backup).

First search in the public.routes table for your project. You can filter by the source_type=Project and path=path/of/your/project. Write down the value in the namespace_id column.

Then look at the public.projects table filtering the namespace_id retreived just before. Write down the id of the project.

Calculating the state storage path​

States are stored with hashed path.

The first part is the hash256 sum of the project id. You can use the following command to calculate it :

echo -n "<project_id>" | openssl dgst -sha256

The second part can be retreived from the database in the table public.terraform_states. Filter by the project_id and state name save the uuid.

The full folder path of the state file is now :

<part_1[0:1]>/<part_1[2:3]>/<part_1>/<part_2>

If you have enabled versionning state are stored with the filename <serial_number>.tfstate

Retreiving the content​

This is the hardest part and you need to be a little familiar with Ruby.

GitLab use a tool named Lockbox to encrypt Terraform state content before storing them. We are going to use the same tool to decrypt our files.

Fill and use the following script :

#################
# Configuration #
#################

# Retreived from GitLab rails secrets
# https://docs.gitlab.com/ee/development/application_secrets.html
# This is a dummy key base. Don't bother using it
db_key_base = "<FILL_DB_KEY_BASE>"

# The project ID in GitLab
project_id = "<FILL_PROJECT_ID>"

# The file to decrypt
input_file = "<FILL_ENCRYPTED_STATE_FILE_PATH>"

# The file where to write the terraform state content
output_file = "<FILL_DECRYPTED_STATE_FILE_PATH>"

#############
# ALGORITHM #
#############

# Compute encryption key
key = OpenSSL::HMAC.digest('SHA256', db_key_base, project_id)

# Generate LockBox tool
# https://github.com/ankane/lockbox
lockbox = Lockbox.new(key: key)

encrypted_state_content = File.binread(input_file)
state_content = lockbox.decrypt_str(encrypted_state_content)
File.write(output_file, state_content)

I'm not an expert in Ruby but here is a simple bash script to setup the correct environment :

#!/bin/bash

rails new myapp
cd myapp/

echo 'gem "lockbox"' >> Gemfile
bundle install

bundle exec rails runner decode.rb

Resources​

• GitLab source code used to encrypt state files
• GitLab issue for documenting the recover procedure

Any developper use or will use Git at a point in is career. Most of the time they will have to work with other people on the same Git repository. To avoid it to be branch and commit battlefield here is a simple guide on how to contribute properly on a Git repository.

Basics​

First of all, the Git repository should have a default branch often called master or main (it may be anything else as long as everyone agree on the name).

Secondly, a second branch used for development purpose. This one will follow the default branch pretty closely. It will be a receptacle for any new development. This branch is the only one allowed to be merged on the default branch.

Finally, all other branch fall into the last category. They are features, bugs fix and other.

Note : Only HotFix branches are allowed to bypass the development branch.

Here is an example of a simple Git repository

Full example of a repository lifecycle​

Initializing a repository​

The first thing to do when creating a new repository, is to initialize the base structure.

Create the Git repository :

mkdir -p myrepository

cd myrepository

git init

You may want to create your first files. Like the README.md/.gitignore and some package file like package.json or pom.xml.

Then create the initial commit of the repository with those files :

git add .
git commit -m "Initial commit"
git tag v0.0.0

Finally create the development branch :

git branch develop

The repository should looks like :

Initializing the development branch​

Now that you have ou base structure on the Git repository, it's time initialize the development branch.

First checkout to the development branch :

git checkout develop

Sometimes you may want to add an initial commit on the development branch with the modification of the current version in your package file package.json, pom.xml or other.

Edit thoses files then create a commit :

git add .
git commit -m "Prepare development version"

The repository should looks like :

Add the first feature​

Now let's create the first feature in our application.

Verify that your are on the development branch :

git branch

Result :

* develop
  master

If not checkout on the development branch.

Then checkout to a feature branch :

git checkout -b feature/my-first-feature

Now write the feature and do commit from time to time with :

git add .
git commit -m "save dev"

The repository should looks like :

Work on a required feature​

You where working on the first feature but you realize that you needed another one to continue.

/!\ Make sure that you don't have any unstaged changes before switching branches

Reproduce the same commands as for the first feature :

• Checkout to the development branch
• Then checkout to a feature branch
• Write your code and create commits

git checkout develop
git checkout -b feature/my-required-feature

The repository should looks like :

Merging the required feature​

Now that you have finished your work on the required feature comes the time to merge the code to the development branch.

First you need to rebase the branch to remove all unecessary commits :

git rebase -i develop

The first commit should always be picked. All other commits can be squashed.

pick fa71872 save dev
squash 5d9a97a save dev

The repository should looks like :

Checkout to the development branch and merge the require feature branch :

git checkout develop
git merge --no-ff feature/my-required-feature

The repository should looks like :

Continuing the work on the first feature​

Now that you have finished the required feature, you want to rebase your current work on the first feature to retrieve the content of the required feature.

• Checkout to the feature branch
• Rebase it on the developement branch

git checkout feature/my-first-feature
git rebase develop

What this command is doing is taking all the commit from the feature branch and apply them at the end of the developement branch.

The repository should looks like :

Merging the first feature​

Now that you have finished your work on the first feature comes the time to merge the code to the development branch.

First you need to rebase the branch to remove all unecessary commits :

git rebase -i develop

The first commit should always be picked. All other commits can be squashed.

pick 717051b save dev
squash 7d39273 save dev
squash 7b700f1 save dev

The repository should looks like :

Checkout to the development branch and merge the require feature branch :

git checkout develop
git merge --no-ff feature/my-first-feature

The repository should looks like :

Hotfix critical issue​

A critical issue have been discovered on the production application an you need to quickly produce a patch. You don't have the time to go through all the release process.

• Checkout from the default branch
• Create a new hotfix branch and checkout to it
• Make the correction
• Merge into the default branch
• Tag your new release

The repository should looks like :

Releasing your work​

It's now time to release all that hard work.

First you need to merge the default branch to retreiv all hotfix corrections.

git checkout develop
git merge --no-ff master

The repository should looks like :

And finaly, merge your development on the default branch.

git checkout master
git merge --no-ff develop

The repository should looks like :

Cleanup​

Let's do some cleanup by removing some unused references like feature and hotfix branches.

git branch -D feature/my-first-feature
git branch -D feature/my-required-feature
git branch -D hotfix/correct-critical-issue  

The repository should looks like :

Notes​

All Git graph have been generated with Bit-Booster app

Using OpenSource software written by unkown people sometimes can be a little scary. Even more when I deploy them I a production environment in my company. On my case, I have created a brand new Kubernetes cluster to host some private services on my local network and I wanted to be sure that they don't do anything malicious on my network.

Isolate applications and services​

First thing to do is to isolate each of the services inside a Kubernetes namespace. This is a core resource of Kubernetes that allow to isolate groups of resources within a single cluster. It then allow to have a more fine control on access, permissions, network on the resources.

A Namespace can be create pretty easily with the following command :

kubectl create namespace <insert-namespace-name-here>

Then I can list Namespaces with :

kubectl get namespaces

Result :

NAME              STATUS   AGE
default           Active   4h52m
kube-system       Active   4h52m
kube-public       Active   4h52m
kube-node-lease   Active   4h52m
tiwabbit-prod     Active   2s

Ignore Namespaces prefixed with kube- who are defined for the Kubernetes control plane

I can now create resources inside my Namespace. Let's start with a Secret for example :

kubectl \
  --namespace tiwabbit-prod \
  create secret generic \
  db-credentials \
  --from-literal=username=tiwabbit \
  --from-literal=password=mysecurepassword

If I list all Secrets in my cluster :

kubectl get secrets --all-namespaces

Result :

NAMESPACE       NAME                  TYPE                                  DATA   AGE
[...]
default         default-token-m59jl   kubernetes.io/service-account-token   3      4h58m
tiwabbit-prod   default-token-8zkr2   kubernetes.io/service-account-token   3      3m15s
tiwabbit-prod   db-credentials        Opaque                                2      49s

In theory, only Pods running inside my Namespace (tiwabbit-prod) can mount this secret and read it.

RBAC securization​

By default all Pods without specific configuration use the default ServiceAccount of the Namespace they are running in. This last one dosn't have any right on the Kubernetes API witch is a great thing and should not be changed.

How ever in some scenarios my application may needs to call the Kubernetes API. For exemple if it need to create batch using a Kubernetes Job. Let's take that last exemple to create a ServiceAccount with those permission an assign it to my application Pod.

First I need to create a new ServiceAccount :

kubectl \
  --namespace tiwabbit-prod \
  create serviceaccount \
  my-application

Then I need a Role that implement the level of permission that my Pod need. here is the manifest :

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-application-batch-creator
  namespace: tiwabbit-prod
rules:
- apiGroups: []
  resources: [ pods, pods/status, pods/log ]
  verbs: [ get, list, watch ]

- apiGroups: [ batch ]
  resources: [ jobs ]
  verbs: [ create, get, list, watch, patch, update, delete ]

Finally, I need to assign the Role to my ServiceAccount using a RoleBinding with the following manifest definition :

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: my-application-permissions
  namespace: tiwabbit-prod
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: my-application-batch-creator
subjects:
- kind: ServiceAccount
  name: my-application
  namespace: tiwabbit-prod

Now let's create a Pod with the ServiceAccount and review the actions allowed by the role. Here is a Deployment manifest :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-application
  namespace: tiwabbit-prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-application
  template:
    metadata:
      labels:
        app: my-application
    spec:
      serviceAccountName: my-application
      containers:
      - command:
        - sleep
        - 1d
        image: alpine:latest
        name: alpine

Geting inside the Pod and follow the Kubernetes documentation to install kubectl.

Let's check if I can list Pods by querying the Kubernetes API :

kubectl get pods

Result :

NAME                             READY   STATUS    RESTARTS   AGE
my-application-df5b5cb75-8twc5   1/1     Running   0          5m21s

Then I should try to create a batch execution with a Job :

kubectl create job my-application-batch --image alpine:latest -- echo "Hello World"

Listing all the Jobs in the Namespace :

kubectl get jobs

Result :

NAME                   COMPLETIONS   DURATION   AGE
my-application-batch   0/1           2s         2s

Listing the Pods in the Namespace :

kubectl get pods

Result :

NAME                             READY   STATUS    RESTARTS   AGE
my-application-df5b5cb75-8twc5   1/1     Running   0          11m
my-application-batch-f2pf6       1/1     Running   0          3s

My Job finished and I can delete it with :

kubectl delete job/my-application-batch

Conclusions​

If the process in my Pod doesn't need to communicate with the Kubernetes API (for creating, querying, deleting ressources), use the default ServiceAccount witch give zero permission to my Pod. In other case, use a custom ServiceAccount for each of my apps and multiple Role and RoleBinding for each of my application use case.

Prevent usage of root user or privileged escalation​

Kubernetes allow by default multiple security options that can be applied to pods and underlaying containers. Most of them can be configured with the SecurityContext block as follow :

apiVersion: v1
kind: Pod
metadata:
  name: my-secure-pod
spec:
  securityContext: {}
  containers:
  - name: my-secure-container
    image: nginx:latest
    securityContext: {}
  - name: my-second-secure-container
    image: busybox:latest
    command: ["sleep", "infinity"]
    securityContext: {}

Obviously such options can be added to a StatefulSet or Deployment template.

Running pod as non root​

The easiest setup to secure the Pod is to overwrite the UID and GID of the user running the main process. This way, what ever the default user in the image, Kubernetes will bypass it. An UID and GID superior or equal to 1000 should be used.

Three parameters can be used :

• runAsUser : Allow to change the UID of the default process
• runAsGroup : Allow to change the GID of the default process
• fsGroup : If specified, the user will also be in that group and all files and directories created will take that GID as owner.
  • This last parameter can increase the mount time of a given external volume because Kubernetes ensure that files are owned by the group defined by fsGroup. Resulting on an chmod -R of all the filesystem.
  • fsGroupChangePolicy can be used to change this behaviour with those values :
    • OnRootMismatch : only check the root directory and change all filesystem if the group mismatch
    • Always : it's in the option name

apiVersion: v1
kind: Pod
metadata:
  name: my-secure-pod
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 2000
    fsGroupChangePolicy: OnRootMismatch
  containers:
  - name: my-secure-container
    image: busybox:latest
    command: ["sleep", "infinity"]
    securityContext:
      runAsUser: 2000
      runAsGroup: 2000

Enforcing running as non root could be achieved with runAsNonRoot parameter :

apiVersion: v1
kind: Pod
metadata:
  name: my-secure-pod
spec:
  securityContext:
    runAsNonRoot: true
  containers:
  - name: my-secure-container
    image: busybox:latest
    command: ["sleep", "infinity"]
    securityContext: {}

Use readonly filesystem​

Preventing all write/update/delete operation on the root filesystem can prevent malicious process (or breached application) to take advantage of the container environment and modify it. The readOnlyRootFilesystem option allow to achieve that :

apiVersion: v1
kind: Pod
metadata:
  name: my-secure-pod
spec:
  securityContext: {}
  containers:
  - name: my-secure-container
    image: busybox:latest
    command: ["sleep", "infinity"]
    securityContext:
      readOnlyRootFilesystem: true

If the application need to write temporary or application data during the runtime, I can still create an EmptyDir volume and mounting it inside the container :

apiVersion: v1
kind: Pod
metadata:
  name: my-secure-pod
spec:
  securityContext: {}
  containers:
  - name: my-secure-container
    image: busybox:latest
    command: ["sleep", "infinity"]
    securityContext:
      readOnlyRootFilesystem: true
    volumeMounts:
    - mountPath: /data
      name: my-data
  volumes:
  - name: my-data
    emptyDir: {}

Prevent priviled escalation​

Linux kernel expose low level function for a process to change is current UID or GID. For example using [setuid][linux-setuid-wikipedia] or setgid privitive functions.

Using the option allowPrivilegeEscalation can prevent this to happens.

apiVersion: v1
kind: Pod
metadata:
  name: my-secure-pod
spec:
  securityContext: {}
  containers:
  - name: my-secure-container
    image: busybox:latest
    command: ["sleep", "infinity"]
    securityContext:
      allowPrivilegeEscalation: false

Dropping all capabilities​

As for privileged escalation, Linux give Capabilities to process allowing them to make high privileged system calls. Like binding network ports under 1024.

We want to only give our process only the necessary capabilities for it to run. In the cas of the exemple we can have :

apiVersion: v1
kind: Pod
metadata:
  name: my-secure-pod
spec:
  securityContext: {}
  containers:
  - name: my-second-secure-container
    image: busybox:latest
    command: ["sleep", "infinity"]
    securityContext:
      capabilities:
        drop:
        - ALL
        add:
        - NET_BIND_SERVICE

Network communication hardening​

In the world of pods, by default every pods in every namespace can talk to each other. If you have multiple application or even multiple clients on the same Kubernetes cluster, you should not allow them to communicate (at least not by default).

That's why, NetworkPolicies must be created with deny all by default. Then open point to point connections if services/clients needs to communicates with each others.

The default deny all config will looks like this :

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  podSelector: {}
  policyTypes:
  - Ingress

Then if you have a third party application that needs to access your application pods :

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
  namespace: my-namespace
spec:
  podSelector:
    matchLabels:
      application: my-application
      component: api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              project: my-friend-namespace
        - podSelector:
            matchLabels:
              application: his-application
      ports:
        - protocol: TCP
          port: 8080

Now that every web service encourage you, more and more, to use MFA to secure your account, one of them is used most than others : Time-based one-time password or TOTP generate a unique code of 6 or more numbers to enter just after typing your password.

The server or web app allowing to setup TOTP give a QRCode to scan (or a Base32 string) to configure in a TOTP generator app like Microsoft Authenticator, Google Authenticator, Bitwarden or more.

We all use it, but how does it work ? Is it secure ? Is my account secure when using third-party TOTP generator ??

The protocol​

I will get deeps into the details of the underlying protocol allowing you to generated TOTP and how the server verify it.

Basics​

The basic idea is pretty straight forward. You and the server shared a secret, a 16 character long Base32 string (sometimes stored in the QRCode). Then you both synchronize your clocks.

Finally when you want to login, you execute a fancy magical function and ultimately you get a TOTP. The server compute the TOTP on his side and ask you the one you generated on your side. If both of the TOTP match, the server allow you in.

TOTP protocol​

Let's dig into the TOTP protocol. According to the RFC 6238 you need :

• A moving counter that must be synchronized between the two parties
• A secret shared between the two parties

The counter is calculated base on epoch time (That way everybody is synchronized) with the following formula :

C = (T - T0) / X

With :

• C the counter value
• T the current Unix time in seconds since epoch
• T0 an arbitrary start Unix time. 0 by default
• X the time step in seconds or the frequency of code update. 30 by default

If we try to make a Python implementation, it will looks like :

t0 = 0
# https://docs.python.org/3/library/time.html#time.time
current_unix_time = time.time()
time_step = 30

counter = int((current_unix_time - t0) / time_step)

current_unix_time: 1651094239.491242


counter => 55036474

Now the TOTP value can be retrieved with the following formula :

TOTP = HOTP(K, C, D)

With:

• C the counter value
• K the secret key
• D the number of digits in our TOTPs
• HOTP the magical function that calculate the OTP

In Python, we get :

counter = 55036474
key = "ABCDEFGHYJKLMNOP"
digits = 6

totp = hotp(key, counter, digits)

totp => 934929

HOTP protocol​

This second protocol is used to calculate a unique value a counter value and the secret key. From the RFC rfc4226, we learn that its based on HMAC hash protocol.

Sit tight because it will begin to be challenging and I hope that your are comfortable with binaries operations.

Handle inputs​

First of all, we need handle the 2 of our 3 input parameters : counter, key

• counter needs to be converted into an unsigned long long and big-endian bytes array value.

In Python :

import struct
# https://docs.python.org/3/library/struct.html#struct.pack
# https://docs.python.org/3/library/struct.html#byte-order-size-and-alignment
# https://docs.python.org/3/library/struct.html#format-characters
bytes_counter = struct.pack('>Q', counter)

counter: 55036474


bytes_counter(bin) => 00000011 01000111 11001010 00111010
bytes_counter(hex) => 03 47 ca 3a

• key should be a valid Base32 string. The RFC 4648 tells us that the string length need to be a factor of 8. So padding can be added with char = at the end if it's too short. For example : AA => AA======, BBBBBBBBBB => BBBBBBBBBB======, CCCCCCCC => CCCCCCCC

padding = '=' * ((8 - len(key)) % 8)
valid_key = key + padding

key: ABCDEFGHIJKLMNOP


valid_key => ABCDEFGHIJKLMNOP

Then we convert the updated key in bytes array value :

import base64

bytes_key = base64.b32decode(key)

valid_key: ABCDEFGHIJKLMNOP


bytes_key(bin) => 10001000 01100100 00101001 10001110 10000100 10101001 01101100 0110101 11001111
bytes_key(hex) => 44 32 14 c7 42 54 b6 35 cf

Calulate HMAC​

Using the RFC 2104 we can understand how the HMAC function work. We need to choose an hash function. HOTP or RFC 4226 use the SHA-1 cryptographic hash function.

Python implement HMAC library :

import hmac

# Calculate the HMAC-SHA1 value
# Produce a 20 bytes (or 160 bites) long string
mac = hmac.new(bytes_key, bytes_counter, "SHA1").digest()

bytes_counter(hex): 03 47 ca 3a
bytes_key(hex): 44 32 14 c7 42 54 b6 35 cf


mac(bin) => 10110100 11010010 01111010 10110100 10111101 00110101 11111110 00100011 11101101 01011001 01111110 10111100 11110000 01111001 11000001 01001010 00000110 01101100 01010001 00101111
mac(hex) => b4 d2 7a b4 bd 35 fe 23 ed 59 7e bc f0 79 c1 4a 06 6c 51 2f

Calculate HOTP​

Following the RFC 4226 section 5.3.

Finding offset​

We first need to find the offset value by getting the least significant 4 bits of the MAC (The 4 on the right) using bitwise AND operation

In Python & can be used to make binary AND operation between two number. With the value 0x0F (in hexadecimal and equivalent to 00001111 in binary) we can extract two 4 last bits. Since offset is an integer stored on 4 bits, the value is between 0 and 15. In Python we can use mac[-1] to get the last byte of the bytes array converted in an integer.

offset = mac[-1] & 0x0F

mac(bin): 10110100 11010010 01111010 10110100 10111101 00110101 11111110 00100011 11101101 01011001 01111110 10111100 11110000 01111001 11000001 01001010 00000110 01101100 01010001 00101111
mac(hex): b4 d2 7a b4 bd 35 fe 23 ed 59 7e bc f0 79 c1 4a 06 6c 51 2f


offset(bin) => 1111
offset(hex) => f
offset(dec) => 15

Finding the dynamic truncation​

Using the previously found offset, we are going to extract the Dynamic Truncation from the MAC. We need to retrieve the 4 bytes at position offset, offset+1, offset+2 and offset+3 in the MAC bytes array.

We got :

dynamic_truncation = mac[offset:(offset+3)+1]

In Python the value after the : is not included. In order to extract offset+3 value, we need to put (offset+3)+1

offset(bin): 1111
offset(hex): f
offset(dec): 15

offset(dec) from 15 to 18

mac(bin): 10110100 11010010 01111010 10110100 10111101 00110101 11111110 00100011 11101101 01011001 01111110 10111100 11110000 01111001 11000001 01001010 00000110 01101100 01010001 00101111
mac(hex): b4 d2 7a b4 bd 35 fe 23 ed 59 7e bc f0 79 c1 4a 06 6c 51 2f


dynamic_truncation(bin) => 10010100 00001100 11011000 1010001
dynamic_truncation(hex) => 4a 06 6c 51