Ryan Day (author of Hands-On APIs for AI and Data Science, O’Reilly) and writer of the Tip Sheet newsletter featured me in his issue #65 last week.

He asked eight questions about my background, my stack, and how I learn.

There were three of them I wanted to expand a bit and share some thoughts with you.

Q4: Why are side projects valuable for people learning data science?

In a side project you set the tone of what you want to build, why it matters, and how to measure if it worked.

You pick a problem you care enough about, you define the requirements, the scope, the technology and the dataset. It forces you to learn and think with a kind of clarity that structured learning rarely does.

I started with football analytics because I wanted to apply what I was learning to something I cared about and found fun enough and it has led me to push my knowledge forwards and collaborate with special people I wouldn’t think I’d meet if I didn’t build these kind of projects.

Fun fact is: this also helped to me work on healthcare data/AI projects on my day job.

Q6: What methods of skill development have you benefited from most?

I told Ryan I combine three: structured learning, project-based learning, and unstructured learning.

In this case my opinion is that the order matters.

Structured learning gives you the map and foundations to get the knowledge that you can then apply to projects. It doesn’t mean you need to learn everything there is about a topic and fall into rabbit holes, but just learning the necessary info to start building and leaving room for experimentation while you research stuff when you’re stuck.

On the other side unstructured learning like newsletters, blog posts, a well-curated social media timeline, keeps you updated to where the field is actually going. It can even give you some ideas to implement them yourself on your projects.

A dangerous trap is staying too long in one mode.

Entering tutorial hell by finishing a course and starting another course with no meaningful output (been there) creates an illusion of learning. Or jumping straight to building without a minimal foundation, spending weeks debugging something a couple hours read (or documentation) would have clarified.

The sweet spot is knowing when to shift from one to another. For example, if you’re new to something or you’re a bit confused, go with structured.

When you feel you got a good initial grasp of that knowledge, go build something even if it’s small, to make it stick.

If you feel you want to see how the field evolves, go with unstructured.

Right now I’m running all three in parallel: taking Agentic AI Engineering course at Towards AI Academy by Paul Iusztin and Louis-François Bouchard, applying it directly to a tool I’m building for myself, and constantly reading newsletters like Ryan’s to stay sharp on what people are shipping.

Q7: What would you tell people who want to keep developing their technical skills?

To make it stick, I’ll put it inside quotes:

Write about what you learn and share it online.

Not for personal branding but because explaining something forces you to actually understand it.

Every post I’ve written about RAG pipelines, MCP, or football data sources has exposed gaps in my own thinking that I didn’t know were there until I tried to put them into words. This makes me want to understand things better.

Th big myth we need to debunk is the need for a big audience. You actually don’t. The “trick” is to be consistent even if you have a “small audience” or you think nobody’s going to listen. There are people out there that will definitely benefit from your point of view.

You can write a bug you fixed, the trade-off you made when deciding on your tech stack, the framework you just learn from one field and how you applied it to solve a problem in another.

Maybe ten people will read it but one of them might be building the exact same thing and needed exactly that post. Or one of them might be writing a newsletter and looking for someone to interview 🤷🏻‍♂️

Your job is to make it exist.

Subscribe

Here’s the link to the full interview: Tip Sheet #65: Data Scientist, Doctor, Football Aficionado

Till next time,

Ricardo.

That led to dashboards and models nobody used or cared.

Then last week in the Twelve Football Context Engineering course, I realized how the concept behind the framework for measuring team and player performance they teach could be used for solving problems in other areas.

Asking questions first, then data

I’ve been taking this course for the past two weeks and the knowledge compounds incrementally from one week to another. The goal is to create a project around qualities, metrics, pitch visuals and time series approach that they emphasize at Twelve.

Their framework starts with a simple instruction:

Start with football. Not data.

The first step is asking a human question like “what you are trying to answer and for whom?”.

What insight can you bring that no-one else has brought before?

An initial question might be: What makes a good midfielder? or What makes a team dangerous in defensive transitions?

Once you’ve answered that in plain football language you move to the data. The idea is to build Qualities → composite scores that capture a specific aspect of player or team performance.

You choose 3 to 10 metrics that capture different aspects of the concept. Assign weights based on how important each one is to the original definition. Compute scores using z-scores and compare each player against the right group.

That last part is highly important.

For example when building a Passing Quality score for midfielders, you can end up with five metrics: Passes (xT), Passes into the final third (xT), Passes in the final third (xT), Crosses (xT), and Creative passes weighted at 30%, 20%, 20%, 20%, and 10% respectively. The result is a single score that tells you how threatening a midfielder’s passing is.

In this image you can see Belgian player Kevin De Bruyne scores near the top but the framework also tells you why, and which specific dimension he excels at versus where he’s merely good compared to other players.

That’s the useful part.

Context is everything

“Metrics are meaningless without context. Choosing the correct comparison group is essential.”

The comparison group (position, age, league, minutes played) is where we need to emphasize the analysis. Posing the problem and thinking clearly about it before writing the first line of code.

For example a midfielder with high progressive passes has a different meaning than an attacking midfield who sits deep in the field.

A 25 y/o player that has an elevated duel success rate has a different interpretation than a 32 y/o player in the same league.

To produce accurate insights, we need to use the same comparison pool of players.

How you can use it in other industries

Take diabetic patient monitoring. A case I’ve been actively working on in my day job.

The human question becomes: How well is this patient managing their blood sugar?

You define it in clinical terms first, consistent glucose levels, avoiding dangerous highs and lows, showing improvement over time. Then you look at what you can measure: HbA1c, time above range, time below range, trend direction, glucose variability.

You weight them by clinical importance to finally compare against patients of similar age, diabetes type, and diagnosis duration.

The output is a sentence a physician can actually use:

“This patient’s glycemic control is below average for their profile, but trending in the right direction over the last six months. The main challenge is post-meal spikes, not overnight lows.”

The focus is on turning data into a decision.

You could also apply this template for industries like customer churn in SaaS or credit risk in finance, by always starting with a human question.

Key Takeaways

The best analytical frameworks are domain-agnostic. They travel between industries.

Start with the human question by defining the concept before you open the dataset, choosing your comparison group deliberately, then let the data answer.

The framework already exists in your field and most people just never look outside their own.

Share The Underdoc

Thanks for reading one more time.

Till next time,

Ricardo.

The numbers and metrics were being computed, the LLM was receiving them, acknowledging them, and then repeating them back to me as if reading from a spreadsheet.

“Home xG was 1.2 and they scored 3 goals. Away PPDA was 4.8, indicating a high press...”

There was no enriched analysis it was just data recitation that didn’t sound like football language.

I was using claude-haiku-4.5 which is capable of writing football language. If you ask it to analyze a match in prose it’ll produce something close to what a scout would write. The model actually knew what a high press looked like, what a low block meant, defensive compactness, progressive passing patterns, etc, but by asking it to recite metrics I just wasn’t letting it “speak” football.

The problem was my prompt not the model

I was passing raw floats directly to the LLM (PPDA of 4.8, expected goals of 1.2, progressive pass counts, compactness scores) and then instructing it to “state only what the metrics directly prove” with “technical terminology and neutral tone.”

David Sumpter, the brain behind Soccermatics and Twelve Football, calls the solution → Wordalisation. The idea is straightforward: before you call the LLM, classify your raw metrics into qualitative English labels. The model never sees the numbers. It only sees what those numbers mean in the language of the domain.

Here’s what the change looked like in practice.

Before, the user prompt contained something like this:

home_ppda: 4.8
home_high_press_count: 12
home_compactness: 33.4
home_field_tilt: 67.9

After applying Wordalisation, the LLM received this instead:

home_press_style: pressed_aggressively_allowing_few_passes
home_defensive_shape: compact_block
home_territorial_dominance: dominated_final_third
home_shot_quality: created_high_quality_chances

Just English labels describing what the team actually did, derived from the real Eredivisie 2025-2026 distributions rather than arbitrary thresholds.

The smoke test output shifted the result for good:

Before:

“Home xG was 1.2 and they scored 3 goals. Away PPDA was 4.8...”

After:

“Heracles set up to frustrate, sitting deep and inviting PEC Zwolle to come at them — a coherent enough plan in isolation, but one that demanded defensive discipline and clinical use of the rare moments they could hurt the visitors on the break. A low block without the finishing to punish opponents on the counter is simply a slow surrender.”

The data from the pipeline was exactly the same, but thanks to this tweak the output changed completely.

The second change: how I formatted the context

I implemented this one from a course I’m working through called Agentic AI Engineering from Towards AI.

The lesson was about Context Engineering, the discipline of deciding what goes into a model’s context window in order to make it follow the right instructions.

One of the practical recommendations was: prefer YAML over JSON when providing structured data as input.

I’d been using Python dicts serialized as JSON. The lesson argued YAML is more token-efficient and reads more naturally alongside prose examples in a prompt.

I ran a quick comparison on my own <tactical_labels> block.

JSON version:

{
  "home_press_style": "pressed_aggressively_allowing_few_passes",
  "home_defensive_shape": "compact_block",
  "home_territorial_dominance": "dominated_final_third"
}

YAML version:

home_press_style: pressed_aggressively_allowing_few_passes
home_defensive_shape: compact_block
home_territorial_dominance: dominated_final_third

The YAML block is shorter (no brackets, no quotes, no commas), and it reads more like prose which matters when it’s sitting next to few-shot examples written in natural language. The model processes both from the same context window. Found out that small token savings compound across many calls, and a cleaner context means less noise for the model to work through.

The other Context Engineering principle I applied was prompt structure.

The lesson covered something called the “lost-in-the-middle” problem: LLMs tend to underweight information placed in the middle of long contexts, so what they remember best is what comes first and what comes last.

The practical implication: you have to put your most important instructions at the START (system prompt) and at the END (<instructions> block). Dynamic content like match labels and retrieved facts goes in the middle.

And for caching: put static content first. That way the system prompt and few-shot examples don’t change between calls. Both go at the top, before any dynamic match data. This allows the provider to cache the static prefix and skips re-processing it on every requests.

My prompt structure now looks like this:

<system_prompt>     ← static, cached
  [scouting style instructions]

<few_shot_examples> ← static, cached
  [two reference match analyses]

<match_context>     ← dynamic per call
  [YAML labels for this match]

<instructions>      ← static, repeated at end
  [core constraints, re-anchored]

The model gets its instructions twice.

Once at the top before anything else and once at the bottom after seeing the data. This is how we can avoid lost-in-the-middle.

A broader principle connecting Wordalisation and Context Engineering

Both are fundamentally about the same thing: giving the model the right kind of information.

Wordalisation says: don’t give the model numbers and ask it to interpret them. Classify the numbers into domain language first, then give the model language it can work with.

Context Engineering: don’t give the model everything you have. Select what’s relevant, compress what’s not, format it so the model can use it without fighting its own architecture.

Since the LLM does not work as a calculator nor a database we need to feed it with the right data: language.

Key Takeaways

The RAG system has a well curated data pipeline but now, at the point where structured data meets the language model, there’s a much needed translation layer that needs to be included.

The data flows like: Numbers → Labels → LLM → football insight.

That’s exactly the principle of Wordalisation applied to Context Engineering.

Hope this gives you an idea on how to translate metrics into text, by leveraging LLM’s text generation capabilities.

Till next time,

Ricardo.

At some point of our lives, someone tells you to pick a lane.

The school, a teacher, family, social media. Maybe your own head at 2am wondering if you’re interested on too many things to ever be taken seriously at any of them.

“Pick one thing, go deep enough and become an expert at that for the rest of your life.”

Although the most interesting people I’ve come across didn’t get there by picking a lane but by refusing to.

Subscribe now

Specialists & Outliers

I’ve been recently reading Dan Koe’s Purpose & Profit and a concept that grasped my attention was that the education system was never designed to help you reach your potential but to turn you into a useful workers in an interchangeable role.

Debatable but I can see the point behind it, if you’re only trained to do one thing you can’t adapt. Specialization isn't bad. But betting your entire identity on a single skill, in a world that's moving this fast, is the riskiest thing you can do.

Your Weird Combination Is the Advantage

I’ll use my own story here because it’s the most honest one I have.

I graduated from med school in Venezuela in 2019. But I felt something was off. Not wrong exactly, I knew I was supposed to be there but I wasn’t feeling fulfilled.

So I started exploring other things on the side: bartending, mobile app development, whatever I could get my hands on.

Then the pandemic hit, and I stumbled upon Python.

Something clicked.

I started applying it to football analytics and got immediately hooked.

I took MOOCs, networked with people in the industry, and eventually joined a non-profit group of analysts working with NWSL and national teams with real data from top providers like StatsBomb, Tracab, Opta.

Then I expanded into data science and machine learning applied to business. This expertise helped me land my first tech role at an AI consultancy, working on health projects and speech recognition.

Along the way, LLMs and RAGs came into the picture, and I started applying all of it to the projects I was already building.

At every step, someone raised an eyebrow.

Why tech? You are a doctor. Why football? You’re a doctor in tech.

I never had a clean answer, but I knew I was shaping the foundations of my identity.

At some point I got honest with myself about where I actually stood. I listed the gaps in my foundations: systems thinking, engineering depth, the ability to build things that work in production and not just in a notebook.

So I looked back at my background as my advantage and started building from it.

I had:

• A medical mindset

• Data/AI engineering foundations

• Football domain knowledge

• AI tools to help me build and mix all of these.

That led me to create my own Venn diagram intersection.

That’s Koe’s point when he says you are the niche.

Most people spend their careers searching for an external niche to fit into (I’ve been guilty of that too). But your specific combination of experience and context it’s already a niche. You're standing in it.

A couple of days ago on a Saturday night, I was building my football RAG system (a project that’s teaching a lot on engineering foundations), and I lost track of time. I ended working from 9pm to 3am. It felt like craft and I think that’s what happens when you stop fighting your weird path and start building from it.

Mapping Your Own Stack

Some self-awareness is necessary here to understand how to mix your skills, (might sound like a self help book advice but trust me, you need to sit with yourself on this). I want to give you something I didn’t have at the beginning of my journey: a starting point.

Start by asking yourself these three questions. Note them down on a notepad, Notion, or somewhere else you can easily go back to:

• Is there something you know that most people in your field don’t? Something specific rather than general. What context do you bring that wasn’t in the job description?

• Where do your interests converge? The things you read on weekends, the problems you notice in one domain that nobody there is thinking about because they don’t have your other domain to compare it to.

• What can you build to work at the intersection of it? Think in projects, tools, analyses, newsletters. Something concrete that only you could have built from exactly where you’re standing.

• An extra: How can you leverage the use of AI to boost your skills?

I would start from there.

May The Weird Ones Prevail

We now know the system was not designed for this but AI is compressing the value of standard skills faster than most people are ready for.

What it can’t compress is taste, context, and the specific lens you’ve built from an unusual combination of experiences.

The people who do well in what’s coming won’t be the deepest specialists or the broadest generalists.

They’ll be the ones who can see across domains, connect things that shouldn’t logically connect, and build things that require a very specific, very strange set of keys to unlock.

Your weirdness is your moat.

This newsletter is about building AI systems, using football analytics as the sandbox. But every now and then it's worth stepping back to talk about why we build what we build, and how we decide what's worth building next.

Till next time,

Ricardo.

Share The Underdoc

In previous articles we’ve covered:

• Engineering the data pipeline: RAG is a Data Engineering Heavy Problem: Building a Football ETL for AI

• Evaluating the prompt and system outputs with EDD: From Vibes to Metrics: How I Evaluated A Football RAG System

Today I’ll be talking about the challenges of designing a user interface to interact with in an effort to make it as comfortable as possible.

This marks the end of these series on how I’ve built this MVP, which I’m scaling into an automated data platform that generates football post match reports and can run on your own computer.

You can find the entire code on the GitHub repository and the new releases: Football RAG Intelligence

What’s Gradio and Why I Chose it

If you haven’t heard of it already, Gradio is an open-source Python framework that allows developers to quickly build and deploy interactive web interfaces for ML models and data pipelines.

In the context of the Football RAG platform, I used it as a lightweight presentation layer to connect the user to all the backend logic (pipelines, LLM orchestration).

At this stage of development, the goal was to keep it simple. I chose Gradio over Streamlit for these reasons:

• Avoid rerunning the script each time there’s retrieval.

• API capabilities: Gradio automatically generates a REST API for your interface, making it easier to treat the UI as modular rather than a monolith.

• LLM Integration: Gradio’s ChatInterface component is purpose-built for the conversational nature of RAG, allowing me to focus on prompt optimization rather than building widgets.

• Fast iteration: working with 108 matches, it allowed for a full database rebuild and UI refresh in seconds.

Although in the future I could move to another frontend, this was enough to validate the idea.

HuggingFace Spaces Deployment

The simplest way to host an AI/ML demo without becoming a full-time DevOps engineer was a using HuggingFace Spaces.

It’s basically a managed hosting service where you can drop your Gradio app.

Some characteristics:

• GitHub-Style Deployment: You push your code to a repository, and Spaces automatically builds the container and serves the app to a public URL.

• Seamless Integration: Since it’s natively built for the ecosystem, it handles the Gradio environment and Python dependencies out of the box.

• Free Tier: It provided the perfect “sandbox” for the MVP, giving me a URL to share it publicly using generous CPU compute.

A Reality Check: MVP vs. Production

Grado helped to build the UI easily, but making it "industrial-grade" meant facing a few technical issues.

1. Deployment Binary Bug

I hit a snag where the specific binaries for my vector store behaved differently on Mac vs. the Linux environment in Spaces.

Standardizing with uv and clean lockfiles was the way out.

2. The $0 Intent Routing Trick:

Saving tokens was necessary from the beginning, so I built a Smart Router:

• Questions like “How did this team play?” or “Why did this team had X metric during that match?” → It goes to the LLM for tactical analysis.

• Visualization commands like “Show passing network plot from the match” → These bypass the LLM and trigger my Matplotlib visualizers at zero cost.

The "Airlock" ETL

Talked about this in previous matches but decoupling the ingestion from the UI helped me caught data bugs like a "Home/Away swap" that was messing up team stats before they reached the user.

Here’s how the frontend looks for now in HF Spaces:

Lesson Learned

The biggest takeaway from building this GUI and deploying the MVP is:

90% is Data Engineering, and the AI part is just the tip.

You need to provide a well crafted plumbing so the system does meaningful retrieval and the user, through a comfortable UI can obtain valuable information.

Precalculating everything (compute 38 tactical metrics during ETL) helps for instant retrieval.

Validate with data contracts before moving the data forward (Pydantic)

Keeping it simple for the GUI, for quick adoption.

What’s Next?

Now that the demo is live and the concept is validated, I’m scaling from 108 matches to 1,000+.

I’ll be documenting that journey. In the next series we’re talking about moving to a proper infrastructure:

• MinIO for object storage (S3-compatible, runs locally)

• Dagster orchestrating the pipelines

• dbt + DuckDB handling the analytical workloads.

• Everything on-premise with Docker services

• Modal free tier for serverless LLM inference on cloud GPUs.

The goal:

An automated data platform that generates post-match tactical reports for any match, running entirely on your own machine, with cheap cloud costs for the heavy lifting.

If you’ve been following along, you now have a blueprint for building a working RAG system with documented engineering decisions.

The next series starts soon.

If there’s a specific part of the architecture you want me to dig deeper into, please let me know in the comments.

Check out the code: Football RAG Intelligence GitHub Repository

Try the live demo: HuggingFace Spaces

Thanks for reading till the end.

See you next time,

Ricardo.

I would write a massive prompt, ask the LLM to analyze a match, read the response, and think, “good enough.”

This approach is not bad at the beginning, but if you want a successful system that generates trustworthy outputs, they should be evaluated from beginning to end both subjectively and objectively.

Moving to EDD

After reviewing different approaches, I adopted the Evaluation Driven Development (EDD), a methodology discussed by in article Escaping POC Purgatory: Evaluation-Driven Development for AI Systems.

Where the core philosophy is:

You cannot optimize what you cannot measure

So instead of treating evaluation as a final step, you build the test harness before you finish building the system.

I’ll walk you through thow I used EDD to engineer my prompts in three steps:

1. The Metrics: Why I chose Retrieval Accuracy, Faithfulness, and Tactical Insights rules.

2. The Optimization Loop: Moving from Prompt V1.0 (1800 tokens, slower, and a bit more expensive) to Prompt V3.5 (360 tokens, more precise to what I wanted).

3. The Results: Critical bugs caught at the beginning.

Golden Answers

In my previous article, I broke down how I built the Golden Dataset (matches_gold.json) since it’s crucial to understand why that dataset exists in the context of evaluation.

In standard software engineering, you write unit tests based on expected inputs and outputs, that’s the traditional software development lifecycle (SDLC).

When working with LLMs, defining “expected output” is harder because these models are non-deterministic, meaning you can give an LLM the same input and get two different outputs.

This leads to a more dynamic process through the entire GenAI application lifecycle. More iterations during your development and your way to production.

In order to battle with that non-deterministic nature, I wanted to grade the system responses. See if they matched with the answers I was expecting.

That meant combining football knowledge with technology capabilities and creating a “Source of Truth” that I trusted 100% → The Golden Dataset. A ground truth that acts as the teacher’s answer key, containing the exact stats (validated by Pydantic before) for every match.

Without this, I would be just guessing and depending entirely on the LLM’s capacity.

Now that I had the answer key in hand, I needed a way to automate the grading of the system.

The Judge: A 3-Metric Framework

Since human evaluation doesn't go that far as LLM can, I adopted the "LLM-as-a-Judge" methodology. In the book AI Engineering, Chip Huyen identifies the AI judge as:

A critical evaluation method, noting its specific utility in measuring complex metrics such as the "factual consistency between the response and the whole context”. (Huyen, 2024).

By implementing this, we can automate the evaluation without only relying on manual review.

The evaluation harness uses the LLM-as-a-Judge pattern:

1. The Analyst: Claude 3 Haiku (Fast, Cheap ~$0.0001 per query) generates the report. It's optimized for speed but since speed comes with trade-offs, Haiku can occasionally miss nuances or hallucinate stats.

2. The Judge: Claude 3.5 Sonnet (Stronger, Slower) grades the report. It catches these issues. It's slower and more expensive, but it only runs during evaluation, not intended for production. This means I can test rigorously without blowing my API budget.

Why Anthropic's models? Honestly, I had a remaining balance there from previous experiments and wanted to blow them away. But I could have used Gemini, OpenAI, whatever. You can use any other "bigger" model for this, even a local one with Ollama.

The 3-Metric System

However, I needed to know why a report failed. If it was hallucination or did it ignore the charts or any other relevant part?

The evaluation framework measures three dimensions:

1. Retrieval Accuracy (30%) - Did we find the right match?

Early on, I noticed reports would analyze the wrong match or mix up team stats. The system would retrieve “Ajax vs. Heracles” when I asked about “Ajax vs. Telstar.” (I’ll show you an error like this at the end).

This metric uses metadata filtering to verify the system found the exact match requested. It’s deterministic in a way so either it got the right match (100%) or it didn’t (0%).

Evaluation Result: 100% ✅

2. Faithfulness (30%) - Are the numbers accurate?

This is the guardrail preventing “hallucinations” where the model invents stats.

Early reports claiming “Ajax had 73% possession” when the actual number was 64.3%. Or “Player #14 made 28 passes” when it was 23.

The judge uses regex to extract every number from the report, then cross-checks it against matches_gold.json. Any number not in the source data = hallucination = failed faithfulness check.

Evaluation Result: 99.4% ✅

3. Tactical Insight (40%) - Is the analysis tactically sound?

I broke it into four measurable components:

• Specificity (10%): Does it cite concrete evidence? Counts specific references like player IDs, exact percentages, tactical metrics.

• Visual Grounding (10%): Does it reference what’s actually in the visualizations? Reports should mention “The Passing Network shows Player #14 as the hub” and not just a generic commentary. This forces the system to tie text to visuals.

• Terminology (10%): Does it use correct football language? The judge verifies terms against tactical_rules.py (e.g., “Field Tilt,” “Progressive Passes,” “PPDA,” “Verticality”) to avoid generic concepts

• Relevance (10%): Does it answer the user’s actual question? If someone asks about pressing strategy, then the response shouldn’t be about passing accuracy.

A report can be factually accurate (Faithfulness = 100%) but tactically shallow (Tactical Insight = 40%).

Evaluation Result: 95.0% ✅

Codifying "Vibes" into Rules

This question was also bugging me from the start: “How do you mathematically prove a report is tactically sound?”

Since what I’m trying to do there is translating stats and tactical dashboards from post match analyses into explainable text, I couldn't let the LLM guess what the stats meant.

I had to define a way of teaching the system the coordinates used in the WhoScored and Fotmob data, how they are calculated, and displayed in the field visualizations with the mplsoccer library.

So, I built a file called tactical_rules.py that reverse-engineers the visualizations into text logic.

For example, I defined Verticality thresholds explicitly:

def interpret_verticality(verticality: float) -> str:
    if verticality > 50:
        return "direct vertical passing - quick transitions"
    elif verticality > 35:
        return "balanced build-up"
    else:
        return "patient possession-based build-up"

• Scenario: Ajax has a verticality of 72%.

• Bad Output: “Ajax played patiently.” (Judge fails this: 72% > 50%, so it must be direct).

• Good Output: “Ajax played a direct game with quick transitions.” (Judge passes this).

If Ajax has 72% verticality and the report says "patient build-up," is a vague description. The judge verifies those numbers exist and it checks if the tactical language matches what tactical_rules.py says those numbers should mean.

Think of `tactical_rules.py` as the rubric, and the judge as the grader applying it.

Of course, in the future this can change depending on what rules you want to add or readapt. This was a first attempt of contributing to the evaluation harness trustworthiness.

The Weighted Scoring Rubric

To bring it all together, I created a weighted rubric that scores every generated report on a 10-point scale.

Weighted Score = (Retrieval × 0.30) + (Faithfulness × 0.30) + (Specificity × 0.15) + (Visual Grounding × 0.15) + (Terminology × 0.15) + (Relevance × 0.15) + (Conciseness × 0.10) 

A report must score ≥ 7.0/10 to be considered production-ready.

Scoring Examples: A Good Analysis But Wrong Retrieval

Here's where evaluation gets interesting. Sometimes the system produces excellent tactical analysis but for the wrong match.

What I asked: “Analyse Feyenoord vs Ajax match”

What the system retrieved: Feyenoord vs FC Volendam

Report Generated:

That might be too small to read, here’s the answer:

“Match Overview: Feyenoord demonstrated superior territorial dominance with a significantly higher average team position (62.4m vs 33.9m). The home team’s lower PPDA of 9.2 indicated more aggressive pressing compared to Volendam’s 15.36, suggesting consistent offensive pressure and a controlled possession strategy.

xG & Shot Efficiency: Feyenoord generated substantially higher shot volume and quality, with 37 shots generating 4.82 xG versus Volendam’s 9 shots and 0.83 xG. The significant xG differential suggests consistent high-quality chance creation, with each Feyenoord shot averaging approximately 0.13 xG compared to Volendam’s 0.09 xG per shot.

Pressing & Progression: Progressive passing metrics (156 vs 114) reinforced Feyenoord’s offensive control. High press events (5 vs 3) further illustrated their tactical superiority in disrupting opponent’s build-up play. The defensive line positioning (13.0m vs 8.6m) confirmed Feyenoord’s ability to maintain more advanced, proactive defensive structure.

Key Tactical Insight: Total pass volume (513 vs 429) combined with superior progressive passing demonstrates Feyenoord’s systematic approach to ball progression and territorial domination, directly translating into offensive efficiency and match control.“

Judge Evaluation:

• Retrieval: 0/10 (30%) - Wrong match retrieved (asked for Ajax, got Volendam)

• Faithfulness: 10/10 (30%) - All numbers verified for Feyenoord vs Volendam match

• Tactical Insight breakdown:

  • Specificity: 10/10 (15%) - Dense with exact metrics (62.4m, 9.2 PPDA, 4.82 xG, 156 progressive passes)

  • Visual Grounding: 9/10 (15%) - References positional data and tactical patterns

  • Terminology: 10/10 (15%) - Perfect use of PPDA, progressive passes, xG/shot, defensive line

  • Relevance: 10/10 (15%) - Excellent explanation of tactical dominance

Conciseness: 10/10 (10%) - 220 words, slightly over but acceptable

Weighted Final Score: 6.7/10 ❌ Failed

This is a dangerous type of failure in production systems.

The analysis is tactically excellent. If you only read the commentary, you’d think the system nailed it.

But it’s analyzing the wrong match.

This is likely a vector database issue, either the embedding similarity between “Feyenoord vs Ajax” and “Feyenoord vs FC Volendam” was high enough to return the wrong match, or the metadata filtering didn’t catch it.

This example shows why without the retrieval check, this report would score 9.5/10 on analysis quality alone. But it’s completely useless because it’s not what the user asked for.

A coach asking about the Ajax match would make wrong decisions based on Volendam data.

The fix: Tighter metadata filtering + explicit team name verification in the retrieval step. This is why matches_gold.json exists, in order to verify we’re analyzing the exact match that was requested.

What Good Evaluation Looks Like and Common Failure Modes

When all three dimensions work correctly (right match retrieved, accurate numbers, insightful analysis) the system produces reports that score high ≥9.0/10:

• Retrieval: 10/10 - Correct match identified

• Faithfulness: 10/10 - Every number verified against matches_gold.json

• Tactical Insight: 9-10/10 across all four checks (Specificity, Visual Grounding, Terminology, Relevance)

• Conciseness: 10/10 - Under 200 words (it’s ok if slightly longer)

The judge validates this automatically across all 108 matches in the dataset, catching failures that manual review would miss.

That away the system surfaced three failure patterns:

1. Wrong retrieval: Feyenoord vs Volendam example above → fix with stricter metadata filtering.

2. Shallow analysis: generic commentary with no specific metrics or insights → tweaking prompt to emphasize specificity.

3. Hallucinated stats → catch errors early in the data engineering pipeline.

Takeaways from this Section

• Turn subjective quality ("does this sound good?") into objective metrics (“does this score ≥7.0/10?“)

• Numerical verification + tactical interpretation allows the system to move from "vibes-based development" to evaluation-driven development.

• A system can sound smart while being completely wrong. This helps you catch errors in an automatic way since the beginning.

The Prompt Optimization Loop: From 1800 to 360 Tokens

With the evaluation harness in place, In parallel I could measure what was actually working.

I tested prompts in Anthropic Console before touching code.

Each iteration was scored against the 10-match eval dataset.

The evolution of iteration was like this:

V1.0: 1800 tokens, 5.0/5.0 quality → Everything explicit, verbose
V2.0: 800 tokens, 3.2/5.0 quality → Removed too much, quality tanked
V3.0: 400 tokens, 3.8/5.0 quality → Added structured sections back
V3.5: 360 tokens, 4.9/5.0 quality → Winner

You can check how I stablished this prompt testing workflow in the repo: ⚽ Football RAG Intelligence

The V3.5 version added explicit metric interpretations directly in the prompt like:

• “Lower PPDA = more aggressive pressing”

• “Higher position values = more advanced positioning”

Also adding the required section headers (Match Overview, xG & Shot Efficiency, Pressing & Progression, Key Tactical Insight).

And limited constraints in the amount of text generated: “180-220 words” instead of just “be concise.”

Result: 80% token reduction, dropping the cost per query from about $0.0018 to $0.0004.

Wrapping Up

When building GenAI systems you need to stick to a set of best practices. I considered this ones:

Evaluate first optimize later. I built the judge before iterating with prompts and before creating the vector database, cause I wanted every change measured and every response in consonance with the data I had.

Prompt-first design pays off. I designed Prompt V3.5 before building the ChromaDB schema, in an attempt to reduce retrieval-generation mismatch. The ideal scenario is to align the metadata with what the prompt needs.

Pre-calculate during ingestion. 38 tactical metrics were computed during ingestion, this helps to make fast rebuilds if the logic changes, a simpler RAG pipeline, fewer tokens, and stable prompts.

Building systems requires evaluation in every step of the journey.

Using Evaluation-Driven Development to cut tokens, go from vibes to data-driven analyses, and automated validation is what helps you get more a trustworthy working product.

Thanks for reading until the end, hope this helps you think of a way to evaluate your own systems.

If so, I’d love to know what other approaches are you considering when evaluating your RAG/LLM/Agentic apps.

See you next time,

Ricardo.

FBRef announced this week they removed all the Advanced Stats provided by Stats Perform by Opta.

Something that took the football analytics community by surprise.

FBRef was maybe one of the first (if not the first) source you could think of to grab data and build high-level analysis. Whether you were a student, analyst, scout, fan, whoever, their web was a must. They these great level of detail in their data that actually made you go down rabbit holes just by looking at the granularity of their metrics.

I used to recommend it as one of the go-to places to get in contact with advanced football data stats → How to Use FBRef for Football Scouting with Python?

This new situation far from being discouraging should make us all football lovers develop our creativity to make the most out of the available sources on the internet and explore other angles. For example: places like SkillCorner Open Tracking & Physical Data or Hudl StatsBomb Event & Physical Open Data are well known data providers where you can obtain top valuable data to run different analyses but with a “detective mind”.

What I mean by that is trying to look at the data as a vehicle to derive close-to-real-world analyses of how technical staff at clubs or pro teams use this data, and also how other fellow colleagues use it.

You can see it on social media. The majority of professionals and data practitioners are active on X or LinkedIn. Perfect places to take a look at how this data is being used and inspire yourself of those signals to craft your own dashboards or reports how they consult other data sources, how they collect, clean, model and even how they present that data.

Having different POVs around how others manage sports data helps you upskill in the data field way faster, which is ultimately what we actually want: be more competent data professionals independently where the data comes from.

I also wouldn’t rule out the use of synthetic data for practicing purposes. With the rise of Generative AI and LLMs, this has become a popular alternative within the industry due to lack of access to closed source data.

Spending some time on crafting comprehensive prompts, using few shot examples, with metrics and well-defined ranges, can help you mimic real-world data for your own practice.

The main thing to remember from this FBRef situation is that it is essential that we adapt by focusing on sharpening skills with what’s available, or by creating our own way of accessing to data sources.

This is a great signal to start creating now.

Till next time,

Ricardo.

RAG efficiency relies on great Data Engineering.

This refers to building a solid ETL process: Extract, Transform, Load — the process that prevents bad retrieval and bad generation, helping the system produce accurate responses.

So, I’ll go over the origin of the data, the sources I used, and the decisions I made to build the pipeline.

Where the Data Comes From

Starting from the main goal of the system: automatically provide concise post match reports using free updated post match data from a league gathered in one place. That way, you can save time hopping from one site to another and focus on what really matters: the analysis.

To build these tactical reports, I wanted to use match event data.

I reused and improved a version of the scrapers I built for my previous dashboard project. The goal was to move from scraping one match for a plot to indexing 108 matches for a system.

The main data sources are:

• WhoScored: with more than 1,300+ events per match: passes, tackles, coordinates.

• Fotmob: source for more specific match data like xG, numbers of shots and shot quality.

Fotmob provides the shot as a number, whereas WhoScored only provides if the action was a shot (True) or not (False). They complement each other.

The first wall I hit in this Data Engineering process was: Entity Resolution.

Source A calls the team “R. Madrid,” Source B calls them “Real Madrid.” I needed to map them, so the RAG system wouldn’t think they are different teams.

Extracting the Data

The scrapers development evolved from a one-off scraping to more matches. In this case I used just 108 matches from the first leg of the Eredivisie league.

Why the Dutch league? It’s a league I’m really curious about, they are one of the best breeding talent leagues in Europe and it is always worth it looking at their emergent talents.

The scrapers followed a simple rule: the Raw information is immutable. I save the JSON exactly as it comes from the web into a “Raw Zone” (local for now). That way if I find a bug in my tactical metrics calculation later, I don’t need to scrape the web again. I just re-run it from the raw files.

Quick summary of how it works:

Scraper WhoScored --(JSON)--> [ Raw Zone /whoscored/ ]
                                            |
 Scraper Fotmob   --(JSON)--> [ Raw Zone /fotmob/   ]
                                            |
                              [ scrape_all_matches.py ]
                                            |
                               ( Orchestration & Logs )

• The Workers: Specialized scrapers (one for WhoScored, another for Fotmob) designed with a single responsibility: extracting raw data.

• The Storage: A local directory that serves as my immutable “Raw Zone.”

• The Orchestrator: A central script (scrape_all_matches.py) that acts as the conductor, identifying missing matches and triggering the workers accordingly.

# scripts/scrape_all_matches.py (A Simplified version of the Orchestrator)

def run_pipeline():
    # 1. Check what we already have (Incremental mode)
    existing_matches = get_local_raw_manifest()

    # 2. Run WhoScored Scraper
    print("🚀 Starting WhoScored extraction...")
    ws_data = scrape_whoscored_season(mode="incremental", exclude=existing_matches)
    save_to_raw_zone(ws_data, source="whoscored")

    # 3. Run Fotmob Scraper
    print("🚀 Starting Fotmob extraction...")
    fm_data = scrape_fotmob_eredivisie(mode="incremental", exclude=existing_matches)
    save_to_raw_zone(fm_data, source="fotmob")

    print("✅ Raw Zone updated. Ready for ETL.")

Pydantic Contracts: the “Airlock” Pattern

To safely store data in a vector database, I needed a Contract. It is a strict, machine-checkable definition of what your data must look like before it’s allowed to move downstream.

In my case, it’s the schema for a “valid match” (required fields, types, constraints). If the scraped payload doesn’t satisfy that contract—missing xG, null team IDs, wrong types—the pipeline fails fast and the match never gets stored in the vector database.

Pydantic is in charge to validate that match, working as an airlock. It defines exactly what a “valid match” looks like.

Pre-calculating +30 Metrics

Dumping 1,300 raw events into the LLM didn’t seem realistic nor cost-effective. It’s expensive in matters of tokens and slow latency, which ultimately confuses the model, making it prone to give back unreliable information.

Instead of the ETL just moving data, it acts as a Factory.

It processes raw, messy coordinates and distills them into Tactical Insights (or Actionable Metrics):

• Instead of 500 raw pass coordinates → Verticality: 0.65

• Instead of 20 defensive events → PPDA: 8.2

The logic is simple: I don’t want the LLM to waste its “brainpower” (and tokens) trying to do math on thousands of x,y points. I want it to focus on what it does best: reasoning over tactical concepts.

By pre-calculating these metrics, I’m handing the AI a “scout’s report” instead of raw notes.

This is the transformation step of the pipeline: where raw events, shots, and mappings are stitched together into a single, validated match profile.

# scripts/process_raw_data.py (simplified version of the transformation logic)

def process_matches():
    # 1. Load the "Glue": Match mappings and raw data
    mappings = load_json("data/match_mapping.json")
    
    for ws_id, mapping in mappings.items():
        # 2. Resolve IDs (The fix for the Home/Away swap bug)
        home_id, away_id = resolve_team_ids(mapping)
        
        # 3. Stitch data sources (WhoScored Events + Fotmob Shots)
        ws_data = load_json(f"raw/whoscored_matches/match_{ws_id}.json")
        fotmob_match = find_match_in_bulk(mapping['fotmob_id'])
        
        # 4. The "Factory": Distill 1,300+ events into 38 tactical metrics
        events_df = pd.DataFrame(ws_data['events'])
        raw_metrics = calculate_all_metrics(events_df, fotmob_match['shots'], home_id, away_id)
        
        # 5. The "Airlock": Validate everything with Pydantic
        match_profile = MatchProfile(
            metadata=MatchMetadata(match_id=ws_id, date=mapping['match_date']),
            home_team=TeamMatchStats(**map_metrics(raw_metrics, "home")),
            away_team=TeamMatchStats(**map_metrics(raw_metrics, "away")),
            home_score=calculate_score(fotmob_match['shots'], "home"),
            away_score=calculate_score(fotmob_match['shots'], "away")
        )
        
        # 6. Save to the Golden Dataset
        save_to_gold(match_profile)

In essence, it resolves team IDs, stitches two different data sources together, calculates tactical metrics, and finally passes everything through a Pydantic "airlock" to ensure the Golden Dataset is 100% reliable.

An issue I found at the beginning

Early in development, the system told me Zwolle dominated a game they actually lost 2-8 (https://www.sofascore.com/football/match/heracles-almelo-pec-zwolle/wjbsCjb)

The LLM was just reading the data I gave it and I thought it was hallucinating.

The ETL had a bug in the team ID mapping that swapped the Home and Away stats.

Something that made me understand that paying attention to the DE part is central.

The Golden Dataset & The Sniper Approach

The final output of the ETL is the matches_gold.json.

This file became the center of the system. Everything downstream: reports, evaluations, and rebuilds started here. That’s why I treat it as the Source of Truth, and not the vector database.

Once the data is in this shape, the next step was loading it into ChromaDB.

To do that, I used a simple 2-chunk architecture when indexing each match:

• Summary Chunk: Contains match metadata: teams, date, score, and competition. Its job is to make sure the system finds the right match.

• Tactical Chunk: Contains the pre-calculated metrics. This is what the LLM actually reads to generate the analysis.

I ended up calling this setup the “Sniper Approach.”

I wanted to avoid relying entirely on semantic search: embed the query, retrieve the closest vectors, and hope the correct match is somewhere near the top. That works, but it always felt a bit loose for this use case.

When a user asks about PSV vs Ajax, I don’t want the system to “kind of” get it right. I already know the exact match I’m looking for, so I use metadata filtering to narrow it down first. Once the match is identified, only then does the LLM see the tactical data.

This way, the vector database serves exactly what I ask for and retrieval becomes deterministic.

[SNIPPET F: scripts/rebuild_chromadb.py – Chunking & Metadata logic]

# scripts/rebuild_chromadb.py 

for match in matches:
    match_id = match['metadata']['match_id']
    home = match['home_team']
    away = match['away_team']
    
    # Chunk 1: Summary (for metadata filtering)
    summary_text = (
        f"{home['team_name']} vs {away['team_name']} "
        f"({match['home_score']}-{match['away_score']}). "
        f"Possession: {home['possession']}% vs {away['possession']}%. "
        f"xG: {home['xg']} vs {away['xg']}."
    )
    
    summary_meta = {
        "match_id": match_id,
        "chunk_type": "summary",
        "home_team": home['team_name'],
        "away_team": away['team_name'],
        "match_date": match['metadata']['match_date']
    }
    
    collection.add(
        documents=[summary_text],
        metadatas=[summary_meta],
        ids=[f"{match_id}_summary"]
    )
    
    # Chunk 2: Tactical Metrics (for LLM generation)
    tactical_meta = {
        "match_id": match_id,
        "chunk_type": "tactical_metrics",
        "home_ppda": home['ppda'],
        "home_verticality": home['verticality'],
        "home_xg": home['xg'],
        "away_ppda": away['ppda'],
        "away_verticality": away['verticality'],
        "away_xg": away['xg'],
        # ... (38 total metrics)
    }
    
    collection.add(
        documents=[f"Tactical metrics for {home['team_name']} vs {away['team_name']}"],
        metadatas=[tactical_meta],
        ids=[f"{match_id}_tactical_metrics"]
    )

Two chunks per match: one for finding it (metadata filtering), one for analyzing it (LLM generation). The vector database doesn't guess; it serves exactly what I ask for.

I also added a sanitize_metadata() function to prevent ChromaDB from crashing on None values. It saved me from silent failures during indexing.

Link to the repo so you can check all the code: Football RAG Intelligence

Bottom Line

At the end of the day, a football analyst needs to trust the numbers, and a developer needs to trust the pipeline.

By building a strict ETL with Pydantic and a Golden Dataset, I managed to reduce the “hallucination surface” before even writing a prompt. The idea was to make the data so clear that the AI didn’t have to guess.

What’s next?

Now that we’ve tackled the data preparation, we need to talk to it.

In the next article, I’ll dive into the Prompt Engineering side of the project.

I’ll show you why I spent a good amount of time designing a Tailored Prompt and how that decision helped me cut token costs by 80%.

If you're interested in the intersection of Data Engineering and Generative AI, subscribe to follow this series.

See you then,

Ricardo.

I won't take much of your time, but before it ends, I wanted to thank you for reading this newsletter during the year.

I wrote about football, AI, RAG systems, agentic IDEs, and a bunch of technical experiments, things I hope helped you apply them to your own work.

So thank you. It really means a lot that you share this space with me.

To wrap it up, I want to share one thing that changed my thinking this year and will serve as my compass for years to come.

It's about how I want to perform my work and develop my career.

I'm sure it can help you start 2026 with a different perspective.

The Book That Made Me Rethink About Purpose

I recently finished reading So Good They Can’t Ignore You by Cal Newport, a book recommended by on a brilliant X/Twitter thread about books every developer should read.

This was the book that really hit me this year.

Why? Cause all my life I’ve thought the popular advice “find your passion and the rest will follow” was real. Too many decisions made using this mantra.

Then, to my surprise, Newport argues the opposite in the chapters of this book.

Telling stories of people from each path and ultimately showing who developed more successful and satisfying lives.

Here’s the deal.

Passion Mindset vs. Craft Mindset

The central subject of the book is these two concepts.

The passion mindset is about finding work that matches a passion you supposedly already have. It’s asking “what can the world offer me?” and assuming passion comes first, then the work follows.

The craft mindset flips this. It’s about getting really good at something rare and valuable. Asking “what can I offer?” instead.

The key difference: passion comes after you develop mastery, not before.

This made me rethink how I approach my own career in tech. Since I pivoted from medicine I’ve been unknowingly following this path, trying to get better at managing data, programming and ultimately building AI systems.

I was lucky enough to apply it to football (which I love) and healthcare, my background.

But the true passion came from pushing through the work and wanting to get better each day.

How Chalamet Pursues Greatness Through Craft

In February, at the SAG Awards ceremony, Timothée Chalamet said something unusual: “I’m really in pursuit of greatness. I want to be one of the greats.”

I felt that quote was really honest because he was actually talking about becoming great through the work.

He mentioned being inspired by Daniel Day-Lewis, Marlon Brando, and Viola Davis, and spending 5.5 years of his life preparing for A Complete Unknown where he portrayed Bob Dylan.

That’s the craft mindset in action.

Greatness is something you build deliberately over years of committed work.

Career Capital: The Currency to Buy What You Want in Your Career

Career capital is the set of rare and valuable skills you’ve accumulated through the years.

Newport uses this economic metaphor deliberately: great jobs (those offering creativity, impact, and control) are scarce and valuable.

To get one, you need to offer something scarce and valuable in return.

Most people want autonomy, interesting projects, and meaningful work from day one (happened to me). But you need currency to buy those things. First you accumulate the capital, then you invest it to negotiate the career you actually want.

You build it by developing skills that are rare and valuable, ones others don’t have and the market needs.

This changed how I choose what to learn.

Before, I chased new technologies that sounded interesting but now I ask myself if it gives me the career capital I want to build. Or is it rare and valuable, more than a commodity.

For example: Python is valuable but not rare. Instead, using Python to create AI systems applied to a niche domain like football analytics or healthcare is a rare combination.

This lens shows if you're building capital or just chasing noise.

What This Means for 2026

For 2026, my plan is simple: keep accumulating capital, investing it by building projects that combine rare and valuable skills, sharing what I learn here, and iterating.

If there’s something you can take from this for your own 2026, let it be this:

Build mastery first. The passion will come on its own.

Have a great year.

Till next time,

Ricardo.

Long time no see

First of all, Merry Christmas 🎄

It’s been a while since I last send a post, but life happens. So, here I am before finishing the year.

I wanted to end 2025 with the same vibe I want to start 2026: building projects and sharing knowledge.

Lately, I’ve been working on something that mixes my interest in AI and my passion for football. I’m really excited to go over its details and how the idea came up.

When I started analyzing football data and creating post match reports, I had a problem that I think a lot of coaches, analysts, and football data practitioners can relate to.

Gathering free match data from various sources to build reports is tedious.

This makes it difficult to create a summarized report from any match without hopping from one web to another.

Say if you want to analyze PSV’s pressing patterns from last week’s Eredivisie match day, you need to go to WhoScored, then Fotmob for shot and xG values, Understat, BeSoccer etc, and obtaining the data manually.

That’s why I wanted to built this system:

One where I could ask “What was PSV’s pressing strategy against Ajax?” in natural language and get an instant tactical report—with real stats, context, and visualizations from multiple free data sources.

I want to show you how it works.

What I Built

This is the live demo hosted at Hugging Face Spaces: Football RAG Intelligence - Live Demo

It’s a RAG (Retrieval-Augmented Generation) system that ingests Eredivisie match data from WhoScored and Fotmob, stores it in a vector database, and generates tactical analysis grounded in actual statistics.

You ask questions in natural language. It retrieves the match data and then generates comprehensive rundowns thanks to the LLM text-generation capability.

Reduced hallucinations and instant visualizations at low cost.

Here’s what it handles:

• 108 Eredivisie matches indexed with event-level data (~1355 events per match).

• Dual-mode interface: Ask a question and get tactical commentary. Request a visualization and get instant plots.

• Multi-provider support: Works with Anthropic’s Claude, OpenAI, and Gemini models.

• $0.50 total cost to test everything.

The visualizations I used were already created using a prior code. I went through it in an earlier article, you can go check it here: Building Your Own Football Analytics Dashboards With Free Data

Why This Matters

Your match data lives in one place so you don’t have to hop around internet to get it. Ask questions in plain language, get answers backed by recent stats. Need a passing network or shot map? Type “show passing network for Feyenoord”.

Who can leverage this?

Coaches, scouts, and analysts at small/medium/big clubs, football fans, developers. Pretty much anyone involved in analyzing football with data.

How It Works

Here’s the high-level flow:

User Query → Router (classify intent)
             ↓
    ┌────────┴─────────┐
    ↓                  ↓
LLM Analysis      Visualization
(Tailored Prompt)     (Keyword-based)
    ↓                  ↓
ChromaDB          pre-built matplotlib
retrieval         plots
    ↓                  ↓
Grounded          $0 cost
Commentary        images

The system does five things:

1. Routes your query: Questions like “What was PSV’s pressing strategy?” go to the LLM pipeline. Commands like “Show shot map” trigger visualizations.

2. Retrieves match data: ChromaDB finds the exact match using team names and metadata filtering instead of semantic search guessing.

3. Generates analysis: The LLM gets the actual stats and writes tactical commentary using a carefully engineered prompt (which I’ll cover in Article 2).

4. Validates output: Every statistic cited in the response is checked against the golden dataset. If it’s not there, it doesn’t make it into the output.

5. Delivers instantly: Text analysis in 6-7 seconds or visualizations in under a second because they bypass the LLM entirely.

Hints on the key architectural decisions I made (I’ll dig deeper into these in future articles):

• Prompt-first design: I designed the Tailored Prompt before building the ChromaDB schema. I wanted to tackle the mismatch between what the LLM expected and what the database provided.

• 2-chunk architecture: Each match gets two documents in ChromaDB. One summary chunk (team names, score, date) for filtering. One tactical metrics chunk (38 pre-calculated features) for LLM generation.

• Pre-calculated metrics: All tactical metrics are computed during data ingestion. PPDA, progressive passes, field tilt, compactness, etc. The RAG pipeline doesn’t calculate anything, it just retrieves and interprets.

• Smart routing: Text questions go to the LLM. Visualization commands go straight to the plots generation with matplotlib. This was to reduce the cost for every viz request.

• Golden Dataset ETL: Raw data from WhoScored and Fotmob flows through a Pydantic validation pipeline. Only clean, validated data makes it into ChromaDB. This “airlock” pattern caught bugs before they could cause hallucinations.

The Results

In initial evaluation testing on 10 diverse matches, these were the obtained results:

Retrieval Accuracy: 100%

Every query found the correct match data. Metadata filtering (team names, dates) acts like a “sniper” approach versus pure semantic search, which only hit 70% accuracy.

Faithfulness: 99.4%

Generated analysis matched source statistics. Every number cited in the commentary exists in the golden dataset. One test case had a minor discrepancy due to rounding, hence 99.4% instead of 100%.

Tactical Insight: 95.0%

The commentary provided actual both tactical depth and data reporting.

I evaluated the system using an LLM-as-a-Judge methodology with specific rubrics and reference answers.

The cost: ~$0.0001 per query with Claude Haiku 4.5.

The system still occasionally generates erroneous team names or statistics, which is why continuous evaluation is important. I'll cover my evaluation approach in Article 4.

What’s Next: The Series Roadmap

This article gives a big picture of what I built and how it works.

In next articles I’ll go deeper into specific aspects of the journey:

• Crafting a Football ETL Pipeline with free event data for RAG: What sources I used, why, how I managed all this data to make it useful for the system, and what is RAG.

• Why Prompt Engineering Came before the vector database and how it helped optimize costs: Designing the Tailored Prompt to avoid retrieval-generation mismatch. I’ll show the many prompt iterations from V1.0 (1800 tokens, slow, vague) to V3.5 (360 tokens, fast, precise).

• Evaluating the system with the Evaluation Driven Development approach: The LLM-as-a-Judge methodology. How I built the three-metric evaluation framework. Creating test datasets that actually matter. Debugging false failures.

• Deploying the App: Hosting an initial version of the app in HF Spaces, a well-known place for AI/ML developers to deploy applications.

• Next steps to Production Scaling: Current MVP limitations. Production stack design following data, software, and AI engineering best practices (CID/CD, pipeline orchestration, LLM cloud hosting and inference, automated evaluations) to go from proof-of-concept to a system that obtains weekly matches and generates fully automated reports on demand hosted on premise.

Try It Yourself

Demo Link: Football RAG Intelligence

Pick a provider (Claude, GPT, or Gemini), paste your API key, and start asking questions.

Example queries to try:

Text analysis:

• “What was PSV’s pressing strategy against Ajax?”

• “How did Heracles build up play against PEC Zwolle?”

• “Explain the tactics in Feyenoord vs Ajax”

Visualizations:

• “Show dashboard for Heracles vs PEC Zwolle”

• “Show passing network for Feyenoord”

• “Show shot map for AZ vs Utrecht”

The code is fully open source: GitHub Repository

Clone it.

Run it locally.

Modify it for your own league or data sources.

I included all the documented architecture decisions to showcase the entire process and next steps.

Bottom Line

The goal of this project is to show that:

A RAG system and an LLM of your choice fed by your own clean data, clear architecture, and careful prompt engineering, helps you solve a real problem like automated post match analysis.

Leveraging AI to create systems or tools is one of the things that excites me the most. There's something incredible about going from idea to working system so fast.

AI helps blur that line between "I want to build this" and "I built this."

I would love to hear some thoughts on this on how useful is this to you.

Til next time,

Ricardo.

For example, you want ChatGPT to analyze files in your S3 bucket, or to help you manage your databases, but you end up copying and pasting data manually.

Every new data source requires its own custom connector same as every AI application needs its own integration code.

There’s a solution which has risen in popularity as new way of how we work with AI and data integration: Model Context Protocol (MCP).

What is an MCP and What is it Used For?

Model Context Protocol (MCP) is an open standard that allows AI models to securely connect to external data sources and tools through a unified interface.

MCP acts as an USB-C port for AI applications.

Just as USB-C provides a standardized way to connect any device to any peripheral without custom cables for each combination, MCP provides a standardized way to connect any AI model to any data source without writing custom integration code.

Before MCP, if you wanted Claude to access your MinIO storage, your GitHub repositories, or your database, you’d need three separate custom integrations. With MCP, you write one MCP server for each data source, and any MCP-compatible AI client (like Claude Desktop, Cursor, or your custom application) can instantly use all of them.

The key is the standardization. Instead of AI models needing to know the specific APIs of hundreds of different services, they just need to understand one protocol: MCP.

General Structure

MCP follows a client-server architecture that separates concerns in the following way:

MCP Host

The user-facing AI application (like Claude Desktop, Cursor, or your custom AI app) that coordinates everything.

MCP Client

Lives inside the host application and manages secure connections to MCP servers. Each server gets its own isolated client connection.

MCP Server

External programs that expose specific capabilities to AI models like connecting to databases, file systems, APIs, or any other data source.

The communication happens through JSON-RPC, a lightweight protocol that enables secure, two-way communication between clients and servers.

Three Core Primitives

MCP organizes functionality into three main types:

Resources: Read-only data that the AI can access (like files, database records, or API responses). We can call them “things the AI can read.”

Tools: Actions the AI can perform (like creating files, sending emails, or running queries). These are “functions the AI can call.”

Prompts: Reusable conversation templates or workflows that provide structured interactions.

The importance of this design is that the AI model doesn’t need to know anything about MinIO’s S3 API, your database schema, or any specific service. It just knows how to ask for resources, call tools, and use prompts through the MCP interface.

Practical Use Cases

MCP is useful in scenarios where you need AI models to work with real, live data:

Development Workflows: AI coding assistants that can read your codebase, access your Git history, and understand your project structure in real-time.

Data Analysis: Models that can query your databases, analyze your files, and generate insights from your actual business data.

Content Management: AI that can access your document repositories, and help you manage and organize content.

Business Intelligence: Models that connect to your analytics platforms, CRM systems, and help you make data-driven decisions.

Infrastructure Management: AI that can interact with your cloud storage, monitor your systems, and help with operational tasks.

The key advantage is real-time access. Instead of feeding static data to AI models, MCP enables dynamic, contextual interactions with live systems.

Practical Example: MinIO MCP Server

Now let’s see MCP in action with a real example: the MinIO MCP Server built in collaboration with my colleague Ginard, a brilliant senior software engineer (Rafa's GitHub and Rafa's LinkedIn) who came up with this idea of connecting your MinIO object storage with any host and client, even a local LLM.

What the MinIO MCP Server Does

The server exposes MinIO object storage operations as MCP tools, allowing any MCP-compatible AI client to manage buckets and objects through natural language.

Repository: https://github.com/Rafitis/minio-mcp-server

Key Features

7 MCP Tools organized by domain:

Bucket Operations:

• list_buckets: Get all available buckets with creation dates

• get_bucket_info: Detailed bucket metadata, policies, object count, and size

• create_bucket: Create new buckets with validation

• delete_bucket: Delete empty buckets or force delete with contents

Object Operations:

• list_objects: List objects with filtering and pagination

• get_object_info: Get detailed object metadata

• delete_object: Remove objects with version support

Architecture Highlights

Built on FastMCP: Uses the FastMCP framework for robust MCP protocol support

Async Architecture: Full async/await support for better performance

Comprehensive Validation: Input validation and error handling with proper HTTP status codes

Modular Design: Separated BucketTools and ObjectTools classes for clean organization

Quick Setup

The setup process demonstrates how streamlined MCP can be:

# Clone and install
git clone https://github.com/Rafitis/minio-mcp-server
cd minio-mcp-server
uv sync

# Configure your MinIO connection
cp .env.example .env
# Edit .env with your MinIO credentials

# Start the MCP server
uv run python src/minio_mcp/server.py

Claude Desktop Integration

To connect with Claude Desktop, you just add this configuration:

{
  “mcpServers”: {
    “minio-mcp-server”: {
      “command”: “/path/to/uv”,
      “args”: [
        “--directory”, “/path/to/minio-mcp-server”,
        “run”, “python”, “src/minio_mcp/server.py”
      ],
      “env”: {
        “MINIO_ENDPOINT”: “localhost:9000”,
        “MINIO_ACCESS_KEY”: “minioadmin”,
        “MINIO_SECRET_KEY”: “minioadmin”,
        “MINIO_SECURE”: “false”
      }
    }
  }
}

After restarting Claude Desktop, you can use natural language commands like:

• “List all buckets in MinIO”

• “Show me information about the ‘analytics-data’ bucket”

• “Delete ‘old-backup.zip’ from the ‘archives’ bucket”

The AI model automatically translates these requests into the appropriate MCP tool calls.

Connecting with Local LLMs and Optimal Prompts

One particular aspect of MCP is its model-agnostic design. You can use the same MCP server with cloud-based models like Claude or with local models like Mistral.

Local LLM Integration

The feature/llm-rest-api-integration branch of our repository shows how to connect MCP servers with local LLMs for maximum privacy and control.

Key Benefits of Local Integration:

• Privacy: Your data never leaves your infrastructure

• Cost Control: No per-token pricing from cloud providers

• Customization: Fine-tune models for your specific domain

• Offline Capability: Works without internet connectivity

Setting Up with Enterprise LLMs

Here’s how you can connect with your company’s enterprise LLM infrastructure:

Enterprise LLM Setup (OpenAI-compatible API):

import asyncio
from openai import OpenAI
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

class EnterpriseMinioMCPClient:
    def __init__(self):
        # Configure your enterprise LLM as an OpenAI-compatible client
        self.enterprise_llm = OpenAI(
            api_key=”your-enterprise-api-key”,  # Your company’s API key
            base_url=”https://llm-service.yourcompany.com/v1”  # Your company’s LLM endpoint
        )
        
    async def connect_to_minio_server(self, server_script_path: str):
        “”“Connect to MinIO MCP server”“”
        server_params = StdioServerParameters(
            command=”python”,
            args=[server_script_path],
            env=os.environ.copy()
        )
        
        # Establish connection to MCP server
        stdio_transport = await stdio_client(server_params)
        self.stdio, self.write = stdio_transport
        self.session = ClientSession(self.stdio, self.write)
        await self.session.initialize()
        
    async def process_query(self, query: str) -> str:
        “”“Process user query through Enterprise LLM + MinIO tools”“”
        
        # Get available MinIO tools
        tools_response = await self.session.list_tools()
        
        # Build tool descriptions for the LLM
        tools_info = “\n”.join([
            f”- {tool.name}: {tool.description}”
            for tool in tools_response.tools
        ])
        
        # Ask enterprise LLM which tool to use
        messages = [
            {
                “role”: “system”, 
                “content”: f”You are a MinIO assistant with these tools:\n{tools_info}\n”
                          “Respond with TOOL_CALL: {{\”tool_name\”: \”name\”, \”parameters\”: {{}}}} or answer directly.”
            },
            {”role”: “user”, “content”: query}
        ]
        
        response = self.enterprise_llm.chat.completions.create(
            model=”your-company-model”,  # Your enterprise model name
            messages=messages,
            max_tokens=500
        )
        
        llm_decision = response.choices[0].message.content.strip()
        
        # Execute tool if LLM decided to use one
        if llm_decision.startswith(”TOOL_CALL:”):
            tool_call = json.loads(llm_decision.replace(”TOOL_CALL:”, “”).strip())
            tool_result = await self.session.call_tool(
                tool_call[”tool_name”], 
                tool_call[”parameters”]
            )
            
            # Format result with enterprise LLM
            format_messages = [
                {
                    “role”: “system”, 
                    “content”: “Format this MinIO tool result into a natural language response.”
                },
                {
                    “role”: “user”, 
                    “content”: f”Query: {query}\nResult: {tool_result.content}”
                }
            ]
            
            final_response = self.enterprise_llm.chat.completions.create(
                model=”your-company-model”,
                messages=format_messages,
                max_tokens=1500
            )
            
            return final_response.choices[0].message.content
        
        return llm_decision

# Usage
client = EnterpriseMinioMCPClient()
await client.connect_to_minio_server(”src/minio_mcp/server.py”)
result = await client.process_query(”Show me all buckets and find the largest files in each one”)

Optimal Prompting Strategies

When working with MCP and local LLMs, these prompting techniques can help deliver a better result:

Be specific about the context:

“Using the MinIO tools, list all buckets and then show me 
the 5 largest objects in the ‘backups’ bucket”

Structure multi-step operations:

“First, check what buckets exist in MinIO. Then, for the 
bucket with the most objects, show me a summary of file 
types and total storage used.”

Provide clear constraints:

“List objects in the ‘logs’ bucket, but only show files 
from the last 7 days and limit results to 20 items”

Use domain-specific language:

“Check the health of our data lake by listing all buckets, 
showing their sizes, and identifying any empty buckets 
that might need cleanup”

The key: leveraging the AI’s natural language understanding while being precise about what data operations you want performed.

Performance Considerations

Local LLMs with MCP work best when you:

• Choose function or tool-calling capable models (Mistral, Llama 3.2, Qwen)

• Optimize context windows for your data size

• Use streaming responses for real-time feedback

• Implement proper error handling for network timeouts

Relevant Resources and Documentation

If you want to dive deeper into MCP, here are the essential resources:

Official Documentation: Model Context Protocol Specification - The complete technical specification

Anthropic’s MCP Guide: MCP Documentation - Practical guides and tutorials

MCP GitHub Organization: GitHub ModelContextProtocol - SDKs, examples, and community contributions

Awesome MCP Servers: A growing collection of community-built MCP servers for popular services

MCP.so: MCP.so - A web to “Find Awesome MCP Servers and Clients”

The ecosystem is expanding rapidly, with new MCP servers being built for everything from Slack and GitHub to specialized databases and custom APIs, even to write your own articles in Substack/Medium.

Key Takeaways

Model Context Protocol helps solving a big challenge in AI development: connecting models to real, live data in a standardized way, providing a universal interface that works across the entire ecosystem.

• For developers, it means more time building intelligent features using AI.

• For organizations, this means building AI applications that can actually work with the nuances of your data infrastructure.

• For the AI ecosystem, this means faster innovation as everyone builds on shared standards and sharing this servers as open source.

The goal of the MinIO MCP Server is to demonstrate how straightforward it can be to expose powerful data operations to AI models, by creating a bridge between your object store and an LLM with just a few hundred lines of code.

Whether you’re using cloud models like Claude or local models like Mistral, the same MCP server works seamlessly.

Besides improving models to make them smarter, the future of AI relies on making models seamlessly interact with the tools and data that power our workflows.

Check out the MinIO MCP Server repository and start building your own AI-data bridges today.

Til’ next time,

Ricardo.

We've all been there.

But fortunately there's a tool that can solve all of these problems in second.

uv is the Python package that's helps setting up and managing Python projects blazingly fast.

Subscribe now

What is uv?

Is a Python package manager built in Rust that promises to be "an extremely fast Python package installer and resolver."

uv was created by the team at Astral (the same people behind Ruff), and it's designed to replace not just pip, but also pipenv, poetry, pyenv, and conda in many use cases. It's a unified tool that handles:

• Package installation and resolution

• Virtual environment management

• Python version management

• Project scaffolding

Think of it as the Swiss Army knife for Python development, but one that actually works smoothly instead of having unstable tools.

Why uv Beats Everything Else

Speed

Where uv really shines. While pip can take minutes to install a complex set of dependencies, uv does it in seconds. We're talking about 10-100x faster installation times.

The first time you use it, you'll think something went wrong because it finished too quickly (happened to me).

Better Than pip

pip is the default, but it's also slow and doesn't handle dependency resolution as elegantly as it could.

uv uses a modern resolver that's both faster and more reliable. Plus, unlike pip, uv creates isolated environments by default.

Simpler Than Poetry

Poetry is great for dependency management, but it has its quirks. The configuration can be complex, and it sometimes feels like overkill for simple projects.

uv gives you the power of Poetry with the simplicity of pip.

More Reliable Than Conda

Conda has its place, it was the first package manager I used, especially for scientific computing and data science. But for pure Python projects, Conda environments can be heavy and sometimes unpredictable, they can easily break and sometimes trying to find the root of that error and fixing it is worse than creating a new environment from scratch.

uv keeps things lightweight and consistent.

What Makes uv Special: The Technical Bits

When you use uv, you get three key files that make your life easier:

1. pyproject.toml

This is your project's configuration file. It contains all your dependencies, project metadata, and build settings in a standardized format. Unlike requirements.txt, it separates development dependencies from production ones and includes version constraints clearly.

When you run uv pip install in a project with a pyproject.toml file, uv will:

1. Read Dependencies: Automatically detect and install the dependencies listed in the [project] and [project.optional-dependencies] sections.

2. Manage Environments: Create and sync a virtual environment with the exact dependencies specified in the uv.lock file (if present) or pyproject.toml.

3. Lock Dependencies: Generate a uv.lock file that pins the versions of all direct and transitive dependencies, ensuring fully reproducible builds.

This makes uv a powerful tool for managing complex projects without needing higher-level workflow managers.

2. uv.lock

It contains the exact versions of every package (and their dependencies) that your project uses. This means when someone else runs your project, they get the exact same package versions you used. Helps to avoid the "but it works on my machine" issues.

3. Automatic .venv Creation

uv automatically creates and manages virtual environments. No more remembering to activate environments or wondering which packages belong to which project.

uv streamlines virtual environment creation and management.

• Creating a virtual environment:

# Creates a .venv directory in the current folder
uv venv

This command automatically finds your system's Python and creates a virtual environment. To use a specific Python version, use the --python flag (e.g., uv venv --python 3.11).

• Activating the virtual environment:

Activation commands remain standard for your shell:

# For bash or zsh
source .venv/bin/activate

Once a virtual environment is active, uv commands like uv pip install will automatically use it.

The Modern Data Stack Connection

If you're working in data science or analytics (for example in football analytics), uv is particularly valuable regarding:

• Reproducible analyses: The lock file ensures your analysis can be reproduced exactly

• Faster iteration: Quick environment setup means less time waiting, more time analyzing

• Clean project structure: Clear separation between development and production dependencies

• Better collaboration: Team members can quickly get identical environments

Real Example: Setting Up My Football Analytics Project

Here’s an example on how to get started with my football analytics tutorials repository using uv.

Step 1: Install uv

# On macOS and Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# On Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

# Or via pip if you prefer
pip install uv

Step 2: Clone and Set Up the Project

git clone https://github.com/ricardoheredia/football-analytics-tutorials.git
cd football-analytics-tutorials

Now, instead of dealing with virtual environments, requirements files, and all that complexity, you just run:

uv sync

And uv will:

• Create a virtual environment automatically

• Install all dependencies from the lock file

• Ensure everything is isolated

• Do it all in seconds

Step 3: Start Working

uv run jupyter notebook

This command runs Jupyter in the project's environment without you having to manually activate it.

The Files That Make It Work

In the repo, you'll see these key files:

pyproject.toml contains our project configuration:

[project]
name = "football-analytics-tutorials"
dependencies = [
    "pandas>=2.0.0",
    "matplotlib>=3.7.0",
    "seaborn>=0.12.0",
    "jupyter>=1.0.0",
    # ... more dependencies
]

[project.optional-dependencies]
dev = [
    "pytest>=7.0.0",
    "black>=23.0.0",
]

uv.lock ensures everyone gets the exact same package versions (this file is quite long, but that's the point – it's comprehensive).

Why This Matters for Your Workflow

Using uv has changed how I approach new projects. Instead of spending minutes setting up environments and installing packages, I spend a few seconds. It helps removing the friction that often prevents us from starting new experiments or sharing our work.

Now, if I want to test a football analytics technique, I can:

1. Create a new project with uv init

2. Add dependencies with uv add pandas matplotlib

3. Start coding immediately

Getting Started Today

If you want to try this out with the example repository:

1. Install uv using one of the methods above

2. Clone the repo: git clone https://github.com/ricardoheredia/football-analytics-tutorials.git

3. Run uv sync

4. Start exploring: uv run jupyter notebook

Check the documentation from the website if you want to go deeper: uv documentation

You'll be up and running under a minute.

The Bottom Line

Python package management has been a pain point for too long.

uv solves the problem and makes the whole experience smooth. Fast installation, reliable dependency resolution, and automatic environment management mean less time fighting with tools.

For football analytics, data science, machine learning, or any Python project, uv is rapidly becoming the standard.

The combination of speed, reliability, and simplicity is hard to beat.

Let me know in the comments if you have already tried it before or what’s your preferred Python package manager.

Check out the tutorials repository and start building your own match analysis dashboards today

Till next time,

Ricardo.

When we think about writing we might believe that we have an inherent obligation of writing something ground breaking, a scientific article, an encyclopedia, etc.

But the reality is simpler.

Writing is a powerful tool that you can use pretty much for any purpose: writing short stories, listing things to remember them, or journaling, for example.

One of my favorite uses besides writing for this newsletter and sharing content online is: tracking my weekly knowledge.

A writing use case I’m sure will help you advance your learning progress, your projects, and your career in general.

Subscribe now

Using a Paper and Digital Notebook

Not so long ago, when I started in this world of technology, especially in data science, in my very first day of work, I was handed a laptop along with a small notebook and a pen. I immediately thought: “What’s the use of this? I’ll have everything inside my IDE, I don’t think I’ll forget anything.”

Boy, I was wrong.

At the same time I was advancing through projects and meetings, that notebook saved my life. It helped me:

• Organize my thoughts before presenting results in a meeting.

• Summarize key ideas from daily standup and sprints.

• Highlight key results and suggestions from my peers.

• Note down next steps.

• And one of my favorites: annotate new things to learn.

The importance of that notebook (and its content) grew more and more in my daily workflow.

Soon, I felt I needed a little “upgrade”, so I decided to expand my note taking to a digital format.

I chose Notion as my main notes storage. This helped me organized my notes in simple categories:

• Work Projects: meeting summaries, progress, next steps, deadlines, pain points.

• Personal Projects

• Learning (courses, new technologies, tools, etc.)

• Newsletter posts ideas

• Notes about books (technical and non-technical)

This way, every note was allocated in a specific category, helping me organize the mess I was accumulating in the physical notebook.

I then realized something: I was noting down things I learned and implemented during the week, but wasn’t actually reflecting about them nor documenting them with detail.

Here, documenting the process you go through when learning something new is important. It helps you be aware of what you’re putting into practice, the failed experiments, and the final success.

That’s why I created a specific category in my notes:

The Data Learning Diary

I fill this diary (a basic separate page in Notion) with the things I’ve learnt through the week.

What I do in 3 steps:

1. I see something I want to learn → note down the topic

2. Go over the topic and use the Feynman technique by writing a summary in simple words to see if I understood → note down that summary (depending on the case I later ask Claude or ChatGPT to help me understand difficult concepts).

3. If it’s relevant to the project, apply that knowledge immediately → document every single step in the way: the goal of the project, first implementation, failed experiments, what worked, what didn't.

I do this daily, so at the end of the week I have a list of all the things I've been going over.

This is an example of last week.

On Friday, I did the exercise of going over each of the things I learned from Monday to Friday. Checking what I documented and rereading it to recall it.

This took me about 20-30 minutes at the end of the day, and I felt it was a powerful way of fixing that knowledge.

Giving room for your mind to process everything it went through the week, may feel overwhelming at first, but it helps your brain settle the information you’ve been learning.

Creating your own note taking methodology can make wonders in the long term. It can make a compound effect and help you advance in areas you feel stuck, understand concepts better, apply them or discard them for others that might be more suitable for your project. It’s your choice.

Key Takeaways

• Whether you’re advancing on your data career or you’re more experienced, writing down and organizing your notes will help you retain the daily information better.

• Choose any tool you want, physical or digital, the important is to have one you feel comfortable with.

• Make the effort of writing things daily, just a few comprehensive lines of what you’ve learnt that day and put aside some minutes at the end of the week to go over it.

• Ask AI to help you ease the way, be it summarizing, understanding, or reframing ideas.

You’ll see the big picture at the end of the week and adjust your methodology for a next occasion.

Hope this idea is valuable to you, and if you have any suggestions on what other note-taking methods are helpful, feel free to comment!

See you next time,

Ricardo.

Writing code has changed in the last a couple of years. When ChatGPT appeared in the map, using a chat window and asking GPT to create something was a new way of interacting with AI.

This definitely changes the way that code and software is created.

According to the great Andrej Karpathy, who talked about this at the AI Startup School lecture in San Francisco called “Software Is Changing (Again)”1, by comparing an old vs. new paradigm on creating software:

“We now spend more time thinking, planning, and reviewing code generated by LLMs than memorizing syntax and programming from scratch”.

With time, LLMs market has risen, different AI labs have developed their own open and closed source ones. New tools have also been created to help developers improve their workflows by leveraging these models, practically embedding them inside development environments, which are known as: agentic IDEs.

What are Agentic IDEs?

They are development environments enhanced with AI that assists with code generation. They are also compared to Agentic CLIs (command line interfaces) like Claude Code and Gemini CLI, which can be used from both the terminal and as a chat window.

These tools offer capabilities like:

• A variety of LLMs from different providers to use.

• Capacity to index your codebase and libraries documentation

• Model Context Protocols (MCP) → open standard that connects AI models to external tools, execute actions, and read data sources, through structured interfaces with LLMs.

• Ability to edit files autonomously, providing tab completions and code suggestions.

• Help in projects from planning phase to execution phase, leveraging the use of contextual documentation to understand the nuances of the project.

• Adaptation to your workflows to help amplify your development in your own domain knowledge.

What are their main challenges?

For the last months I’ve been using Cursor and a couple weeks ago I started using Trae, one that looks like an affordable competitor.

These tools are intuitive at first and you can get proficient at using them them pretty quickly, especially if you come from VSCode.

But like Uncle Ben says:

“With great power comes great responsibility" 🕷️

Things can go sideways pretty fast if you are not on the steering wheel.

Some of the main challenges I’ve faced:

• Understanding how the different LLMs work, and which one excels at which task.

• Rate limit: tracking how much of your monthly quota are you consuming. This helps you understand how to prompt better to achieve results in less iterations.

• Testing different prompt formats, to see what helps to achieve better results with each model.

• Knowing how to leverage the context window of each model.

• Keeping the leash tight in Agent mode, trying not to let it go rogue and make extreme codebase changes without supervision.

• Integrating MCPs using them as extra tools that improves my workflow. Especially the ones that are suitable for my project.

You’'ll see how to tackle these challenges with a framework I used and how I applied it in two football-related use cases.

How can you leverage them for Football Analytics?

One of the things I like the most about AI is that it is here to augment our capabilities and help us create opportunities within our domain knowledge.

In this case, I followed a general framework that you can replicate for any project, it works for both Cursor and Trae.

1. As soon as you open your project, create three files:

   1. rules.mdc: this one sets the context of the rules2 that the agent must follow in each project when generating code. In any conversation, it will consider anything you want your project to have: role the AI must adopt, your Python version, the libraries you use, your coding style, design pattern, etc.

   2. project_context.md: you’ll entire context, this is the document where you set your project’s main idea goal, characteristics of the project, steps to achieve it, anything. It’s an opportunity to describe clearly and concisely what you’re working on.

   3. tasks.md: granular description of your project tasks. You can use it as a log so models understand what you’ve done and your progress.

2. Version Control initiating Git from the beginning of the project: commit changes before accepting the code suggestions from the Agent, if you’re satisfied then test and proceed.

3. Index only necessary folders, files, and libraries documentations at the beginning, this will help the Agent pay attention to your codebase structure and understand the last version of the documentation you’re using, for example the last version of Pandas or Matplotlib.

In summary:

Football Examples

Example 1: six‑plot match dashboard

For this I needed to do a bunch of stuff:

• Scraping match event data from different sources.

• Cleaning and unifying pitch coordinates and IDs

• Precalculating the metrics

• Generate the plotting functions.

You can refer the last article so see what the project is about: Building Your Own Football Analytics Dashboards With Free Data

And I needed to go over each step with a structure.

1. Files

   1. rules.mdc: emphasized the generation of the code in functions and simplified code, sometimes LLMs go way over their head and make up stuff.

      

   2. project_context.md: I went from broad to specific explanation of my project. Main goa, key components, steps to create it, plots characteristics, color and aesthetic I wanted to implement.

      

   3. project_log2025.md: this is the task document where I only set the specific task to accomplish and next steps.

   

2. Version control with Git

3. Index files and folders: in Trae, the context indexes automatically, in Cursor you need to indicating it at the beginning but then files are automatically synced.

From this point onwards, is a back and forth iteration, tweaking the documents, and adapting the prompts to obtain satisfactory outputs from the LLMs.

Some of the challenges: the model confused coordinate systems (0–100 vs 120×80), misnamed some columns, or repeated different Matplotlib functions. Had to pass the dimensions explicitly in a document to be used as context.

Example 2: on‑ball sequences with GPT‑5 in Cursor (Trae didn’t have it yet)

Sequence helps you not only analyze what happened in a match (player passed a ball in a specific direction and zone of the field) but also ‘how’ it happened before and after that, it adds context.

I asked GPT‑5 to find sequences based on my data from Salzburg vs. Real Madrid in the CWC, and gave it freedom to pick any event and any player. I did it in 3 steps:

• Asked it to adopt the role of a PhD Level Football Machine Learning Engineer and that it needed to understand my codebase and recent progress.

• Indexed my event data, so it could understand the type of events and specifications of the plots colors.

• I asked it to plot coordinates X and Y of any event and any player.

It chose a pass where Dean Huijsen receives deep on the left in his own half and carries into the attacking half on the same flank.

Then plotted two examples with a dark mplsoccer theme and label the player.

It filtered events, computed distance gained (about 39–43m in two cases), and drew the trajectory of the sequence.

The takeaway: when your instruction is specific and concrete, agentic IDEs translate it into working code quickly.

How to get reliable results from the models

• Be specific with inputs: a well-described tasks, specific code format (generate functions).

• Paste snippets of your information (e.g. rows of your data frame) and state plotting specifications.

• Index your context: point the Agent to the exact files or docs you want it to use.

• Have it explain the logic in plain language before you run anything.

Quick comparison between Cursor vs Trae based on my experience

Cursor

Is one of the most populars, I personally find these specs useful:

• Agent mode in Claude Sonnet 4: best model I’ve tested inside Cursor, it provides well structured code and understands the codebase when you give it the entire context.

• MCP integration: most of the MCPs have integration for Cursor, for example setting Supabase was easier here than in Trae which is newer.

• Haven’t tried but looks interesting: Background Agents → edit and run code in a remote environment.

• Broader range of models: for small tasks the Cursor small own model helped me for writing specifications for project and I also liked using Deepseek v3.1 for coding, it was pretty solid.

• Something I didn’t like: the rate limit. With pro plan ($20 a month), you were suppose to have unlimited requests but one day they changed it quietly and people started to hit rate limits all of a sudden.

Trae

• Is cheaper: first month $3, next bill $10.

• Has two modes:

  • Builder: to execute routine and end-to-end tasks.

  • Builder with MCP: use MCP servers to run tasks.

• MCP: setting them is easier than in Cursor and they have a marketplace where you can grab and the ones you need.

• Haven’t tried but looks interesting: Custom Agent → use MCP tools and built-in ones from Trae to use a specific Agent for specific tasks.

• Needs more popularity but they are clear with the limits you have 600 a month.

• Something I didn’t like: they have a SOLO builder mode which is more autonomous and you can see the development of your app during the whole process: backend and frontend, but you have to join a waitlist to get an invitation codes that hasn’t arrived yet.

To help you evaluate them deeply you can check these two sources to help you decide:

Trae AI: A Guide With Practical Examples

Cursor AI: A Guide With 10 Practical Examples

Final thoughts

Agentic IDEs can help you move fast from any question you have to working code.

Start small: aim for iteration and then repeatable workflows. Keep guardrails in place: index your documents, do sanity‑checks after every iteration, have the IDE generate simple tests, and ask models to state assumptions before coding.

As you polish prompts and set rules, you’ll see the improvements.

Is a comprehensive visual report that provides a quick-analytical overview of a football match.

If you missed a match or watched the 90+ minutes and you want to review the statistics quickly, a dashboard helps you get an overview of the tactical and performance insights in one place.

In case you're tracking a league as an analyst, as a scout to flag a player, or even as fan that wants to know how your favorite team performed this weekend, building them can make a difference in how fast you retrieve that information.

I built this dashboard using:

• Python 3.10

• Beautiful Soup

• Pandas

• NumPy

• Pandas

• Matplolib

• Mplsoccer

The goal was to build a template with just 6 key analytical plots:

• Passing Networks: Shows team shape, player positions, and passing combinations.

• Defensive Actions: Reveals pressing intensity, defensive positioning, and team compactness.

• Progressive Passes: Analyzes forward progression and attacking build-up patterns

• Shot Map: Comprehensive shooting analysis with xG, shot types, and quality assessment.

• xT Momentum: Tracks dangerous situation creation, goals, and match momentum shifts over time.

• Match Statistics: Side-by-side comparison of key performance metrics between teams.

Where to get the data from?

This is the most important part.

In this case I needed event data for only one match, using coordinates from different situations in the offensive and defensive phases like: Passes, Progressive Passes, Interceptions, Blocks, etc.

These coordinates are represented as X and Y, paired with its end_x and end_y. This helps understanding where and when a situation started, same as where it ended on the pitch.

WhoScored was my go to place to obtain the majority of them. But I was still missing the shots coordinates. Fotmob has them, though it’s a tricky place to obtain them.

I suggest you to visit the GitHub repo and give the scraping logic a look: https://github.com/ricardoherediaj/football-analytics tutorials/blob/main/notebooks/cwc_scraping_test.ipynb

Plotting the Dashboard

After obtaining the data sources, I transformed the raw match events into insights.

In this case I saved the data in CSV and JSON format, you’ll see it at: https://github.com/ricardoherediaj/football-analytics tutorials/blob/main/data

• df_events contains the main events and coordinates I used for the plotting

• match_events.json and matchdict.json contain players numbers and shirts

• shots_df includes the shots data from Fotmob in the match

• xT_grid.csv has the values to understand the highest and lowest threat actions a team makes during a match on the pitch.

Assisted by Claude Sonnet 4 inside the Cursor IDE, I designed this template around six plots that I found interesting for the dashboard.

I selected just six to try not to overcrowd the report. Each plot shows specific characteristics of team’s performances.

1. Passing Networks

Shows the team shape, players average positions, and passing combinations.

I also calculated measures for verticality, to see how direct a team's passing is. I'm also tracking the defensive line, forward line, and team median position to understand how compact or stretched a team's shape is.

When you visualize them size-by-size, Salzburg's network might show a compact, possession-based approach with short passing triangles, while Real Madrid could display a more vertical, counter-attacking structure with longer passing connections.

I converted WhoScored coordinates (0-100 scale) to StatsBomb format (0-120x0-80) for consistency across all visualizations.

The bidirectional pass combinations reveal which player partnerships are most active during the match.

2. Defensive Actions

This heat map shows where each team applies pressure and wins possession.

I tracked every tackle, interception, ball recovery, and blocked pass, then creating a KDE (kernel density estimation) heat map that shows the intensity of defensive actions across the pitch.

The Defensive Action Height metric tells you how high up the pitch a team typically wins the ball.

Combined with the compactness percentage, you can immediately see if a team is playing a high press or sitting deep.

3. Progressive Passes

Passes that significantly advance the ball toward the opponent's goal at least 30%.

The pitch is divided by zones where you can see if a team builds attacks primarily through the left flank, through the center, or via the right side.

Example: If Salzburg completed 23 progressive passes with 65% coming from the left side, you can deduct they're targeting Real Madrid's right defensive lane.

4. Shot Map

It shows where shots were taken and incorporates Expected Goals (xG), big chances, and different shot outcomes to give you a complete attacking picture.

Each shot is sized by its xG value and color-coded by outcome. Green circles are goals, hatched circles are saves, and larger markers indicate big chances. This shows you which team created better opportunities, regardless of the final score.

For example: a team with 8 shots worth 0.3 xG total had a very different performance than a team with 4 shots worth 1.2 xG total.

5. xT Momentum

Expected Threat (xT) momentum tracks which team is creating more dangerous situations throughout the match, smoothed over time to show momentum shifts.

I'm calculating the xT value for every successful pass and carry, then using a weighted rolling window to show how momentum builds and shifts. When you see the line spike upward, that team is creating dangerous situations. When it dips, they're losing control.

Momentum spikes often correlate with goals, substitutions, or tactical changes.

6. Match Statistics

The final central plot is a comparative view of key performance metrics.

Possession percentage, field tilt (percentage of touches in the attacking third), passing accuracy, defensive actions, and PPDA (Passes Per Defensive Action - a pressing intensity metric).

Notes on the Development Journey

I've left my experimental attempts visible in the notebook because I wanted to show the messy development process before the final dashboard code.

Experiment 1: data integration challenges, mixing WhoScored events and Fotmob shot coordinates. Since they’re different platforms they use different coordinate systems and player ID formats but still the plotting was poor.

Experiment 2: used the data preprocessing from experiment 1, builds the layout and makes the visualizations work as subplots rather than standalone plots. This required making three functions to be "subplot-friendly”.

Experiment 3: improved the “dual ID system” issue, since WhoScored uses different team IDs than Fotmob, so I needed to map between them correctly for the shot map while keeping everything else consistent.

Experiment 6: the final version I'm satisfied with (for now). It did the preprocessing process more straightforward and plotted all six plots into a meaningful dashboard.

Next Steps

This template is designed to be the foundation for a larger end-to-end football analytics pipeline. The next steps would be:

• Modularizing the code into separate Python scripts for easier maintenance

• Database integration in Supabase to store scraped data persistently

• Automated data collection and reports visualization using GitHub Actions or similar

• Individual players performance reports

• Obtaining data for more leagues and matches

Once you have the template working for one match, you can generate these reports for any match in any league, as long as you get the underlying event data.

The goal of this is to think it a systematic approach to football analysis that combines the best of data science with tactical understanding.

Each visualization answers specific questions that matter to different stakeholders:

• Coaches can see their team's tactical execution and identify areas for improvement

• Scouts can quickly assess player and team performance across multiple matches

• Analysts can spot patterns and trends that aren't visible from watching alone

• Developers/Fans can understand the tactical nuances of matches beyond just the final score

I invite you to grab a cup of coffee, subscribe to my newsletter, and give it a check to the repo: https://github.com/ricardoherediaj/football-analytics-tutorials/blob/main/notebooks/match_dashboard_template.ipynb

You might learn something new, discover something to improve, or ideas to make your own football plots!

Subscribe now

References

Special mention to the main reference I based on:

https://github.com/adnaaan433/Post-Match-Report-2.0/tree/main

It is one of the best places to start when you don’t have access to paid providers and want to build your own players profiles for scouting, you can leverage this data and flag specific players from different leagues.

This gives you the opportunity to filter players and then check through video that the player you selected actually meets your criteria in the context of the team your scouting for.

Metrics you can find here:

• Attacking stats: passing accuracy, possession,

• Defensive stats: tackles, blocks, interceptions, clearances, errors.

• Goal and Shot-creation actions.

• A basic Scouting report compared with other players from similar leagues.

This can give you an idea of the player performance from a given season.

I’ll show you how I scraped data from Championship, League One, and League Two to find a hypothetical signing for Sheffield United.

I also integrated the FBRef data with the Transfermarkt data, a process I described in my last post:

My Approach for Scraping FBRef

I basically did a two-step process: manual then automatic using only the requests, pandas, and pyjanitor packages.

1. First, I manually checked the page of the Championship league for the 2024–2025 season and went tab to tab to get familiar with the metrics I actually needed (they’re a lot, and most of the time you don’t need that many). I did this for this league since it basically has the same metrics as League One and League Two.

   Since I was interested on midfielders and forwards I only used data from all of the tabs except Goalkeepers.

2. Web scraping: to do this I need to to look at the HTML and tried to find the tabs I wanted to obtain data from. You do this by opening the FBRef URL in Chrome and righ-clicking “Inspect”, to check the HTML code. Then, you get this beautiful Matrix-like mess.

The idea is to obtain the information from a table like this:

I used this function for that:

def get_clean_table(url, table_id):
    """
    Fetches and cleans a table from fbref.com given a URL and a table ID.

    Args:
        url (str): The URL of the webpage containing the table.
        table_id (str): The 'id' attribute of the HTML table.

    Returns:
        pd.DataFrame: The cleaned and ready-to-use DataFrame.
    """
    # Make the request and get the HTML content
    response = requests.get(url)
    html = response.text.replace('<!--', '').replace('-->', '')

    # Read the specific HTML table
    df = pd.read_html(html, attrs={'id': table_id})[0]

    # Clean the columns with clean_columns from pyjanitor
    # It converts the name of the columns as lower case
    df = clean_columns(df)

    # Fill missing values and reset the index
    df = df.fillna(0).reset_index(drop=True)

    return df

I then used this other function for some data cleaning regarding mutlti-index and columns names so I could get a decent result:

def clean_columns(df):
    """
    Cleans and flattens the columns of a DataFrame with multi-level indices.

    Args:
        df (pd.DataFrame): The original DataFrame with multi-level column indices.

    Returns:
        pd.DataFrame: The DataFrame with flattened and cleaned columns.
    """
    # Flatten the columns by joining the levels
    df.columns = [' '.join(col).strip() if isinstance(col, tuple) else col for col in df.columns]

    # Rename the columns if necessary
    new_columns = []
    for col in df.columns:
        if 'level_0' in col:
            new_col = col.split()[-1]
        else:
            new_col = col
        new_columns.append(new_col)
    df.columns = new_columns

    return df

def process_dataframe(df):
    """
    Applies additional processing to a DataFrame.

    Args:
        df (pd.DataFrame): The DataFrame to be processed.

    Returns:
        pd.DataFrame: The processed DataFrame.
    """
    # Check for the existence of columns before applying operations
    if 'Age' in df.columns:
        df['Age'] = df['Age'].astype(str).str[:2]
    else:
        df['Age'] = None

    if 'Pos' in df.columns:
        df['Position_2'] = df['Pos'].astype(str).str[3:]
        df['Position'] = df['Pos'].astype(str).str[:2]
    else:
        df['Position_2'] = None
        df['Position'] = None

    if 'Nation' in df.columns:
        df['Nation'] = df['Nation'].astype(str).str.split(' ').str.get(1)
    else:
        df['Nation'] = None

    # Remove columns if they exist
    df = df.drop(columns=['Rk', 'Matches'], errors='ignore')

    # Replace abbreviations with full names
    position_mapping = {'MF': 'Midfielder', 'DF': 'Defender', 'FW': 'Forward', 'GK': 'Goalkeeper'}
    df['Position'] = df['Position'].replace(position_mapping)
    df['Position_2'] = df['Position_2'].replace(position_mapping)

    return df

And boring but quick manual part: selecting the URLs from each tab I wanted to gather info from and then turning them into dataframes, so I could merge the data from all tabs and visualize all the metrics.

# URLs and table IDs
tables_info = [
    ('https://fbref.com/en/comps/10/stats/Championship-Stats', 'stats_standard'),
    ('https://fbref.com/en/comps/10/shooting/Championship-Stats', 'stats_shooting'),
    ('https://fbref.com/en/comps/10/passing/Championship-Stats', 'stats_passing'),
    ('https://fbref.com/en/comps/10/passing_types/Championship-Stats', 'stats_passing_types'),
    ('https://fbref.com/en/comps/10/gca/Championship-Stats', 'stats_gca'),
    ('https://fbref.com/en/comps/10/defense/Championship-Stats', 'stats_defense'),
    ('https://fbref.com/en/comps/10/possession/Championship-Stats', 'stats_possession'),
    ('https://fbref.com/en/comps/10/playingtime/Championship-Stats', 'stats_playing_time'),
    ('https://fbref.com/en/comps/10/misc/Championship-Stats', 'stats_misc'),
]

# Dictionary to store the processed DataFrames
dataframes = {}

# Iterate over each URL and table ID
for url, table_id in tables_info:
    # Fetch and clean the table
    df = get_clean_table(url, table_id)
    # Apply the additional processing
    df = process_dataframe(df)
    # Store the DataFrame in the dictionary using the table ID as the key
    dataframes[table_id] = df

Resultant dataframe up till this point:

To select the relevant metrics for midfielders and forwards I used a dictionary that included key-value pairs of the lists of variables I needed:

columns_to_retain = {
    'stats_standard': ['Player', 'Nation', 'Squad', 'Age', 'Born', 'Playing Time 90s', 'Position_2', 'Position'],
    'stats_passing_types': ['Player', 'Squad', '90s', 'Pass Types TB','Pass Types Sw', 
                            'Pass Types Crs', 'Outcomes Cmp', 'Outcomes Blocks'],
    'stats_passing': ['Player', 'Squad', '90s', 'Total Cmp', 'Total Att', 
       'Total Cmp%', 'Short Cmp', 'Short Att', 
       'Short Cmp%', 'Medium Cmp', 'Medium Att',
       'Medium Cmp%', 'Long Cmp', 'Long Att', 
       'Long Cmp%', 'Ast', '1/3', 'PPA', 'PrgP'],
    'stats_playing_time': ['Player', 'Squad', 'Playing Time 90s', 'Playing Time MP','Playing Time Min'],
    'stats_shooting': ['Player', 'Squad', '90s', 'Standard Gls', 'Standard Sh', 'Standard SoT', 'Standard SoT%',
       'Standard Sh/90', 'Standard SoT/90'],
    'stats_gca': ['Player', 'Squad', '90s', 'SCA SCA','GCA GCA'],
    'stats_defense': ['Player', 'Squad', '90s', 'Tackles Tkl',
       'Tackles TklW', 'Tackles Def 3rd', 'Tackles Mid 3rd', 'Tackles Att 3rd', 'Blocks Blocks', 'Blocks Sh', 'Blocks Pass', 'Int', 'Clr', 'Err'],
    'stats_possession': ['Player', 'Squad', '90s', 'Touches Touches', 'Touches Mid 3rd', 'Touches Att 3rd', 'Touches Att Pen',
       'Take-Ons Att', 'Take-Ons Succ', 'Take-Ons Succ%', 'Carries Carries', 'Carries PrgC', 'Carries 1/3', 'Carries CPA',
       'Carries Mis', 'Carries Dis', 'Receiving Rec', 'Receiving PrgR'],
    'stats_misc': ['Player', 'Squad', '90s', 'Performance Off',
       'Performance Crs', 'Performance Int', 'Performance TklW',
       'Performance Recov', 'Aerial Duels Won', 'Aerial Duels Lost',
       'Aerial Duels Won%']
}

Final function to obtain the merged dataframe with all the selected variables:

def merge_dataframes(dataframes_dict, columns_dict):
    """
    Merges the dataframes from `dataframes_dict` by filtering the required columns from `columns_dict`.

    Args:
        dataframes_dict (dict): Dictionary of processed dataframes.
        columns_dict (dict): Dictionary with the columns to retain for each dataframe.

    Returns:
        pd.DataFrame: Merged DataFrame with the specified columns.
    """
    filtered_dfs = []
    
    # Filter the columns to retain from each dataframe, only if the columns exist
    for key, df in dataframes_dict.items():
        if key in columns_dict:
            print(f"Columns available in '{key}': {df.columns.tolist()}")  # Debug: print available columns
            columns_to_retain = [col for col in columns_dict[key] if col in df.columns]
            print(f"Columns to retain for '{key}': {columns_to_retain}")  # Debug: print columns to retain
            df_filtered = df[columns_to_retain].copy()
            df_filtered.columns = [f"{col}_{key}" if col not in ['Player', 'Squad'] else col for col in df_filtered.columns]
            filtered_dfs.append(df_filtered)
    
    # Concatenate all dataframes on 'Player' and 'Squad'
    merged_df = pd.concat(filtered_dfs, axis=1)
    
    # Remove duplicate columns for 'Player' and 'Squad'
    merged_df = merged_df.loc[:, ~merged_df.columns.duplicated()]
    
    return merged_df


merged_dataframe = merge_dataframes(dataframes, columns_to_retain)

The dataframe still had a lot of NaN values that needed to be cleansed, but the process to obtain data for other leagues (League One and League Two in this case) is good to go.

I need to mention that I used these references as inspiration to drive my own analysis:

How to scrape and personalize data from FBref with Python: A guide to unlocking Football Insights

The Super Easy Way to Scrape FBREF for Free Soccer Data

There are also other approaches like this one from Henrik who kindly referenced mine:

Scraping Fbref —Creating a Pipeline

So, if you’re into scouting/football analytics and what to try it for yourself, give it a go!

There’s more than one way to get to a solution.

I encourage you to find yours and see how it works.

Web scraping has always intrigued me. In my opinion, is one of the best skills you can have in your data science/ML toolbox. It is useful for when you want to build your own custom databases for personal projects and you don’t have access to paid data providers.

Using Web Scraping For A Project

I’ll show you how I approached a past project using web scraping. I wanted to identify two players profiles:

• A central midfielder with a mixed profile (good in both defense and attack)

• A striker with good aerial game and ball retention.

I focused on finding five players for each position that fit the profiles based on different metrics from FBRef, the team’s playing style and the economic context of the team, where Transfermarkt helps a lot with players value market.

For this, I randomly chose Sheffield United from the Championship to carry on with the analysis.

To get to know Sheffield’s playing style and what they needed I analyzed three matches and identified positions that could benefit from these signings.

For more insights on how to analyze a team performance you can read this post about how I applied this framework for Como 1907 from Serie A: Como 1907 Team Analysis for Data Scouting

How to Know Players Value Market?

To achieve this, I needed to gather data from Transfermarkt.

For those unfamiliar with this web page, it is a place that offers a variety of data points that are helpful to identify potential players:

• Age.

• Country.

• Contract

• Positions

• Market values.

To do this efficiently, I needed an automated way to extract player information directly from Transfermarkt, which led me to build a web scraper.

Designing the Scraper: Building Process and Core Components

In order to build the scraper, I first needed to understand what and how I was going to do it. So I wanted to see if somebody else did it first and fortunately, someone did. These two Medium articles were my go-to references to help me go through this task:

https://blog.devgenius.io/get-to-know-the-greatest-football-league-in-the-world-with-python-and-transfermarkt-25a7cfa6b136

https://medium.com/@conalhenderson/how-to-build-a-custom-web-scraper-to-extract-premier-league-player-market-data-3b8e5378cca2

I specially followed Conal’s comprehensive tutorial to build my own custom web scraper, as he shows how to do it exactly but with Liverpool’s and Premier League data, so it was a matter of going through it, running the code and understanding the process that happens under the hood when scraping Transfermarkt.

It was challenging, honestly. But after a few days I managed to replicate the exact same process for my goal: obtaining England’s tier 2, 3 and 4 data.

The bright side, is that I learned a new way of web scraping and improved capabilities on Python coding, which is necessary for when you want your code to be more readable and scalable as your project grows.

I won’t get into each detail of the code though, if you’re looking to do some web scraping, I highly recommend following the tutorial to better understand step by step where everything comes from and the final output, since it requires some attention if you don’t know German (all of Transfermarkt source code is in German).

I’ll focus on the core parts I included in my pipeline.

So, to make the scraper as modular and efficient as possible, and using Python flexible capabilities, I designed it with some key components:

1. Abstract Parser Class: I used an Abstract Base Class (Parser) as a template for all specific parsers which helped obtain different types of player data.

   As you’ll see in the image I focused on getting these data points from all the players: names, ages, countries, market values, and positions (plus the team and the season), because these were the ones I could get, somehow, I couldn’t manage to obtain other datapoints from the HTML source (got to look into that), but I believe these were the most helpful for my goal.

By breaking the parsing into smaller, independent classes, you can maintain a cleaner code and made modifications easier in the feature without affecting other components.

2. Dataclasses for Structured Data: To organize my code and ensure type safety, I used Python’s dataclass to create a Team class. This helped me encapsulate relevant information such as the team ID and name, and it was an efficient way to pass data around the scraper.

3. The Scraper Class: It takes a team, a list of parsers, and then extracts data from Transfermarkt. The structure of the URL allowed for programmatic generation by passing in the team ID and name, making the whole process automated and highly scalable.

This part of the code shows the main Scraper class I mentioned, which fetches data for a given team and year, and allows you to scale the scraping across different teams by utilizing a uniform URL structure.

It uses BeautifulSoup to retrieve the HTML and httpx with custom headers to mimic a browser. Also, using ‘time.sleep(5)’ is critical to use to avoid overwhelming the server, getting kicked out and blocked in the future.

The Result: Structured CSV file ready to use for scouting analysis

The final dataset included the information I meant to scrape: player names, ages, countries, positions, and market values, which I saved into CSV files for further analysis.

Additionally, I uploaded the data to Supabase in different SQL tables for each league to make it easily accessible for querying and integration with other tools afterwards.

This allows me to have a central structured storage and to query this data quickly, be it inside Supabase or using a connection to that database and clean the data with Pandas in another development environment.

Key Takeaways

• Handling the intricacies and best practices of web scraping is tough, but necessary to execute repetitive processes effectively.

• Breaking the code into modular pieces using classes and parsers, makes it easier to extend the scraper and add new features as needed.

• Storing the scraped data in a database helps to maintain a centralized storage for further analyses.

Here’s the link to project in case you want to reference the code: Data Driven Scouting Project

I’d love to hear your thoughts or discuss any challenges you’ve faced when web scraping data.

Leave a comment

Quick Recap

To recap, there are 3 steps I’ve taken to carry out this analysis:

1. Full Team Analysis: I’ve analyzed Como’s performance in Serie A 24/25 and how they performed in offensive and defensive phases. I also compared them with the other teams in the league. With this, I wanted to spot areas of improvements within the squad.

2. Profile Identification : based on what the team needs, I identified specific profiles that can fulfill that necessity.

3. Search & Comparison: For that, I used Driblab’s data hub to shortlist interesting players based on specific criteria.

I’ll develop the last 2 points and final conclusions on the players I shortlisted.

Profile Identification

• Attacking Left Back: an attacking fullback with good physical ability, dribbling, and crossing skills to provide more width and depth on the left flank. Alberto Moreno’s contract ends in June 2025, so it’s key to anticipate his possible departure and secure a quality replacement.

• Attacking Right Back: an attacking fullback who can strengthen the right side and create synergy with the right winger. Ideally he can carry the ball forward and deliver crosses into the box, but also has good passing quality.

• Right Winger: there’s a need for more dribbling and quality crosses from the right wing to diversify the attack and create more goal-scoring chances. Ability to get in behind and provide depth. Would add competition and alternatives to Strefezza.

• Center Forward: there’s a need for more physical presence and effectiveness in the box to make the most of crosses and increase chances conversion. A striker who wins aerial duels and is clinical, giving the team more finishing options.

Search & Comparison

After I had identified the profiles, it was time to search for the players.

Driblab has a ‘Scout’ tab that allows to search players worldwide, filtering by different parameters. I only had access to the top 5 leagues, but still it was an awesome setting to test this feature.

With this, you can select players and make shortlists, build radar plots of each selected player and compare them with the ones from your squad and pretty much build your own database of selected players.

To search players, I considered: team performance, specific KPIs, and the economic context, something important for clubs, because there’s a budget sporting directions need to stick to. Aspects I looked into:

• Age of recent signings → young players with an average of 26 y/o. Veteran players (over 30) with leadership, signed as free agents or at low cost, mixing experience with youth in the squad.

• Players on loan with an option to buy, or direct transfers/purchases.

• Market value → range from € 100K - € 10M

• Year End Contract → Summer 2025 (preferably)

• Top 5 Leagues: → Serie A, Bundesliga, Premier League, LaLiga, Ligue 1.

• Specific Positions → i.e. Attacking Midfielder Left - AML

• KPIs→ Driblab scores (>45), crosses, passing, minimum minutes played (450’), xG, xA, etc.

Shortlisted Players

Players Radars and Comparison With Como Players

Javi Galán Summary

Left back with great physical ability and the capacity to advance with the ball down the wing. He provides depth on the left, with a high percentage of successful passes and dribbles. Reliable in offensive progression, though he can be impulsive defensively, bringing speed, dribbling, and solidity to the left side.

Alvaro Tejero Summary

Attacking and versatile right back. He crosses well, creates chances, and is reliable in building out from the back. While he doesn’t stand out defensively, he would add attacking depth and quality crosses to diversify play down the flanks.

Junya Ito Summary

Fast and productive right winger, excellent at beating defenders and delivering crosses. He provides assists and scores from deep positions, also contributing defensively. Although his goal contribution is low, he offers speed and unpredictability on the wing.

Mbala N’Zola Summary

Center Forward with a strong presence in the box, excels in physical duels and aerial play. He stands out for his finishing, ability to shield the ball, and pressing high up the pitch. While he doesn’t contribute much in creation or dribbling, he adds finishing, power, and is a key attacking reference in the opponent’s area.

Final Conclusions

• Diagnosis and Strategic Solution: Como 1907’s play is overly centralized, with a heavy reliance on Nico Paz and low efficiency in the opponent’s box. The proposed signings directly address these needs by strengthening the flanks and improving the team’s attacking aerial game.

• Expected Tactical Impact of the Signings:

  • Javi Galán and Álvaro Tejero: would provide depth and width on both flanks, diversifying attacking options and increasing the quality of crosses into the box.

  • Junya Ito: would complement Strefezza, adding unpredictability, speed, and the ability to dominate the right wing.

  • Mbala N'Zola: will bring physical presence in the box, improving offensive aerial play and finishing.

• Economic Viability: The selected players represent a sustainable investment (range €1.8M–€6M) with potential for increased value, aligned with the club’s recent transfer policy and with contracts that allow for medium-term planning.

This is my personal approach on tackling a data scouting analysis. Always considering data as an asset that complements the on-site and/or video remote scouting, for a well-rounded informed process.

Signing Players In the Modern Way

If you love sports, you’ve probably heard the term Moneyball. First a book, then a Brad Pitt movie, Moneyball tells the story of how the Oakland Athletics built a playoff team on a low budget. What was the secret? Data. Instead of chasing big names, they found undervalued players, signed them cheap, and sold high using math and statistics.

Why does this matter for football?

Because the same principles are changing the game in Europe since not so long ago. Clubs like Brentford and Midtjylland are using data to spot hidden gems and build squads that perform above their level. As Rasmus Ankersen, Brentford’s former sporting director, put it: “We look for players with potential and analyze their context—why haven’t they reached it yet? It’s like finding an undervalued stock, buying low, and selling high.”

Modern scouting is not only about watching games but using data to see what others miss and turning that insight into results.

Starting With the Right Questions: Club Analysis

Data is an asset. It helps you filter out noise and focus on the players who actually fit your team’s needs. That means smarter decisions, fewer wasted transfers, and a bigger impact for every euro/pound/dollar spent.

Let’s see this with an example I’ve been working on. It’s a case study with Como 1907—a club on the rise in Serie A. My goal was to: use performance metrics to break down their current squad, understand their transfer strategy, and spot which players from Europe’s top 5 leagues could improve their level.

It all starts with the right questions:

• What’s the team’s game model? How do they play in different phases?

• Which metrics matter most for their style?

• How do they stack up against the rest of the league?

• What’s their transfer history and scouting philosophy?

• And most importantly: Who could you sign to level up the squad?

I’ll walk you through how I tackled each of these, using data from Driblab Pro—a professional scouting platform I access thanks to my Scouting360 course with Objetivo Analista.

Key Performance Indicators: Using Metrics That Matters

For Como 1907, I started with a general overview of them, as a presentation card, and then a general variable analysis, looking at the roster age profile and several metrics like own expected goals (xG) and against, attack zones, shot zones, own assists and against.

• Como 1907 is a young team, with an average age of 26.

• The club made significant investments in the squad over the last two transfer windows.

• Under Cesc Fábregas’s attacking philosophy, they finished 10th in the league.

• The team’s preferred formation was 4-2-3-1.

• They have potential to qualify for European competitions.

• Top 3 players with most minutes played: Jean Butez, Nico Paz, and Kempf.

Variables Analysis

For a quick overview to see how the team attacks, we can look at the attacking zone maps and xG per Shot maps. They attack mostly through the center, with 66.5% of possessions ending in that zone.

Defensively, Como had a mix of performances throughout the season. In several matches, they created more and better chances than they conceded, but there were also games where they were outperformed defensively. The team was capable of both attacking with danger and having vulnerable defensive moments, with many balanced matches in between.

To dig deeper, I broke down the analysis by phases of play in order to compare them with other opponents peers in Serie A:

• Offensively, Como builds up through the center, relying on key creators like Nico Paz, and struggles with xG per shot, they make more shots, but those shots are low quality. Though they show an average position in moving the ball into dangerous areas (the final third of the pitch) compared to other teams in the league.

How do they compare to the rest of Serie A? Como sits around the league average in most metrics, but stands out for ball progression, shots taken and touches in the opposition box. However, their aerial duels and set-piece threat are other areas for improvement.

• Defensively, Como behaves like this compared to the other teams:

  • Press high and recover balls in advanced areas, ranking among the best in Serie A for pressing intensity (PPDA). Though this intensity drops in the second half, leading to more goals conceded late in games.

  • Concede few shots overall, but the chances they allow are often high quality (higher xG per shot conceded).

  • Very effective at defending set pieces, conceding the lowest xG from set pieces in the league.

Key Takeaways

I started broad, then zoomed in. Using general metrics to spot trends, then breaking them down by phase and comparing to the league to find where the team stands out or falls short.

Squad Evaluation: Finding Gaps in Their Game

To better understand what the team needs, is important to understand how they play and identify the areas that need to be improved. Again, it was helpful to make come up with some questions and separate the analysis by game phases.

Offensive Phase

Possession & Build Up

• Como builds from the back, relying on defenders and midfielders (Goldaniga, Da Cunha, Nico Paz), which centralizes play.

Progression and chance creation

• Most progression happens through the center; wide players like Strefezza, Diao, and Fadera are underused for dribbling and crossing.

• The team could improve by creating more danger from the flanks.

Danger in the box & Shot Quality

• Cutrone gets the most touches in the box, but Strefezza is more efficient with fewer touches.

• Only Gabrielloni and Douvikas consistently take high-quality shots; most starters are below average for shot quality.

Defensive Phase

Pressing & Recoveries

• Como presses high and recovers balls in advanced areas, led by Nico Paz, Strefezza, and Da Cunha.

• They concede more goals in the second half, likely due to fatigue and a drop in pressing intensity.

Defensive Solidity & Defense-to-Attack Transitions

• Goalkeepers (Reina, Butez) and central defenders (Dossena, Enghardt) are strong in aerial duels. Goldaniga and Dossena excel at interceptions.

• Goldaniga is key for starting attacks from the back progressing the ball to higher areas of the pitch, with support from Da Cunha.

What’s Next?

With all this analysis, we’ve seen Como’s strengths: solid build-up, intense pressing, and real danger creation. But they still need more width, better shot conversion, and more aerial power in the box. The data points us where to improve.

Next up: finding the who.

I’ll use this data to build a shortlist of transfer targets, matching each pick to the positions and criteria that fit Como’s game model.

Curious to see who makes the cut?

I’ll break it all down in the next article!

Leave a comment

Today I bring 5 tips that helped me get acquainted with how football data is used in real-world settings.

1. Start Where the Data Is

You don't need special credentials to start. High-quality datasets are readily available on public websites.

Some of them:

• StatsBomb: Granular event data for advanced analysis of top competitions and players (Messi’s career, Euro, Copa America, Leverkusen unbeaten Bundesliga 2023/2024). StatsBomb Free Data

• FBRef: Aggregated stats for team and player comparisons, trend tracking, and scouting. FBRef

• Transfermarkt: The go-to for player profiles, transfer histories, market values, and club finances. Transfermark

• WhoScored: Detailed match and player stats, including event-by-event breakdowns and player ratings. WhoScored (Opta Data)

• For Video Analysis: Ben Griffis Streaming sheet is the go to: Ben Griffis Free Streaming Sheet

Spend time exploring these resources, ask yourself real football questions, and let the data guide your analysis.

2. Free YouTube Channels

If I had to pick three channels to binge-watch, it would be:

• Friends of Tracking: Community-driven project with free educational content, code, and data. Led by David Sumpter one of the data pioneers in the field. Friends of Tracking

• McKay Johns: sTutorials, code, and guides on using Python and data science for football analysis. McKay Johns Channel He has football analytics courses that are well crafted and contains everything you need to know to start creating quality projects.

• LanusStats: this one is in Spanish, the creator Federico Rábanos teaches how to use Python to grab stats, make dashboards, and understand metrics. It's great if you speak Spanish and want to learn how to analyze soccer with data and code. LanusStats

3. Improve Your Football Knowledge

It's easy to overestimate our football knowledge. Coaches, analysts, and scouts use specific tactical concepts daily.

For that, I recommend:

• Watch games with a tactical focus: observe team behavior in attack, defense, build-up play, and player technique. Go beyond just following the ball. You can watch free matches in these leagues: Ben Griffis Google Sheet, a list of leagues that upload their matches put up together by another great analyst Ben Griffis who also shares awesome visualizations and resources ( Ben Griffis X Account)

• Online resources: watch YouTube channels that talk about tactics, read blogs in Medium and Substack or even books that go deep in these concepts.

• At some point and if its suitable for your pocket start with affordable video analysis courses that can help you level up your game understanding. For this I’ve taken two courses (in Spanish):

  • AFOPRO (Spanish): an online academy that teaches a wide variety of courses in scouting, video analysis, physical preparation at affordable prices. AFOPRO

  • ObjetivoAnalista (Spanish): one of the best academies on Spain created by Dani Pérez, they offer three signature course: Analista360, Scouting360 and PowerBI 360. I’ve personally taken Analista360 and Scouting360. The first one help me A LOT to improve my game understanding and applying this knowledge to a collaborative opportunity I had as a remote videoanalysis for a friend’s team in the US. I’m currently finishing Scouting360 because I want to get deeper in scouting analytics. ObjetivoAnalista

  • APFA (English): an organization focused on developping analysis skills, foster a community through webinars and content, provide career progression resources like job opportunities and mentorship. APFA

Technical skills matter but there’s something that’s really important to consider: knowing the game matters too.

Picture it like this: you think in data, but speak in football.

4. Start Building Projects: From Simple to End-to-End Solutions

Building projects is the best way to learn. You don't need to know everything upfront; start with what you know..

It doesn’t matter if your first project is simple. A simple visualization from one match or a scouting report from one player. You improve by building tiny projects until you gain confidence through creating. In time, you’ll build more complex data pipelines or modeling.

The important is to:

• Show your entire process: data collection, cleaning, modeling, and visualization.

• Communicate your insights in a clean and concise way.

• Translate everything into football terms.

  

5. Share Your Work, Build Your Network

Sharing my work was a challenge at first. I had accounts on X and LinkedIn, but rarely posted. It's easy to feel hesitant.

I started being a little bit more constant. Since there’s value in sharing what we know. It’ll probably going to help somebody else realize they like football analytics and creating content online like I did back when I did my first Python course with DataCamp (a story for another day).

The are plenty of platforms you can use but I recommend using:

• X: Share quick insights, visualizations, and engage with the community.

• LinkedIn: Showcase professional projects, network with potential employers, and share career progress.

• Medium/Substack (obviously): Share in-depth articles, build a community, and connect with readers. Substack is a recent one for me but I’m digging it.

This is a compact curated list with the sources I’ve tried since the beginning. Something I love I had back in the day.

Now it's your turn! Explore these resources, start building, and share your journey. I'm excited to see what you create!