The situation: why migrate?
To understand what was at stake, it's worth recalling that a Milter (Mail Filter) is a program that sits between the receiving server (MTA) and the recipient to analyse, filter and modify emails on the fly (SPF checks, anti-spam, signing, and so on).
Until now, our architecture relied on a monolithic model hosted on dedicated physical servers. While it worked, this approach was beginning to show its limits:
- Costly vertical scaling: adding resources meant purchasing new physical servers, a slow and inflexible process.
- Complex deployments: updating a Milter required heavy-handed interventions, lengthening maintenance windows and increasing the risk of error.
- Operational silos: this "in-house" infrastructure made maintenance difficult for SREs outside the Mail team.
The goal was clear: to turn this stack into an agile, scalable platform aligned with the company's standards.
The technical strategy: a modern, unified Python stack
Our migration to Kubernetes was the opportunity to modernise our codebase.
The Python revival
We rewrote most of our in-house Milters by drawing on the modern Python ecosystem. Adopting asyncio was central to handling intensive network I/O, making it possible to process thousands of simultaneous connections with a reduced memory footprint. To ensure both quality and speed, we equipped ourselves with a set of standard tools:
uv: for ultra-fast package management and runtime environments.ruff: for instant linting and consistent formatting.Pydantic: for strict data validation and greater safety at runtime.httpx: for high-performance asynchronous API calls to our external services.
Integrating third-party filters
This modernisation extended to our critical third-party filters: Antivirus, AntiSpam and Rspamd. Packaging them into standardised containers lets us orchestrate them in exactly the same way, giving a unified view of the entire filtering chain.
Cross-team collaboration: towards an "internal PaaS" approach
The success of this migration rests on close collaboration. We now deploy our services on Kubernetes clusters managed by the Kube-Infra team. This lets us delegate cluster resilience, scaling and security so we can focus on our core business: filtering.
The CI/CD pipelines were set up in tandem with the DevOps team. While Merge Requests (MRs) were already in place, this migration was the catalyst for tightening the discipline around them: systematic code review, shared standards and tool interoperability. We have moved from a "server management" mindset to an "internal Platform as a Service (PaaS)" one, making the underlying infrastructure transparent to developers.
The engine of change: CI/CD and code quality
CI (Continuous Integration) has become the cornerstone of our workflow:
- Strict validation pipeline: every commit triggers style checks (
ruff), unit tests and a Docker image build. - A culture of review: the migration reinforced our requirement for peer code reviews, encouraging knowledge sharing and the early detection of bugs.
- This rigorous CI pipeline is the foundation of our Continuous Deployment strategy. By ensuring every image is validated, tested, and secure before it even reaches the registry, we gain the confidence to fully automate production releases. The CI now acts as the gatekeeper that makes automated CD safe and reliable.
The routing architecture: why HAProxy?
The centerpiece of our network architecture is HAProxy. Why this choice over Nginx, Traefik, or a native Ingress Controller? While solutions like Nginx (with its mail module) are technically capable of handling TCP mail traffic with similar efficiency, our decision was driven by organizational expertise and standardization.
We have a dedicated internal team specializing in HAProxy who maintain our core routing infrastructure. By aligning our mail stack with their expertise and existing tooling, we benefit from their deep knowledge, established monitoring patterns, and rapid support. Rather than introducing a new technology stack that the Mail team would have to maintain in isolation, we leveraged the company's existing "HAProxy standard." This ensures our routing layer is supported by experts and integrated seamlessly into our broader infrastructure ecosystem.
We set up a resilient grid of reverse proxies:
- Multi-DC redundancy: in each data center, two reverse proxies announce IP addresses via BGP (one primary, one secondary).
- Intelligent routing:
- In production: routing to the Kube clusters is handled through ACLs based on the source IP, directing traffic from the MTA (Postfix) to the right cluster.
- In staging: ACLs based on IP or URL route traffic to different backends for A/B testing.
HAProxy therefore acts as a dynamic, secure switch between Postfix and our Milter containers.
Observability and maintenance
Enhanced observability
As the number of containers grew, observability became critical. We deployed a centralised stack (logs, metrics, traces) that lets us visualise in real time the health of each Milter, the rejection rate, processing latency and resource usage. Every alert is now correlated with a specific service rather than with a physical server.
Maintenance and progressive traffic ramp-up
Our deployment is duplicated across multiple clusters, allowing for maintenance without interruption. The update process includes a progressive traffic ramp-up (Canary deployment):
- Putting the target backend into maintenance via HAProxy.
- Deploying the new version on the isolated cluster.
- Weighted ramp-up: We first reinject 5% of traffic, then 20%, 50%, and finally 100%, monitoring error metrics at each step.
- Instant rollback: If an anomaly occurs, reverting to the previous version is instantaneous through a simple HAProxy switchover.
Impact & Metrics
Before diving into the technical hurdles we faced, it’s worth looking at the tangible results. The migration delivered immediate efficiency gains that justified the effort:
- Legacy Monolith: Consumed 144 vCPUs and 768 GB of RAM across physical servers. Despite this massive footprint, we were already hitting resource ceilings during peaks, with no room for elastic growth.
- New Kube Architecture: We now operate on just 80 vCPUs and 300 GB of RAM.
This represents a ~44% reduction in compute and a massive ~61% drop in memory usage. More importantly, thanks to Horizontal Pod Autoscaling (HPA), these resources are now dynamic. We shifted from allocating fixed, often idle hardware to a model where capacity scales instantly with demand, freeing up significant physical resources for other workloads.
However, achieving these numbers wasn't just a matter of "lifting and shifting". The path to this efficiency was paved with specific technical challenges.
Technical challenges and lessons learned
A migration of this scale inevitably comes with its share of problems. Beyond the architectural benefits, we had to solve complex issues tied both to the language itself and to third-party libraries.
1. The discipline imposed by asyncio
The move to asynchronous code was decisive for performance, but asyncio proved unforgiving when it came to managing the task lifecycle. We ran into two major problems, perfectly described in this analysis from Sailor.li:
Cancellation is broken: cancelling tasks during reloads or timeouts didn't work as expected, leaving ghost processes to consume resources.Task was destroyed but it is pending: orphaned tasks were destroyed by the garbage collector before they had finished running, leading to incomplete processing or silent errors. These errors forced us to deeply refactor how we handle coroutines, to ensure that every task is properly awaited (await) and cancelled.
2. Hardening purepythonmilter
For the interface with Postfix, we relied on the open-source library purepythonmilter. While it gave us a solid foundation, it is no longer maintained and couldn't withstand our initial production load without changes:
- Connection leaks: we observed TCP sockets that weren't closing properly, gradually leading to exhaustion of the available ports.
- Essential fixes: we had to fork the project to implement robust handling of connection closures, even in the event of a critical error. Mastering the network stack right down to the low level proved essential to the stability of the service.
3. Upfront investment vs operational simplicity
The learning curve was steep. The initial setup of the Kubernetes orchestration, the fine-tuning of the HAProxy ACLs and the polishing of the CI/CD pipelines all required a significant investment of time. That design effort is now paying off in full, however. Once the platform had stabilised, the complexity was absorbed by automation: what used to take hours of manual handling now comes down to merging a Merge Request. We have shifted the complexity away from day-to-day operations and into platform engineering, a choice that today translates into complete operational peace of mind.
Conclusion: towards zero downtime
This migration marks a decisive turning point. Where our old updates required weekly maintenance windows with service interruptions, we now operate with zero downtime.
Beyond the technical performance, it's an organisational success. By breaking down the silos between the Mail, DevOps and Kube-Infra teams, we have built a resilient platform that lets us innovate faster. We no longer manage servers, we deliver a service.