COBOL 仍是金融應用的核心引擎嗎？TigerBeetle 原型測試其替代可行性

This is the first post in a series about lessons I've learned over my career. Today I work as an engineering manager at Google (This is a personal project. The views expressed here are my own and do not represent those of my employer, Google.), but for more than 15 years I was a consultant, working with companies across many industries. Along the way I ran into problems I couldn't fully solve at the time. Many times I had to deliver solutions that worked, but with limitations, because there wasn't enough time, technology or people.

With AI I no longer have those limitations. I can explore different solutions and make decisions based on data, and I'd like to finally solve these problems and share what I learned, even if in the end the one reading this is another AI.

The financial engine nobody "should" touch

I want to start with a problem I ran into many times. When we built new features on top of a financial company's core engine, the users always told me the same thing: we can't change the engine, nothing but COBOL can handle the number of transactions we have.

And for a long time that was true. IBM builds the mainframe hardware and COBOL runs on top of it; they are tightly coupled, but very efficient. To give you an idea, IBM says that CICS, the transaction system where COBOL runs, processes more than 1.1 million transactions per second when you add up all its customers worldwide (TechTarget).

The problem is that mainframes are expensive and need constant payments to stay up to date. For a fintech, a small bank or a lender, a mainframe doesn't pay off, so they end up running COBOL on x86 servers, and that's where transactions per second drop dramatically. In a test AWS published with Micro Focus in 2022, a 2-CPU server reached about 400 transactions per second, a 16-CPU server 3,200, and to go past 4,000 they had to run nine servers in parallel (AWS).

That's when companies running COBOL on x86 hit a wall with their financial engine: they want to grow and they can't. I know what you're thinking: you can scale by adding nodes and partitioning. Yes, COBOL scales that way too; that's exactly what AWS did in that test. But if you're going to partition anyway to reach a few thousand transactions per second, why not replace it? Why keep using it when there are fewer and fewer COBOL experts every year?

That's where my research started.

I didn't want another Hello World

To test a COBOL replacement, I wanted to prove you can process transactions with a more realistic architecture. I didn't want to build a Hello World and just brag about the transactions per second of a very simple example. I wanted something that looked like a real business.

So I worked with a fictional company that manages pension funds. As you may know, many companies in Latin America use private pension funds; in Mexico these private plans also come with some tax benefits. Once a month employers deposit into the fund, and employees can also contribute their own money. Each employee contributes a different amount, often depending on their salary. All of it is pooled, sent to a low-risk investment fund, and that's how it earns returns. At the end of each day, the fund calculates how much every employee of every participating organization has in their account.

This seemed like an interesting case to me because the transaction volume is high, especially when employers send contributions at the end of the month. It gets even harder when a new employer signs up, because a fund like this is always looking for new clients, and suddenly you have to add 10,000 or 15,000 new employees.

This business runs on daily and monthly cycles; to see results I would have had to wait months. So I built a small hack I found interesting: a financial clock that doesn't follow the wall clock. I decide when a business day passes, and with one command I move the fund to the next one: contributions arrive, the day's value is calculated and the day is closed. That way I could see in minutes what would take weeks in real life.

What I had to learn, and where AI helped

I have to admit I didn't know much about this kind of business, and AI helped me first to understand it and then to model it. The key piece is something called NAV, the net asset value.

Every contribution becomes a number of units. Those units don't change when the fund's value goes up or down; what changes is the price of each unit. Every day the fund's value is divided by the units in issue, and that's the day's NAV. To know how much your account is worth in pesos, you multiply your units by the NAV.

It took me a couple of days and several conversations with AI to really get it. The best part is that when the fund gains or loses, a single number changes, and everyone's balance in pesos updates on its own. You don't have to calculate each employee's interest, or keep posting transactions all the time to get it. A new contribution doesn't move the price: more pesos come in and more units go out, in the same proportion.

The architecture

Since I wanted a realistic application, I built it with containerized microservices; each one lives in its own folder under services/. There's an API where new employers sign up and contributions are received. Then a process converts contributions into units, and those units are stored in individual accounts in the ledger.

The ledger is the financial engine, exactly the COBOL I'm replacing. I chose TigerBeetle: it's open source, and the team behind it has dedicated itself to tuning this database at the CPU level. They constantly revisit their sorting and insertion algorithms to win microseconds of performance. In that sense it's similar to COBOL in how close it sits to the hardware, but with a more modern architecture.

Scaling TigerBeetle works the same way, by partitioning and adding instances. In this exercise each partition is an independent fund, so a fund is managed entirely in one instance, never split in two, to avoid distributed transactions. Imagine this fictional company decides one day to go after a big contract with an employer that has more than 200,000 or even a million employees, but this client demands to be isolated and have its own fund. That scenario fits the design perfectly: you give it its own fund, put it on another TigerBeetle instance and scale it separately from the rest of the application.

In TigerBeetle there are accounts for each worker, in units, with their credits and debits. There are also accounts in pesos for each employer, where contributions are held and then distributed among all its workers. Pesos and units live in separate ledgers inside TigerBeetle, and a transfer from one to the other is impossible by design. The ledger contract is in libs/ledger-client and the TigerBeetle implementation in libs/ledger-tigerbeetle.

This is what the infrastructure looks like: where a request comes in, how each connection is protected, and which pieces scale by adding replicas and which by adding CPU and RAM.

The APIs scale on their own with load. Background processes are scaled manually, because you have to watch how many clients connect to the ledger and how the queues are split. The databases scale vertically (more CPU and RAM); TigerBeetle also grows with a new cluster per fund.

The cache that can't lie

Since I don't want to query the financial engine all the time and put unnecessary load on it, I built a projection, a kind of cache. You have to be very careful here, because we're talking about money: nobody likes an app that shows them less money just because the cache is stale.

That's why the services don't fill the projection. TigerBeetle itself publishes every movement to a RabbitMQ queue, and a worker reads those events and updates the projection. An event can arrive more than once, so processing it twice doesn't change the balance (idempotency). And if an event can't be processed, the process doesn't keep going: the event goes to an error queue and the worker stops, because a gap in the movements is worse than downtime. In the end, the balance is always read from the ledger.

That projection lives in Postgres, and that's also where I generate account statements. Something I didn't anticipate is that generating statements is one of the most expensive processes. Building the end-of-day reports required by regulators is costly too. Statements are stored in an object store very similar to Amazon buckets (same S3 API).

This is the full path of a contribution, and what you see on the dashboard at each step:

The contribution goes down to TigerBeetle, and from there it takes two paths: the balance, which balance-api reads directly from the ledger, and the history, which travels through RabbitMQ to the projection and from there to the statements. The panels on the right are the ones I watch in Grafana: if "Queued messages" grows and doesn't come down, the projection is falling behind; if "Projection vs ledger" isn't zero, something doesn't add up.

Open source and portable

I tried to make the architecture as portable as possible to any cloud, and I was careful with licenses to avoid surprises. Redpanda was left out because its license (BSL) isn't open source, and I already needed RabbitMQ because it's the queue TigerBeetle publishes its events to. Redis was left out too because it changed its license in 2024, so I used Valkey, the Linux Foundation fork, instead. For files I started with MinIO and ended up with SeaweedFS when MinIO's images stopped being freely downloadable; since what I use is the S3 API, the impact was small.

Everything runs as Docker images deployed to Kubernetes: TigerBeetle, PostgreSQL, RabbitMQ, Valkey, SeaweedFS, Keycloak for identity, and Prometheus, Loki, Tempo and Grafana to see what's going on. And the code is public, under the MIT license: it's on GitHub as Fondaria.

Security: no secrets in the repo

Since I was going to publish the code, I made sure there wasn't a single secret in the repository. Local development passwords and keys are generated when you bring up the cluster (local-secrets.sh), and in production they come from a secrets manager; if one is missing, the service doesn't start. It never falls back to a default value.

On the outside, every request comes in through a gateway (Envoy) that terminates TLS, validates the token against Keycloak and limits how many requests each client can make. On the inside, services talk to each other over mTLS with Linkerd, and nobody can call a service unless they're authorized; that includes Postgres, RabbitMQ, Valkey and the object store, which only accept the services that actually use them. TigerBeetle is the exception: I didn't give it a sidecar so there's no proxy in the ledger's path, so it sits outside the mesh and is protected by NetworkPolicies, where everything is blocked by default and only the paths that are needed are opened. On top of that, traffic between nodes is encrypted with WireGuard.

The numbers

I ran the performance tests on my local machine, which is a bit old by now: the whole Kubernetes cluster runs on a single node with 10 CPUs and 16 GB of RAM. Even so, I reached about 4,000 contributions per second with three replicas of the process that allots units, and more than 6,000 with four in another run. Each contribution is two movements in the ledger, so in TigerBeetle that's on the order of 8,000 to 12,000 transfers per second.

The most interesting part is where the bottleneck is. It isn't TigerBeetle: each batch of 8,188 transfers ran in 30 to 130 milliseconds, and TigerBeetle never went above 0.2 CPUs. The limit was Postgres, which stores the evidence for every allotment, and then the path that feeds the projection. Allotting 100,000 contributions and having the projection caught up took 5.6 minutes end to end. In other words, the financial engine can take much more; what needs work is everything around it.

Then I ran the full test on Google Cloud (GKE) with a more production-like topology: 9 nodes spread across three availability zones (us-east1-b, c and d), with 6 TigerBeetle replicas, 3 Postgres instances with synchronous replication and a 3-node RabbitMQ cluster. So I wouldn't leave expensive machines running by mistake, I wrote the scripts (infrastructure/gke) with a dead man's switch in Cloud Scheduler that destroys the cluster and unlinks billing automatically if I forget to shut it down. The whole test ran for 39 minutes.

In the cloud I raised the scale to 300,000 employees across 12 companies (600,000 investment accounts and 1.2 million ledger events per business day). And there I learned something I would never have seen on my local machine: in the first runs, the daily close took 10 to 12 minutes even though the servers had CPU to spare. The problem wasn't TigerBeetle or Postgres. It was cross-zone latency combined with two queue settings: a very short batch timeout (100 ms) that sent half-empty batches of 24 rows to Postgres instead of 200, and a Spring AMQP flag that forced RabbitMQ to make a synchronous cross-zone call for every delivered message, capping throughput at 1,600 messages per second.

After raising the timeout to 2 seconds and using per-consumer prefetch, the 3 projection workers split the 16 queues between them and kept an average of 199.6 rows per batch (out of a maximum of 200), projecting more than 7,300 records per second. With that, signing up the 300,000 employees took 2 minutes 5 seconds, receiving contributions from the 12 companies took 1 minute 10 seconds, and the full end-of-day close dropped to 5 minutes 56 seconds. That close covers allotting units, projecting the 1.2 million movements, reconciling the 600,000 accounts against TigerBeetle in 4.8 seconds without a single difference, and saving the signed snapshot of 300,000 positions.

I didn't build a web page or a web app, because that's not the problem I wanted to solve, and I think with AI that part is pretty easy to do. The problem is the financial engine and how transactions are distributed; that's why the only visual you'll see is the Grafana dashboard. I also built some tools to generate fake data: the fictional company, fictional employees, contributions and the financial clock, to see how the fund gains or loses on its investments every day, how that affects the fund's total value and, in turn, the value of the units each employee holds. If you want to run it yourself, the steps are in the demo runbook.

So, can COBOL be replaced?

In short, the answer is yes: COBOL can be replaced under these conditions. Obviously, if you're a large company and can afford a mainframe, it still seems to be the best option. But if you're a small company that wants to start with a few clients and grow, you can have elasticity and scale to thousands of transactions per second (TPS), the user experience will stay good, and you'll avoid a large upfront investment.

References

• Scaling for performance with AWS Mainframe Modernization and Micro Focus, AWS, December 2022: COBOL/CICS on EC2, ~400 transactions per second per c5.large, 3,200 on c5.4xlarge, ~4,000 with 9 instances.
• What is CICS?, TechTarget, 2021: "According to IBM, CICS processes over 1.1 million transactions per second" (all installations worldwide).
• Fondaria: the code behind this post on GitHub, MIT licensed. It includes the services, the Kubernetes infrastructure and the demo tools.