Testing data isn’t optional anymore. When models grow, teams scale, and releases get faster, having the right testing tool becomes critical.
At one of our enterprise clients in Switzerland, we compared two approaches side by side:
🟦 NBi –a framework based on XML
🟪 CAT – an automated testing tool built for speed, reuse, and simplicity
Here’s what we learned — based entirely on real numbers from a working solution.
1. NBi vs. CAT – What They Are?
NBi
A framework used with MS Visual Studio. It relies on XML and C# and requires multiple files for each model. Tests must be copied, mapped, and managed per model. Pipelines are isolated and must be launched one by one.
CAT
A standalone tool that works with YAML. You can use it with PowerShell, command-line and Python. Tests live in one place and apply to any model with a matching structure. There’s also a UI for business users.
2. Performance comparison
One recent release used both tools:
- NBi: 25 models × 23 minutes = 575 minutes total
(plus manual work to monitor and review results)
- CAT: 2,500 tests = 23 minutes
(one pipeline, one report, zero extra steps)
The gap is clear. While NBi runs model by model, CAT runs once for everything.
3. Maintenance
Tests are reusable. You write them once. They apply to all fitting models.
With NBi, each test must be created per model. There’s no way to avoid duplication, which makes updates harder and riskier over time.
4. Logs and Reports
CAT can export logs to many formats like JSON, Excel, YAML, SQL, JUnit, and more. It has built-in templates for reports, which teams can reuse. That saves hours — or even days — after each cycle.
NBi produces only TRX logs. To get a complete picture, someone must manually compile results, often taking 1–2 days per report. Or you must invest in the automation of this (CAT comes with many forms of outputs out-of-the-box).
5. Test Creation by Business Users
CAT Studio lets business analysts build and run tests with no code. This gives control back to the people closest to the data.
NBi is too technical. Even basic tests require developer support.
6. Product Support
CAT is actively maintained. New features are shipped based on real feedback. During our project, several updates were delivered within weeks — all based on client needs.
NBi has limited updates. It depends on a single contributor and lacks dedicated support.
7. Development of Tests
CAT is far easier to implement first tests and upkeep them. Developers learn CAT relatively fast due to extensive documentation and simple language (Powershell, YAML). Developing locally can be done with a simple tool such as Visual Studio Code.
NBi, due to the fact it is a C# bootstrap project, is hard for a majority of the teams to even start with. The XML is far more restrictive then YAML, harder to read and upkeep. Users also have hard time to develop and run tests locally, as specific versions on NBi need to run in Visual Studio (not VS Code). From experience of ALL developers that used both NBi and CAT, CAT has been selected in 100% of the cases as preferred tool by the Developers.
8. Cost vs. Impact
Here’s what the numbers look like:
Category | NBi | CAT |
License | Free | €12,000/year |
Test Creation | Manual per model | Very simple test templating |
Test Runs | One-by-one | One pipeline for all |
Business User Access | Not available | Built-in with UI |
Estimated Yearly Savings | €0 | ~€20,000 |
The calculation behind that savings:
- ~11 MDs saved per release due to shorter test runs
- ~10 MDs saved on writing tests
- ~10 MDs saved by removing manual checks done by analysts
At €650 per MD, that’s ~€20,000/year, even with license included. These are real numbers, based on real project work. And they’re conservative. CAT is far easier to learn and use than NBi.
Final Takeaway
CAT isn’t just faster. It’s built to keep teams focused and testing under control — no matter how many models or environments you’re dealing with.
This isn’t a pitch. It’s what we saw on a live project with a Swiss customer.
Want to See CAT in Action?
If your current testing process feels slow, complex, or repetitive — CAT might be the change you need. Let’s talk. We’re happy to show you how CAT can work in your ecosystem, no slides needed.
