Custom Software & Systems Engineering
Architecture & Reliability Audit for production software systems.
You may be seeing events that go missing between services, payments recorded twice, timeouts that appear only under load, or two systems that disagree about the same order.
When a system fails in ways your team cannot fully explain, or is about to carry far more load than it was built for, an independent review shows which risks are serious and which to deal with first.
An example of one payment request followed end to end. The request comes from the checkout, is validated, and is processed, including a call to the payment provider. The call times out, which is the point an audit investigates. Recovery retries the request without charging the customer a second time.
When an audit makes sense
An audit is most useful after a pattern of problems that nobody can fully explain, or before a change that would be expensive to get wrong.
Problems you can already see
Incidents keep coming back
The same kind of failure returns after each fix, and the cause is still not certain.
Events go missing or arrive twice
Messages between services are lost, duplicated or processed in the wrong order.
Systems disagree about the same data
An order, a payment or a stock level looks different depending on where you check.
Timeouts appear under load
Everything works in testing, then slows down or fails when traffic rises.
Changes ahead
A growth step is coming
More customers, a new market or a large integration will put more load on the system than it was built for.
A rewrite or migration is being discussed
You want to know what is wrong before deciding to rebuild, split or move the system.
A new team has taken over the system
The people who built it have moved on, and the team now running it needs an accurate picture of how it works.
What we examine
Every audit is scoped to your system and the questions you need answered. These are the areas an audit covers, with the kind of question we ask in each.
- 01
Service boundaries and coupling
Do the boundaries between services follow the business, or the history of the code? For one customer action to succeed, which services all have to be available, and does the design acknowledge that dependency?
- 02
Queues, messaging and event delivery
Can a message be lost between being published and being processed? What happens to messages that keep failing? Does any part of the system assume an order of events that the infrastructure doesn't guarantee?
- 03
Retries and timeouts
When a call times out but the operation actually succeeded, what happens next? Retries at several layers can multiply the load during an incident, and timeouts along a chain of calls need to fit together; we check both.
- 04
Idempotency
Can every operation that might be retried or redelivered safely run twice? This matters most where money, billing or stock is involved. An idempotent operation has the same effect whether it runs once or several times.
- 05
Transactions and data consistency
Where a business operation spans more than one database or service, we look at what keeps the parts in agreement when one step fails halfway through.
- 06
Resilience and failure modes
What does the system do when a dependency is slow rather than down? Is degraded behaviour designed, or does it happen by accident?
- 07
Performance and scaling
We look for the place where the next step in growth will hit first: a database, a shared lock, a long chain of synchronous calls, or a limit at a third-party provider.
- 08
Observability and operational risk
Can your team follow what happened to a single request or event using logs, metrics and traces? Do alerts point to real impact on users? Which parts of the system depend on one person's knowledge?
- 09
Code, where it answers a question
Some questions can only be answered in the code, such as how a client retries a payment request or when a consumer acknowledges a message. Where that is part of the agreed scope, we read the relevant code. It is a targeted review, not a line-by-line audit of the whole codebase.
What you receive
The audit ends with a written report your team can act on, and a session to go through it together.
- 01
Current-state assessment
How the system works today, including the flows that were never written down. For a team that has inherited a system, this alone can be useful.
- 02
Evidence-backed risks
Each finding names what we observed, such as a configuration, a code path, a log pattern or an incident timeline, and explains why it matters. Risks are rated by how likely they are and how much harm they could do.
- 03
Prioritised recommendations
What to change, in what order, and why that order. Changes that remove serious risk at low cost are marked as such, so the list stays useful even if only part of it gets done.
- 04
Improvement roadmap
A realistic sequence of work, sized for your team's capacity, that your engineers or we can carry out.
- 05
Findings session
A working session with your team to walk through the findings, test our conclusions against what they know, and answer questions.
Example finding, written for this page
Imagine an online shop with separate payment and order services.
P1A payment request that times out is retried and can be charged twice
- Areas
- Idempotency; Retries and timeouts
- Evidence
- In this example, the checkout retries the payment call after a timeout without an idempotency key, so the payment provider's log can show two charges for one order.
- Why it matters
- A customer can be charged twice, and each case has to be found and refunded by hand.
- Recommendation
- Send an idempotency key with every payment request, and store the provider's reply against it.
Low cost, removes a serious risk
- P2Messages that keep failing are retried without limit and can hold up the queue behind them
- P3One order can be hard to follow across services in the logs
How the audit runs
Each audit follows the same sequence. How long it takes, and how much of your team's time it needs, is agreed during scoping.
- 01
Scoping call
We agree the questions the audit has to answer, which systems are in scope, and whether reading code is part of it.
- 02
Access and confidentiality
How we get access, and how your information is protected, is agreed before anything is granted.
- 03
Interviews and evidence
We talk to the people who build and run the system, and review its architecture, configuration, logs, metrics and incident history, plus code where it is in scope.
- 04
Findings session
We go through the findings with your team, test our conclusions against what they know, and answer questions.
- 05
Written report
The assessment, the evidence-backed risks, the prioritised recommendations and the roadmap, in one document your team keeps.
- 06
Follow-up, if you want it
Your engineers can carry out the roadmap, or we can, as separate work through Custom Software Engineering.
Who carries out the audit
The audit draws on the founder's experience in distributed systems, reliability and fault tolerance. Unbelievable Software was founded in 2022 by Bartłomiej Różycki, whose background is backend systems in fintech and e-commerce.
What the audit does not do
It does not promise that the system will never fail, or any particular outcome. It shows where the serious risks are and which to deal with first.
Fixing what it finds is separate work, which your team can do, or we can.
Code is read only where it answers a question in the agreed scope. Not every audit includes code review, and none covers the whole codebase.
Questions before an audit
What access do you need?
It depends on the agreed questions. The evidence we review is architecture, configuration, logs, metrics and incident history, and code only where code review is in scope. Access is agreed before anything is granted.
How long does it take, and how much of our team's time?
Both are agreed during scoping, once we know the size of the system and the questions you need answered.
Is the audit confidential?
Confidentiality is agreed as part of scoping, before any access to your systems or information.
Do you review all of our code?
No. Where a question can only be answered in the code, and code review is part of the agreed scope, we read the relevant code. It is not an exhaustive review.
What happens after the report?
Your team can work through the roadmap on its own. If you want help, we can carry out the changes as a separate piece of work.
Start with the questions you need answered
Describe the system and what worries you about it. We reply by email, usually with questions or a suggested call.