NULLSYSnot a vendor
not an agency
no billable bodies©

Ticket releases are the opposite of a surprise. The on sale time is published weeks ahead, so at the exact minute it opens roughly a year of ordinary traffic arrives at once. The shop was Drupal on ECS behind an application load balancer, and it behaved the way that stack behaves against a vertical wall of requests. Target tracking autoscaling reactedarrived late to CPU, which means new tasks were still pulling images while the queue was already deep.

Most of the fix sat in front of PHP rather than inside it. During a release the overwhelming majority of requests are for a handful of URLs: the fixture page, the release landing page and the assets around them. Those moved onto CloudFront cache policies narrow enough that an authenticated supporter with a session cookie still hit the origin while everyone browsing was served from the edge. Cache keys were cut down to the query parameters that actually change the response, because a cache key that includes tracking parameters is a cache that never hits.

The traffic was never the surprise. The origin was.

Capacity was then placed before the event instead of chasing it. Scheduled scaling actions raised the ECS desired count ahead of every on sale window, sized from the previous fixture rather than a guess, and dropped it afterwards. Warm tasks and a pre pulled image turn a cold start problem into a non event.

Aurora was doing two unrelated jobs on one endpoint. Catalogue browsing and search were competing with the writes that take money. Read traffic moved to the reader endpoint and checkout stayed on the writer, so a surge of people looking could no longer exhaust connections for the smaller number of people buying. Connection limits and pool sizes were set from the instance class rather than left at framework defaults.

Everything was described in Terraform so a release configuration could be reviewed and rolled back like code. Before each fixture a k6 run reproduced the real shape of the traffic, a cliff rather than a ramp, because a gentle ramp passes and tells you nothing. That run caught two regressions before supporters did. Origin requests during a drop fell to a fraction of what they had been, and the architecture did not become clever. It became predictable, which under a scheduled wall of traffic is the same thing as fast.

A gentle ramp passes the test and tells you nothing.

Around seven hundred developers shipping services used by roughly two and a half million people, with no shared path to production. Every group had solved the same problem slightly differentlyeleven ways, and nobody could say which pipeline was authoritative.

The platform itself went under version control first. The GitLab installation, its groups, protected branches, runner registration and membership were declared in Terraform against the GitLab provider. CI was centralised the same way: instead of seven hundred hand written .gitlab-ci.yml files, one set of shared templates that projects include and override only where they genuinely differ.

A question like who granted this access should have a commit and a reviewer.

Secret scanning ran inside the pipeline rather than as a report somebody reads later. trufflehog executes as a job on every push, with verified detections failing the run, because in an estate this size the only control that works is the one nobody can forget to invoke.

Network visibility was the more interesting piece. Service to service traffic was observed with eBPF programs attached at the kernel, so there were no sidecars, no application changes and nothing to ask four hundred developers to instrument. What you get is the actual flow graph, which is reliably different from the architecture diagram, including the calls nobody documented and the ones still pointing at a decommissioned host.

A separate OpenShift cluster was configured on the operator's cloud purely for user acceptance testing, with its own projects, quotas and ingress, so the environment where the business signs off stopped being where developers experiment. Ansible handled node configuration, and F5 virtual servers and pools came from the same repositories rather than from tickets.

The last piece is the one nobody expects to be technical. Power BI reports lived on laptops and were emailed around. They moved into Git and shipped through a pipeline: automatic publish to development, promotion to test and production only after a named reviewer approves. A dashboard became a versioned artefact with a history, like everything else.

The real flow graph is always different from the diagram.

Development, staging and acceptance environments all ran on physical servers in an office, alongside fifteen projects of very different ages. Some had been deployedcopied by hand for years. At least one had no build step worth the name.

The migration ran project by project rather than as an event. Each application was containerised first, with a Dockerfile and a compose definition that reproduced its real dependencies, while the old environment kept running beside it. That is slower on paper and much faster in practice, because a failed cutover affects one client instead of forty engineers. Puppet and Ansible carried the hosts that could not be containerised yet, and Terraform described the target from the first project onward rather than being retrofitted later.

A failed cutover should cost one client, not forty engineers.

Genuinely stateless workloads went to ECS Fargate behind an application load balancer, with task definitions and service autoscaling in code. Several were not stateless: applications writing to local disk, cron jobs assuming a fixed hostname, PHP sessions on the filesystem. Those went to dedicated EC2 instances and stayed there. Forcing an old application into an architecture it cannot support is how migrations acquire a reputation.

The account structure was rebuilt alongside. Instead of one account holding everything, projects and environments were separated into their own accounts under an AWS Organization, with cross account roles for deployment and no long lived keys. The immediate benefit is blast radius. The unglamorous benefit is that consolidated billing finally answers what each client actually costs.

Monitoring consolidated onto self hosted Wazuh with a three node indexer cluster, agents on every instance, and file integrity and vulnerability detection enabled per group. Alerts route into Slack where people already are, and dashboards into Grafana. Central monitoring only works when the alert arrives where the person is, not in a console somebody has to remember to open.

With the estate in code, accounts separated and logging centralised, the information security audit stopped being archaeology and became a matter of exporting what already existed.

Consolidated billing finally answered what each client costs.

Agents entered the daily work of business analysis, quality assurance, operations and solution architecture. The interesting decision was not which model to use. It was deciding, up front and in writing, that a person stays in the middle of every one of those flows. A person still signs.

That constraint changes what gets built. An agent that drafts a test plan for a human to approve is a different system from one that executes it unattended, and only the first is honest about where accountability sits. In practice the agent removes the blank page and the tedious first eighty percent, and the person who puts their name on the output still reads it.

An agent that drafts is a different system from one that decides.

Prompts, tool definitions and system instructions live in Git, versioned and reviewed like application code, because an agent that drafts well on Monday can drift by Friday when somebody edits an instruction to fix an unrelated case. Treating agent configuration as code is the difference between a toolhabit and a rumour.

The second half of the problem was cost. Model spend behaves the way cloud spend did fifteen years ago: trivial to start, invisible until the invoice, impossible to attribute afterwards. A tracking agent watches usage across the company broken down by model and by team, so the question of whether something is worth running gets answered with numbers instead of opinions.

The pattern is the same one as everywhere else on this page. The value is not in the clever part. It is in making the boring part reliable, attributable and visible, so the clever part can be trusted enough to use. Nothing here replaced anybody. It removed the part of the day that was never the reason anyone took the job.

Model spend behaves the way cloud spend did fifteen years ago.

Ransomware reached more than sixty workstations and the virtual machines running beside them on a bare metal Hyper-V host: the domain controller carrying DNS and DHCP, the file server, and the accounting database the business runs on. Shares were encrypted, shadow copies were gone, and the backup target was reachable from the same credentials that had been used to spread with it.

The decision that decided the week was made in the first hour. Nothing would be disinfectedtrusted and nothing reused. A cleaned domain controller is a domain controller you will be arguing about for two years, because you cannot prove what is still in SYSVOL, in a scheduled task or in a service account. A new forest was built from nothing, workstations were rejoined rather than repaired into it, and the old environment was left isolated and intact for examination rather than fixed in place.

A cleaned domain controller is one you argue about for two years.

Data came back from what backups existed. The accounting database was restored and then reconciled with the finance team against paper records rather than handed back and assumed correct. Full operation resumed inside three days.

The years afterwards were the more instructive part, and none of it was dramatic. Twenty thousand documents moved off the Windows file share into SharePoint Online, which sounds simple until the paths are measured. Years of nested folders had produced paths well past two hundred and fifty characters, beyond what the sync client would accept, so the migration had to flatten and remap as it went rather than lift and shift. Permissions were rebuilt from groups instead of inherited from the mess that existed.

Sixty machines were then upgraded in place from an end of life build, with domain profiles carried across so nobody arrived on Monday to an empty desktop. Around sixty mailboxes moved to Exchange Online in staged batches with mailbox moves rather than PST export, keeping history and calendar permissions intact.

Worth saying plainly, because it comes up every time. Whether to pay never came up, and the reason it never came up is that a clean rebuild was possible. That option existed because of decisions made long before the incident.

Whether to pay never came up, because a clean rebuild was possible.

An Exchange server running as a VMware guest went down and did not come back. There was no backup of the machine itself. What survived was a copy of the mailbox databases and their logs, sitting behind a database availability group that everyone involved had quietly filed under the heading of backupwishful.

They are not the same thing, and this is the failure mode that proves it. A DAG protects you from one copy failing. It does not protect you when the thing that failed is the platform holding every copy, and it certainly does not help when there is no image of the server to bring back.

A DAG protects you from one copy failing. Not from this.

So the server was rebuilt. A fresh Windows Server install, then Exchange reinstalled at a version and cumulative update level compatible with the databases that survived, which is the detail that quietly decides whether any of this works at all. Schema and organisation objects still existed in Active Directory, so the rebuild was done with a recovery installation that reuses the existing server object rather than creating a new one. A mailbox database without its matching directory entries is a file, not a mailbox.

The databases were then checked before anybody was allowed near them. State was confirmed as clean shutdown, and where it was not, logs were replayed to bring it there rather than forcing a repair, because a hard repair discards pages and you do not find out which ones until somebody opens an old thread. Only then were they mounted and mailboxes reconnected.

Mail came back with history intact. Two things made that possible and neither was clever: the exact build number was written down somewhere, and the directory had survived. Lose either and the conversation changes from a long night to a very different discussion about how much mail is gone.

A mailbox database without its directory entries is a file.

Rolling endpoint protection across an office is a project. Rolling it across every cash machine a bank operates in a country is a logistics exercise wearing a security project as a disguise. Every ATM in the network, none of which can be rebooted at convenientany moment.

An ATM is not a workstation. Locked down Windows embedded build, fixed disk image, constrained and often metered link, limited RAM, and a vendor supported configuration you are not free to change. There is no convenient window, only a less inconvenient one.

A scan at the wrong moment is not slow. It is out of service.

Before anything shipped, somebody had to establish what was actually out there: how many machines, on which OS build and service pack, behind which link, managed by which team. That inventory did not exist in one place, and assembling it honestly took longer than the rollout.

Deployment was staged rather than pushed. A pilot group first, in locations an engineer could physically reach, left running long enough to be boring. Then region by region, each wave gated on the previous one reporting clean.

Policy was the more delicate half. Default settings assume a general purpose computer with spare capacity and somebody sitting at it. On a cash machine an on demand scan at the wrong moment is not a performance complaint, it is a device out of service. Exclusions were built around the vendor application paths and the transaction journal, scheduled scans moved to genuine low traffic hours per region, and definition updates were staggered so a whole region did not pull at once over links that could not take it. Tamper protection stayed on, and everything reported to a central console so coverage could be evidenced rather than asserted.

The result is the least interesting sentence possible, which is the goal. Full coverage, centrally reportable, with no interruption anyone noticed.

The inventory took longer to assemble than the rollout took to run.

Software asset management sounds like a procurement task and is actually an archaeology task. Before you can say whether an organisation is correctly licensed you have to establish what it actually has, and no large organisation knows. The largest exercise of its kind done in the country at the time.

Across more than six and a half thousand devices the work started with discovery rather than contracts. Agent based inventory where an agent could be deployed, agentless scans for the rest, then reconciliation against DHCP and directory records to find the machines that answer to neither. InstalledDiscovered and licensed are different numbers, and installed and actually used is a third, usually smaller than both.

Installed, licensed and actually used are three different numbers.

Then the metrics, which is where this work is either accurate or worthless. Per device, per user, per core and per processor are counted differently, and a virtualised server counted on the wrong metric can be out by an order of magnitude. Server products licensed by physical core needed the host topology, not the guest count, and clusters with live migration enabled had to be counted against every host a workload could move to.

Entitlement records live in purchase orders, in email, in vendor portals and in the memory of one person who has since left. Matching those against discovered installations produces two lists nobody enjoys: things being paid for that are not used, and things being used that were never bought. Both cost money. Only one of them is embarrassing.

The deliverable was not a number. It was an effective licence position per product, reconciled to entitlement, in a form that survives the next audit and the next renewal. Discovery is not a one time scan: devices come and go, images get rebuilt, a trial quietly becomes production. What is worth the money is a process that keeps the number true.

Counted on the wrong metric, a virtualised server is out by an order of magnitude.

Encrypted email fails for the same reason most controls fail. It works perfectly and nobody uses it, because using it asks the sender to stop, think and do an extra thing at the exact moment they are trying to finish a message.

The design removed that moment. Messages carrying the high importance flag, a control that already exists in every mail client and that people in a bank already use out of habit, were routed out through an encryption gateway. A transport rule matched on the importance header and redirected to the gateway connector; the gateway applied the key and handled delivery. The sender ticked the box they were already ticking, without doing anything new.

The best control is the one nobody has to remember.

Behind that simplicity sits the work. The rule had to catch flagged traffic without catching everything else, because a rule that is too broad becomes a rule somebody disables. Key management integrated with the directory so recipients resolve from existing objects and keys do not become a spreadsheet. Recipients outside the organisation, who have no client and no key, needed a path that does not end in an attachment they cannot open, so external delivery fell back to a secure web portal with expiry and identity verification.

Connector routing had to be explicit in both directions so that mail did not loop between the gateway and the mail platform, and the gateway had to fail closed: if it cannot encrypt, the message queues rather than going out in clear. Journaling and message tracking were kept on so that a given message could be shown to have been encrypted, because in a regulated institution a control that works but cannot be evidenced is only half a control.

What made it succeed was not the cryptography, which is a solved problem. It was choosing a trigger that already lived in the sender's hands, so adoption required no trainingno memory, no campaign and no reminders.

A control that works but cannot be evidenced is half a control.

Eighteen years of keeping other people's estates alive, across a manufacturer, a trading group, a cultural institution, a professional chamber and a public agency. Domains, mail servers, firewalls, racks and thin clients, kept running while they were replaced underneath.

Active Directory carried forward through four generations of Windows Server without a rebuild. Functional levels raised only once every controller supports them, FSMO roles moved rather than seized, SYSVOL replication migrated from FRS to DFSR before it became a blocker, old controllers demoted rather than switched off. Mostly refusingdeclining shortcuts that become somebody else's problem.

It mostly consists of refusing shortcuts that become somebody else's problem.

Mail moved through four major Exchange versions, each one a coexistence exercise rather than a switch: new version installed alongside, namespaces and autodiscover repointed, mailboxes moved in batches, public folders and connectors migrated last, legacy decommissioned only once nothing still referenced it.

Perimeter security migrated off a legacy proxy product onto FortiGate. That is less a swap than a re-derivation of every rule somebody wrote a decade earlier and never documented, rebuilding publishing rules as virtual IPs and policies, and finding out which rule nobody understands is the one holding a line of business together.

One site moved its entire user base off desktops onto thin clients, with sessions and profiles held centrally on Citrix and the device on the desk reduced to a screen and a network port. The hardware saving is the headline. The real change is that a broken device stops being a lost day and becomes a five minute swap.

One small business server went from physical hardware into a virtual machine, was upgraded there, then deliberately put back onto physical hardware, because that was what the site could support afterwards. And one greenfield build: ProLiant servers and Cisco switching racked from empty, a hundred and fifty users onto a new domain over a weekend. All judged by the same standard, how few people noticed.

A broken thin client stops being a lost day and becomes a five minute swap.

Three cloud builds close together in time that deliberately ended up with three different architectures. Putting them side by side is the clearest argument against having a favourite one. Deliberately not the same shape.

The first was a gaming software platform on conventional infrastructure: EC2 instances in an autoscaling group behind an application load balancer, static assets and media on S3 served through CloudFront, everything described in Terraform with Ansible handling what runs inside the instances. Load is predictable and steady, processes are long lived and hold state in memory. The boringcorrect architecture is the right one here, and reserved capacity actually pays for itself when the floor never drops.

Paying for idle capacity eleven months a year to serve one is the wrong answer.

The second was a global corporate step challenge, which is the opposite problem. Traffic is enormous, concentrated into a few weeks a year, and close to nothing the rest of the time. That was built fully serverless and production grade: functions behind an API gateway, a document store rather than a relational instance, queues absorbing the write bursts so the ingest path degrades by getting slower rather than by falling over. Paying for idle capacity for eleven months to serve one month is the definition of the wrong answer.

The third was a public sector application on Azure, one of the first of its kind in the country, on a .NET stack. The constraint there was neither cost nor scale. It was compliance, data residency, procurement and the fact that public sector platform choices are decided long before an architect arrives.

The lesson is the one argued about most and demonstrated least. There is no default. Load shape, spend profile, team skills and procurement rules choose the architecture, and any two of those pointing in different directions is a conversation rather than a preference. What the three did share is the reason for writing infrastructure down: a year later somebody who was not in the room asks why a component exists and what breaks if they remove it, and only the builds with their infrastructure in code can answer without archaeology.

There is no default. The load shape picks the architecture.

A legal practice built from an empty office. Tenant and identity, document storage structured around how the practice files matters rather than how the software wants to, collaboration, endpoint protection and a full Windows 11 estate joined to the tenant and managed from it.

The interesting decision in a law firm is never the technology. It is the document structure, because a structure that does not match how the firm thinks about a matter is abandoned within a month and replaced by attachments in email. SharePoint Online holds the matter libraries with metadata driven views rather than deep folder trees, OneDrive holds work in progress, and the boundary between the two is stated rather than assumed. Sharing defaults were tightened and external sharing set to require verification, because in this sector one misdirected link is the whole risk.

Certification tests your documents, not your systems.

Alongside that, two organisations taken through information security certification, one on each major cloud. The honest observation from both is worth stating plainly: certification does not test whether your systems are securedocumented. It tests whether you can evidence, in writing, that you decided something, implemented it, reviewed it, and would notice if it stopped being true.

The technical prerequisite is real but not sufficient. Conditional access and multi factor enforcement, centralised logging with a defined retention, tested restores, change control with an approval trail, a risk register that is actually maintained and an incident process somebody has walked through. Then all of it written up as a statement of applicability with each control mapped to evidence, and kept current afterwards rather than assembled in a panic every year.

One practical note from both audits. The controls that fail are almost never the technical ones, because those are visible and somebody owns them. They are the ones that depend on a person remembering: the quarterly access review, the annual restore test, the register nobody has touched since the last audit. Anything that runs on memory eventually does not run. That gap, between the system that exists and the document describing it, is the most common finding in every audit, and it is the entire service on the other page.

Anything that runs on a person remembering will eventually not run.