One EC2 box to a load balancer, RDS and S3

2,059 words, about 9 min read

The client’s entire platform was one m5.large in ap-southeast-1 with an Elastic IP, provisioned by hand in 2018 by a contractor nobody at the firm had ever met. I picked it up in June. What forced the decision was not traffic, and not the architecture diagram I had been asked to draw. It was a Tuesday afternoon in July when the box stopped answering for nineteen minutes because an unrotated log had filled the root volume, and there was nothing else to send traffic to.

Nginx, PHP-FPM 7.4, MySQL 5.7, Redis, the crontab and 61 GB of user uploads all lived on that one volume. The application is Symfony 4.4, a booking and invoicing back office, about 40 requests per second at the daytime peak and almost nothing between midnight and six. Two months later it runs on two instances behind an Application Load Balancer, with RDS for MySQL, ElastiCache for sessions, and uploads in S3. The monthly bill went from roughly 108 dollars to roughly 470. I will come back to that.

I had done this once before, on a larger fleet at my previous job. Doing it again from an empty console showed me which parts I had been carried through the first time.

The health check was the first thing I got wrong

An ALB does not send traffic to instances. It sends traffic to a target group, and a target group only forwards to targets that pass its health check. I registered the first instance, set the health check path to / because that is the obvious answer, and got this in the console two minutes later:

Target.ResponseCodeMismatch
Health checks failed with these codes: [302]

The root route redirects anonymous visitors to /login. The default success matcher is 200, and only 200, so a perfectly healthy application was marked unhealthy. You can widen the matcher to 200-399, and I did that for about an hour before deciding it was the wrong fix. A redirect proves the web server is up. It proves nothing about PHP, and nothing about the database.

So the app got a real endpoint: a controller that runs SELECT 1 against the write connection, does a PING against Redis, and returns 200 with a short JSON body or 503. Nothing else. No templating, no session start, no auth. That last part matters, because a health check that starts a session creates a session row every thirty seconds forever.

Two settings from that documentation page are worth internalising before you scale anything. The interval defaults to 30 seconds and UnhealthyThresholdCount defaults to 2, so a dying instance keeps taking requests for up to a minute. And deregistration_delay.timeout_seconds defaults to 300, which means a deploy that terminates an instance waits five minutes for in-flight requests to drain. Our longest request is a PDF batch that takes about 40 seconds, so I set it to 60 and stopped watching deploys crawl.

Sessions: ten minutes of config, a week of argument

PHP’s default session handler writes files under session.save_path. With two instances and no stickiness, every second request lands on a box that has never heard of your session ID, so users get logged out at random. The tempting fix is the load balancer’s own stickiness.enabled attribute, which pins a client to one target with a cookie. I argued against it and lost the first round.

Stickiness is not free. It pins users to a box, so a scale-in event logs out everyone on that box, an uneven hash means one instance does more work than the other, and every deploy that replaces an instance is a mass logout. It also reserves the AWSALB, AWSALBAPP and AWSALBTG cookie name prefixes, which is a small thing until somebody’s cookie audit asks what they are. We used it for two weeks as a bridge while I got Redis provisioned, then took it off.

The actual change is this file, dropped into the FPM image, with a single cache.t3.micro ElastiCache for Redis node behind it:

; /etc/php.d/50-session.ini
session.save_handler = redis
session.save_path = "tcp://app-sessions.abcdef.0001.apse1.cache.amazonaws.com:6379?read_timeout=2.5"
session.gc_maxlifetime = 43200
session.cookie_secure = 1
session.cookie_httponly = 1
session.cookie_samesite = Lax

; the default is off, and the app writes to $_SESSION from two ajax endpoints
redis.session.locking_enabled = 1
redis.session.lock_expire = 30

Set read_timeout. The phpredis session handler documentation lists it as an option on session.save_path, and without it a stalled Redis connection can hang the request for as long as your socket timeout allows. The locking settings are the other half: phpredis leaves session locking off by default, and we have two concurrent ajax endpoints that both write to the session, so we turned it on and capped the lock at 30 seconds.

The alternative I costed was the DynamoDB session handler in the AWS SDK for PHP, which is a genuinely nice piece of work: it registers through session_set_save_handler(), garbage collects through a TTL attribute called expires, and has no instance to patch. It also ships with locking off, and the docs are blunt that turning it on “can become a performance bottleneck and drive up costs”. We already needed Redis for the application cache, so a second store would have been a second thing to monitor for no gain.

Uploads: S3, and the stream wrapper I took back out

Every uploaded document was going to /var/www/app/public/uploads. On two boxes, half of them become 404s.

My first version was three lines, because the S3 stream wrapper lets you keep using file_put_contents:

$s3->registerStreamWrapper();
file_put_contents('s3://'.$bucket.'/'.$key, $contents);

It works. It also swallows failures, and the documentation says so plainly: write errors are only returned when you call fflush, they are not returned from an unflushed fclose, and they are not returned from file_put_contents at all because of how PHP implements it. I found that out on staging when a bucket policy was wrong and eleven uploads reported success while nothing arrived. So the wrapper came out and the calls went through one small service:

namespace AppStorage;

use AwsExceptionAwsException;
use AwsS3S3Client;
use SymfonyComponentHttpFoundationFileUploadedFile;

final class UploadStore
{
    private $s3;
    private $bucket;

    public function __construct(S3Client $s3, string $bucket)
    {
        $this->s3 = $s3;
        $this->bucket = $bucket;
    }

    public function put(UploadedFile $file, string $key): string
    {
        try {
            $this->s3->putObject([
                'Bucket' => $this->bucket,
                'Key' => $key,
                'SourceFile' => $file->getPathname(),
                'ContentType' => $file->getMimeType(),
                'ServerSideEncryption' => 'AES256',
            ]);
        } catch (AwsException $e) {
            throw new UploadFailed(
                sprintf('put %s failed: %s', $key, $e->getAwsErrorCode()),
                0,
                $e
            );
        }

        return $key;
    }
}

The client is built once with ['version' => 'latest', 'region' => 'ap-southeast-1'] and no keys, because the instance profile supplies credentials. Backfilling the old files was an aws s3 sync that took four hours and eleven minutes over a weekend, followed by nine days where reads checked S3 first and fell back to local disk. The fallback is deleted now. Downloads go out as presigned URLs valid for ten minutes rather than streaming through PHP, which is where the upload endpoint’s p95 improvement came from.

Cron still runs on one box, and I am not happy about it

The crontab had nine entries. Copy that host to two instances and the nightly invoice run sends every invoice twice. This is the failure mode nobody warns you about, because it does not look like an outage, it looks like a support ticket three days later.

We did the boring thing. There is a third, smaller instance that is not in the target group, is not in the Auto Scaling group, and runs nothing but the crontab and the message consumers. It is a single point of failure, the jobs are idempotent for exactly the wrong subset of cases, and if it dies at 23:00 nobody finds out until morning. The honest version of this article is that I traded a clean design for a shipping date and wrote it on the runbook as the next thing to fix, probably with a database lock so any instance can take the work.

RDS buys you a standby and charges you write latency

MySQL 5.7 moved to a db.m5.large with Multi-AZ and a 14 day backup retention. Two things about Multi-AZ that people get wrong. The standby is synchronous and it is not a read replica: the documentation says outright that you cannot use it to serve read traffic. And the same page warns that Multi-AZ instances “can have increased write and commit latency compared to a Single-AZ deployment” because of that synchronous replication. That is not a footnote. Our nightly invoice run, which is thousands of small inserts in a loop, went from 41 seconds to 58. Batching the inserts got most of it back, but the tax is real and you should budget for it before somebody notices.

The parameter group is the other early lesson: you cannot edit a default parameter group at all. Create your own before you create the instance, because half of what you want to change is a static parameter and static parameters only apply after a manual reboot. Ours carries time_zone, max_allowed_packet, long_query_time and slow_query_log, and I still had to schedule a reboot to pick up two of them. Automated backups are a storage volume snapshot of the whole instance, incremental after the first one, kept in S3, and they let you restore to any point inside the retention window. That capability alone was worth the migration. The old box had a mysqldump cron writing to the same volume as the database.

What it cost, and what I did not build

Logs used to be two files on one box. Now they are in CloudWatch Logs through the CloudWatch agent, which tails the FPM error log and the Symfony log into one group per stream, so I stopped guessing which instance served the request I am reading about. Four alarms: ALB 5xx count, target response time p95, UnHealthyHostCount above zero, and RDS free storage under 20 percent. The storage one has fired twice. Both times it was the same batch export writing to /tmp on the database. Neither time did it take the site down, which is the whole point.

Four and a half times the money. I want to be direct about that, because the write-ups that skip the invoice are the reason people expect this to be free. You are now paying for a standby database that serves no traffic, a load balancer, a Redis node, a second application instance, a third instance for cron, and NAT. What you bought is a box you are allowed to terminate.

Two things I deliberately did not build. No containers: no ECS, no EKS, no Kubernetes. This is a Symfony 4.4 monolith with one deployment unit and no scaling problem an orchestrator would solve. And none of it is in Terraform. It was built in the console, and what exists instead is a 2,000 word runbook and an Auto Scaling launch template that at least makes the instances reproducible. That is not good enough and I know it. If the second region conversation ever happens, the first sprint goes to codifying what is already there.

Sources

• Health checks for Application Load Balancer target groups: the default matcher of 200, the 30 second interval, the unhealthy threshold of 2, and the Target.ResponseCodeMismatch reason code.
• Target groups for your Application Load Balancers: target group attributes including stickiness.enabled, the reserved AWSALB cookie prefixes, and the 300 second default deregistration delay.
• Multi-AZ DB instance deployments: the synchronous standby cannot serve reads, and the warning about increased write and commit latency.
• Parameter groups for Amazon RDS: default parameter groups cannot be modified, and static parameters need a manual reboot.
• Introduction to backups: incremental snapshots stored in S3 with point-in-time restore inside the retention window.
• Amazon S3 stream wrapper: why write errors do not surface from file_put_contents.
• DynamoDB session handler: the option we costed, including its default of locking off.
• phpredis PHP session handler: session.save_handler, the save_path parameters, and the session locking INI settings.