MJ Convento

/

Ten weeks of PHP 7 in production

1,893 words, about 8 min read

The deploy that put PHP 7.0.2 on all six of our app servers finished at 02:41 on a Saturday in January. I did not check latency first. I sat watching the php-fpm error log for about eleven minutes, because the thing that scared me was never slowness. It was the category of failure that used to be a plain fatal error and is now a thrown object our code was not catching.

Ten weeks have passed since 7.0.0 shipped on 3 December. Eight of those weeks we have had PHP 7 serving real traffic, the last month with all six boxes on it. We are still on Symfony 2.8 LTS, still on MySQL 5.6, still nine developers, still Jenkins. Only the runtime changed, and that turned out to be the interesting part: most of the work was mechanical, and the small remainder was genuinely hard.

Why we waited for 7.0.2

We had RC 8 on a staging box in late November and the suite was green, which is the kind of evidence that makes people want to ship on day one. We did not. One app server went to 7.0.1 on 21 December, and the other five stayed on 5.6 until 16 January.

Reading the 7.0 ChangeLog is a decent argument for that patience. 7.0.1 (17 December) fixed a segfault with opcache.huge_code_pages=1 (bug #70977), try { } finally { } creating infinite chains of exceptions (#70944), and yield from incorrectly marking a valid generator as finished (#70904). 7.0.2 (7 January) fixed a segfault with return type hinting (#71092) and a performance degradation in ArrayIterator with large arrays (#71153). That last one is not academic here. The COD settlement report walks roughly 400,000 rows through an iterator every night.

7.0.3 landed on 4 February with the bundled PCRE upgraded to 8.38 and a set of phar and WDDX fixes carrying CVE numbers, so that went out the Monday after. If you are still on 5.6, my advice is to skip the .0 of a major version and let other people find the opcache crashes. Two patch releases of lag cost us nothing.

The mechanical half

Most of the diff was boring, which is the best thing you can say about a language migration. The removed extensions list is short: ereg, mssql, sybase_ct, mysql. Three of those we never had. The fourth we did, in exactly the place you would guess.

PHP Fatal error:  Uncaught Error: Call to undefined function mysql_real_escape_string()
 in /var/www/app/legacy/reports/CodSettlement.php:118

That file dates from 2013 and predates the Symfony app around it. It opened its own connection because whoever wrote it (me) wanted a second cursor and did not want to think about the ORM. Eleven call sites across four files, all moved to PDO. The manual’s API comparison page treats mysqli and PDO_MySQL as equally recommended, and we chose PDO because the rest of the codebase already had. Two hours.

The rest of the mechanical work, in rough order of how many files it touched:

  • A courier webhook reading $HTTP_RAW_POST_DATA, which is gone along with the always_populate_raw_post_data setting. Now php://input.
  • One admin template using <script language="php">. ASP tags and script tags are both removed. It was a print stylesheet nobody had opened since 2014.
  • Two vendored libraries with PHP 4 style constructors, which now emit E_DEPRECATED when they are the only constructor in a class. Our logs gained about 4,000 deprecation lines an hour until we upgraded them.
  • Four func_get_args() calls, which now report the current parameter value rather than the one passed in. One of them mutated its parameter first, and it was a logging helper, so the symptom was a wrong log line rather than a bug.

None of that needed judgement. Grep, patch, run the suite, move on.

The Error hierarchy ate our job runner

Here is the part that cost a real evening. PHP 7 converts many fatal and recoverable fatal errors into thrown objects, and those objects do not extend Exception. They extend Error, and both implement the new Throwable interface. The Throwable RFC is explicit about why the split exists: the names were chosen so that catch (Exception $e) visibly does not catch them.

Our queue worker had this shape, and had had it since 2014:

<?php
// src/AppBundle/Worker/JobRunner.php
try {
    $handler->handle($job);
} catch (Exception $e) {
    $this->logger->error('job failed', ['job' => $job->id(), 'err' => $e->getMessage()]);
    $this->queue->retry($job);
}

Under 5.6, a bad argument into a type hinted method was a recoverable fatal error that our error handler turned into an ErrorException, which that block caught, which meant the job got retried and logged. Under 7.0 the same call throws a TypeError, the catch does not match, and the worker dies:

PHP Fatal error:  Uncaught TypeError: Argument 1 passed to AppBundleRateCalculator::forWeight()
 must be of the type integer, string given, called in
 /var/www/app/src/AppBundle/Worker/RateHandler.php on line 61 and defined in
 /var/www/app/src/AppBundle/Rate/Calculator.php:24

Total damage: 1,900 jobs stuck over about 40 minutes, because supervisord restarted the worker, the worker picked the same poisoned job, and died again. The fix is one line, catch (Throwable $e), but finding it meant understanding that the Error hierarchy is deliberately separate. We log and rethrow Error now instead of retrying, because a TypeError is a bug in our code and retrying a bug forever is not error handling.

The same shape bit us again in a place I would not have thought to look. A default handler registered with set_exception_handler() that type hints Exception causes a fatal error when an Error is thrown, which the manual spells out. Ours was in a bundle we wrote, and it had never once been wrong in two years.

The other change that needed actual thought was the uniform variable syntax. Indirect access now evaluates strictly left to right, so $foo->$bar['baz'] means ($foo->$bar)['baz'] and not $foo->{$bar['baz']}. Our CSV exporter had a field mapper doing exactly that:

<?php
// 5.6 read this as $row->{$map['source']}
$value = $row->$map['source'];

// what it has to say now
$value = $row->{$map['source']};

No error. No warning. A column of empty strings in a partner’s reconciliation file, noticed by the partner. Braces are valid in both 5.x and 7, so make that change before you upgrade, while you still have a working comparison.

Numbers from our own boxes

The release announcement claims PHP 7 is up to twice as fast as PHP 5.6, and the phpng benchmark page carries the numbers behind it: the same 1,000 request WordPress 3.6 run went from 26.756 seconds on the pre-refactor master to 10.398 seconds at the 7.0.0 release, with CPU instructions for 100 requests falling from 9.4 billion to 2.37 billion. A real measurement of a real application, and also not your application.

Ours, last four weeks against the four before the cutover, same six boxes, same traffic shape, peak around 310 requests per second:

MeasurePHP 5.6PHP 7.0.3
p95, booking search840 ms470 ms
p95, tracking lookup210 ms115 ms
p99, booking search2.6 s2.1 s
Mean php-fpm worker RSS47 MB26 MB
App server CPU at peak68%39%

Roughly 1.8x on the endpoints that are mostly PHP, and very little on p99, because our p99 is MySQL waiting on a query with a bad index rather than the engine doing work. That is the honest summary of what the new engine buys you: the compacted zval and hashtable layouts and the reduced allocation churn make PHP code cheaper, and they do nothing for a slow join. We had budgeted two more app servers for Q2 and cancelled them, raising pm.max_children from 40 to 70 per box instead.

Coercive or strict, the argument we had twice

Scalar type declarations were the only feature that produced a real disagreement. The RFC gives you two modes: coercive by default, or declare(strict_types=1); as the first statement in a file, which makes calls in that file strict. Three people wanted strict everywhere. Two wanted nothing, on the grounds that coercion is what PHP has always done.

What settled it was reading the scoping rules properly. For parameters, the mode is decided by the file making the call. For return types, by the file defining the function. So a strict file calling into a coercive library is still strict at the boundary, and int to float widening is the one conversion strict mode still allows. That means you can adopt it file by file without a flag day, and without asking vendors for anything.

<?php
declare(strict_types=1);

namespace AppBundleRate;

class Calculator
{
    public function forWeight(int $grams, float $multiplier): int
    {
        return (int) ceil($grams * $multiplier / 1000) * 100;
    }
}

My recommendation, which is now our rule: strict_types=1 at the top of every new file, and no retrofitting. Coercive mode will happily turn "12kg" into 12 with a notice, and a notice is something we have 40,000 of a day. Under strict mode that same call stops at the door with a message naming the argument and the caller. Retrofitting old files is the trap. Every one you convert can turn a silent coercion somewhere upstream into a thrown TypeError, and you find out in production, not in the suite.

What we left alone

We use ?? everywhere already, because it removes a real class of undefined index notice, and <=> in every comparator we own. intdiv() replaced three hand written floor() divisions in the rate table. Those are free.

Return type declarations are on concrete classes only, not on interfaces. Adding one to an interface forces every implementor to match, including two in a partner client library that still runs on their PHP 5.5, and that library is why one directory of our own repository cannot use PHP 7 syntax at all. Anonymous classes are in tests and nowhere else. Generator delegation would clean up the importer, and I am not touching it until the memory leak with consecutive yield from fixed in 7.0.3 has been out a while longer.

We looked hard at HHVM last year and did not take it. HHVM 3.11.0 was tagged on 9 December and it is genuinely fast. It is also a second runtime with its own extension story, and with nine developers I could not justify production running something different from our laptops. PHP 7 let us keep one runtime, worth more to us than the last stretch of speed.

Still open: eleven files in the reporting module have no tests worth the name, so I do not actually know the PDO conversion is correct beyond the two reports finance checks weekly. And I want to know why our p99 barely moved. If that is all MySQL, the next quarter is an indexing quarter, and PHP 7 will have been the cheap win that bought the time for it.

Sources