The request came from someone who spends the day approving instructions in a queue, and it was one sentence: tell me when something lands, so I can stop keeping the tab in front of me. What we had instead was a poll. Every open tab in the back office asked /admin/queue/count every fifteen seconds, which is four requests a minute per tab, all day, against a query that counts rows in a table with 2.1 million of them. p95 on that endpoint is 31 ms and it is still the single most requested URL we own, by a factor of six.
So we spent six weeks putting browser push into an internal admin. Most of it worked. Two parts of it went wrong in ways that took production traffic to discover, and both of those are worth more than the happy path.
Registration is the easy half
A push subscription needs a service worker, because the push message is delivered to a worker that the browser can start when no page is open. The Service Workers specification, currently a Working Draft dated 25 June 2015, gives a worker six states: parsed, installing, installed, activating, activated and redundant, and a registration holds up to three workers at once, an installing worker, a waiting worker and an active worker. That three way split is the source of every surprise later on.
Registration itself is a handful of lines, with three checks in front of it. MDN’s guide is blunt about the constraints: the page must be served over HTTPS, the worker script must be on the same origin, and the script’s location caps its scope, so a worker at /js/sw.js can never control /admin/. Ours sits at the document root for that reason alone. The Push API draft of 15 December 2015 is equally blunt in section 5: user agents must implement the Push API as HTTPS only. Our admin was already behind TLS, which is one advantage of building this inside a bank.
// web/js/push.js, loaded on every admin page
(function ($) {
'use strict';
if (!('serviceWorker' in navigator) || !('PushManager' in window)) {
return;
}
if (!('showNotification' in ServiceWorkerRegistration.prototype)) {
return;
}
if (Notification.permission === 'denied') {
$('.js-push-toggle').prop('disabled', true).text('Notifications blocked');
return;
}
navigator.serviceWorker.register('/sw.js', { scope: '/' })
.then(function (registration) {
registration.update();
return navigator.serviceWorker.ready;
})
.then(function (registration) {
return registration.pushManager.getSubscription().then(function (existing) {
return existing || registration.pushManager.subscribe({ userVisibleOnly: true });
});
})
.then(function (subscription) {
return $.ajax({
url: '/admin/push/subscriptions',
method: 'POST',
data: { endpoint: subscription.endpoint }
});
})
.catch(function (error) {
window.console && console.warn('push setup failed', error);
});
}(jQuery));Notification.permission is checked first because a denial is permanent until the user goes into browser settings and undoes it, so one badly timed prompt costs you that person forever. We learned that on ourselves: the first build asked on page load, three of us clicked the wrong button out of reflex, and the only way back was chrome://settings/content. The prompt comes from a button in the header now.
userVisibleOnly: true is the only member of PushSubscriptionOptions in the Push API draft, and it declares that the subscription will only be used for messages whose effect is visible to the user. In Chrome it is not optional in practice. MDN’s Push API page puts the difference plainly: Firefox allows a quota of push messages per application, while Chrome applies no limit but requires that every push message causes a notification to be displayed.
The last piece is the one nobody expects, because it has nothing to do with either specification. Chrome subscribes through Google Cloud Messaging, and for that it wants a gcm_sender_id in a web app manifest, linked from every page with <link rel="manifest" href="/manifest.json">. The value is the project number from a Google Developer Console project with Google Cloud Messaging for Android enabled, and without it the subscribe call fails with Registration failed - no sender id provided. I spent an hour on that error convinced it was our TLS chain.
Which browsers this actually covers
Chrome has had the Push API and the Notification API since version 42, announced in March 2015, and that is the browser our operations floor mostly uses. Firefox shipped push in 44, released on 26 January this year, on desktop only, and with one caveat MDN’s compatibility table spells out in a footnote: push messages are only delivered while Firefox is running. There is no support in Internet Explorer, and Microsoft lists the Push API as not implemented in Edge. Safari has none of this either.
Safari does have notifications, and they share nothing with the code above. Apple’s guide for websites describes a separate mechanism, available since OS X v10.9, that runs over the Apple Push Notification service: a Website Push ID registered with Apple, a signed zip called a push package containing a website.json and an iconset, a device token handed out through window.safari.pushNotification.requestPermission(), and a binary interface on gateway.push.apple.com port 2195. A second implementation, not a fallback, and we did not build it.
Practically, this feature covers the part of the floor that runs Chrome, and the poll stays for everyone else. The one that stung: our standard Firefox package is the Extended Support Release, and MDN notes that push and service workers are disabled in the Firefox 45 ESR. So the Firefox users we do have get nothing, and the answer is not a code change, it is a conversation with the desktop team.
Sending it from PHP
The subscription hands you an endpoint, and that endpoint is a capability URL: anyone who has it can push to that browser, so it lives in a table nobody but the sender reads. For Chrome the endpoint looks like https://android.googleapis.com/gcm/send/<registration id>, and you cannot simply POST to it. You split off the registration id, POST to https://android.googleapis.com/gcm/send with an Authorization: key= header carrying the server API key, and put the ids in a JSON body. Google’s own advice is to detect the GCM endpoint and special case it, because nothing else works that way.
<?php
// src/AppBundle/Push/GcmSender.php
namespace AppBundlePush;
class GcmSender
{
const GCM_URL = 'https://android.googleapis.com/gcm/send';
private $apiKey;
private $subscriptions;
public function __construct($apiKey, SubscriptionRepository $subscriptions)
{
$this->apiKey = $apiKey;
$this->subscriptions = $subscriptions;
}
/**
* @param string[] $endpoints at most 1000 per call
*/
public function send(array $endpoints)
{
$gcm = array_values(array_filter($endpoints, function ($endpoint) {
return 0 === strpos($endpoint, self::GCM_URL);
}));
if (!$gcm) {
return;
}
$ids = array_map(function ($endpoint) {
return substr($endpoint, strlen(self::GCM_URL) + 1);
}, $gcm);
$ch = curl_init(self::GCM_URL);
curl_setopt_array($ch, array(
CURLOPT_POST => true,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_TIMEOUT => 10,
CURLOPT_HTTPHEADER => array(
'Authorization: key='.$this->apiKey,
'Content-Type: application/json',
),
CURLOPT_POSTFIELDS => json_encode(array(
'registration_ids' => $ids,
'time_to_live' => 300,
)),
));
$body = curl_exec($ch);
$status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
if (200 !== $status) {
throw new GcmRequestFailed(sprintf('GCM returned HTTP %d', $status));
}
$decoded = json_decode($body, true);
foreach ($decoded['results'] as $i => $result) {
if (!isset($result['error'])) {
continue;
}
if (in_array($result['error'], array('NotRegistered', 'InvalidRegistration'), true)) {
$this->subscriptions->removeByEndpoint($gcm[$i]);
}
}
}
}The HTTP connection server reference is worth reading once in full, because the failure modes are not in the status code. A request can return 200 with a body saying "failure": 3, and the per recipient errors sit in results, in request order. A 401 means the API key or the whitelisted sender IP is wrong, and anything in the 500 range means retry with exponential backoff, honouring Retry-After.
Our messages carry no data. That was not a principled choice at first, it was the only option: push payloads did not exist in Chrome until version 50. Google’s write up of payload encryption explains why the wait was long. Payloads must be encrypted client key to server, using the p256dh and auth values from the subscription, ECDH on P-256, HKDF with SHA-256, AES128 GCM, and three extra headers on the request. The Chrome team published a library for Node and said more languages were coming. There is no PHP one from them, and I am not writing elliptic curve code against a draft to save one HTTP request in an internal tool. So the worker gets an empty push and fetches the details itself:
// web/sw.js
self.addEventListener('install', function () {
self.skipWaiting();
});
self.addEventListener('push', function (event) {
event.waitUntil(
fetch('/admin/push/pending', { credentials: 'include' })
.then(function (response) { return response.json(); })
.then(function (data) {
return self.registration.showNotification(data.title, {
body: data.body,
icon: '/img/notify-192.png',
tag: 'queue-' + data.queue,
data: { url: data.url }
});
})
);
});Two things about that fetch. It needs credentials: 'include' or it goes out without the session cookie and comes back as a login page, which is a confusing half hour. And it is a second network request on a machine that may be on a slow link, which is exactly the weakness the payload work exists to remove.
On authentication, there is an Internet-Draft, Voluntary Application Server Identification for Web Push, last revised on 31 January this year, which would let an application server identify itself with its own key pair instead of a sender id and an API key. It is an individual draft with no formal standing, and Chrome today still wants the GCM key. Worth tracking, not worth building on.
The worker that would not die
We shipped a bug in the push handler on a Wednesday evening, fixed it in twenty minutes, deployed, and then watched the old behaviour continue on three machines for the next three days.
The service worker is a file, and files are cached. Our asset configuration sets Cache-Control: max-age=31536000 on everything under the web root that looks static, because that is correct for fingerprinted assets, and sw.js is not fingerprinted, because it cannot be: the registration points at one fixed URL. The spec is explicit that the update check compares the fetched script with the installed one byte for byte, and that registration.update() fetches “without consulting caches”, which is described as the same operation the user agent performs at most once every 24 hours. A year long max-age on the one file that describes your background behaviour is a trap with a slow spring.
Three changes fixed it. The worker script is now served with Cache-Control: no-cache, by an explicit rule ahead of the static rule. The page calls registration.update() on load, which you can see in the registration code above. And the worker calls self.skipWaiting() during install, so a new worker does not sit in the waiting state behind tabs that have been open since Monday. Even then a document keeps the worker it started with, so an open page keeps the old behaviour until it is reloaded, which is worth saying out loud to whoever is testing with you.
Subscriptions expire without telling you
The second failure is quieter and I do not think it ever fully goes away. A subscription is not a user record. It belongs to a browser profile on a machine, and it can be invalidated without anything happening in your application: the Push API draft says that when a permission is revoked, all push subscriptions created with that permission must be deactivated, and that a deactivated endpoint must never be reused. Google’s guide adds the operational version, that subscriptions drift out of sync with your server when a network request to save one fails, and recommends checking getSubscription() on page load and syncing.
Ours drift more than I expected. In the first six weeks we deleted 47 endpoints because GCM answered NotRegistered, which the reference says to treat as permanent: remove it and stop sending. Profile resets on shared machines are most of it. The ones that worry me are the endpoints that never come back with an error and never produce a notification either, because GCM accepting a message tells you nothing about whether anyone saw it.
The specification has an answer we cannot use yet. There is a pushsubscriptionchange event, fired at the worker when a subscription has been invalidated or is about to be, and the draft says a service worker should attempt to resubscribe while handling it. We have the handler written. In six weeks of production it has fired zero times in Chrome, so the thing that actually keeps our table clean is the page load sync and the NotRegistered sweep.
The poll is still there, at 60 seconds instead of 15, because the people on Internet Explorer still need the badge to move. That change alone cut the requests on that endpoint by three quarters, which is a better number than anything push produced. Next is the notification click handler opening the right queue item instead of the dashboard, and after that I want to know whether Chrome will speak the Web Push protocol directly, so the GCM special case in GcmSender can be deleted rather than maintained.
Sources
- Service Workers, W3C Working Draft of 25 June 2015, for the worker states, the installing, waiting and active split,
skipWaiting(), the byte for byte update comparison and the note aboutupdate()bypassing caches. - Push API, W3C Working Draft of 15 December 2015, for the HTTPS only requirement,
userVisibleOnly, deactivation of subscriptions when permission is revoked, and thepushsubscriptionchangeevent. - MDN’s Push API page, for the browser compatibility table (Chrome 42, Firefox 44 on desktop, nothing in Internet Explorer, Edge or Safari), the ESR footnote, and the difference between Chrome’s visible notification requirement and Firefox’s quota.
- MDN on using service workers, for the HTTPS restriction, the same origin rule, how scope is capped by the script’s location, and the fact that a document is only controlled after a reload.
- Push Notifications on the Open Web, for Chrome 42 shipping push, the
gcm_sender_idmanifest field, the “Registration failed – no sender id provided” error, the endpoint format and the Authorization header. - Web Push Payload Encryption, for payloads arriving in Chrome 50, the encryption scheme they require and the absence of a PHP library.
- GCM HTTP connection server reference, for the 1000 id limit,
time_to_live, the response body fields, andNotRegisteredandInvalidRegistrationmeaning delete the token. - Apple’s Notification Programming Guide for Websites, for Safari’s separate APNs based scheme, the signed push package and the binary gateway.