- Published on |
- 11 min read
Feature Flags: Ship to Production Without Breaking It
The $440 Million Kill Switch That Didn't Exist
On August 1, 2012, Knight Capital deployed new trading software. Within 45 minutes, a bug executed 4 million bad trades, buying 440 million—bankrupting the company.
The fix was a simple code change. But there was no way to disable the faulty feature without a full rollback, which would take down their entire trading system.
If they had a kill switch—a feature flag to instantly disable the new code—they could have stopped the bleeding in seconds.
What's a Feature Flag?
It's just an if statement that you can control without redeploying:
// The scary way: deploy = release
public PaymentResult processPayment(PaymentRequest request) {
return newPaymentProcessor.process(request);
}
// The safe way: deploy ≠ release
public PaymentResult processPayment(PaymentRequest request) {
if (featureFlags.isEnabled("new-payment-processor", user)) {
return newPaymentProcessor.process(request);
}
return legacyPaymentProcessor.process(request);
}
The difference matters at 2 AM when something breaks. Without flags, you're rolling back the entire deployment (20-60 minutes of downtime, maybe more if you panic). With flags, you flip a toggle and the new code stops running. Instantly.
Types of Flags
There are really just four kinds:
Release flags hide unfinished work. Your half done checkout flow can sit in production behind a flag while you iterate on it. Delete these once the feature ships.
Ops flags are kill switches. Email provider down? Disable outbound emails without a deploy. These stick around forever.
Experiment flags are for A/B testing. "Does the green button convert better?" Run the test, pick a winner, delete the flag.
Permission flags gate features by plan. "Only enterprise customers get SSO." These are basically entitlements.
The lifespan matters. Release and experiment flags should die within weeks. Ops and permission flags are permanent. If you don't track this, you'll end up with 500 flags and no idea which ones are safe to remove.
Building One
Here's a basic interface:
public interface FeatureFlagService {
boolean isEnabled(String flagKey);
boolean isEnabled(String flagKey, User user);
String getVariant(String flagKey, User user);
}
And the flag itself:
@Entity
public class FeatureFlag {
private String key;
private boolean enabled;
private int rolloutPercentage;
private Set<String> userWhitelist;
private Set<String> userBlacklist;
private List<TargetingRule> rules;
private Instant staleAfter;
}
The evaluation logic has a specific order. Get this wrong and you'll confuse yourself:
public boolean evaluate(String flagKey, User user) {
FeatureFlag flag = getFlag(flagKey);
// Kill switch trumps everything
if (!flag.isEnabled()) return false;
// Explicit "never this user"
if (flag.getUserBlacklist().contains(user.getId())) return false;
// Explicit "always this user" (useful for testing)
if (flag.getUserWhitelist().contains(user.getId())) return true;
// Rules like "plan = enterprise"
for (TargetingRule rule : flag.getTargetingRules()) {
if (matchesRule(rule, user)) return rule.isEnabled();
}
// Fall back to percentage rollout
return isInRolloutPercentage(flagKey, user.getId(), flag.getRolloutPercentage());
}
The percentage rollout is the interesting bit:
private boolean isInRolloutPercentage(String flagKey, String userId, int percentage) {
String hashInput = flagKey + ":" visitorId;
int bucket = Math.abs(hashInput.hashCode() % 100);
return bucket < percentage;
}
This gives every user a consistent "bucket" from 0-99. Set rollout to 10%, and users in buckets 0-9 get the feature. The same user always lands in the same bucket, so they don't randomly flip between the old and new experience. That would be jarring.
Gradual Rollouts
Don't flip features on for everyone at once. That's how you get Knight Capital situations.
Percentage rollout
Start at 1%. Watch your error rates. Nothing on fire? Bump to 5%. Then 10%. Then 25%. Then 50%. Then everyone.
@PatchMapping("/{flagKey}/rollout")
public FeatureFlag updateRollout(String flagKey, int newPercentage) {
FeatureFlag flag = flagRepository.findByKey(flagKey);
flag.setRolloutPercentage(newPercentage);
flagRepository.save(flag);
auditService.log("ROLLOUT_UPDATE", flagKey, newPercentage);
return flag;
}
If something breaks at 5%, only 5% of users are affected. Toggle it off, fix the bug, start over. Much better than "everyone's checkout is broken."
Ring based rollout
Microsoft does this. Instead of random percentages, you roll out to specific groups:
public boolean isEnabledForRing(String flagKey, User user) {
RolloutRing ring = getFlag(flagKey).getCurrentRing();
return switch (ring) {
case CANARY -> isInternalTeam(user);
case EARLY_ADOPTERS -> user.isBetaUser();
case SLOW_RING -> isInRollout(flagKey, user, 10);
case FAST_RING -> isInRollout(flagKey, user, 50);
case GA -> true;
};
}
Canary is your team. You eat your own dogfood first. Early adopters are users who've opted into beta—they expect some rough edges. Slow ring is 10% of production. Fast ring is 50%. GA is everyone.
The idea is that bugs hit people who can handle them first. Your team can debug. Beta users signed up for this. By the time it hits general availability, you've already found the weird edge cases.
Attribute based targeting
Sometimes you want to target by user properties, not random selection:
// Enterprise customers pay more, they get features first
new TargetingRule("plan", "equals", List.of("enterprise"), true)
// Legal compliance: US only until we figure out GDPR
new TargetingRule("country", "equals", List.of("US"), true)
// Mobile app isn't ready yet
new TargetingRule("deviceType", "equals", List.of("desktop"), true)
Kill Switches
This is the whole point. When something breaks, you need to turn it off fast.
public void killFeature(String flagKey, String reason, AdminUser admin) {
FeatureFlag flag = flagRepository.findByKey(flagKey);
int previousPercentage = flag.getRolloutPercentage();
flag.setEnabled(false);
flag.setRolloutPercentage(0);
flagRepository.save(flag);
cacheInvalidator.invalidate("flag:" + flagKey);
alertService.sendUrgent("KILL SWITCH: " + flagKey + " by " + admin.getEmail());
auditService.log("KILL_SWITCH", flagKey, reason, previousPercentage);
}
Notice the cache invalidation. If you're caching flag values (you probably should be), you need to bust that cache immediately. A kill switch that takes 5 minutes to propagate isn't much of a kill switch.
You can also automate this:
@Scheduled(fixedRate = 60_000)
public void checkFeatureHealth() {
for (FeatureFlag flag : getEnabledFlags()) {
HealthMetrics health = metrics.getFlagHealth(flag.getKey());
if (health.getErrorRate() > 0.05 && health.getSamples() > 100) {
killFeature(flag.getKey(), "AUTO: Error rate " + health.getErrorRate());
}
}
}
If error rate spikes above 5% with enough samples to be statistically meaningful, kill it automatically. You can argue with the machine later. The important thing is that it stopped the bleeding.
The Cleanup Problem
Here's what nobody tells you about feature flags: they accumulate.
Year one, you have 50 flags. All clean, all documented. Year two, 200 flags, some of them stale. Year three, 500 flags, and nobody knows what half of them do. By year four, engineers are afraid to delete anything because "what if it breaks something?"
Uber reportedly had over 5,000 flags at one point.
You need to detect stale flags proactively:
@Scheduled(cron = "0 0 9 * * MON")
public void detectStaleFlags() {
for (FeatureFlag flag : flagRepository.findAll()) {
boolean isStale = false;
String reason = "";
// If it's been at 100% for a month, the flag is pointless
if (flag.getRolloutPercentage() == 100 &&
flag.getUpdatedAt().isBefore(Instant.now().minusDays(30))) {
isStale = true;
reason = "100% rollout for 30+ days";
}
// Nobody's even evaluating this flag anymore
if (accessLog.getLastAccess(flag.getKey())
.isBefore(Instant.now().minusDays(30))) {
isStale = true;
reason = "No usage in 30 days";
}
if (isStale) {
notifyOwner(flag.getCreatedBy(), flag.getKey(), reason);
}
}
}
When you remove a flag, keep only the winning branch. If the flag was at 100%, delete the else branch. If it was at 0%, delete the if branch. Update tests. Delete the flag from the database after the code ships.
Common Mistakes
Checking flags in a loop:
// This hits the database N times
for (Item item : items) {
if (flags.isEnabled("new-pricing", user)) { ... }
}
// Check once
boolean useNewPricing = flags.isEnabled("new-pricing", user);
for (Item item : items) {
if (useNewPricing) { ... }
}
Nesting flags:
// This has 8 possible states. Good luck testing.
if (flags.isEnabled("a")) {
if (flags.isEnabled("b")) {
if (flags.isEnabled("c")) { }
}
}
Magic strings everywhere:
// Typo. Silent bug.
if (flags.isEnabled("chekcout-v2")) { }
// Use constants
static final String CHECKOUT_V2 = "checkout-v2";
if (flags.isEnabled(CHECKOUT_V2)) { }
Bad naming:
// What feature? Which John?
"new-feature"
"johns-flag"
"disable-thing" // Double negative: is it enabled or disabled?
// Better
"release.checkout-v2"
"ops.elasticsearch-enabled"
"experiment.pricing-annual-discount"
Testing Both Paths
This is easy to forget. You need to test both branches:
@ParameterizedTest
@ValueSource(booleans = {true, false})
void shouldWorkRegardlessOfFlag(boolean flagEnabled) {
when(flags.isEnabled("checkout-v2", any())).thenReturn(flagEnabled);
CheckoutResult result = service.checkout(request, user);
assertThat(result.isSuccessful()).isTrue();
}
If you only test with the flag on, you'll find out the hard way that the old path broke at some point.
Build vs Buy
If you have fewer than 50 simple flags and basic needs, build it yourself. It's not that hard.
If you need rich targeting, A/B testing with statistical significance, audit logs for compliance, or you just don't want to maintain this forever—use a service. LaunchDarkly and Split are the big ones. Flagsmith and Unleash are self hosted options.
The Point
Feature flags shift your relationship with production:
| Before | After |
|---|---|
| Deploy = Release | Deploy ≠ Release |
| Bugs mean rollback | Bugs mean toggle off |
| Launch day is scary | Launch day is Tuesday |
| All or nothing | 1% → 5% → 10% → 100% |
Knight Capital lost $440 million because they couldn't flip a switch. You can learn that lesson without paying the price.
Pick your next feature. Wrap it in a flag. Deploy it off. Roll out to 1%. Watch the metrics. If something looks wrong, toggle it off. If everything's fine, bump to 10%. Then everyone.
That's how you ship to production without breaking it.
Further Reading
Mustafiz Kaifee
@mustafiz_kaifee