The continuous profiling decision record: why the profiling data collection model you chose determines your performance regression detection latency and your infrastructure overhead budget

A 32-person B2B SaaS company built project management software for creative agencies. Their engineering organization had five engineers, a platform team of two, and a monitoring setup that matched what most teams at their size had: CloudWatch metrics covering CPU, memory, and request counts per endpoint; a custom Grafana dashboard showing p50, p95, and p99 latency broken out by endpoint; and Sentry for error tracking. They did not have profiling. Their approach was reactive: when an endpoint was slow, someone added rack-mini-profiler to the staging environment, reproduced the slow condition, identified the bottleneck, and removed the profiler after the fix shipped. It had worked well for the first 10 months because every performance issue they had encountered was a discrete, reproducible problem — an N+1 query in the dashboard controller, a missing index on a foreign key, an unoptimized ActiveRecord scope that fetched unnecessary associations.

In month 11, they shipped an activity feed feature: a paginated list of recent workspace events — comments, file uploads, status changes, assignments — showing the last N actions across a customer's workspace. The initial implementation used LIMIT/OFFSET pagination. At launch, with customers accumulating 200–400 workspace events in their history, the query ran in 22ms average, 45ms p95. The feature was used heavily — more heavily than the team expected — and the customer base grew faster than the team's performance mental model. By week 7 post-launch, the top 20% of customers by workspace activity had accumulated 2,000–4,500 events in their activity history. The OFFSET pagination query's cost scales with the offset value: to return page 15 of 25 results requires the database to scan 350 rows before returning the 25 requested. At 4,000 events and typical browsing patterns that reached pages 10–18, the query was scanning 250–450 rows to return 25, at 38ms average. The latency was acceptable in isolation, but the high usage volume meant 160,000 activity feed queries per hour during peak, and the database CPU sat at 74% during the 2pm–5pm window.

The on-call engineer got paged at 3:40pm on a Tuesday: p99 latency on the activity feed endpoint had crossed 2,100ms. She opened the Grafana dashboard. API server CPU was normal — 45%. Database CPU was at 81%, elevated but not critically. The CloudWatch slow query log showed the activity feed query averaging 38ms — not in the slow query threshold, not individually concerning, just executing at extremely high volume. She could see that the database CPU was the constraint. She could not see which specific query pattern was causing the slow requests, whether it was the pagination behavior, a lock contention issue, or a different code path that had recently changed. The most recent deployment had been two days prior — a UI-only change to the feed's display formatting. She added rack-mini-profiler to the staging environment and began reproducing the slow-path queries. Reproducing the issue required a customer account with 4,000+ events and a browsing pattern that reached page 12 or higher. Production data could not be copied to staging without a data handling process that required 4 hours of lead time. She built synthetic test data — 4,200 events — and began reproducing. It took 90 minutes to reconstruct the conditions. The profiler showed the bottleneck in 3 minutes: the OFFSET clause was the issue, clearly visible in the query execution time breakdown. Connect this to the observability sampling decision record: the profiling investigation latency and the trace sampling decision are governed by the same principle — you need representative data from the time the problem is occurring, not from a synthetic reproduction after the fact; just as a trace sampling strategy that drops slow-path traces before the on-call engineer requests them forces reproduction, a profiling strategy that does not collect data before incidents forces reproduction; the cost of reproduction is the core investigation latency in both cases.

The fix was cursor-based pagination: replace OFFSET N with WHERE id < cursor ORDER BY id DESC LIMIT 25. The query became O(1) regardless of page depth. Writing and deploying the fix took two hours. Total incident duration from first page to resolution: 8 hours and 20 minutes — 90 minutes of reproduction, 3 minutes of profiling analysis, 2 hours of fix development and deployment, and 4 hours of the investigation sitting blocked while the engineer waited for test data construction. The post-incident review produced a clear finding: continuous profiling with a 48-hour rolling retention window would have held flame graphs from before and during the incident. The regression onset would have been visible as the activity feed endpoint's database time per request trending upward over 6 weeks as customer event counts grew. The investigation on the incident day would have been 10–15 minutes of comparing the pre-incident and incident flame graphs — the OFFSET scan behavior would have been visible in both, confirming it as the cause. Instead of 8 hours, the investigation would have been 30 minutes. The team added Pyroscope to their incident list of remediation items. They also estimated the cost: Pyroscope self-hosted on their existing infrastructure, at 2MB of profile data per service instance per hour, would have added roughly 4GB of storage per day across their 7 service instances — approximately $0.12/day in S3 storage. The 8-hour incident had cost approximately 3 engineer-hours, a VPs attention, and 4 hours of degraded service for the top 20% of customers.

A 55-person B2B SaaS company built a content operations platform for media companies. Their engineering organization ran 11 services — 8 Python/FastAPI microservices and 3 Go services — deployed on Kubernetes on GKE. After reading a Pyroscope blog post about continuous profiling reducing MTTR, their platform engineering team proposed adding continuous CPU profiling across all services. The proposal was approved, and one platform engineer was assigned to the rollout. She read the Pyroscope documentation, identified the Python profiling agent (pyroscope-io), and chose the configuration from the "Getting Started" guide: 100Hz sampling rate, 50-frame maximum stack depth, push-mode agent that reported to a central Pyroscope server every 10 seconds. The documentation noted this configuration was appropriate for development and staging environments. The platform engineer, whose focus was the rollout rather than performance tuning, did not flag this caveat.

She deployed the Pyroscope agent as a sidecar container to 3 of the 8 Python services in a staged rollout: the article ingestion service, the content transformation service, and the search indexing service. The three services were running at 62–68% CPU utilization under typical load. After 24 hours of monitoring, the platform engineer reviewed the Pyroscope server UI, confirmed profiles were being collected, and presented the results to the engineering team. The flame graphs were useful and immediately surfaced three unexpected hotspots that had not been visible in any metrics. The team approved expanding the rollout to all 8 Python services. The rollout completed on a Wednesday morning.

By Wednesday afternoon, the Kubernetes HPA (Horizontal Pod Autoscaler) had scaled the article ingestion service from 4 replicas to 7. By Thursday, the content transformation service had scaled from 3 replicas to 5. The infrastructure cost increase over the prior week was 28%. The platform engineer investigated: the Pyroscope Python agent at 100Hz sampling rate was adding 14–17% CPU overhead to each service instance. CPython's profiling signal handler must acquire the GIL to inspect the stack; at 100Hz, the signal arrives 100 times per second and each signal handler acquisition serializes with the main application thread's GIL release cycle. For GIL-contended services — services making frequent I/O calls and native library calls that release and reacquire the GIL — the profiling signal's GIL contention adds overhead that scales nonlinearly with the service's GIL contention level. The article ingestion service and content transformation service both made heavy use of Pillow for image processing and lxml for XML parsing — both release the GIL during the C extension execution, creating the GIL contention pattern that amplified the profiling overhead. Connect this to the capacity planning decision record: the profiling overhead budget is a capacity planning input in exactly the same sense as a feature's memory growth or a background job's CPU spike; a capacity planning process that does not include the profiling infrastructure's overhead in its resource envelope for each service will trigger autoscaler responses that appear to be load-driven but are profiling-driven, making the autoscaler's behavior harder to interpret and the infrastructure cost harder to attribute.

The team's immediate response was to add a CPU utilization check to the Pyroscope agent configuration: if CPU utilization exceeded 75%, reduce sampling rate to 20Hz; if it exceeded 85%, disable profiling entirely. This solved the autoscaler problem — with adaptive sampling, the infrastructure cost returned to baseline. But the adaptive sampling policy recreated the failure pattern the platform engineer had been trying to solve: the system collected profiles at 100Hz during baseline load (when performance was normal) and either reduced to 20Hz or disabled profiling during elevated load (when performance regressions were most likely to manifest). The three flame graph hotspots the team had been planning to investigate were in code paths that only appeared at the load levels where profiling was now being disabled. The platform engineer presented this finding to the engineering team: the configuration they had landed on was providing profiling value during normal operation but not during performance incidents. She recommended reconfiguring the Python profiling at 10Hz and measuring the overhead at that rate in a staging environment before re-deploying. At 10Hz, preliminary testing showed 2–4% overhead on the same services. The engineering team accepted the tradeoff: 10Hz profiling would miss short-lived method calls consuming less than 100ms per second, but would correctly capture code paths consuming 10% or more of CPU time — which was the class of regression they cared about detecting.

A 44-person fintech startup built a lending platform for small business financing. Their transaction processing service handled loan application evaluation: it computed creditworthiness scoring, verified identity data against external providers, ran anti-fraud checks, and — critically — calculated a set of financial projections including compound interest schedules for each loan term option. The service was written in Java 17 with Spring Boot. Month 12 of the service's production life introduced a new regulatory requirement: loan term projections had to display monthly payment amounts accurate to the nearest cent for all 36 months, computed with compound interest precision. The previous implementation used a simplified approximation formula that was fast but not regulatory-compliant for the new disclosure requirements. The engineering team replaced the approximation with an exact compound interest calculation using BigDecimal arithmetic — regulatory precision required BigDecimal rather than double because IEEE 754 floating-point rounding errors would accumulate across 36 months of multiplication.

The BigDecimal implementation was a straightforward translation of the formula: a loop over 360 months (12 months × 30 years, the maximum term), computing the outstanding balance after each month's principal and interest payment using BigDecimal.multiply(), BigDecimal.subtract(), and BigDecimal.divide() with MathContext.DECIMAL128. The loop ran 360 BigDecimal arithmetic operations per loan option, and each loan application evaluated up to 5 term options, producing 1,800 BigDecimal operations per transaction. The implementation was correct, passed all unit tests, and was deployed in a routine release. Three days after deployment, the platform's performance monitoring showed transaction p95 latency increasing from 180ms to 420ms. The increase was not dramatic enough to page the on-call engineer; it was flagged at the weekly engineering metrics review. The team decided to profile the service to identify the cause.

They deployed YourKit Java Profiler to a production-adjacent staging environment, replayed production traffic, and captured CPU profiles. YourKit's default profiling mode uses JVMTI GetAllStackTraces, which is safepoint-sampling: the profiler can only capture stack frames when the JVM is at a safepoint — a specific point in the bytecode where it is safe for the garbage collector to stop threads. YourKit reported the top CPU-consuming methods as: Jackson ObjectMapper.readValue() at 13.4% of sample time; Spring's DispatcherServlet.processDispatchResult() at 8.1%; a fraud detection API client's HTTP connection pool acquisition at 7.3%; and a variety of framework overhead methods totaling 22%. The BigDecimal compound interest loop did not appear in the top 20 methods. It appeared at position 47 as BigDecimal.multiply(), 0.4% of sample time. Connect this to the distributed tracing decision record: distributed tracing captures the elapsed time of a request across service boundaries and within service spans, and would have shown the transaction's compound interest calculation span duration increasing from <1ms to 55ms after the BigDecimal change — a direct signal that the new code was the regression source; profiling then provides the code-path attribution within that span; the two tools are complementary: tracing identifies which span regressed, profiling identifies which function within that span is responsible; when profiling alone is the investigation tool, a misleading flame graph produces a false attribution that tracing with correct instrumentation would have prevented by pointing at the new span before the profiler was asked to find the cause within it.

The team spent 3 weeks investigating the profiler's reported hotspots. They optimized the Jackson deserialization path: cached the ObjectMapper as a singleton rather than constructing it per request, reduced the loan application request schema depth by flattening two nested objects, and added Jackson's FAIL_ON_UNKNOWN_PROPERTIES=false to skip schema validation on fields the service did not use. These were real improvements — Jackson ObjectMapper construction is genuinely expensive when not cached — and the p95 latency improved from 420ms to 388ms. But the 180ms pre-change baseline remained out of reach. A new hire who had joined 3 weeks into the investigation had come from a team that had encountered JVM safepoint bias on a similar BigDecimal-heavy financial calculation service. She asked whether the team was using a safepoint-free profiler. They were not familiar with the term. She introduced async-profiler — a safepoint-free JVM profiler that uses AsyncGetCallTrace to capture stack frames asynchronously, without waiting for a safepoint. She configured it, replayed the same production traffic replay, and captured a flame graph. The compound interest loop appeared at position 1: BigDecimal.multiply() at 31.2% of CPU time, BigDecimal.divide() at 8.4%, BigDecimal.subtract() at 3.1%. The safepoint-biased YourKit profile had shown 0.4% because the BigDecimal tight loop contained no method calls or object allocations that triggered safepoints; the loop ran entirely between safepoints and was statistically invisible to JVMTI GetAllStackTraces. The fix — using double arithmetic for intermediate calculations and rounding to BigDecimal precision only for the monthly payment output values — reduced transaction p95 latency from 388ms to 172ms. The regulatory requirement was still satisfied: the rounding occurred correctly at the output boundary where the disclosure number was written, and the intermediate calculations in double were within precision tolerance for the subsequent BigDecimal rounding. The combined improvement from the Jackson optimization and the BigDecimal fix was 172ms from a 180ms baseline — effectively recovered.

Structural properties set by the profiling decision

Three structural properties are determined by the founding decisions about whether profiling is always-on, how its sampling rate is calibrated, and whether the chosen profiler technology correctly models the language runtime's execution. None of these properties are visible in your latency percentile dashboards or your APM tool's service map until a performance regression forces the investigation that profiling should enable. They are set by whether you have pre-incident flame graph history when a regression occurs, whether your profiler is collecting data during the load conditions where regressions manifest, and whether your flame graph accurately reflects where CPU time is actually spent.

Property 1: The performance regression detection latency. The performance regression detection latency is the elapsed time between a regression occurring in production and the engineering team being able to name the responsible code path with enough specificity to write a fix. It has two components: the alerting latency (how quickly a latency metric threshold fires a page after the regression starts) and the investigation latency (how long it takes the on-call engineer to move from the alert to a named code path). Continuous profiling reduces the investigation latency by providing pre-incident flame graph history for comparison. Without it, the investigation latency is dominated by reproduction time: the engineer must reconstruct the conditions under which the regression manifests, instrument the service, and reproduce the condition. Reproduction time is bounded by the complexity of the regression's triggering conditions — a regression that appears only with specific data distributions (the first failure pattern's 4,000+ event customer accounts), specific traffic compositions (a Friday payroll batch), or specific concurrency levels may require hours to reproduce in a staging environment. With continuous profiling, the investigation time is the time to compare two flame graphs — typically 10–30 minutes. Connect this property to the observability sampling decision record: the profiling baseline requirement is the same problem as the trace retention requirement — you need historical observability data from the pre-incident window to make incident investigation a comparison exercise rather than a reproduction exercise; the sampling rate for traces and the sampling rate for profiles both determine what fraction of the pre-incident evidence survives to the investigation window, and both are subject to the same fundamental tradeoff: higher sampling rates capture more evidence but consume more storage and add more overhead; the profiling decision and the trace sampling decision should be made with the same evidence-preservation goal: retaining enough pre-incident signal to diagnose the most frequent regression classes without reproduction.

Property 2: The infrastructure overhead budget. The infrastructure overhead budget is the CPU and memory consumed by the profiling infrastructure running in production, measured as a percentage of each service instance's resource envelope. The overhead budget has a critical threshold above which the operational response is to disable or throttle profiling under load — creating a load-correlated blind spot. The blind spot is structurally worse than the absence of profiling: a team that has never deployed profiling knows their performance investigation tooling is incomplete and invests in reproduction infrastructure to compensate; a team whose profiler is disabled under load may believe they have profiling coverage and skip the reproduction investment, arriving at an incident with neither pre-incident profiles nor a reproduction capability. The overhead budget is language-runtime-specific and must be measured empirically rather than assumed from documentation. JVM services at 100Hz async-profiler sampling typically add 1–3% CPU overhead — manageable at 70% base utilization. CPython services at 100Hz statistical sampling typically add 8–15% CPU overhead due to GIL interaction — not manageable at 65–70% base utilization. eBPF-based profilers (Parca, Polar Signals Cloud, Pixie) add 0.5–1% overhead regardless of language runtime because sampling occurs at the OS level via kernel BPF programs rather than via language runtime signal handlers; the tradeoff is reduced visibility into JVM-internal heap and GC behavior that JVMTI provides. The overhead budget decision should be made per language runtime, measured in a staging environment at representative load before production deployment, and never adapted downward under production load. Connect this property to the capacity planning decision record: the profiling overhead is a permanent addition to each service instance's resource envelope that must be reflected in the autoscaler's target utilization threshold; a 15% profiling overhead that is not reflected in the autoscaler configuration produces autoscaler responses that appear to be load-driven, which are harder to interpret during incidents and obscure the infrastructure cost attribution that would otherwise surface the overhead calibration problem.

Property 3: The flame graph interpretation surface. The flame graph interpretation surface is the set of code paths that are statistically visible in the profiler's output at the chosen sampling rate, given the profiler's sampling model. It is not equivalent to the set of code paths that consume significant CPU — the sampling model determines which code paths are observable, and sampling model constraints can render a code path that consumes 30% of CPU nearly invisible in the flame graph while accurately representing code paths that consume 5%. Three sampling model constraints produce systematic blind spots. Safepoint bias in JVM profilers: code in tight loops between safepoints (BigDecimal arithmetic, primitive array sorting, regex matching in tight iterations) is systematically undersampled by profilers using JVMTI GetAllStackTraces; the fix is async-profiler using AsyncGetCallTrace, which captures frames asynchronously at arbitrary points in execution rather than waiting for a safepoint. GIL serialization in CPython profilers: high-frequency signal-handler-based sampling acquires the GIL on each signal, which serializes with the application's GIL acquisition pattern and distorts the flame graph's representation of GIL-contended code paths; the fix is to use lower sampling rates (10–20Hz) or eBPF-based profilers that observe CPython's stack from the kernel without language runtime interaction. PTRACE permission constraints in containerized environments: py-spy in blocking mode requires CAP_SYS_PTRACE; in non-blocking mode, it cannot observe native stack frames below Python wrapper calls, making NumPy, SciPy, and compiled extension bottlenecks invisible in the flame graph; the fix is to grant CAP_SYS_PTRACE in the container security context (if the security model permits), or to use an eBPF-based profiler that can observe both the Python stack and the native stack without PTRACE. Connect this property to the observability platform decision record: profiling is the highest-resolution layer of the observability stack — traces identify which service and span regressed, metrics identify when the regression began, profiling identifies which code path within the span is responsible; the profiling tool selection is therefore a direct complement to the trace instrumentation and metrics instrumentation decisions, and the blind spots produced by sampling model constraints are the profiling layer's equivalent of the instrumentation gaps in the tracing and metrics layers — they are the code paths where the observability system cannot answer "what changed here."

The continuous profiling ADR: five sections

Section 1: Profiling trigger model. The profiling trigger model specifies whether profiling runs continuously, on-demand during incidents, in a staged configuration (always-on in dev and staging, on-demand in production), or event-triggered (always-on with extended retention on incident events). The trigger model is the primary determinant of whether pre-incident flame graph history exists when a regression occurs. The on-demand model — the first failure pattern's approach — has one failure mode: reproduction latency. When a regression manifests, the on-call engineer must either reproduce the condition under profiling in staging (hours of work) or deploy profiling to production during the incident (adds overhead to an already-degraded service). The staged model addresses the "no production data" failure but not the "no pre-incident history" failure: always-on in staging provides profiles from the test environment, not from production where the regression is occurring. The event-triggered model is the recommended default for most production systems: run continuous profiling always-on in production with a 24–72 hour rolling retention window; when an incident event is detected (p95 latency threshold exceeded, error rate spike, manual incident declaration), tag the current rolling window for indefinitely extended retention and continue collecting. This model provides pre-incident flame graph history for every regression that triggers an alert and keeps storage costs bounded by the rolling window size. The 24–72 hour rolling window covers the most common regression classes: regressions introduced in the current deployment (detectable immediately), regressions that emerge as data grows (detectable within the growth curve), and regressions that manifest during specific traffic patterns (detectable when the pattern recurs within the window). Connect this section to the service observability ownership decision record: the profiling trigger model and the observability ownership model interact directly — if the observability platform team owns the central profiling infrastructure (the Pyroscope server, the profile storage, the profile query interface), then the trigger model configuration applies uniformly across all services and the platform team owns the rolling window duration and event-triggered retention policy; if profiling is decentralized (each service team configures and pays for their own profiling), the trigger model will vary per team, typically defaulting to on-demand because the cost of always-on is visible per-team and the benefit of pre-incident history is invisible until the first incident; the observability ownership decision determines whether the profiling trigger model is a platform-level policy or a per-team configuration that individual teams optimize for local cost rather than cross-team incident investigation value.

Section 2: Sampling rate and overhead budget. The sampling rate calibration process must work backward from an overhead budget constraint, not forward from a fidelity requirement. Define the overhead budget first: for most production services, the profiling overhead should stay below 2% CPU to ensure it does not affect autoscaler behavior or service latency. For services running at base utilization above 75%, reduce the budget to 1% or choose an eBPF-based profiler with sub-1% overhead. With the budget defined, measure empirically: deploy the profiling agent to a staging environment at representative load, measure CPU utilization at 10Hz, 50Hz, and 100Hz, and choose the highest rate that keeps total utilization (service plus profiler) below the autoscaler's scale-out threshold minus a 15% margin. For JVM and Go services at 50–70% base utilization, 100Hz is typically within the 2% budget. For CPython services at 50–70% base utilization, start at 10Hz and measure — the GIL interaction overhead at 100Hz is 5–15× the overhead that documentation written for non-Python runtimes suggests. For CPython services above 70% base utilization, use an eBPF-based profiler (Polar Signals Cloud's parca-agent, Pixie, or Continuous Profiling for Python using eBPF from Pyroscope's eBPF agent) to avoid GIL interaction entirely. Document the chosen sampling rate and the overhead measurement data in the ADR so that the rationale is visible when the rate is revisited. The most important constraint to specify explicitly: no adaptive sampling that reduces rate or disables profiling under load. Adaptive sampling inverts the profiling value proposition: it collects profiles when the service is performing normally and stops collecting when the service is under stress — which is the condition under which regressions manifest. If the overhead budget cannot be met with always-on sampling at any useful rate, the correct response is to switch to an eBPF-based profiler or to reduce the number of concurrently profiled service instances, not to configure adaptive sampling. Connect this section to the observability sampling decision record: the sampling rate calibration process for profiling and the sampling rate calibration for distributed traces follow the same overhead-budget-first logic; the observability sampling decision record's recommendation to measure tail-sampling overhead at representative load rather than assuming a fixed overhead from vendor documentation applies identically to profiling agent configuration; both decisions are calibrated against the same capacity constraint — available CPU headroom in production — and both require empirical measurement rather than documentation-based estimation.

Section 3: Profiler technology selection. Select the profiler for each language runtime after explicitly evaluating the sampling model constraints for that runtime. The selection should not be based on dashboard UI quality, integration ecosystem, or cost alone; the sampling model constraint is a correctness property — a profiler with a misleading sampling model for your runtime produces flame graphs that will direct engineering time to the wrong optimization targets. For JVM services (Java, Kotlin, Scala): use async-profiler. It uses AsyncGetCallTrace to capture stack frames asynchronously without waiting for a safepoint, eliminating safepoint bias. It also captures native frames (JNI calls, GC internals) that JVMTI-based profilers miss. The known risks are that AsyncGetCallTrace is an unsupported extension that can produce incorrect results in corner cases involving GC and JIT compile state at the moment of the async signal; verify compatibility with your specific JVM version and JVM flags before production deployment. Avoid YourKit, JProfiler, and Java Flight Recorder in sampling mode as production always-on profilers — all use safepoint-biased sampling for their CPU profiling modes. For CPython services: use a lower sampling rate (10–20Hz) with the standard py-spy in blocking mode if CAP_SYS_PTRACE is available in the container security context; the blocking mode captures native frames and avoids the non-blocking mode's native frame blindness. If CAP_SYS_PTRACE cannot be granted, use Pyroscope's eBPF agent or Polar Signals' parca-agent — both observe the Python stack from the kernel via BPF without requiring PTRACE, capturing native frames at OS overhead levels. For Go services: use the built-in net/http/pprof endpoint combined with a scrape-and-forward agent (Pyroscope's Go agent, Grafana's go-pprof-to-pyroscope), or use eBPF-based profiling; Go's runtime profiling uses goroutine sampling with precise attribution and has no safepoint bias because Go's GC does not use safepoint semantics. For Rust, C, and C++ services: eBPF-based profilers are the correct choice for production always-on profiling — perf with eBPF backend, Polar Signals, or Pixie provide accurate stack frames at very low overhead without language runtime interaction. Connect this section to the distributed tracing decision record: the profiler technology selection decision and the distributed tracing instrumentation decision are complementary rather than competing observability investments; tracing identifies which span across which service regressed — it has end-to-end request scope but limited intra-service code path resolution; profiling identifies which function within a service is consuming CPU — it has deep code path resolution but no cross-service context; the combination of span duration anomalies from tracing and flame graph attribution from profiling produces a faster investigation path than either alone; when selecting a profiling technology, verify that it can be correlated with your trace spans via trace ID propagation, so that profiles captured during the incident window can be filtered to the spans that are regressing rather than requiring the on-call engineer to identify the regressing code path from a whole-service flame graph that may include unrelated CPU consumers.

Section 4: Data retention and baseline management. Specify the rolling window duration for always-on profiles, the event-triggered extended retention policy, and the baseline management approach for automated regression detection. The rolling window duration should be long enough to cover the regression classes you care about detecting. A 24-hour window covers deployment-induced regressions (visible immediately) and regressions that manifest within hours of a data growth threshold. A 72-hour window covers weekend-to-Monday traffic pattern shifts and regressions that manifest only on specific days of the week. Longer windows are rarely cost-justified for always-on profiling because the storage cost scales linearly and the regression detection value of week-old profiles is low — most regressions manifest within days of the change that introduced them. Event-triggered retention is not "store everything"; it is "tag the current 24–72 hour window for extended retention when an incident is declared." Store incident-tagged profiles for 90 days — long enough to support the postmortem investigation and the cross-incident comparison that identifies recurring performance regressions. For baseline management: create deployment-tagged profile snapshots — a profile collected in the first 30 minutes after each production deployment, retained for 30 days. Deployment-tagged baselines enable automated regression detection: compare the current service's function-level CPU share distribution against the baseline for the most recent deployment, alert when any function's share increases by more than 15% relative to baseline. This detection model catches regressions introduced by the deployment before they manifest as user-visible latency changes, providing the earliest possible regression signal. The automated alert threshold (15% relative change) is a starting point; tune based on false positive rate in the first 30 days. Connect this section to the on-call rotation design decision record: the profiling data retention decision determines whether the on-call engineer has pre-incident flame graph history available during the incident response window; the on-call rotation design decision — specifically the runbook content and the tooling access policies — determines whether the on-call engineer knows to look at the profiling data and has the access to query it under incident time pressure; the highest-value integration between profiling and on-call design is a runbook section for each service's common performance incident types that specifies exactly which profiling query to run and what to look for in the flame graph — this converts the profiling data from a raw resource that requires profiling expertise to interpret to a checklist step that any on-call engineer on the rotation can execute within 5 minutes of being paged.

Section 5: Flame graph ownership and on-call integration. Specify who can access profiling data, which on-call runbooks include profiling as a first-look step, and how automated profiling anomalies are routed in the incident response workflow. The access policy for profiling data is a security and privacy decision: flame graphs contain function names and call stacks from production code, and in services that process user data, the call stacks may inadvertently reflect user data patterns in their execution shape. Most engineering organizations grant all engineers read access to profiling data for their own services — the value of democratized profiling access in shortening incident investigation time outweighs the risk of profiling data misuse for internal engineering teams. Restrict cross-service profiling access to on-call rotation members and senior engineers to reduce the risk of profiling data being used to reverse-engineer a competitor's service implementation if contractors or vendors have system access. For on-call integration: add profiling as a step to the performance incident section of every service's runbook. The step should specify the exact profiling query to run (service name, time window, metric type), the comparison baseline to check against (deployment baseline, prior-week baseline, or prior-month baseline for seasonal services), and the decision rule for escalating from the flame graph to a code review (a function that has increased its CPU share by more than 20% relative to baseline is the regression target; start the investigation there). The runbook integration converts profiling from a specialized expertise tool to a routine incident response action. Without runbook integration, profiling data is used only by engineers who already know it exists, know how to query it, and have profiling expertise — typically 2–3 engineers per organization, who become bottlenecks in performance incidents because their colleagues don't know to look at profiling data or don't know what to look for. For automated anomaly routing: treat profiling-based regression alerts (function CPU share increased by more than 15% since the last deployment baseline) as a deployment health signal rather than an on-call page. Route them to the deployment author's team channel as a deployment review notification with the flame graph delta attached. Escalate to on-call page only when the profiling anomaly coincides with a user-visible latency increase. This routing model filters the automated detection signal through a second confirmation step before it creates on-call load, reducing false positive page rate without losing the early regression detection value of deployment-baseline comparison. Connect this section to the open-source extractor: the continuous profiling decisions — whether to use async-profiler or YourKit for JVM services, whether 100Hz or 10Hz is the right sampling rate for CPython services given the GIL interaction overhead, whether to grant CAP_SYS_PTRACE for native frame visibility or use eBPF as the alternative, how to configure event-triggered retention without making storage costs unbounded — are exactly the kind of multi-variable decisions that happen in AI chat conversations with a performance engineering consultant or a colleague who has hit this problem before; the written output is the profiling configuration in infrastructure-as-code, which records the sampling rate and the retention window but not the reasoning behind the safepoint-free profiler choice, the GIL interaction measurement that set the CPython sampling rate, or the decision to use eBPF for the high-utilization services; when a new platform engineer joins and asks why the CPython services are configured at 10Hz when the Pyroscope documentation says 100Hz, the reasoning is not recoverable from the configuration files — and the risk of a well-intentioned reconfiguration to the documented default is exactly the second failure pattern.

FAQ

Further reading

• Observability sampling decision record — the profiling baseline requirement and the trace sampling retention requirement are governed by the same principle: you need pre-incident observability data to make incident investigation a comparison exercise rather than a reproduction exercise; just as a trace sampling strategy that drops slow-path traces before the on-call engineer can request them forces reproduction, a profiling strategy that does not collect data before incidents forces reproduction; both decisions are calibrated against the same overhead budget constraint and both require empirical measurement rather than documentation-based estimation at the target load.
• Distributed tracing decision record — tracing and profiling are complementary layers of the same investigation stack; tracing identifies which service and which span regressed (end-to-end scope, limited intra-service code path resolution); profiling identifies which function within the span is consuming CPU (deep code path resolution, no cross-service context); when profiling alone is the investigation tool, a misleading flame graph due to sampling model bias produces a false code path attribution that a correctly-instrumented span with a duration anomaly would have avoided by pointing at the new code before the profiler was asked to find the cause within it; select a profiling tool that supports trace ID propagation so profiles can be filtered to regressing spans.
• Observability platform decision record — profiling is the highest-resolution layer of the observability stack; the observability platform decision — what is centrally owned versus per-team owned — determines whether the profiling trigger model and sampling rate calibration are platform-level policies or per-team configurations that individual teams optimize for local cost; when profiling is decentralized, teams default to on-demand profiling because the always-on cost is visible per-team and the pre-incident history value is invisible until the first incident; when profiling is a platform-owned capability, the overhead calibration and event-triggered retention policy benefit from cross-team data and the cost of always-on profiling is amortized across all services at the platform level.
• Capacity planning decision record — the profiling agent's overhead is a permanent addition to each service instance's resource envelope that must be reflected in the autoscaler's target utilization threshold; a capacity planning process that does not include profiling overhead in the resource envelope for each service will produce autoscaler behavior that appears to be load-driven but is profiling-driven, making the autoscaler's responses harder to interpret and obscuring the cost attribution that would surface the overhead calibration problem; the profiling overhead is the second failure pattern's root cause — not the profiling decision itself, but the failure to include the overhead in the capacity plan before deploying to production services running at 65–70% base utilization.
• On-call rotation design decision record — the profiling data retention policy determines whether on-call engineers have pre-incident flame graphs available during incidents; the on-call runbook design determines whether they know to look at profiling data and have the access to query it under time pressure; the highest-value integration between profiling and on-call design is a runbook section for each service's common performance incident types that specifies the exact profiling query to run and what to look for in the flame graph, converting profiling from a specialized expertise requirement to a checklist step that any rotation member can execute within 5 minutes of being paged.
• Open-source extractor — the continuous profiling decisions — whether to use async-profiler or YourKit for JVM services given safepoint bias, whether 100Hz or 10Hz is the right sampling rate for CPython services given GIL interaction overhead, whether to grant CAP_SYS_PTRACE for native frame visibility or use eBPF as the production-safe alternative, how long a rolling retention window provides adequate pre-incident coverage without storage costs exceeding budget — are exactly the kind of multi-variable technical decisions that happen in AI chat conversations with a performance engineering colleague who has encountered these failure modes before; the written output is the profiling configuration in infrastructure-as-code, which records the chosen sampling rate and retention window but not the safepoint bias measurement that determined the JVM profiler selection, the GIL interaction overhead measurement that set the CPython sampling rate, or the decision to prohibit adaptive sampling; when a new platform engineer reconfigures the CPython sampling rate to the documented default because the documentation says 100Hz and the configuration says 10Hz, the reasoning behind that choice is not recoverable from the configuration file alone.