Record Metrics for Reminder #4831

Vyom-Yadav · 2024-10-27T17:44:16Z

Summary

Provide a brief overview of the changes and the issue being addressed.
Explain the rationale and any background necessary for understanding the changes.
List dependencies required by this change, if any.

Fixes #(related issue)

Change Type

Mark the type of change your PR introduces:

Bug fix (resolves an issue without affecting existing features)
Feature (adds new functionality without breaking changes)
Breaking change (may impact existing functionalities or require documentation updates)
Documentation (updates or additions to documentation)
Refactoring or test improvements (no bug fixes or new functionality)

Testing

Outline how the changes were tested, including steps to reproduce and any relevant configurations.
Attach screenshots if helpful.

Review Checklist:

Reviewed my own code for quality and clarity.
Added comments to complex or tricky code sections.
Updated any affected documentation.
Included tests that validate the fix or feature.
Checked that related changes are merged.

evankanderson

A bunch of comments, mostly aiming to help simplify your life. 😁

evankanderson · 2024-10-28T18:44:56Z

internal/reminder/metrics/metrics.go

+	// For tracking average calculation
+	// TODO: consider persisting this to avoid reset on restart (maybe)
+	totalBatches   int64
+	totalReminders int64


Rather than storing these as int64s, why not use an Int64Counter? That will automatically record the total, but also that the metric is cumulative over time.

evankanderson · 2024-10-28T18:46:18Z

internal/reminder/metrics/metrics.go

+	// Current number of reminders in the batch
+	BatchSize metric.Int64Gauge


Why not use an Int64Histogram for this, which will also give you the Average metric you're tracking next (approximately, but with more-detailed frequency information, which is probably interesting).

evankanderson · 2024-10-28T18:48:05Z

internal/reminder/metrics/metrics.go

+	0,    // immediate
+	10,   // 10 seconds
+	20,   // 20 seconds
+	40,   // 40 seconds
+	80,   // 1m 20s
+	160,  // 2m 40s
+	320,  // 5m 20s
+	640,  // 10m 40s
+	1280, // 21m 20s


While powers of 2 are generally handy, I'd tend to align this more with human units, e.g.:

0, 10, 30, 60, 120, 300, 600, 900, 1800, 3600, 7200

I'm not sure what frequency we'll run reminder at, but 2 hours seems reasonably pessimistic for now -- it's really annoying when you run into problems and your frequency distribution has always been "it's not so good".

Given that our expected value for latency is 1/2 scan interval, I could even be convinced that < 1 minute is not particularly interesting (which could trim a couple buckets if you're worried about it -- I'm not!).

Updated with:

var delayBuckets = []float64{ 60, // 1 minute 300, // 5 minutes 600, // 10 minutes 1800, // 30 minutes 3600, // 1 hour 7200, // 2 hours 10800, // 3 hours 18000, // 5 hours 25200, // 7 hours 36000, // 10 hours }

evankanderson · 2024-10-28T18:52:34Z

internal/reminder/metrics/metrics.go

+	// Update running average
+	m.totalBatches++
+	m.totalReminders += size


You should aim to get the metrics code to do this for you automatically -- in particular, if we end up with two reminder processes running, getting a global average from reminder_avg_batch_size is impossible without knowing the numerator and denominator for each process. Exporting them as two cumulative totals allows aggregation across multiple processes.

Good point, I think this should be doable now for multiple reminder processes as we can aggregate metrics which are being exposed.

evankanderson · 2024-10-28T18:55:35Z

internal/reminder/reminder.go

+				remMetrics.SendDelay.Record(ctx, (time.Now().Sub(reminderLastSent.Time) - r.cfg.RecurrenceConfig.MinElapsed).Seconds())
+			} else {
+				// TODO: Should the send delay be zero if the reminder has never been sent?
+				remMetrics.SendDelay.Record(ctx, 0)


This will produce a bunch of +Inf values in the histogram bucket, which will disturb any sort of Nth percentile.

It's probably better to either export nothing here, or export it under some different counter metric ("new reminders").

Exposing a new metric now.

evankanderson · 2024-10-28T19:11:48Z

internal/reminder/metrics/provider.go

+	otel.SetMeterProvider(mp)
+
+	// Create metrics
+	meter := mp.Meter("reminder-service")


Suggested change

meter := mp.Meter("reminder-service")

meter := mp.Meter("reminder")

evankanderson · 2024-10-28T19:14:07Z

internal/reminder/metrics/provider.go

+}
+
+// Shutdown gracefully shuts down the metrics server
+func (p *Provider) Shutdown(ctx context.Context) error {


If you make this method private, then cancelling the context to Start or NewProvider becomes the only way to reach this code, which simplifies your lifecycle over having multiple callers.

(among other things, you can guarantee that p is not nil, because you don't call the shutdown method unless p is non-nil.)

evankanderson · 2024-10-28T19:17:35Z

internal/reminder/metrics/provider.go

+func (p *Provider) Metrics() *Metrics {
+	return p.metrics
+}


My main suggestion is to remove the Provider object in favor of a static setup function with cancellation, but if you do keep it, I'd suggesting making Metrics exported, rather than needing to add an accessor function.

evankanderson · 2024-10-28T19:18:32Z

internal/reminder/reminder.go

@@ -143,6 +163,11 @@ func (r *reminder) sendReminders(ctx context.Context) error {
 		return fmt.Errorf("error creating reminder messages: %w", err)
 	}

+	remMetrics := r.metricsProvider.Metrics()
+	if remMetrics != nil {
+		remMetrics.RecordBatch(ctx, int64(len(repos)))


If you make batches a histogram, this can just be a Record() on that histogram.

evankanderson · 2024-10-28T19:20:45Z

internal/reminder/metrics/metrics.go

+func (m *Metrics) RecordSendDelay(ctx context.Context, delaySeconds float64) {
+	m.SendDelay.Record(ctx, delaySeconds)
+}


It looks like this function isn't doing anything -- you're already exposing SendDelay, and there's not much semantic benefit to foo.RecordSendDelay(...) over foo.SendDelay.Record(...). That also saves you from needing to write a comment.

If you transform BatchSize into a histogram as well, then you might be able to get rid of all the methods on Metrics, which would be nice.

github-actions · 2024-11-28T02:42:53Z

This PR needs additional information before we can continue. It is now marked as stale because it has been open for 30 days with no activity. Please provide the necessary details to continue or it will be closed in 30 days.

Vyom-Yadav · 2025-01-03T07:24:29Z

Opening this, I'm working on this.

coveralls · 2025-01-03T12:39:51Z

coverage: 54.92% (-0.2%) from 55.145%
when pulling 6dc6aa3 on Vyom-Yadav:add-metrics-to-reminder
into a6d9b22 on mindersec:main.

Signed-off-by: Vyom Yadav <[email protected]>

Vyom-Yadav

@evankanderson Because we reused eventer code in a6d9b22, which created pubWithMetrics using minder namespace, there are some non-reminder metrics being outputted by reminder (e.g. minder_eventer_publish_time_seconds_bucket).

I think this can cause problems, but I assume them to use the same infra, so not sure about how big of a problem that is. (Is this something we want to tackle in this PR?)

Also updated the code to remove provider and have a fire and forget setupMetricServer function

Vyom-Yadav · 2025-01-04T17:28:48Z

internal/reminder/reminder.go

+		if r.metrics != nil {
+			// sendDelay = Now() - ReminderLastSent - MinElapsed
+			reminderLastSent := repo.ReminderLastSent
+			if reminderLastSent.Valid {
+				r.metrics.SendDelay.Record(ctx, (time.Since(reminderLastSent.Time) - r.cfg.RecurrenceConfig.MinElapsed).Seconds())


I'm still not sure about this metric. Let's say a repo has been frequently updated every hour (due to GH webhook events) and r.cfg.RecurrenceConfig.MinElapsed is 2 hours. In this case, let's say after sending a reminder 18 hours back, every hour the repo was updated by GH webhook and wasn't eligible for a reminder. Now when it becomes eligible for the reminder SendDelay would be recorded as 18h which seems to be very wrong in this case. (I think we discussed this previously too)

Vyom-Yadav · 2025-01-04T17:29:49Z

internal/reminder/reminder.go

+				// Recording metric for first time reminders (after how long was the initial reminder sent)
+				newSendDelay := time.Since(repoToLastUpdated[repo.ID]) - r.cfg.RecurrenceConfig.MinElapsed
+				r.metrics.NewSendDelay.Record(ctx, newSendDelay.Seconds())


Added a new metric for first time reminders. (I'm thinking this is the definition of sendDelay and what we have above is wrong)

evankanderson reviewed Oct 28, 2024

View reviewed changes

github-actions bot added the Stale label Nov 28, 2024

github-actions bot closed this Dec 29, 2024

Vyom-Yadav reopened this Jan 3, 2025

Vyom-Yadav removed the Stale label Jan 3, 2025

Vyom-Yadav force-pushed the add-metrics-to-reminder branch from 20f38c1 to b92b0de Compare January 3, 2025 12:27

Vyom-Yadav force-pushed the add-metrics-to-reminder branch from b92b0de to ccde7be Compare January 4, 2025 17:21

Record Metrics for Reminder

6dc6aa3

Signed-off-by: Vyom Yadav <[email protected]>

Vyom-Yadav force-pushed the add-metrics-to-reminder branch from ccde7be to 6dc6aa3 Compare January 4, 2025 17:24

Vyom-Yadav commented Jan 4, 2025

View reviewed changes

Vyom-Yadav marked this pull request as ready for review January 4, 2025 17:32

Vyom-Yadav requested a review from a team as a code owner January 4, 2025 17:32

Vyom-Yadav requested a review from evankanderson January 4, 2025 17:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Record Metrics for Reminder #4831

Record Metrics for Reminder #4831

Vyom-Yadav commented Oct 27, 2024 •

edited

Loading

evankanderson left a comment

evankanderson Oct 28, 2024

Vyom-Yadav Jan 4, 2025

evankanderson Oct 28, 2024

Vyom-Yadav Jan 4, 2025

evankanderson Oct 28, 2024

evankanderson Oct 28, 2024

Vyom-Yadav Jan 4, 2025

evankanderson Oct 28, 2024

Vyom-Yadav Jan 4, 2025

evankanderson Oct 28, 2024

Vyom-Yadav Jan 4, 2025

evankanderson Oct 28, 2024

evankanderson Oct 28, 2024

evankanderson Oct 28, 2024

evankanderson Oct 28, 2024

evankanderson Oct 28, 2024

evankanderson Oct 28, 2024

github-actions bot commented Nov 28, 2024

Vyom-Yadav commented Jan 3, 2025

coveralls commented Jan 3, 2025 •

edited

Loading

Vyom-Yadav left a comment •

edited

Loading

Vyom-Yadav Jan 4, 2025

Vyom-Yadav Jan 4, 2025

		// Current number of reminders in the batch
		BatchSize metric.Int64Gauge

	meter := mp.Meter("reminder-service")
	meter := mp.Meter("reminder")

Record Metrics for Reminder #4831

Are you sure you want to change the base?

Record Metrics for Reminder #4831

Conversation

Vyom-Yadav commented Oct 27, 2024 • edited Loading

Summary

Change Type

Testing

Review Checklist:

evankanderson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 28, 2024

Vyom-Yadav commented Jan 3, 2025

coveralls commented Jan 3, 2025 • edited Loading

Vyom-Yadav left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Vyom-Yadav commented Oct 27, 2024 •

edited

Loading

coveralls commented Jan 3, 2025 •

edited

Loading

Vyom-Yadav left a comment •

edited

Loading