Migrate trusty eval engine to Trusty v2 API. #5013

blkt · 2024-11-20T17:41:29Z

Summary

This change migrates trusty evaluation engine to use Trusty v2 API. Most of the changes apply to the intermediate representation we recently added to decouple Trusty from Minder, but some additional changes were required due to some fields becoming optional and nullable.

Note: this change is again meant to be bug-compatible with the current evaluation engine, so we treat the lack of "score" as a score of 0, thus triggering false positives as done by the current engine. The idea is to build on top of this to fix known issues of the engine before migrating it to Data Sources.

Fixes stacklok/minder-stories#77

Change Type

Bug fix (resolves an issue without affecting existing features)
Feature (adds new functionality without breaking changes)
Breaking change (may impact existing functionalities or require documentation updates)
Documentation (updates or additions to documentation)
Refactoring or test improvements (no bug fixes or new functionality)

Testing

Updated unit tests, ran smoke tests locally.

Review Checklist:

Reviewed my own code for quality and clarity.
Added comments to complex or tricky code sections.
Updated any affected documentation.
Included tests that validate the fix or feature.
Checked that related changes are merged.

go.mod

internal/engine/eval/trusty/actions.go

coveralls · 2024-11-21T08:37:15Z

coverage: 54.582% (-0.05%) from 54.633%
when pulling 026b68d on feat/trusty-engine-v2-api
into 0c315ea on main.

evankanderson

A few small comments, but nothing to block you if you want to go ahead as-is.

evankanderson · 2024-11-22T18:56:20Z

internal/engine/eval/trusty/trusty.go

+	respSummary, err := trustyClient.Summary(ctx, input)
+	if err != nil {
+		return nil, fmt.Errorf("failed getting summary: %w", err)
+	}
+
+	respPkg, err := trustyClient.PackageMetadata(ctx, input)
+	if err != nil {
+		return nil, fmt.Errorf("failed getting package metadata: %w", err)
+	}
+
+	respAlternatives, err := trustyClient.Alternatives(ctx, input)


Not required, but you might want to issue these calls in parallel, since they may be around a second each. It's a bunch more code, though, so it's also fine to just leave a comment for now:

Suggested change

respSummary, err := trustyClient.Summary(ctx, input)

if err != nil {

return nil, fmt.Errorf("failed getting summary: %w", err)

}

respPkg, err := trustyClient.PackageMetadata(ctx, input)

if err != nil {

return nil, fmt.Errorf("failed getting package metadata: %w", err)

}

respAlternatives, err := trustyClient.Alternatives(ctx, input)

summary := make(chan *v2types.PackageSummaryAnnotation, 1)

metadata := make(chan *v2types.TrustyPackageMetadata, 1)

alternatives := make(chan *v2types.PackageAlternatives, 1)

errors := make(chan error)

defer go func() {

close(summary)

close(metadata)

close(alternatives)

close(errors)

}

go func() {

resp, err := trustyClient.Summary(ctx, input)

errors <- err

summary <- resp

}()

go func() {

resp, err := trustyClient.PackageMetadata(ctx, input)

errors <- err

metadata <- resp

}()

go func() {

resp, err := trustyClient.Alternatives(ctx, input)

errors <- err

alternatives <- resp

}

for (int i = 0; i < 3; i++) {

err := <- errors

if err != nil {

return nil, fmt.Errorf("Trusty call failed: %w", err)

}

}

respSummary := <- summary

respPkg := <- metadata

respAlternatives := <-alternatives

I integrated this suggestion.

evankanderson · 2024-11-22T18:57:41Z

internal/engine/eval/trusty/trusty.go

 	}

-	res := makeTrustyReport(dep, resp)
+	respProvenance, err := trustyClient.Provenance(ctx, input)


(This would follow the same pattern above)

evankanderson · 2024-11-22T18:57:57Z

internal/engine/eval/trusty/trusty.go

 	}

-	res := makeTrustyReport(dep, resp)
+	respProvenance, err := trustyClient.Provenance(ctx, input)


Do you know whether the Trusty client is emitting metrics on these calls that we could use to track performance over time?

Yes, it emits the following metrics via headers, but they're not yet exposed by the client library.

x-ratelimit-limit: 480, 480;w=3600 x-ratelimit-remaining: 479 x-ratelimit-reset: 2879

I'm adding an issue to stacklok/trusty-sdk-go to expose them.
stacklok/trusty-sdk-go#57

I'd actually meant the HTTP client-observed latency distribution and error rate

evankanderson · 2024-11-22T19:03:28Z

internal/engine/eval/trusty/actions.go

+const (
+	better packageComparison = iota
+	worse
+)


Can we use the same contract as e.g. slices.SortFunc and use positive numbers to indicate a > b, and negative numbers for a < b?

That would also allow us to use slices.SortFunc elsewhere for lists of packages with the comparison function.

Unfortunately, comparePackages arguments are of different types, so it would not be possible to use that as argument to slices.SortFunc.

I'm inclined to keep this implementation because

it's unclear whether smaller-is-better or greater is

the score is deprecated, so this comparison doesn't really make sense anymore

we're moving pr_trusty_check rule to Data Sources and Rego evaluation type

evankanderson · 2024-11-22T19:18:22Z

internal/engine/eval/trusty/actions.go

@@ -428,8 +448,12 @@ func newSummaryPrHandler(
 	}, nil
 }

-func preprocessDetails(s string) string {
-	scanner := bufio.NewScanner(strings.NewReader(s))
+func preprocessDetails(s *string) string {


Why change this to take a *string? (If so, this looks like the correct behavior)

Because its argument maliciousInfo.Details is now a pointer to string.
I can change this, but I'd have to move the nil check to the caller.

evankanderson · 2024-11-22T19:24:02Z

internal/engine/eval/trusty/trusty.go


 	return res
 }

-func makeScoreComponents(descr map[string]any) []scoreComponent {
+func addSummaryDetails(res *trustyReport, resp *trustytypes.PackageSummaryAnnotation) {


Can we get a nil annotation response without an error? If not, maybe pass through the concrete type or document that it can't be nil.

evankanderson · 2024-11-22T19:27:04Z

internal/engine/eval/trusty/trusty.go

-			Score:       &alt.Score,
+			Score:       alt.Score,


👍 to removing pointer types where possible; it reduces the number of times we need to remember nil checks.

I'll add an issue to stacklok/trusty-sdk-go to improve quality by removing pointer types where possible.
stacklok/trusty-sdk-go#58

This change migrates `trusty` evaluation engine to use Trusty v2 API. Most of the changes apply to the intermediate representation we recently added to decouple Trusty from Minder, but some additional changes were required due to some fields becoming optional and nullable. Note: this change is again meant to be bug-compatible with the current evaluation engine, so we treat the lack of `"score"` as a score of `0`, thus triggering false positives as done by the current engine. The idea is to build on top of this to fix known issues of the engine before migrating it to Data Sources. Fixes stacklok/minder-stories#77

blkt · 2024-11-25T12:26:16Z

@evankanderson I should have addressed all comments.
I'll be traveling today, could you kindly merge this if you think it's ok?

blkt self-assigned this Nov 20, 2024

blkt commented Nov 20, 2024

View reviewed changes

go.mod Outdated Show resolved Hide resolved

internal/engine/eval/trusty/actions.go Outdated Show resolved Hide resolved

blkt force-pushed the feat/trusty-engine-v2-api branch from 5432bcd to c4601e0 Compare November 21, 2024 08:25

blkt force-pushed the feat/trusty-engine-v2-api branch 2 times, most recently from 6b37f59 to 026b68d Compare November 22, 2024 12:09

blkt marked this pull request as ready for review November 22, 2024 12:23

blkt requested a review from a team as a code owner November 22, 2024 12:23

evankanderson previously approved these changes Nov 22, 2024

View reviewed changes

blkt dismissed evankanderson’s stale review via ed006e4 November 25, 2024 12:22

blkt added 3 commits November 25, 2024 13:24

Refactor package comparison in a separate routine.

1b2a180

Address PR comments.

bae5027

blkt force-pushed the feat/trusty-engine-v2-api branch from ed006e4 to bae5027 Compare November 25, 2024 12:25

evankanderson approved these changes Nov 25, 2024

View reviewed changes

evankanderson merged commit 3fbc363 into main Nov 25, 2024
25 checks passed

evankanderson deleted the feat/trusty-engine-v2-api branch November 25, 2024 14:43

blkt mentioned this pull request Dec 6, 2024

Missing package alternatives when Minder comments on a PR #4945

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate trusty eval engine to Trusty v2 API. #5013

Migrate trusty eval engine to Trusty v2 API. #5013

blkt commented Nov 20, 2024

coveralls commented Nov 21, 2024 •

edited

Loading

evankanderson left a comment

evankanderson Nov 22, 2024

blkt Nov 25, 2024

evankanderson Nov 22, 2024

evankanderson Nov 22, 2024

blkt Nov 25, 2024

evankanderson Nov 25, 2024

evankanderson Nov 22, 2024

blkt Nov 25, 2024

evankanderson Nov 22, 2024

blkt Nov 25, 2024

evankanderson Nov 22, 2024

evankanderson Nov 22, 2024

blkt Nov 25, 2024

blkt commented Nov 25, 2024

-	respSummary, err := trustyClient.Summary(ctx, input)
-	if err != nil {
-		return nil, fmt.Errorf("failed getting summary: %w", err)
-	}
-	respPkg, err := trustyClient.PackageMetadata(ctx, input)
-	if err != nil {
-		return nil, fmt.Errorf("failed getting package metadata: %w", err)
-	}
-	respAlternatives, err := trustyClient.Alternatives(ctx, input)
+	summary := make(chan *v2types.PackageSummaryAnnotation, 1)
+	metadata := make(chan *v2types.TrustyPackageMetadata, 1)
+	alternatives := make(chan *v2types.PackageAlternatives, 1)
+	errors := make(chan error)
+	defer go func() {
+		close(summary)
+		close(metadata)
+		close(alternatives)
+		close(errors)
+	}
+	go func() {
+		resp, err := trustyClient.Summary(ctx, input)
+		errors <- err
+		summary <- resp
+	}()
+	go func() {
+		resp, err := trustyClient.PackageMetadata(ctx, input)
+		errors <- err
+		metadata <- resp
+	}()
+	go func() {
+		resp, err := trustyClient.Alternatives(ctx, input)
+		errors <- err
+		alternatives <- resp
+	}
+	for (int i = 0; i < 3; i++) {
+		err := <- errors
+		if err != nil {
+			return nil, fmt.Errorf("Trusty call failed: %w", err)
+		}
+	}
+	respSummary := <- summary
+	respPkg := <- metadata
+	respAlternatives := <-alternatives

Migrate trusty eval engine to Trusty v2 API. #5013

Migrate trusty eval engine to Trusty v2 API. #5013

Conversation

blkt commented Nov 20, 2024

Summary

Change Type

Testing

Review Checklist:

coveralls commented Nov 21, 2024 • edited Loading

evankanderson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blkt commented Nov 25, 2024

coveralls commented Nov 21, 2024 •

edited

Loading