OpenAPI spec for the REST API

[taras]

Context

An organization that I'm working with is considering adopting Backstage as their developer portal. One of the hard(ish) requirements that they have is that all services that we consume provide an API specification. This requirement exists because the infrastructure team needs to automate communication with services. If a specification is available, then they can use code generators to generate API clients for these services.

Feature Suggestion

It would be very helpful if there was a way to ensure that services integrated into Backstage come automatically with specifications. .NET, Rails & FastAPI all provide a mechanism that automatically generates an OpenAPI definition from route definition. I'm not aware of any tools that work like this in Node.js world.

Possible Implementation

express-openapi is a middleware that generates some specification from routes. It doesn't produce complete specifications - I don't even know if this is possible.

This request could also be satisfied by a GraphQL API that wraps services that Backstages consumes. The only consideration is making sure that the design of the APIs covers plugin services as much as possible.

[andrewthauer]

I ran across this project the other day that could be of some use when using a code first approach: https://github.com/bee-travels/openapi-comment-parser.

However, I agree that using annotations or a middleware would be more preferable.

[stefanalund]

Does it have to be automated from the beginning, or could we start with a manually edited file?

[taras]

It doesn't have to be automated. Having an OpenAPI spec for services included out of the box could be a good start.

[stefanalund]

@taras wanna take a stab at it?

[taras]

I wouldn't mind but we need a way to ensure that they stay current. Any ideas?

[kaos]

I think it looks like express-openapi indeed produces a complete specification, although you will have to write the spec for each route manually. The upside to that vs a completely manual spec on the side is that you get the spec closer to the implementation. You may also get validation middlewares for your routes (both for request and response data) amongst others, if desired.. ;)

[andrewthauer]

I tried out https://github.com/bee-travels/openapi-comment-parser recently and it will generate the openapi spec through JSDoc comments. It's still manual, but the spec ends up being right right above the code. I haven't tried it, but I suspect you could use some of the existing openapi tools like validation in conjunction with this approach.

[kaos]

looked a bit more at express-openapi, and I must say that I was stumped to learn that you had to put your implementation along with the spec.. I don't fancy the idea that the spec drives/dictates the implementation, it should be the other way around (the spec reflects the registered routes)

[taras]

@kaos I agree - spec and implementation should be separate.

We've been experimenting with this inside of our Backstage instance for integrating with our platform. We've been able to decouple the frontend from backend development by using OpenAPI spec in the middle. We generate a client with autorest and use Prism to mock service while in development. We're going to put it through the grind internally and then make an RFC when we're happy with the workflow.

[kaos]

Thanks for the update, looking forward to it :)

[freben]

Just bumping this to say that there's definite interest in this still! :)

[OrkoHunter]

Hey @taras,

We're going to put it through the grind internally and then make an RFC when we're happy with the workflow.

Just curious if there is anything you'd like to share on your experience so far on this setup that you mention!

[taras]

@OrkoHunter yep, the setup that we have right now is this,

1. each service that we're consuming in Backstage has an OpenAPI spec
2. we generate clients for these services with autorest
3. the generated code is wrapped in a use*API hook which uses the Backstage's ref API
4. we use the use*API hooks in Backstage
5. we use Prism to create a mock server in development - this allow us to build UI before backend microservice is implemented

This is pretty spiky at the moment because Autorest is quite heavy. We could create a leaner generator that generates hooks automatically and eliminate glue. Prism is also not exactly what we want to because it allows mocking by matching requests but not a high fidelity simulation. There is an issue to make Prism more realistic.

I would like to see and contribute to is having OpenAPI spec for each service in Backstage. Then generate clients wether at built time or runtime from the OpenAPI spec.

[alexcurtin]

@taras I'm very interested in seeing how you generated the OpenAPI spec for the catalog API, do you mind sharing your implementation somewhere? Can you push up a branch showcasing what you've built so far? Our end goal is to be able to build a catalog-client from the OpenAPI spec for the /api/catalog endpoint. Thanks!

[taras]

👋 @alexcurtin we haven't integrated with the catalog so far. We only used the clients that we created to integrate services that already have OpenAPI specs into Backstage. I might have an opportunity to do it this week, but I'm trying to figure out how to validate the implementation.

[taras]

I'm looking at using json-schema-to-openapi-schema to convert at least a part of https://github.com/backstage/backstage/blob/master/packages/catalog-model/src/schema/kinds/API.v1alpha1.schema.json to YAML. From there, I'm going to try to create an OpenAPI spect that includes the routes and references the generated files.

[freben]

Nice. Yeah if we could reduce the duplication of defining the entity shape it'd be nice. But why API? I'd have expected Entity instead, since that's the actual return type. Unless you wanna get fancy and do oneOf each of the specific kinds, but the value of that is probably very limited especially since it won't account for custom kinds etc

[taras]

But why API? I'd have expected Entity instead, since that's the actual return type.

You're right. I referenced the wrong file. It was the last file opened in my tab instead of the proper file :) https://github.com/backstage/backstage/blob/master/packages/catalog-model/src/schema/Entity.schema.json is the right file.

[taras]

@freben I started creating a @backstage/openapi package. It'll contain the script and generated YAML files. Does that sound alright?

[freben]

Uh, I am not sure. @Rugvip @benjdlambert @jhaals any input on that?

Gut feeling says that the catalog API, for one, is a thing of its own and should go in the @backstage/catalog-client package with the other client functionality. And that an overall mega-api-definition for a backend, rather than individual plugins, might be of limited use.

[regicsolutions]

@regicsolutions If your goal is a client Go application for your Backstage instance, I would recommend vending that internally through a Go package and not relying on a template. If you're looking to create a Go application from just any OpenAPI 3 spec, this isn't the right issue. The link you shared should work, but I would be careful to check for OpenAPI 3.1 vs OpenAPI 3.0 compatibility as many tools don't yet work with the new 3.1 spec format.

Thanks @sennyeya my goal is to be able to scaffold a repository/go application based on a OpenAPI 3 spec. The link requires running cli commands wasn't sure how to best handle that in a template.

[sennyeya]

@freben @taras Wanted to follow up on this as I have some free time and #15667 was finally merged ❤️.

Since we have a spec for the catalog now, I want to take a look at generating the catalog-client. I'm going to focus primarily on openapi-generator with openapi-generator-plus as a backup. We'll definitely want to deep dive into generated code and what not once I have example clients.

[bforbis]

Hello, just wanted to mention a bit of awkwardness on the decision to generate OpenAPI 3.1 specifically for the catalog-backend as opposed to 3.0.

I just tried loading the generated YAML into the backstage catalog so that users can explore the REST API definition within backstage itself and noticed that backstage swagger-ui cannot render OpenAPI 3.1 yet as detailed in #8090.

Is there another solution in the meantime to make the backstage REST API also support generating version 3.0 until the built in swagger UI can be updated? I would love to be able to "dogfood" the tools within backstage to show how you can document backstage with backstage, including the REST API.

[sennyeya]

@bforbis Stay tuned for #17535, I ended up switching down to OpenAPI 3.0 for that work for the same reasons you're mentioning. If you want to handle that on your side more quickly, you can pull the changes that I made to the spec to translate it from 3.1 to 3.0.

[sennyeya]

Quick update for those following this,

1. [PRFC] API Docs for the whole backend #17535 has been merged and is part of the 1.20.0-next.0 release. This adds a new catalog provider to show a merged OpenAPI spec for your backend. The set of plugins it pulls from is manually defined and very few plugins have specs so far (limited to just catalog, search and todo rn).
2. As part of [PRFC] API Docs for the whole backend #17535, plugins that use the createValidatedOpenApiRouter method from @backstage/backend-openapi-utils will now have a /api/${pluginId}/openapi.json route that serves the full OpenAPI spec for this plugin, ie all spec paths are correctly mounted to /api/${pluginId}, etc.
3. feat(openapi): Add a new OpenAPI schema linting command to repo-tools. #18185 and feat: add support for validating OpenAPI specs against test runs #19955 have been merged and can be used as part of 1.20.0-next.0. The first PR adds some basic linting of OpenAPI specs to the project using Stoplight and the second PR adds test -> spec validation (and optionally, automatic updating) using Optic.
4. feat(catalog-client): Support generating a front end client based on an OpenAPI spec #17470 is working and building finally! There were some hiccups while trying to find a good path forward, but I think we've overcome those. Will be trying to push on this now that the OpenAPI router work has mostly died down.

Looking forward, there are a couple of interesting areas I'd like to start exploring,

1. JSON schema ref deduplication. Answering, how can we have a consistent Entity and Error definition across plugins?
2. Notification of breaking changes when updating the spec. Optic has some pretty nice support for this, should be straightforward to add.
3. Starting to remove a lot of types from catalog-backend, search-backend, and todo-backend to start and replacing them with their generated counterparts.
4. API fuzzing. Building off of feat(openapi-tooling): Add support for request validation. #17875 and feat: add support for validating OpenAPI specs against test runs #19955. This one solves two problems,
   a. How can we continue to protect against bad input and ensure stability of our APIs? Mainly looking at 500 errors and bad requests.
   b. How can we keep our specs up to date without manually iterating through all of the request/responses, either in test cases or in the spec itself?

For a more up to date version of the above, check out #17482

Additionally, if folks are interested in writing API specs for their own plugins, I've started some documentation at https://github.com/backstage/backstage/blob/master/docs/openapi/test-case-validation.md and will be adding more soon.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tudi2d]

Hej @aramissennyeydd - do you think this should stay open?

[aramissennyeydd]

@tudi2d I think we're good to close this. Tracking the work in #17482.

[freben]

Thanks for your service 🖖