Deluan Quintão 0e5b9e3263
feat(plugins): share plugin DTOs via a types package (#5655)
* refactor(plugins): remove Python PDK generation from ndpgen

* feat(plugins): parse Go type aliases distinctly in ndpgen

* feat(plugins): resolve shared-type aliases against a registry in ndpgen

* fix(plugins): resolve host-service shared aliases package-wide

Mirror the capability approach in ParseDirectoryWithShared: do a first
pass over all package files to build a package-wide alias map, then pass
it into parseServiceFile so that a shared-type alias declared in a sibling
file is visible when resolving types in the service interface file.

Add a focused test that writes the alias in one file and the hostservice
in another, confirming RED before the fix and GREEN after. Also
strengthens the existing Task 3 test with an ArtistRef.Target assertion.

* feat(plugins): add ndpgen -shared-types mode for the Go types package

* feat(plugins): generate the nd-pdk-types Rust crate from -shared-types

* feat(plugins): inject types import and emit deprecated aliases in Go output

* feat(plugins): emit deprecated Rust aliases to the shared types crate

* feat(plugins): inline shared-type shapes into XTP schemas

* feat(plugins): add nd-pdk-types crate and wire dependents

* feat(plugins): move shared capability types to plugins/types with deprecated aliases

* fix(plugins): point Rust deprecated-alias note at the replacement type

* fix(plugins): include shared aliases in KnownStructs so Rust fields keep their type

Capability.KnownStructs() and Service.KnownStructs() previously only
registered names from .Structs. After the shared-types migration, types
like ArtistRef/TrackInfo/SongRef live in .SharedAliases instead, so
ToRustTypeWithStructs could not find them and fell back to serde_json::Value
for every struct field referencing a shared type.

Add the shared-alias names to the knownStructs map in both methods.
Regenerate the Rust capability files; track/song/artist fields now render
as their named types (TrackInfo, SongRef, ArtistRef, etc.).

Add a regression test that verifies a struct field whose type is only in
SharedAliases renders as the named type and not serde_json::Value.

* docs(plugins): remove stale Python references from ndpgen and plugins READMEs

ndpgen no longer has a -python flag; remove it from the usage synopsis,
flags table, and defaults note in ndpgen/README.md. Delete the "Python
Client Library" section that described its output.

plugins/README.md referenced plugins/pdk/python/host/ (deleted) as the
source for Python host-service stubs. Remove that paragraph; Python plugins
still work via the XTP-schema / extism-py path (see examples/*-py).

* refactor(plugins): dedupe ndpgen helpers and tidy shared-type codegen

* docs(plugins): restore Python as a supported XTP schema target

The ndpgen-generated Python PDK was removed, but the XTP YAML schemas are
language-neutral and the XTP CLI still generates Python bindings from them
(as the extism-py examples demonstrate). Only the ndpgen Python output was
dropped, not Python support itself.

* test(plugins): use the shared types package in test plugins

The test fixtures referenced the now-deprecated capability aliases
(sonicsimilarity.SongRef, metadata.ArtistRef/SongRef). Point them at the
canonical types package so our own fixtures don't depend on symbols slated
for removal.

* refactor(plugins): use the shared types package in host adapters

Replace deprecated capabilities.TrackInfo, capabilities.ArtistRef, and
capabilities.SongRef aliases with the canonical types.TrackInfo,
types.ArtistRef, and types.SongRef from plugins/types.

* fix(plugins): reference shared types by canonical path in generated Rust

Previously the generator emitted `pub field: SongRef` (the local deprecated
alias) for struct fields whose type came from SharedAliases. Refactored
ToRustTypeWithStructs into a private toRustType that accepts a shared map,
and added ToRustTypeWithShared which resolves shared-alias names to their
canonical nd_pdk_types::X path before falling through to the knownStructs
check. Both rustCapabilityFuncMap and rustFuncMap now build the shared map
from SharedAliases and use it for fieldRustType, so the generated capability
files reference nd_pdk_types::SongRef / nd_pdk_types::TrackInfo directly.
The deprecated pub type aliases remain in place as the external back-compat
surface. Deprecation warning count from cargo build drops to 0.

* fix(examples): implement missing Scrobbler.playback_report in Rust examples

The Scrobbler trait gained a playback_report method but the two Rust example
plugins (webhook-rs and discord-rich-presence-rs) were not updated, causing
E0046 compile errors. Added the missing fn playback_report to both: webhook-rs
logs and returns Ok(()) mirroring its now_playing handler; discord-rich-presence-rs
is a no-op since Discord presence does not need playback reports. make all-rust
now exits 0.

* refactor(plugins): point Rust deprecation notes at the nd_pdk::types umbrella path

Plugin authors depend on the nd-pdk umbrella crate, which re-exports
nd_pdk_types as 'types', so the migration target they should type is
nd_pdk::types::X. The alias target stays nd_pdk_types::X (the real path
inside nd-pdk-capabilities).

* fix(plugins): error when a shared-type alias can't be resolved against the registry

* refactor(plugins): parse each Go source file once in ndpgen

* fix(plugins): correct ndpgen review nits (flag name, unused dep, docs)

* refactor(plugins): drop the now-unused path param from parseServiceFile

* refactor(plugins): use shared types directly, rename TrackInfo to Track

Capability interfaces now reference the shared `types` package by qualified
name (types.Track, types.SongRef, types.ArtistRef) instead of the package-local
deprecated aliases, and the shared TrackInfo type is renamed to Track to match
its role as the plugin-facing projection of a library media file.

The deprecated bare aliases (scrobbler.TrackInfo, metadata.ArtistRef,
sonicsimilarity.SongRef, etc.) are kept as re-exports so existing plugins keep
compiling, with a deprecation warning steering them to the canonical types.

To support this, ndpgen now resolves qualified types.X references: it collects
them during type discovery, maps each used canonical type back to its declared
deprecated alias for re-export, emits nd_pdk_types::X paths in Rust, and names
the XTP schema components by their canonical type. Regenerated the Go and Rust
PDK and the XTP schemas, and added generator tests covering the qualified-ref
path. Also adds clarifying doc comments to the shared types.

* refactor(plugins): extract shared types selector into a named const

Replace the "types." string literal that detects and strips the shared types
package selector with a single sharedTypesPrefix constant across the ndpgen
generator (parser, types, generator, xtp_schema), giving the package one source
of truth for the selector.

Also restore the single reused scratch map (cleared each iteration) in the
resolveSharedAliases BFS instead of allocating a fresh map per shared-struct
field, matching the prior implementation.

Pure cleanup from a /simplify pass: regeneration produces byte-for-byte
identical Go, Rust, and XTP output.

* refactor(plugins): keep TrackInfo in the capability package for now

Move the track type back out of the shared plugins/types package: it is again
defined inline as TrackInfo in plugins/capabilities/scrobbler.go and referenced
directly by the scrobbler and lyrics capabilities, reverting the rename to
types.Track. The host helper is renamed back to mediaFileToTrackInfo and now
returns capabilities.TrackInfo. SongRef and ArtistRef stay in the shared types
package; TrackInfo keeps using types.ArtistRef for its artist lists.

This type is expected to be reshaped in upcoming work, so leaving it in the
capability package avoids churning the shared types twice. Regenerated the Go
and Rust PDK and the XTP schemas accordingly.

* fix(plugins): emit the Go types import for direct shared-type refs

ndpgen's Capability/Service.ImportsSharedTypes only reported a shared-types
dependency when a deprecated re-export alias (type X = types.X) was declared. A
struct field referencing the canonical form directly (e.g. types.SongRef) with
no such alias produced an empty SharedAliases slice, so the Go templates skipped
the import while still emitting fields/signatures using types.SongRef — leaving
generated PDK code for new shared DTOs uncompilable unless an otherwise
unnecessary alias was added.

ImportsSharedTypes now also returns true when any struct field references the
types. package by qualified name, via a new structsReferenceSharedTypes helper
that reuses collectReferencedTypes (so []types.X and map[...]types.X are covered
too).

* fix(plugins): preserve base64 encoding for shared byte fields in Rust

The Rust shared-types crate template rendered a []byte field as a plain Vec<u8>
without the base64_bytes serde override used by the capability/client templates.
Go's encoding/json serializes []byte as a base64 string, so a Rust plugin using
nd_pdk::types would have serialized an array of numbers instead of the wire
format the Go/server side expects.

GenerateSharedTypesRust now registers the base64_bytes partial and passes a
HasByteFields flag (new anyFieldIsByteSlice helper); types.rs.tmpl emits the
base64_bytes module and a #[serde(with = "base64_bytes")] attribute on []byte
fields, mirroring the capability template.

* fix(plugins): include directly-referenced shared types in XTP schemas

buildSchemas registered shared types into the schema components only by iterating
cap.SharedAliases, which records deprecated re-export aliases. A capability that
referenced a shared DTO solely as types.Foo (no declared alias) therefore never
got Foo into the component set, so the self-contained XTP schema rendered the
field as a generic object (or emitted a dangling $ref), breaking the direct
shared-type use case enabled by -shared.

resolveSharedAliases now also returns the resolved shapes of every used shared
type (alias or not); these are carried on the new Capability.SharedTypes field
and registered by buildSchemas alongside SharedAliases. Validated end-to-end with
the xtp CLI: a direct types.Foo reference now produces a proper component plus a
$ref, so xtp generates a typed struct instead of an untyped serde_json::Map.

* fix(plugins): resolve renamed shared aliases to canonical schema refs

When a deprecated alias renames its canonical type (e.g. type TrackInfo =
types.Track) and a capability field is typed with the alias name (TrackInfo),
buildProperty emitted a $ref to #/components/schemas/TrackInfo. Components are
keyed by the canonical name (Track), so no TrackInfo component was emitted,
leaving a dangling reference that crashes the xtp code generator.

buildSchemas now builds an alias->canonical map; buildProperty (and the slice
item path) resolves $ref targets through it, and a used alias name marks its
canonical component used so it is emitted. Validated with the xtp CLI: the
renamed-alias schema previously crashed xtp and now generates cleanly.

* fix(plugins): detect shared types used directly in method signatures

ImportsSharedTypes only inspected struct fields, so a capability method using a
shared type directly in its signature (e.g. types.SongRef as input/output rather
than inside a local struct) was not detected. The generated Go templates still
rendered the provider/export signatures with types.SongRef, so the capability
package omitted the types import and failed to compile; the same gap applied to
service params/returns.

ImportsSharedTypes now also scans capability method input/output types and
service method params/returns, via a typeReferencesSharedTypes helper that reuses
collectReferencedTypes (covering pointer/slice/map wrappers).

* fix(plugins): add base64 dependency to the shared Rust types crate

When a shared DTO has a []byte field, ndpgen emits the base64_bytes serde helper
and use base64::... imports into nd-pdk-types/src/lib.rs, but the crate manifest
declared only serde. In that case make gen produced a crate that failed to
compile with 'unresolved module base64'.

Add base64 = "0.22" (matching nd-pdk-capabilities) so the generated shared types
crate compiles whenever a []byte field is present. Verified by generating a
shared crate with a []byte field and confirming cargo check fails before and
passes after.

* fix(plugins): translate shared method types in generated Rust

A capability method using a shared DTO directly as input/output (e.g.
types.SongRef) was passed through rustOutputType unchanged, so the Rust template
emitted invalid trait and extism_pdk::Json<$crate::pkg::types.SongRef> signatures
that do not compile.

Method input/output types now resolve through the shared registry: trait
signatures use rustTraitType (shared -> nd_pdk_types::X, locals stay bare) and the
export macros use rustMethodType (fully qualified: shared -> nd_pdk_types::X,
primitives -> Rust, locals -> $crate::<pkg>::X). Verified end-to-end by compiling
a generated capability that takes types.SongRef directly against the real
nd-pdk-types crate.

* fix(plugins): canonicalize XTP export refs for renamed shared aliases

buildSchemas canonicalized alias-to-canonical references for struct-field $ref
targets, but buildExport built export input/output $refs straight from
fieldBaseType. A capability whose export used a renamed deprecated alias
directly (e.g. type TrackInfo = types.Track with NowPlaying(TrackInfo)) emitted
$ref: #/components/schemas/TrackInfo, while the component is emitted under the
canonical name Track — a dangling export reference.

Lift the alias-to-canonical map into GenerateSchema (buildAliasToCanonical) and
apply it to export refs via canonicalRefName, the same resolution already used
for field properties.

* fix(plugins): route shared macro types through $crate for plugin builds

When a capability method used a shared type directly, the generated export macro
named the type as nd_pdk_types::SongRef. The macro expands in the downstream
plugin crate, which depends on the umbrella nd-pdk crate and not on nd-pdk-types
directly, so that path is unresolvable there and the plugin fails to build.

rustMethodType (macro-facing) now emits $crate::types::X, and the generated
nd-pdk-capabilities lib.rs re-exports nd_pdk_types as types so $crate resolves
it. Trait signatures keep nd_pdk_types::X since they live in nd-pdk-capabilities,
which has the direct dependency. Verified end-to-end: a plugin crate depending
only on the umbrella that uses a capability with a direct types.X method now
compiles via the macro.

* fix(plugins): add nd-pdk-types dependency to the Rust host crate

When a host service uses a shared type, ndpgen emits nd_pdk_types::X into the
generated nd-pdk-host client wrappers, but the host crate's manifest did not
depend on nd-pdk-types, so the crate failed to compile with 'unresolved module
nd_pdk_types'. Host client wrappers are plain functions resolved in the host
crate's own context (not macros expanded downstream), so a direct dependency is
the right fix.

Add nd-pdk-types = { path = "../nd-pdk-types" } to nd-pdk-host, mirroring
nd-pdk-capabilities. Found while auditing all Rust paths against the realistic
crate topology after the capability-side $crate fix; verified by generating a
host service with a shared-type return and confirming cargo check fails before
and passes after.

* fix(plugins): resolve shared aliases in Rust host signatures

The Rust host client rendered method params and returns through
RustTypeWithStructs, which only consults KnownStructs. A host service using a
shared alias in a signature (e.g. type Track = types.Track plus
MatchSongs(...) ([]Track, error)) therefore emitted a bare Vec<Track>, but the
client template emits no Track alias or import, so the generated nd-pdk-host
crate did not compile. Only struct fields went through the shared map.

rustType/rustParamType now use the shared map too (RustTypeWithShared /
RustParamTypeWithShared), so an aliased param/return resolves to its canonical
nd_pdk_types::X path, matching field handling. Verified by generating a host
service returning a shared alias and confirming cargo check fails before and
passes after.
2026-06-29 21:20:33 -04:00

1043 lines
30 KiB
Go

package internal
import (
"fmt"
"go/ast"
"go/parser"
"go/token"
"maps"
"os"
"path/filepath"
"regexp"
"slices"
"strings"
)
// Annotation patterns
var (
// //nd:hostservice name=ServiceName permission=key
hostServicePattern = regexp.MustCompile(`//nd:hostservice\s+(.*)`)
// //nd:hostfunc [name=CustomName]
hostFuncPattern = regexp.MustCompile(`//nd:hostfunc(?:\s+(.*))?`)
// //nd:capability name=PackageName [required=true]
capabilityPattern = regexp.MustCompile(`//nd:capability\s+(.*)`)
// //nd:export name=ExportName
exportPattern = regexp.MustCompile(`//nd:export\s+(.*)`)
// key=value pairs
keyValuePattern = regexp.MustCompile(`(\w+)=(\S+)`)
)
// parsedGoFile pairs a source path with its already-parsed AST.
type parsedGoFile struct {
path string
file *ast.File
}
// parseGoFiles returns the eligible Go source files in dir, each parsed once.
func parseGoFiles(dir string, fset *token.FileSet) ([]parsedGoFile, error) {
paths, err := goSourceFiles(dir)
if err != nil {
return nil, err
}
out := make([]parsedGoFile, 0, len(paths))
for _, p := range paths {
f, err := parser.ParseFile(fset, p, nil, parser.ParseComments)
if err != nil {
return nil, fmt.Errorf("parsing %s: %w", filepath.Base(p), err)
}
out = append(out, parsedGoFile{path: p, file: f})
}
return out, nil
}
// goSourceFiles returns the Go source file paths in dir, excluding generated,
// test, and doc files.
func goSourceFiles(dir string) ([]string, error) {
entries, err := os.ReadDir(dir)
if err != nil {
return nil, fmt.Errorf("reading directory: %w", err)
}
var paths []string
for _, entry := range entries {
name := entry.Name()
if entry.IsDir() || !strings.HasSuffix(name, ".go") {
continue
}
if strings.HasSuffix(name, "_gen.go") || strings.HasSuffix(name, "_test.go") || name == "doc.go" {
continue
}
paths = append(paths, filepath.Join(dir, name))
}
return paths, nil
}
// ParseDirectory parses all Go source files in a directory and extracts host services.
func ParseDirectory(dir string) ([]Service, error) {
return ParseDirectoryWithShared(dir, nil)
}
// ParseDirectoryWithShared parses all Go source files in a directory, resolving any
// type aliases that reference the shared `types` package against the provided registry.
func ParseDirectoryWithShared(dir string, shared map[string]StructDef) ([]Service, error) {
fset := token.NewFileSet()
parsed, err := parseGoFiles(dir, fset)
if err != nil {
return nil, err
}
// First pass: collect all type aliases from every file so that an alias
// declared in one file is visible when resolving types in a sibling file.
pkgAliasMap := make(map[string]TypeAlias)
for _, pf := range parsed {
for _, a := range parseTypeAliases(pf.file) {
pkgAliasMap[a.Name] = a
}
}
// Second pass: parse services using the package-level alias map.
var services []Service
for _, pf := range parsed {
svcList, err := parseServiceFile(pf.file, pkgAliasMap, shared)
if err != nil {
return nil, fmt.Errorf("parsing %s: %w", filepath.Base(pf.path), err)
}
services = append(services, svcList...)
}
return services, nil
}
// ParseCapabilities parses all Go source files in a directory and extracts capabilities.
func ParseCapabilities(dir string) ([]Capability, error) {
return ParseCapabilitiesWithShared(dir, nil)
}
// ParseCapabilitiesWithShared parses all Go source files in a directory, resolving any
// type aliases that reference the shared `types` package against the provided registry.
func ParseCapabilitiesWithShared(dir string, shared map[string]StructDef) ([]Capability, error) {
fset := token.NewFileSet()
parsed, err := parseGoFiles(dir, fset)
if err != nil {
return nil, err
}
// First pass: collect all structs, type aliases, and const groups.
pkgStructMap := make(map[string]StructDef)
pkgAliasMap := make(map[string]TypeAlias)
var allConstGroups []ConstGroup
for _, pf := range parsed {
for _, s := range parseStructs(pf.file) {
pkgStructMap[s.Name] = s
}
for _, a := range parseTypeAliases(pf.file) {
pkgAliasMap[a.Name] = a
}
allConstGroups = append(allConstGroups, parseConstGroups(pf.file)...)
}
// Second pass: parse capabilities using the package-level type maps.
var capabilities []Capability
for _, pf := range parsed {
capList, err := parseCapabilityFile(pf.path, pf.file, pkgStructMap, pkgAliasMap, allConstGroups, shared)
if err != nil {
return nil, fmt.Errorf("parsing %s: %w", filepath.Base(pf.path), err)
}
capabilities = append(capabilities, capList...)
}
return capabilities, nil
}
// LoadSharedTypes parses every struct defined in dir (the shared `types` source
// package) and returns them keyed by name. dir == "" yields an empty map.
func LoadSharedTypes(dir string) (map[string]StructDef, error) {
result := map[string]StructDef{}
if dir == "" {
return result, nil
}
paths, err := goSourceFiles(dir)
if err != nil {
return nil, fmt.Errorf("reading shared types directory: %w", err)
}
fset := token.NewFileSet()
for _, path := range paths {
f, err := parser.ParseFile(fset, path, nil, parser.ParseComments)
if err != nil {
return nil, fmt.Errorf("parsing %s: %w", filepath.Base(path), err)
}
for _, s := range parseStructs(f) {
result[s.Name] = s
}
}
return result, nil
}
// parseCapabilityFile parses a single Go source file and extracts capabilities.
func parseCapabilityFile(path string, f *ast.File, structMap map[string]StructDef, aliasMap map[string]TypeAlias, allConstGroups []ConstGroup, shared map[string]StructDef) ([]Capability, error) {
var capabilities []Capability
for _, decl := range f.Decls {
genDecl, ok := decl.(*ast.GenDecl)
if !ok || genDecl.Tok != token.TYPE {
continue
}
for _, spec := range genDecl.Specs {
typeSpec, ok := spec.(*ast.TypeSpec)
if !ok {
continue
}
interfaceType, ok := typeSpec.Type.(*ast.InterfaceType)
if !ok {
continue
}
// Check for //nd:capability annotation in doc comment
docText, rawDoc := getDocComment(genDecl, typeSpec)
capAnnotation := parseCapabilityAnnotation(rawDoc)
if capAnnotation == nil {
continue
}
// Extract source file base name (e.g., "websocket_callback" from "websocket_callback.go")
baseName := filepath.Base(path)
sourceFile := strings.TrimSuffix(baseName, ".go")
capability := Capability{
Name: capAnnotation["name"],
Interface: typeSpec.Name.Name,
Required: capAnnotation["required"] == "true",
Doc: cleanDoc(docText),
SourceFile: sourceFile,
}
// Parse methods and collect referenced types
referencedTypes := make(map[string]bool)
for _, method := range interfaceType.Methods.List {
if len(method.Names) == 0 {
continue // Embedded interface
}
funcType, ok := method.Type.(*ast.FuncType)
if !ok {
continue
}
// Check for //nd:export annotation
methodDocText, methodRawDoc := getMethodDocComment(method)
exportAnnotation := parseExportAnnotation(methodRawDoc)
if exportAnnotation == nil {
continue
}
export, err := parseExport(method.Names[0].Name, funcType, exportAnnotation, cleanDoc(methodDocText))
if err != nil {
return nil, fmt.Errorf("parsing export %s.%s: %w", typeSpec.Name.Name, method.Names[0].Name, err)
}
capability.Methods = append(capability.Methods, export)
// Collect referenced types from input and output
collectReferencedTypes(export.Input.Type, referencedTypes)
collectReferencedTypes(export.Output.Type, referencedTypes)
}
// Recursively collect all struct dependencies
collectAllStructDependencies(referencedTypes, structMap)
// Resolve shared-type aliases against the registry
sharedAliases, sharedTypes, err := resolveSharedAliases(referencedTypes, aliasMap, shared)
if err != nil {
return nil, err
}
capability.SharedAliases = sharedAliases
capability.SharedTypes = sharedTypes
// Build a set of names already covered by SharedAliases so we don't
// emit them again in TypeAliases (which would cause a redeclaration).
sharedAliasNames := make(map[string]bool, len(capability.SharedAliases))
for _, sa := range capability.SharedAliases {
sharedAliasNames[sa.Name] = true
}
// Sort type names for stable output order
sortedTypeNames := slices.Sorted(maps.Keys(referencedTypes))
// Attach referenced structs to the capability
for _, typeName := range sortedTypeNames {
if s, exists := structMap[typeName]; exists {
capability.Structs = append(capability.Structs, s)
}
}
// Attach referenced type aliases (skip those already in SharedAliases)
for _, typeName := range sortedTypeNames {
if sharedAliasNames[typeName] {
continue
}
if a, exists := aliasMap[typeName]; exists {
capability.TypeAliases = append(capability.TypeAliases, a)
}
}
// Also attach type aliases prefixed with interface name (e.g., ScrobblerError for Scrobbler interface)
// This supports error types that are not directly referenced in method signatures
interfaceName := typeSpec.Name.Name
for _, typeName := range slices.Sorted(maps.Keys(aliasMap)) {
if sharedAliasNames[typeName] {
continue
}
a := aliasMap[typeName]
if strings.HasPrefix(typeName, interfaceName) && !referencedTypes[typeName] {
capability.TypeAliases = append(capability.TypeAliases, a)
referencedTypes[typeName] = true // Mark as referenced for const lookup
}
}
// Attach const groups that match referenced type aliases
for _, group := range allConstGroups {
if group.Type == "" {
continue
}
if referencedTypes[group.Type] {
capability.Consts = append(capability.Consts, group)
}
}
if len(capability.Methods) > 0 {
capabilities = append(capabilities, capability)
}
}
}
return capabilities, nil
}
// resolveSharedAliases determines which shared `types` package structs a host
// service or capability uses and returns the deprecated re-export aliases to emit
// for them.
//
// A shared type counts as used when a field references it by qualified name
// (e.g. types.Track) or via a declared alias used by bare name (e.g. a field of
// type TrackInfo where `type TrackInfo = types.Track`). The shared struct's own
// fields are followed transitively so nested shared types are picked up too. For
// every used canonical type, each declared `type X = types.Canonical` alias is
// emitted as a SharedAlias so the generated PDK keeps re-exporting it for
// backwards compatibility.
//
// It returns the deprecated re-export aliases to emit and the resolved shapes of
// every used shared type (alias or not, for schema inlining).
//
// Returns an error if a referenced shared type cannot be found in the shared registry.
func resolveSharedAliases(referenced map[string]bool, aliasMap map[string]TypeAlias, shared map[string]StructDef) ([]SharedAlias, []StructDef, error) {
// Index declared shared aliases by the canonical type they target, e.g.
// "Track" -> [TrackInfo]. A canonical type may have more than one alias.
aliasesByCanonical := map[string][]TypeAlias{}
for _, a := range aliasMap {
if a.IsSharedAlias() {
canonical := strings.TrimPrefix(a.Type, sharedTypesPrefix)
aliasesByCanonical[canonical] = append(aliasesByCanonical[canonical], a)
}
}
// Walk the referenced types, following nested shared references inside the
// shared structs, to find the set of canonical shared types used.
used := map[string]bool{}
var queue []string
for name := range referenced {
if c, ok := seedSharedCanonical(name, aliasMap); ok {
queue = append(queue, c)
}
}
fieldRefs := map[string]bool{}
for len(queue) > 0 {
canonical := queue[0]
queue = queue[1:]
if used[canonical] {
continue
}
def, ok := shared[canonical]
if !ok {
return nil, nil, fmt.Errorf(
"shared type %q could not be resolved: pass -shared=<dir> pointing at the shared types package, and ensure %s is defined there",
canonical, sharedTypesPrefix+canonical,
)
}
used[canonical] = true
for _, f := range def.Fields {
clear(fieldRefs)
collectReferencedTypes(f.Type, fieldRefs)
for t := range fieldRefs {
if c, ok := nestedSharedCanonical(t, shared); ok {
queue = append(queue, c)
}
}
}
}
var out []SharedAlias
var usedDefs []StructDef
for canonical := range used {
usedDefs = append(usedDefs, shared[canonical])
for _, a := range aliasesByCanonical[canonical] {
out = append(out, SharedAlias{Name: a.Name, Target: a.Type, Doc: a.Doc, Def: shared[canonical]})
}
}
slices.SortFunc(out, func(a, b SharedAlias) int { return strings.Compare(a.Name, b.Name) })
slices.SortFunc(usedDefs, func(a, b StructDef) int { return strings.Compare(a.Name, b.Name) })
return out, usedDefs, nil
}
// seedSharedCanonical maps a type token referenced by a capability/service field
// to the canonical shared type it denotes. It recognizes qualified references
// (types.X -> X) and declared shared aliases used by bare name (X where
// `type X = types.Y` -> Y). A bare name that is not a declared shared alias is
// not treated as shared, so a local struct sharing a name with a shared type is
// never misclassified.
func seedSharedCanonical(name string, aliasMap map[string]TypeAlias) (string, bool) {
if rest, ok := strings.CutPrefix(name, sharedTypesPrefix); ok {
return rest, true
}
if a, ok := aliasMap[name]; ok && a.IsSharedAlias() {
return strings.TrimPrefix(a.Type, sharedTypesPrefix), true
}
return "", false
}
// nestedSharedCanonical maps a type token found inside a shared struct's own
// fields to a canonical shared type. Within the shared package, types reference
// each other by bare name (e.g. Track.Artists is []ArtistRef), so any bare name
// present in the shared registry counts.
func nestedSharedCanonical(name string, shared map[string]StructDef) (string, bool) {
if rest, ok := strings.CutPrefix(name, sharedTypesPrefix); ok {
return rest, true
}
if _, ok := shared[name]; ok {
return name, true
}
return "", false
}
// collectAllStructDependencies recursively collects all struct types referenced by other structs.
func collectAllStructDependencies(referencedTypes map[string]bool, structMap map[string]StructDef) {
// Keep iterating until no new types are added
for {
newTypes := make(map[string]bool)
for typeName := range referencedTypes {
if s, exists := structMap[typeName]; exists {
for _, field := range s.Fields {
collectReferencedTypes(field.Type, newTypes)
}
}
}
// Check if any new types were found
foundNew := false
for t := range newTypes {
if !referencedTypes[t] {
referencedTypes[t] = true
foundNew = true
}
}
if !foundNew {
break
}
}
}
// parseExport parses an export method signature into an Export struct.
func parseExport(name string, funcType *ast.FuncType, annotation map[string]string, doc string) (Export, error) {
export := Export{
Name: name,
ExportName: annotation["name"],
Doc: doc,
}
// Capability exports have exactly one input parameter (the struct type)
if funcType.Params != nil && len(funcType.Params.List) == 1 {
field := funcType.Params.List[0]
typeName := typeToString(field.Type)
paramName := "input"
if len(field.Names) > 0 {
paramName = field.Names[0].Name
}
export.Input = NewParam(paramName, typeName)
}
// Capability exports return (OutputType, error)
if funcType.Results != nil {
for _, field := range funcType.Results.List {
typeName := typeToString(field.Type)
if typeName == "error" {
continue // Skip error return
}
paramName := "output"
if len(field.Names) > 0 {
paramName = field.Names[0].Name
}
export.Output = NewParam(paramName, typeName)
break // Only take the first non-error return
}
}
return export, nil
}
// parseServiceFile parses a single Go source file and extracts host services.
// pkgAliasMap is the package-wide alias map built from all files in the package.
func parseServiceFile(f *ast.File, pkgAliasMap map[string]TypeAlias, shared map[string]StructDef) ([]Service, error) {
// Collect all struct definitions in the file.
allStructs := parseStructs(f)
structMap := make(map[string]StructDef)
for _, s := range allStructs {
structMap[s.Name] = s
}
var services []Service
for _, decl := range f.Decls {
genDecl, ok := decl.(*ast.GenDecl)
if !ok || genDecl.Tok != token.TYPE {
continue
}
for _, spec := range genDecl.Specs {
typeSpec, ok := spec.(*ast.TypeSpec)
if !ok {
continue
}
interfaceType, ok := typeSpec.Type.(*ast.InterfaceType)
if !ok {
continue
}
// Check for //nd:hostservice annotation in doc comment
docText, rawDoc := getDocComment(genDecl, typeSpec)
svcAnnotation := parseHostServiceAnnotation(rawDoc)
if svcAnnotation == nil {
continue
}
service := Service{
Name: svcAnnotation["name"],
Permission: svcAnnotation["permission"],
Interface: typeSpec.Name.Name,
Doc: cleanDoc(docText),
}
// Parse methods and collect referenced types
referencedTypes := make(map[string]bool)
for _, method := range interfaceType.Methods.List {
if len(method.Names) == 0 {
continue // Embedded interface
}
funcType, ok := method.Type.(*ast.FuncType)
if !ok {
continue
}
// Check for //nd:hostfunc annotation
methodDocText, methodRawDoc := getMethodDocComment(method)
methodAnnotation := parseHostFuncAnnotation(methodRawDoc)
if methodAnnotation == nil {
continue
}
m, err := parseMethod(method.Names[0].Name, funcType, methodAnnotation, cleanDoc(methodDocText))
if err != nil {
return nil, fmt.Errorf("parsing method %s.%s: %w", typeSpec.Name.Name, method.Names[0].Name, err)
}
service.Methods = append(service.Methods, m)
// Collect referenced types from params and returns
for _, p := range m.Params {
collectReferencedTypes(p.Type, referencedTypes)
}
for _, r := range m.Returns {
collectReferencedTypes(r.Type, referencedTypes)
}
}
// Resolve shared-type aliases against the registry. Host-service schemas
// are not generated (the -schemas pass is capability-only), so the resolved
// shared shapes are not needed here.
sharedAliases, _, err := resolveSharedAliases(referencedTypes, pkgAliasMap, shared)
if err != nil {
return nil, err
}
service.SharedAliases = sharedAliases
// Attach referenced structs to the service (sorted for stable output)
for _, typeName := range slices.Sorted(maps.Keys(referencedTypes)) {
if s, exists := structMap[typeName]; exists {
service.Structs = append(service.Structs, s)
}
}
if len(service.Methods) > 0 {
services = append(services, service)
}
}
}
return services, nil
}
// parseStructs extracts all struct type definitions from a parsed Go file.
func parseStructs(f *ast.File) []StructDef {
var structs []StructDef
for _, decl := range f.Decls {
genDecl, ok := decl.(*ast.GenDecl)
if !ok || genDecl.Tok != token.TYPE {
continue
}
for _, spec := range genDecl.Specs {
typeSpec, ok := spec.(*ast.TypeSpec)
if !ok {
continue
}
structType, ok := typeSpec.Type.(*ast.StructType)
if !ok {
continue
}
docText, _ := getDocComment(genDecl, typeSpec)
s := StructDef{
Name: typeSpec.Name.Name,
Doc: cleanDoc(docText),
}
// Parse struct fields
for _, field := range structType.Fields.List {
if len(field.Names) == 0 {
continue // Embedded field
}
fieldDef := parseStructField(field)
s.Fields = append(s.Fields, fieldDef...)
}
structs = append(structs, s)
}
}
return structs
}
// parseTypeAliases extracts all type alias definitions from a parsed Go file.
// Type aliases are non-struct type declarations like: type MyType string
func parseTypeAliases(f *ast.File) []TypeAlias {
var aliases []TypeAlias
for _, decl := range f.Decls {
genDecl, ok := decl.(*ast.GenDecl)
if !ok || genDecl.Tok != token.TYPE {
continue
}
for _, spec := range genDecl.Specs {
typeSpec, ok := spec.(*ast.TypeSpec)
if !ok {
continue
}
// Skip struct and interface types
if _, isStruct := typeSpec.Type.(*ast.StructType); isStruct {
continue
}
if _, isInterface := typeSpec.Type.(*ast.InterfaceType); isInterface {
continue
}
docText, _ := getDocComment(genDecl, typeSpec)
aliases = append(aliases, TypeAlias{
Name: typeSpec.Name.Name,
Type: typeToString(typeSpec.Type),
Doc: cleanDoc(docText),
IsAlias: typeSpec.Assign.IsValid(),
})
}
}
return aliases
}
// parseConstGroups extracts const groups from a parsed Go file.
func parseConstGroups(f *ast.File) []ConstGroup {
var groups []ConstGroup
for _, decl := range f.Decls {
genDecl, ok := decl.(*ast.GenDecl)
if !ok || genDecl.Tok != token.CONST {
continue
}
group := ConstGroup{}
for _, spec := range genDecl.Specs {
valueSpec, ok := spec.(*ast.ValueSpec)
if !ok {
continue
}
// Get type if specified
if valueSpec.Type != nil && group.Type == "" {
group.Type = typeToString(valueSpec.Type)
}
// Extract values
for i, name := range valueSpec.Names {
def := ConstDef{
Name: name.Name,
}
// Get value if present
if i < len(valueSpec.Values) {
def.Value = exprToString(valueSpec.Values[i])
}
// Get doc comment
if valueSpec.Doc != nil {
def.Doc = cleanDoc(valueSpec.Doc.Text())
} else if valueSpec.Comment != nil {
def.Doc = cleanDoc(valueSpec.Comment.Text())
}
group.Values = append(group.Values, def)
}
}
if len(group.Values) > 0 {
groups = append(groups, group)
}
}
return groups
}
// exprToString converts an AST expression to a Go source string.
func exprToString(expr ast.Expr) string {
switch e := expr.(type) {
case *ast.BasicLit:
return e.Value
case *ast.Ident:
return e.Name
default:
return ""
}
}
// parseStructField parses a struct field and returns FieldDef for each name.
func parseStructField(field *ast.Field) []FieldDef {
var fields []FieldDef
typeName := typeToString(field.Type)
// Parse struct tag for JSON field name and omitempty
jsonTag := ""
omitEmpty := false
if field.Tag != nil {
tag := field.Tag.Value
// Remove backticks
tag = strings.Trim(tag, "`")
// Parse json tag
jsonTag, omitEmpty = parseJSONTag(tag)
}
// Get doc comment
var doc string
if field.Doc != nil {
doc = cleanDoc(field.Doc.Text())
}
for _, name := range field.Names {
fieldJSONTag := jsonTag
if fieldJSONTag == "" {
// Default to field name with camelCase
fieldJSONTag = toJSONName(name.Name)
}
fields = append(fields, FieldDef{
Name: name.Name,
Type: typeName,
JSONTag: fieldJSONTag,
OmitEmpty: omitEmpty,
Doc: doc,
})
}
return fields
}
// parseJSONTag extracts the json field name and omitempty flag from a struct tag.
func parseJSONTag(tag string) (name string, omitEmpty bool) {
// Find json:"..." in the tag
for _, part := range strings.Split(tag, " ") {
if strings.HasPrefix(part, `json:"`) {
value := strings.TrimPrefix(part, `json:"`)
value = strings.TrimSuffix(value, `"`)
parts := strings.Split(value, ",")
if len(parts) > 0 && parts[0] != "-" {
name = parts[0]
}
for _, opt := range parts[1:] {
if opt == "omitempty" {
omitEmpty = true
}
}
return
}
}
return "", false
}
// collectReferencedTypes extracts custom type names from a Go type string.
// It handles pointers, slices, and maps, collecting base type names.
func collectReferencedTypes(goType string, refs map[string]bool) {
// Strip pointer
if strings.HasPrefix(goType, "*") {
collectReferencedTypes(goType[1:], refs)
return
}
// Strip slice
if strings.HasPrefix(goType, "[]") {
if goType != "[]byte" {
collectReferencedTypes(goType[2:], refs)
}
return
}
// Handle map
if strings.HasPrefix(goType, "map[") {
rest := goType[4:] // Remove "map["
depth := 1
keyEnd := 0
for i, r := range rest {
if r == '[' {
depth++
} else if r == ']' {
depth--
if depth == 0 {
keyEnd = i
break
}
}
}
keyType := rest[:keyEnd]
valueType := rest[keyEnd+1:]
collectReferencedTypes(keyType, refs)
collectReferencedTypes(valueType, refs)
return
}
// Qualified reference to the shared `types` package (e.g. types.Track).
// These start with a lowercase package selector, so they must be collected
// before the uppercase check below would skip them.
if strings.HasPrefix(goType, sharedTypesPrefix) {
refs[goType] = true
return
}
// Check if it's a custom type (starts with uppercase, not a builtin)
if len(goType) > 0 && goType[0] >= 'A' && goType[0] <= 'Z' {
switch goType {
case "String", "Bool", "Int", "Int32", "Int64", "Float32", "Float64":
// Not custom types (just capitalized for some reason)
default:
refs[goType] = true
}
}
}
// toJSONName is imported from types.go via the same package
// getDocComment extracts the doc comment for a type spec.
// Returns both the readable doc text and the raw comment text (which includes pragma-style comments).
func getDocComment(genDecl *ast.GenDecl, typeSpec *ast.TypeSpec) (docText, rawText string) {
var docGroup *ast.CommentGroup
// First check the TypeSpec's own doc (when multiple types in one block)
if typeSpec.Doc != nil {
docGroup = typeSpec.Doc
} else if genDecl.Doc != nil {
// Fall back to GenDecl doc (single type declaration)
docGroup = genDecl.Doc
}
if docGroup == nil {
return "", ""
}
return docGroup.Text(), commentGroupRaw(docGroup)
}
// commentGroupRaw returns all comment text including pragma-style comments (//nd:...).
// Go's ast.CommentGroup.Text() strips comments without a space after //, so we need this.
func commentGroupRaw(cg *ast.CommentGroup) string {
if cg == nil {
return ""
}
var lines []string
for _, c := range cg.List {
lines = append(lines, c.Text)
}
return strings.Join(lines, "\n")
}
// getMethodDocComment extracts the doc comment for a method.
func getMethodDocComment(field *ast.Field) (docText, rawText string) {
if field.Doc == nil {
return "", ""
}
return field.Doc.Text(), commentGroupRaw(field.Doc)
}
// parseHostServiceAnnotation extracts //nd:hostservice annotation parameters.
func parseHostServiceAnnotation(doc string) map[string]string {
for _, line := range strings.Split(doc, "\n") {
line = strings.TrimSpace(line)
match := hostServicePattern.FindStringSubmatch(line)
if match != nil {
return parseKeyValuePairs(match[1])
}
}
return nil
}
// parseHostFuncAnnotation extracts //nd:hostfunc annotation parameters.
func parseHostFuncAnnotation(doc string) map[string]string {
for _, line := range strings.Split(doc, "\n") {
line = strings.TrimSpace(line)
match := hostFuncPattern.FindStringSubmatch(line)
if match != nil {
params := parseKeyValuePairs(match[1])
if params == nil {
params = make(map[string]string)
}
return params
}
}
return nil
}
// parseCapabilityAnnotation extracts //nd:capability annotation parameters.
func parseCapabilityAnnotation(doc string) map[string]string {
for _, line := range strings.Split(doc, "\n") {
line = strings.TrimSpace(line)
match := capabilityPattern.FindStringSubmatch(line)
if match != nil {
return parseKeyValuePairs(match[1])
}
}
return nil
}
// parseExportAnnotation extracts //nd:export annotation parameters.
func parseExportAnnotation(doc string) map[string]string {
for _, line := range strings.Split(doc, "\n") {
line = strings.TrimSpace(line)
match := exportPattern.FindStringSubmatch(line)
if match != nil {
return parseKeyValuePairs(match[1])
}
}
return nil
}
// parseKeyValuePairs extracts key=value pairs from annotation text.
func parseKeyValuePairs(text string) map[string]string {
matches := keyValuePattern.FindAllStringSubmatch(text, -1)
if len(matches) == 0 {
return nil
}
result := make(map[string]string)
for _, m := range matches {
result[m[1]] = m[2]
}
return result
}
// parseMethod parses a method signature into a Method struct.
func parseMethod(name string, funcType *ast.FuncType, annotation map[string]string, doc string) (Method, error) {
m := Method{
Name: name,
ExportName: annotation["name"],
Doc: doc,
}
// Parse parameters (skip context.Context)
if funcType.Params != nil {
for _, field := range funcType.Params.List {
typeName := typeToString(field.Type)
if typeName == "context.Context" {
continue // Skip context parameter
}
for _, name := range field.Names {
m.Params = append(m.Params, NewParam(name.Name, typeName))
}
}
}
// Parse return values
if funcType.Results != nil {
for _, field := range funcType.Results.List {
typeName := typeToString(field.Type)
if typeName == "error" {
m.HasError = true
continue // Track error but don't include in Returns
}
// Handle anonymous returns
if len(field.Names) == 0 {
// Generate a name based on position
m.Returns = append(m.Returns, NewParam("result", typeName))
} else {
for _, name := range field.Names {
m.Returns = append(m.Returns, NewParam(name.Name, typeName))
}
}
}
}
return m, nil
}
// typeToString converts an AST type expression to a string.
func typeToString(expr ast.Expr) string {
switch t := expr.(type) {
case *ast.Ident:
return t.Name
case *ast.SelectorExpr:
return typeToString(t.X) + "." + t.Sel.Name
case *ast.StarExpr:
return "*" + typeToString(t.X)
case *ast.ArrayType:
if t.Len == nil {
return "[]" + typeToString(t.Elt)
}
return fmt.Sprintf("[%s]%s", typeToString(t.Len), typeToString(t.Elt))
case *ast.MapType:
return fmt.Sprintf("map[%s]%s", typeToString(t.Key), typeToString(t.Value))
case *ast.BasicLit:
return t.Value
case *ast.InterfaceType:
// Empty interface (interface{} or any)
if t.Methods == nil || len(t.Methods.List) == 0 {
return "any"
}
// Non-empty interfaces can't be easily represented
return "any"
default:
return fmt.Sprintf("%T", expr)
}
}
// cleanDoc removes annotation lines from documentation.
func cleanDoc(doc string) string {
var lines []string
for _, line := range strings.Split(doc, "\n") {
trimmed := strings.TrimSpace(line)
if strings.HasPrefix(trimmed, "//nd:") {
continue
}
lines = append(lines, line)
}
return strings.TrimSpace(strings.Join(lines, "\n"))
}