TinyGo API Design: "Header" files

[soypat]

I'd like to discuss a design I really enjoy for creating HALs that are well documented and searchable with go doc. This pattern where the pins_exported.go ("header" file) can be applied throughout all embedded Go projects, example(s) below. The design consists of two categories of files, the "Header" files and the implementation files.

"Header" or HAL

The "Header" files contain ALL exported API. Ideally these do not have build tags. See Grove example for an example of a no-build-tag HAL that can compile on any target, including native Go.

// machine/pins_exported.go

//go:build esp32 || rp2 || stm32
// Pin type defined in hardware files. 
// Define hardware which implements it in build tag above
package machine

// Low ...
func (p Pin) Low() {
   p.low()
}

// High ...
func (p Pin) High() {
   p.high()
}

// Get ...
func (p Pin) Get() (level bool) {
   return p.get()
}

Implementation

The implementation files contain unexported functions, including those wrapped by Header/HAL.

// machine/rp2.go

//go:build rp2
package machine

type Pin uint8

func (p Pin) low() {
    rp2.SIO_CLR.Set(1<<p)
}

func (p Pin) high() {
    rp2.SIO_SET.Set(1<<p)
}

func (p Pin) get() (level bool) {
    return rp2.SIO_GET.Get() & (1<<p) != 0
}

Example(s):

• For the Seeed Grove HAL API I have 3 layers, a "header", "subheader/implementation" and the implementation at lowest level: Header and implementation (subheader) and hardware specific implementation. Notice how the header file does not have hardware specific imports!

Why concrete type and not an interface ? (edited)

We need concrete types for HAL definitions for the following reasons:

• Performance: Virtual method calls can have very detrimental effects as discussed in the Pin HAL issue in tinygo drivers repo. We ended up going the function type definition route due to this issue type PinOutput func(level bool)
• Ability to provide additional capabilities: What if your I2C supports an additional capability not supported by the interface? To avoid growing an interface indefinetely for every hardware capability we can use a struct and have hardware specific fields and methods for those who want non-HAL functionality.
• Idiomatic Go route: Implementations are concrete! We use interfaces at the contract level, not the implementation level. i.e: tinygo drivers repo requires interfaces because that is the contract abstraction level for hardware.

Why I want this

This would serve to better standardize APIs. If we have a "header" file then we don't have competing documentation and function signatures for the same method. Standardization is what the world's infrastructure is built on. This is extremely valuable.

It would allow trivial PR reviewing- if someone adds support for a new MCU it is trivial to know whether they implemented the functionality correctly at the API level (functionality may or may not be complete, but at least we guarantee that if it compiles it follows the TinyGo API.

Furthermore, this is an issue today! We have several broken APIs in TinyGo TODAY!! Several Configure methods on certain SPI implementations DO NOT CONFORM to the expected TinyGo API and will break for other targets. We wish to keep TinyGo compatible with all supported architectures since it completely feasible to do so. This proposal simplifies the path to API compliance between targets.

[digitalentity]

I guess you're trying to enforce the same API on all platforms, so the Pin has a well-defined public API.

I suggest we do something based on interfaces rather than wrappers to avoid creating one more level of indirection.
Creating a wrapper around something that is not well-defined (private methods) just shift the problem one level down - into the machine package.

A Pin in your example is an interface implementation, so we need to simply validate that the implementation satisfies the "public contract":

interface GPIOPin {
  func Set(high bool)
  func Get() bool
  func Low()
  func High()
}

interface GPIOPinInterrupt {
  func SetInterrupt(change PinChange, callback func(Pin)) error
}

A Pin implementation can be easily verified at compile time to satisfy the interface:

var _ GPIOPin = (*Pin)(nil) 

[soypat]

Heya! I was quick to write out the propsal and not talk about the why of it. Thank you for your comment, I've edited the proposal with 2 new subsections amendments addressing your concerns. Please let me know if this answers your concerns regarding the why of the header file.

[digitalentity]

The API conformance is exactly what go interfaces are for and tinygo linker should be smart enough to devirtualize the interface if there is only one implementation compiled in (exactly the case with machine APIs).

We can also use the interface as a "contract enforcer" - e.g. Pin is still a direct implementation, interface check merely enforces compliance with the API.

Unfortunately I don't buy the argument about additional capabilities - this is exactly the recipe to end up with incompatible implementations of the same capability in different boards.

[soypat]

The API conformance is exactly what go interfaces are for

Yes. In an ideal world we'd have interfaces.

We can also use the interface as a "contract enforcer" - e.g. Pin is still a direct implementation, interface check merely enforces compliance with the API.

This is a good idea and it has been floated a couple of times. There are ways to use interfaces as compile-time guarantee. This solves the API contract fulfillment guarantee but not the standardized documentation. You'd still have to copy-paste documentation every time you implement the API.

tinygo linker should be smart enough to devirtualize the interface if there is only one implementation compiled in (exactly the case with machine APIs).

The conversation regarding using interfaces spans many, many, many weeks. The consensus we arrived at is that:

• Virtualization can lead to arbitrary overhead
• Virtualization is not always avoidable
• Pins is a great example of a HAL that would have several implementations present
• And other reasons

You can relive the conversation here:

• Original function pointer conversation and virtualization benchmarks
• Rejected Pin interface PR conversation
• Final Consolidated PR using function pointers

Unfortunately I don't buy the argument about additional capabilities - this is exactly the recipe to end up with incompatible implementations of the same capability in different boards.

It is the way hardware works. When a manufacturer adds additional functionality not present on other hardware you'll run into situations where code valid on one target is invalid on another. Another disadvantage of not living in an ideal world. I'm trying to look for a pragmantic tradeoff. TinyGo already has this "incompatible implementation" pattern strewn across the whole project. If you can figure out a better solution, I'd love to hear it.

Finally: What I propose is nothing new. We already do "Header" files in TinyGo and these are the most succesful SDK parts of tinygo. See the src/machine/rp2* family of files. We managed to share a buttload of logic between RP2040 and RP2350 architectures thanks to this pattern. I'd even go as far as to say our SDK is in a better state than Raspberry Pi's own RP2 C SDK. This has real, tangible benefits. Other architectures in tinygo have duplicated code and lead to more maintenance burden. If you have an alternative path please share it. For now I put forth this idea which I believe brings many benefits for little to no tradeoff.

[digitalentity]

Pins is a great example of a HAL that would have several implementations present

I gave it a thought and the public API + implementation seems to be the cleanest solution indeed.

An API contract

It has build tags for every platform that has this API (or no build tags at all if the API is present everywhere)

// machine/pins.go
//go:build esp32 || rp2 || stm32
package machine

// Low ...
func (p Pin) Low() {
   p.low()
}

// High ...
func (p Pin) High() {
   p.high()
}

A machine-specific implementation:

// machine/mach_rp2_pins.go
//go:build rp2
package machine

type Pin uint8

func (p Pin) lowImpl() {
    rp2.SIO_CLR.Set(1<<p)
}

func (p Pin) highImpl() {
    rp2.SIO_SET.Set(1<<p)
}

@soypat essentially this is exactly what you propose, with a few caveats:

1. I don't call the main file "header" (this is very confusing for gophers and that's actually what triggered me here).
2. The "API contract" file doesn't have a suffix, keeping the name short and clean.
3. The "Implementation" file has a prefix mach_<architecture>_ - keeping all machine-specific files colocated in the file listing.

What do you think?