Address Dexcom and Abbott data ingestion issues

[darinkrauss]

• Address Dexcom and Abbott data ingestion issues
• Retry Dexcom API calls with exponential fallback during task run
• Do not specify last sync time for Dexcom data range API, per Dexcom
• Handle oddities in Dexcom API response
• Drain and close HTTP response bodies
• Allow refresh of provider session, meaning particular to provider
• Do not persist expired token to allow retry refresh
• Increase summary update running maximum duration to four minutes
• Use platform parser for token data
• Redact provider session for logging
• Refactor UnstickTasks to report tasks unstuck
• Allow clearing error from data source and task
• Handle error marshaling failures due to invalid Meta field
• Add debug logging for unusual data source and provider session conditions
• Fix issues with provenance collection
• Rename structure NotParsed to ReportNotParsed
• Add NotParsed to return all not parsed fields
• Add total_insulin tool
• Various minor updates
• Add and update tests
• https://tidepool.atlassian.net/browse/BACK-3927
• https://tidepool.atlassian.net/browse/BACK-3613
• https://tidepool.atlassian.net/browse/BACK-3973
• https://tidepool.atlassian.net/browse/BACK-3964

[Copilot]

Pull request overview

This PR focuses on improving reliability and correctness of third-party data ingestion (notably Dexcom, plus some Abbott-related dependency updates) and tightening request/response handling across services. It introduces a generic retry mechanism, improves parsing/“not parsed” reporting, adds provider-session refresh support, and addresses a number of operational issues such as draining HTTP response bodies.

Changes:

• Add a reusable request retrier with exponential backoff and wire it into the Dexcom client; adjust Dexcom fetch logic to match Dexcom guidance (e.g., omit lastSyncTime) and handle response oddities.
• Refactor parsing utilities to return unparsed fields and separate “report not parsed” behavior; improve error serialization robustness (e.g., marshal failures due to invalid Meta).
• Add provider-session refresh capability (API + client + provider hook), improve provenance collection utilities, and standardize draining/closing HTTP response bodies.

Reviewed changes

Copilot reviewed 83 out of 87 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
work/store/structured/mongo/mongo.go	Logging/error-handling cleanups around Mongo operations.
work/service/coordinator.go	Adjust error message for invalid process result.
test/rest/rest.go	Add Response interface + go:generate for mocks.
test/rest/rest_mocks.go	New generated gomock for Response.
task/task.go	Use Serializable.HasError() when checking task error presence.
task/store/store.go	Change UnstickTasks to return task list instead of count.
task/store/mongo/mongo.go	Refactor UnstickTasks implementation and type-filter selectors.
task/service/service/service.go	Inject Dexcom retrier into Dexcom client construction.
task/queue/queue.go	Update unstick/dispatch/complete logging and dispatch flow.
summary/task/updaterunner.go	Increase max summary update duration to 4 minutes.
structure/parser/object_parser.go	Switch to any, add NotParsed map + ReportNotParsed behavior.
structure/parser/object_parser_test.go	Update tests for NotParsed/ReportNotParsed semantics and any.
structure/parser/array_parser.go	Switch to any, add NotParsed map + ReportNotParsed behavior.
structure/parser/array_parser_test.go	Update tests for NotParsed/ReportNotParsed semantics and any.
structure/parser.go	Update parser interfaces for any + NotParsed/ReportNotParsed split.
services/tools/total_insulin/total_insulin.go	Add CLI tool to compute total insulin totals by type/subtype.
service/middleware/auth.go	Update token hashing calls to new crypto hash signatures.
request/values_parser.go	Switch to any, add NotParsed map + ReportNotParsed behavior.
request/retrier.go	New generic retrier with exponential backoff + jitter helpers.
request/retrier_test.go	Tests for retrier behavior (success/failure/backoff/jitter/logging).
request/request.go	Add DrainAndClose helper for safely draining/clsoing bodies.
request/parser.go	Use DrainAndClose on responses; switch to ReportNotParsed.
provider/test/provider.go	Add provider test hook for OnRefresh.
provider/provider.go	Extend provider interface with OnRefresh.
provider/debug.go	Add debugger for data sources/provider sessions with redaction.
plugin/abbott/go.sum	Add go.sum entries for new Abbott plugin dependency.
plugin/abbott/go.mod	Add indirect dependency github.com/gowebpki/jcs.
oura/webhook/webhook.go	Use fmt.Appendf to avoid string formatting allocation patterns.
oura/service/api/v1/event.go	Comment tweak for provider sessions pluralization.
oura/provider/config_test.go	Remove AuthStyleInParams assignment in test setup.
oauth/work/mixin.go	Avoid persisting expired token to allow future refresh retries.
oauth/work/mixin_test.go	Update tests to match new ExpireToken behavior (no persistence).
oauth/test/oauth_mocks.go	Extend mock provider with OnRefresh.
oauth/provider/provider.go	Add no-op OnRefresh; use fmt.Appendf in state hashing.
errors/test/errors.go	Use math/rand/v2; switch helper maps/meta to any.
errors/errors.go	Add errors.As/LastCause, Serializable.HasError, safer marshal handling, any conversions.
errors/errors_test.go	Update tests for NewSerializable(nil) behavior and serialized outputs.
dexcom/provider/provider.go	Add provider OnRefresh implementation to trigger task availability.
dexcom/fetch/runner.go	Adjust Dexcom range fetch and unauth retry cause detection; avoid persisting expired tokens.
dexcom/fetch/runner_test.go	Update expectations for data range behavior and call counts.
dexcom/event.go	Handle Dexcom health/value oddities; default missing blood glucose unit.
dexcom/event_test.go	Update/extend tests for new Dexcom event validation behavior.
dexcom/dexcom_test.go	Naming cleanup for transmitter ID constant.
dexcom/client/client.go	Add retrier support + request duration threshold constant; retry Dexcom API requests.
dexcom/client/client_test.go	Update constructor tests for new retrier argument.
data/source/test/source.go	Use crypto test helper for md5 hashes.
data/source/store/structured/mongo/mongo.go	Use Serializable.HasError for error persistence decisions.
data/source/source.go	Always retain parsed Serializable (even if nil error) when error field exists.
data/source/source_test.go	Update expected parsed Error values under new Serializable semantics.
data/service/api/v1/v1.go	Add provenance extraction helper + improved XFF selection helper.
data/service/api/v1/v1_test.go	Add tests for provenance/XFF selection helpers.
data/service/api/v1/users_datasets_create.go	Switch to any and use ReportNotParsed + new provenance helper.
data/service/api/v1/summary.go	Drain/close response bodies on certain error paths; rename clinicID variable.
data/service/api/v1/summary_test.go	New test file for realtime request query parsing.
data/service/api/v1/datasets_data_create.go	Switch to any, use new provenance helper, import cleanup.
data/service/api/v1/datasets_data_create_provenance_test.go	Remove old provenance collection tests (replaced by v1_test.go).
data/raw/store/structured/mongo/mongo.go	Logging cleanup; fix some error returns and post-update re-fetch.
data/raw/store/structured/mongo/mongo_test.go	Add missing-context and invalid-document tests for raw store APIs.
data/data_set.go	Use ParseDataSetClient helper instead of manual parsing.
crypto/test/crypto.go	Update hashing helpers to pass bytes to crypto hash APIs.
crypto/crypto.go	Change hex hash APIs to accept bytes to avoid implicit conversions.
crypto/crypto_test.go	Update tests for new byte-based crypto hash APIs.
consent/loader/loader_test.go	Remove unnecessary .Times(1) expectations.
clinics/test/clinics.go	Refactor imports/types; add RandomClinicID helper.
client/client.go	Replace local drainAndClose with request.DrainAndClose helper.
blob/service/api/v1/v1.go	DrainAndClose blob content bodies.
blob/client/client.go	DrainAndClose bodies on header parse errors.
auth/token.go	Add OAuthToken.Redacted for logging-safe token representation.
auth/test/provider_session.go	Add ProviderSessionRefresh test helpers + object builders.
auth/test/provider_session_client.go	Add RefreshProviderSession test client support.
auth/test/auth.go	Rename New* helpers to Random* (keep deprecated wrappers).
auth/test/auth_mocks.go	Add RefreshProviderSession method to auth client mock.
auth/service/test/service_mocks.go	New generated gomock for auth service interfaces.
auth/service/service/client.go	Add RefreshProviderSession service method calling provider OnRefresh.
auth/service/service.go	Add go:generate for service mocks.
auth/service/api/v1/provider_session.go	Add POST /refresh endpoint for provider sessions.
auth/service/api/v1/provider_session_test.go	Add tests for provider session refresh endpoint behavior.
auth/providersession/test/provider_session_mocks.go	Add RefreshProviderSession to provider session client mock.
auth/provider_session.go	Add ProviderSessionRefresh model + ProviderSession.Redacted/ProviderSessions.Redacted.
auth/provider_session_test.go	Add parse/validate/serialize tests for ProviderSessionRefresh.
auth/client/external.go	Parse token validation response via TokenData using structure parser; DrainAndClose serverlogin responses.
auth/client/client.go	Add client method for RefreshProviderSession endpoint.
auth/auth_test.go	Fix nil context test to use context.Context(nil).
appvalidate/palm_tree_secrets.go	DrainAndClose partner API response body.
appvalidate/coastal_secrets.go	DrainAndClose partner API response body.

Files not reviewed (3)

• auth/providersession/test/provider_session_mocks.go: Generated file
• auth/service/test/service_mocks.go: Generated file
• auth/test/auth_mocks.go: Generated file

Comments suppressed due to low confidence (1)

data/service/api/v1/summary.go:129

• In the clinics.ErrorCodeClinicClientFailure branch, res may be nil (or the meta may not be a *http.Response), but the code unconditionally uses res.StatusCode/res.Body, which can panic. Guard the Reader call and fall back to InternalServerError when the response meta is missing.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 99 out of 103 changed files in this pull request and generated 4 comments.

Files not reviewed (3)

• auth/providersession/test/provider_session_mocks.go: Generated file
• auth/service/test/service_mocks.go: Generated file
• auth/test/auth_mocks.go: Generated file

[Copilot]

Pull request overview

Copilot reviewed 99 out of 103 changed files in this pull request and generated 2 comments.

Files not reviewed (3)

• auth/providersession/test/provider_session_mocks.go: Generated file
• auth/service/test/service_mocks.go: Generated file
• auth/test/auth_mocks.go: Generated file

[toddkazakov]

LGTM

[toddkazakov]

I have found id tokens useful for debugging purposes in the past - consider stripping the id token signature and keeping the rest intact

[darinkrauss]

A couple of thoughts:

1. I'd rather go the whitelist route for the ID token claims. (In general, there are a variety of standard OIDC claims that expose personal info that wouldn't be useful for debugging. For example, full name, phone number, gender, DOB.) Do you have any specific, non-personal claims that you use for debugging purposes? (I could see the require claims; namely, iss, sub, aud, exp, iat.)
2. The ID token is currently stored as a raw string at this level. It is only parsed in provider-specific code. We can certainly parse it here, whitelist certain claims, and the serialize it again for logging, if you want.

Thoughts?

[toddkazakov]

I've used sub and email to verify that the minted token is for the intended account, iat and exp for determining whether a tokens was considered valid at the time a request was made.

The base branch was changed.