Skills evaluation#38
Open
Tellili wants to merge 14 commits into
Open
Conversation
…ing) Two-skill workflow mirroring the prescriptive pattern: - rai-predictive-modeling: concepts, Snowflake loading, task relationships, graph edges, and PropertyTransformer features. Includes node-classification, link-prediction, and regression examples with generic concept names (no domain lock-in). - rai-predictive-training: GNN constructor, fit, predictions, evaluation, register/load. Includes regression-specific sanity checks guidance (epoch count, R^2 < 0 interpretation, target-profiling). Discovery integration (minimal additions on top of latest main): - rai-discovery/SKILL.md: add rai-predictive-modeling to the Formulation skill list. - rai-discovery/references/predictive.md: replace "Future" platform-status language with a two-modes description (pre-computed vs GNN training); cross-reference the new skills. - rai-graph-analysis/SKILL.md: one-line cross-reference to rai-predictive-modeling for GNN graph construction. Rebased from branch tip onto origin/main; prior 19 commits of pre-split history dropped in favor of a single clean commit against latest main.
…rce skill Pulls in content present in the rai-predictive source skill (PyRel repo) that hadn't reached the rai-predictive-training target. SKILL.md: - Predictions section: add Pattern 1 (attribute bind) / Pattern 2 (Python variable bind) explanation with [Duplicate relationship] warning - Predictions section: add gnn.prediction_concept direct-access subsection - Evaluation & Debugging: add gnn.dataset.metadata_dict, basic visualize_dataset() variant, and pydot note - Common Pitfalls: add 7 missing rows (select fragments to train/validation, reassigning Source.predictions, model_name/version_name with spaces, duplicate (model_name, version_name), register_model on load-mode, fit on load-mode, load on fit-mode) references/task-types-and-metrics.md: - Restore note that @k is optional (target had reduced this to a single example, dropping the "omit for no top-k cutoff" guidance) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sync rai-predictive-training with source skill (gap fixes)
- Replace Beer.churn placeholder with generic Source/predictions in the new prediction_concept Directly snippet (matches Source/Target convention used elsewhere in the skill). - Drop the wrong "Passing select(...) fragments to train=/validation=" Pitfalls row: PyRel's GNN constructor accepts both Relationship and Fragment (verified in source: train: Optional[Relationship | Fragment | Chain] with explicit isinstance check). The "Alternative: select() fragments" section in task-relationships.md correctly advertises this. - Drop duplicate "Source.predictions twice" Pitfalls row -- already covered by the Pattern 1 / Pattern 2 prose in the Predictions section. - Compress the "Accessing prediction_concept Directly" subsection from a full H3 with code block to an inline note (one-line code). - Compress the inspect-the-dataset code block: keep one visualize_dataset variant (with show_dtypes=True) instead of two. SKILL.md: 517 -> 496 lines, back under the 500 cap.
The "Resume suspended GPU pools before runs" paragraph and the matching "gnn.fit() hangs with no error" Pitfalls row were added based on a single test report. The predictive team can't reproduce the behavior in their own runs, so the guidance is removed pending root-cause clarity to avoid steering users toward a workaround for an issue that may not be universal. Auto-suspend, multi-engine sizing, and GPU pairing notes (separate concerns) are unchanged.
- rai-predictive-training/SKILL.md: add a "By user intent" bullet list in the Summary so an agent reading the skill can quickly route to the right sections based on goal (train+val, train+predict+downstream, or train+register+reload). Same Model carries through all three; only the gnn.* call sequence and which sections you exercise differ. - rai-predictive-modeling/SKILL.md: add a "User-input boundary" callout at the top of Define and Populate Concepts -- the user-input boundary is the 3 prompts in auto-discovery.md (source FQNs, task FQNs, experiment artifact location); auto-derive everything else from INFORMATION_SCHEMA / DESCRIBE TABLE. - rai-predictive-modeling/references/auto-discovery.md: rename the "What to Auto-Discover" section to "What to Auto-Discover (and what NOT to ask)" with explicit "Do not ask the user" framing -- column names, PKs, FKs, label/target columns, timestamp columns, task type, feature types are all auto-derivable; asking the user creates friction (often they don't know without checking the schema). Both SKILL.md files stay under the 500-line cap (modeling 307, training 497).
Made-with: Cursor
Replace blanket GRANT ALL ON SCHEMA with the specific grants the predictive reasoner needs: USAGE on the schema and CREATE EXPERIMENT. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cafzal
requested changes
May 1, 2026
cafzal
left a comment
cafzal
left a comment
Collaborator
There was a problem hiding this comment.
Hi, thanks for this contribution. Please discuss adding to https://github.com/RelationalAI/rai-agent-evals repo with @larf311 instead.
cafzal
force-pushed
the
predictive_skills_structured
branch
3 times, most recently
from
May 4, 2026 22:53
931d82d to
b556191
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.