Skip to content

Replace String with Ulids to store ids #1345

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

chinmaym07
Copy link

@chinmaym07 chinmaym07 commented Jun 12, 2025

Fixes #1125 .

Description


This PR has:

  • been tested to ensure log ingestion and log query works.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added documentation for new or modified features or behaviors.

Summary by CodeRabbit

  • Refactor
    • Updated various user and resource identifier fields across the application to use ULID (Universally Unique Lexicographically Sortable Identifier) instead of generic strings. This impacts user IDs, client IDs, node IDs, request IDs, group IDs, and similar fields in multiple areas such as authentication, streaming, permissions, and filtering.
    • Improved consistency and uniqueness of identifiers throughout the platform.

Copy link
Contributor

github-actions bot commented Jun 12, 2025

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

@chinmaym07
Copy link
Author

I have read the CLA Document and I hereby sign the CLA

@chinmaym07 chinmaym07 force-pushed the rep-string-ulid-1125 branch from 02f8a12 to eae9f5e Compare June 12, 2025 19:42
nitisht added a commit to parseablehq/.github that referenced this pull request Jun 12, 2025
@chinmaym07 chinmaym07 force-pushed the rep-string-ulid-1125 branch from eae9f5e to 5daf562 Compare June 12, 2025 19:42
@chinmaym07 chinmaym07 marked this pull request as ready for review June 13, 2025 05:56
Copy link
Contributor

coderabbitai bot commented Jun 13, 2025

"""

Walkthrough

The changes systematically replace the use of String types for various ID fields and parameters with the ulid::Ulid type across multiple modules. This affects struct fields, method signatures, and type aliases related to identifiers, ensuring IDs are now represented as ULIDs. No logic or control flow is altered.

Changes

File(s) Change Summary
src/cli.rs, src/oidc.rs Changed OIDC-related ID fields from String to ulid::Ulid.
src/connectors/kafka/config.rs Changed Kafka config fields (client_id, group_id, group_instance_id) from String to ulid.
src/correlation.rs Changed CorrelationId and UserId type aliases from String to ulid::Ulid.
src/handlers/http/kinesis.rs Changed Message struct's request_id field from String to ulid.
src/handlers/http/modal/mod.rs Changed NodeMetadata struct's node_id field and constructor parameter from String to ulid.
src/handlers/http/rbac.rs Changed User struct's id field from String to ulid.
src/livetail.rs Changed id parameter and fields in LiveTail methods from String to ulid.
src/parseable/streams.rs Changed Stream struct's ingestor_id and related method parameters from Option<String> to Option<ulid>.
src/prism/home/mod.rs Changed Resource struct's id field from String to ulid.
src/rbac/mod.rs Changed UsersPrism struct's id field from String to Ulid.
src/rbac/user.rs Changed OAuth struct's userid field from String to ulid.
src/storage/mod.rs Changed Owner and Permisssion structs' id fields and constructors from String to ulid.
src/users/filters.rs Changed id fields in FilterBuilder, FilterRules, and Rules structs from String to ulid.

Sequence Diagram(s)

sequenceDiagram
    participant Caller
    participant Module
    participant Ulid

    Caller->>Module: Call method with id: Ulid
    Module->>Ulid: Validate/Generate/Use Ulid
    Ulid-->>Module: Return Ulid value
    Module-->>Caller: Processed result with Ulid id
Loading

Assessment against linked issues

Objective Addressed Explanation
Replace use of String to store IDs with Ulid across all relevant locations (#1125)

Assessment against linked issues: Out-of-scope changes

No out-of-scope changes found.

Suggested labels

for next release

Suggested reviewers

  • nikhilsinhaparseable

Poem

A hop, a leap, a ULID hop,
No more strings for IDs—let’s swap!
Across the code, unique and neat,
Each bunny byte now skips a beat.
With every field, a ULID shines,
Hopping through those data lines!
🐇✨
"""


📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b617229 and e2d4ae2.

📒 Files selected for processing (14)
  • src/cli.rs (2 hunks)
  • src/connectors/kafka/config.rs (4 hunks)
  • src/correlation.rs (2 hunks)
  • src/handlers/http/kinesis.rs (1 hunks)
  • src/handlers/http/modal/mod.rs (3 hunks)
  • src/handlers/http/rbac.rs (2 hunks)
  • src/livetail.rs (3 hunks)
  • src/oidc.rs (2 hunks)
  • src/parseable/streams.rs (4 hunks)
  • src/prism/home/mod.rs (3 hunks)
  • src/rbac/mod.rs (2 hunks)
  • src/rbac/user.rs (2 hunks)
  • src/storage/mod.rs (2 hunks)
  • src/users/filters.rs (2 hunks)
🚧 Files skipped from review as they are similar to previous changes (14)
  • src/users/filters.rs
  • src/handlers/http/rbac.rs
  • src/rbac/mod.rs
  • src/livetail.rs
  • src/handlers/http/kinesis.rs
  • src/handlers/http/modal/mod.rs
  • src/rbac/user.rs
  • src/oidc.rs
  • src/prism/home/mod.rs
  • src/correlation.rs
  • src/cli.rs
  • src/storage/mod.rs
  • src/connectors/kafka/config.rs
  • src/parseable/streams.rs
✨ Finishing Touches
  • 📝 Generate Docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Explain this complex logic.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 23

🔭 Outside diff range comments (9)
src/handlers/http/rbac.rs (1)

54-57: ⚠️ Potential issue

Type mismatch: assigning String to an Ulid

user.username() returns a &str, but the id field is now Ulid; this will not compile. Decide whether
a) the field should remain a String, or
b) you must convert the username to a Ulid (likely wrong conceptually).

If you really need a ULID:

-            id: user.username().to_owned(),
+            id: Ulid::from_string(user.username())
+                .expect("username must be a valid ULID"),

…but double-check the domain model—usernames and IDs are usually distinct.

src/users/filters.rs (1)

91-95: ⚠️ Potential issue

Wrong type for Rules.id

-    pub id: ulid,
+    pub id: Ulid,
src/connectors/kafka/config.rs (2)

21-30: ⚠️ Potential issue

Type mismatch & invalid default for client_id.

pub client_id: ulid,

  1. ulid is not a type – use Ulid.
  2. default_value_t = String::from("parseable-connect") no longer matches the new type and will not compile.
  3. clap doesn’t derive ValueParser for Ulid; you need a custom value_parser (or keep this field as String and convert later).

Minimal compile-fix:

-    #[arg(
-        long = "client-id",
-        env = "P_KAFKA_CLIENT_ID",
-        required = false,
-        default_value_t = String::from("parseable-connect"),
-        value_name = "client_id",
-        help = "Client ID for Kafka connection"
-    )]
-    pub client_id: ulid,
+    #[arg(long = "client-id", env = "P_KAFKA_CLIENT_ID", value_parser = parse_ulid)]
+    pub client_id: Ulid,

and add

fn parse_ulid(s: &str) -> Result<Ulid, String> {
    Ulid::from_string(s).map_err(|e| e.to_string())
}

(or revert to String).
Similar corrections are needed for every field switched to ULID.


72-85: ⚠️ Potential issue

group_id now Ulid → downstream breakage.

ConsumerConfig::validate still calls self.group_id.is_empty() (line 616) and apply_to_config does set("group.id", &self.group_id). Both APIs expect a &str, not an Ulid. Convert to string (or revert to String) and drop the is_empty check.

-    pub group_id: ulid,
+    pub group_id: Ulid,
...
-        if self.group_id.is_empty() {
+        if self.group_id == Ulid::nil() {  // or custom check / remove

and in apply_to_config

-            .set("group.id", &self.group_id)
+            .set("group.id", &self.group_id.to_string())

Also applies to: 80-83

src/storage/mod.rs (1)

187-201: ⚠️ Potential issue

Permisssion struct – multiple type errors and logic bug.

pub id: ulid,
...
pub fn new(id: ulid) -> Self {
    Self {
        id: id.clone(),
        group: id,

Issues:

  1. ulidUlid.
  2. group is String but you assign an Ulid.
  3. Calling .clone() on Ulid already Copy (unnecessary).

Suggested patch:

-    pub id: ulid,
+    pub id: Ulid,
...
-    pub fn new(id: ulid) -> Self {
-        Self {
-            id: id.clone(),
-            group: id,
+    pub fn new(id: Ulid) -> Self {
+        Self {
+            id,
+            group: id.to_string(),
src/livetail.rs (3)

29-33: ⚠️ Potential issue

Import typo blocks compilation.

use ulid: Ulid;

should be

use ulid::Ulid;

35-50: ⚠️ Potential issue

Type inconsistency between pipes and registry.

new_pipe(id: ulid, …) passes an Ulid but SenderPipe.id is still String (type Id = String).
Subsequent .retain(|x| x.id != self.id) compares String with Ulid, causing compile errors and logic flaws.

Decide on a single type:

-type Id = String;
+type Id = Ulid;

Update SenderPipe, ReceiverPipe, registry map key, and all usages to Ulid (or keep everything as String and revert the function signatures).

Also applies to: 83-90


108-125: 🛠️ Refactor suggestion

channel returns mismatched types & redundant clone.

SenderPipe.id expects Id (String) but receives Ulid, then .clone() is called. Remove the clone and align types per previous comment.

Additionally change:

-            id: id.clone(),
+            id,

once the Id alias is fixed.

src/parseable/streams.rs (1)

104-118: ⚠️ Potential issue

Lower-case ulid and optional default collide with new type.

pub ingestor_id: Option<ulid>,
...
ingestor_id: Option<ulid>,
  1. Use Ulid not ulid.
  2. Tests/instantiation still pass None or strings; ensure callers supply Option<Ulid>.
-    pub ingestor_id: Option<ulid>,
+    pub ingestor_id: Option<Ulid>,
...
-        ingestor_id: Option<ulid>,
+        ingestor_id: Option<Ulid>,
♻️ Duplicate comments (2)
src/connectors/kafka/config.rs (1)

105-113: group_instance_id – same issues as above.

Lower-case ulid, invalid default value string, and missing conversion when passed to rdkafka (set("group.instance.id", …)). Fix identically to group_id.

src/parseable/streams.rs (1)

768-769: Streams::get_or_create param type needs update.

ingestor_id: Option<ulid>Option<Ulid> to stay consistent with Stream::new.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 970a5a5 and 5daf562.

📒 Files selected for processing (14)
  • src/cli.rs (2 hunks)
  • src/connectors/kafka/config.rs (4 hunks)
  • src/correlation.rs (2 hunks)
  • src/handlers/http/kinesis.rs (1 hunks)
  • src/handlers/http/modal/mod.rs (3 hunks)
  • src/handlers/http/rbac.rs (2 hunks)
  • src/livetail.rs (3 hunks)
  • src/oidc.rs (2 hunks)
  • src/parseable/streams.rs (4 hunks)
  • src/prism/home/mod.rs (2 hunks)
  • src/rbac/mod.rs (2 hunks)
  • src/rbac/user.rs (2 hunks)
  • src/storage/mod.rs (1 hunks)
  • src/users/filters.rs (2 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
src/handlers/http/kinesis.rs (1)
src/utils/json/flatten.rs (3)
  • flatten (58-93)
  • generic_flattening (269-328)
  • has_more_than_max_allowed_levels (335-348)
🔇 Additional comments (3)
src/oidc.rs (1)

64-66: API expectation mismatch for openid::Client::discover

discover expects client_id: String, but you’re now passing an Ulid.
Convert to a string or keep the original type.

-        DiscoveredClient::discover(self.id, self.secret, redirect_uri.to_string(), self.issuer)
+        DiscoveredClient::discover(self.id.to_string(), self.secret, redirect_uri.to_string(), self.issuer)
src/parseable/streams.rs (2)

29-33: Good: correct import of Ulid.

use ulid::Ulid; is syntactically correct.


173-178: String concatenation with Ulid produces wrong filename.

hostname.push_str(id);

push_str expects &str, but id is Option<Ulid>. Convert properly:

-if let Some(id) = &self.ingestor_id {
-    hostname.push_str(id);
+if let Some(id) = &self.ingestor_id {
+    hostname.push_str(&id.to_string());
 }

@chinmaym07 chinmaym07 force-pushed the rep-string-ulid-1125 branch 3 times, most recently from cb74dd7 to b617229 Compare June 13, 2025 07:42
@chinmaym07 chinmaym07 force-pushed the rep-string-ulid-1125 branch from b617229 to e2d4ae2 Compare June 13, 2025 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

replace use of String to store IDs with Ulid
1 participant