fix: convert branches and users child streams to INCREMENTAL replication by atttiwari · Pull Request #56 · singer-io/tap-gitlab

atttiwari · 2026-07-15T12:48:15Z

Description of change

This PR updates the branches and users child streams to use INCREMENTAL replication (via ChildBaseStream) by propagating the parent project’s updated_at into each emitted record, along with schema/test updates and a version bump. SAC-31625

Changes:

Convert Branches and Users streams from FullTableStream to ChildBaseStream and switch replication method to INCREMENTAL with replication_keys = ["projects_updated_at"].
Add updated_at to the branches and users JSON schemas and add unit tests covering the new behavior.
Bump major package version 2.0.0.

Manual QA steps

Discovery: Running
Sync: Running
Unit Tests: Running
Integration test: Running

Risks

Rollback steps

revert this branch

AI generated code

https://internal.qlik.dev/general/ways-of-working/code-reviews/#guidelines-for-ai-generated-code

this PR has been written with the help of GitHub Copilot or another generative AI tool

Copilot

Pull request overview

This PR updates the branches and users child streams to use INCREMENTAL replication (via ChildBaseStream) by propagating the parent project’s updated_at into each emitted record, along with schema/test updates and a version bump.

Changes:

Convert Branches and Users streams from FullTableStream to ChildBaseStream and switch replication method to INCREMENTAL with replication_keys = ["updated_at"].
Add updated_at to the branches and users JSON schemas and add unit tests covering the new behavior.
Bump package version from 1.1.2 to 1.2.0.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/unittests/test_projects_stream.py	Adds unit tests asserting Branches/Users are `ChildBaseStream`, INCREMENTAL, and set `project_id` + `updated_at` from the parent.
tap_gitlab/streams/users.py	Converts Users stream to INCREMENTAL and sets `updated_at` from the parent project in `modify_object`.
tap_gitlab/streams/branches.py	Converts Branches stream to INCREMENTAL and sets `updated_at` from the parent project in `modify_object`.
tap_gitlab/schemas/users.json	Adds `updated_at` field to support incremental replication key on emitted user records.
tap_gitlab/schemas/branches.json	Adds `updated_at` field to support incremental replication key on emitted branch records.
setup.py	Updates package version to `1.2.0`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

atttiwari · 2026-07-15T12:56:07Z

    def modify_object(self, record, parent_record=None):
-        """Adding project_id and last_committed_date to the record"""
-        if isinstance(record, dict):
-            if parent_record and isinstance(parent_record, dict):
-                record["project_id"] = parent_record.get("id")
+        """Add project_id and parent project's updated_at (used as replication key) to each record."""
+        if isinstance(record, dict) and parent_record and isinstance(parent_record, dict):
+            record["project_id"] = parent_record.get("id")
+            record["updated_at"] = parent_record.get("updated_at")



not required.

atttiwari · 2026-07-15T12:56:19Z

+    def modify_object(self, record, parent_record=None):
+        """Add project_id and parent project's updated_at (used as replication key) to each record."""
+        if isinstance(record, dict) and parent_record and isinstance(parent_record, dict):
            record["project_id"] = parent_record.get("id")
+            record["updated_at"] = parent_record.get("updated_at")



not required

RushiT0122 · 2026-07-20T10:02:05Z

+        "updated_at": {
+            "type": [
+                "null",
+                "string"
+            ],
+            "format": "date-time"
+        },


Why do we need to add this field? If it to bookmark then ideally we should bookmark child stream on parent bookmark.

IMO we don't need to add it.

…eplication-method-for-child-streams

RushiT0122 · 2026-08-07T07:53:16Z

 setup(
    name='tap-gitlab',
-    version='1.1.2',
+    version='1.2.0',


Please make this a major release.

RushiT0122 · 2026-08-19T08:03:17Z

branches and users now obey start_date, contrary to the integration contract.

IncrementalStream.sync() gets the parent bookmark, defaults it to start_date, and only emits records whose projects_updated_at is at or after that value (abstracts.py:171-208). The new child streams use that path and populate projects_updated_at from the parent project (branches.py:32-37, users.py:32-37).

However, the integration expectations still explicitly mark both streams as OBEYS_START_DATE: False (base.py:40-48, base.py:75-83). A project whose updated_at is before the configured start date will therefore produce neither its branches nor users, even though the tests still expect those child streams not to obey the start date.

Update the integration contract to OBEYS_START_DATE: True and verify the intended behavior

RushiT0122 · 2026-08-19T08:06:07Z

updated_since is sent to endpoints that do not use the parent timestamp as a child filter.

Every incremental stream adds updated_since to self.params before calling get_records() (abstracts.py:173-182). For the new streams, that means requests such as:

GET /projects/{id}/repository/branches?updated_since=... GET /projects/{id}/users?updated_since=...

The GitLab branches and project-users endpoints are list endpoints whose filtering parameters do not include updated_since. If GitLab ignores unknown parameters this is harmless, but if it rejects or changes behavior for unknown query parameters, every incremental child sync can fail or return incorrect data.

More importantly, the actual filtering is performed against the parent project timestamp in Python, so sending updated_since to the child endpoint provides no benefit. The child implementation should either avoid adding this parameter for these endpoints or explicitly document and test that GitLab ignores it.

RushiT0122 · 2026-08-19T08:26:45Z

Before this PR, child extraction was already parent-driven because only incrementally eligible projects invoked the child streams. The proposed change correctly labels branches and users as incremental only if projects.updated_at is a valid change cursor for their child data. Since the new projects_updated_at field is copied unchanged onto every child record, this remains parent-driven replication rather than true child-level incremental replication. Please confirm that GitLab advances the parent project’s updated_at for every branch/user mutation; otherwise the PR mainly changes metadata, state, and duplicate suppression without improving child change detection.

As per the guidelines, if the child does not have a replication key available in the API response, we should use the parent-driven replication key.

SAC-31624: Update to fetch groups independently while syncing projects

aacff8c

atttiwari requested a review from Copilot July 15, 2026 12:48

Copilot started reviewing on behalf of atttiwari July 15, 2026 12:48 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

fix: convert branches and users child streams to INCREMENTAL replication

93d9c54

atttiwari force-pushed the SAC-31625/fix-replication-method-for-child-streams branch from 193938b to 93d9c54 Compare July 15, 2026 12:59

RushiT0122 reviewed Jul 20, 2026

View reviewed changes

atttiwari added 3 commits July 21, 2026 10:21

SAC-31624: Addressed reivew comments

1c1aa6d

Merge branch 'SAC-31624-fix-data-count-mismatch' into SAC-31625/fix-r…

6b49fca

…eplication-method-for-child-streams

SAC-31625: fixed reivew comments

3f03384

atttiwari force-pushed the SAC-31625/fix-replication-method-for-child-streams branch from 477c7e3 to 3f03384 Compare July 21, 2026 08:51

Base automatically changed from SAC-31624-fix-data-count-mismatch to master July 30, 2026 05:01

RushiT0122 requested changes Aug 7, 2026

View reviewed changes

atttiwari requested a review from RushiT0122 August 10, 2026 11:04

Updated setup.py and changelog.md file

5c2b872

atttiwari force-pushed the SAC-31625/fix-replication-method-for-child-streams branch from e6bc2ea to 5c2b872 Compare August 11, 2026 05:20

RushiT0122 requested changes Aug 19, 2026

View reviewed changes

akkumar-qlik and others added 4 commits August 24, 2026 13:44

Added review comment changes

4c2c6d4

Added review comment changes

2e3b1fe

Added version change for circle-ci workflow

3ed0c5b

reversted previous changes

f70a0c1

RushiT0122 approved these changes Sep 1, 2026

View reviewed changes

Navigation Menu

Uh oh!

fix: convert branches and users child streams to INCREMENTAL replication - #56

fix: convert branches and users child streams to INCREMENTAL replication#56
atttiwari wants to merge 10 commits into
masterfrom
SAC-31625/fix-replication-method-for-child-streams

atttiwari commented Jul 15, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

atttiwari Jul 15, 2026

Uh oh!

atttiwari Jul 15, 2026

Uh oh!

RushiT0122 Jul 20, 2026

Uh oh!

RushiT0122 Jul 20, 2026

Uh oh!

RushiT0122 Aug 7, 2026

Uh oh!

atttiwari Aug 10, 2026

Uh oh!

RushiT0122 Aug 19, 2026

Uh oh!

RushiT0122 Aug 19, 2026

Uh oh!

RushiT0122 Aug 19, 2026

Uh oh!

atttiwari Aug 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

atttiwari commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of change

Manual QA steps

Risks

Rollback steps

AI generated code

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

atttiwari commented Jul 15, 2026 •

edited

Loading