fix(search): isolate Google crawl failures and bound embedding writes

Author: waleedlatif1

.github/workflows/test-build.yml

@@ -233,6 +233,7 @@ jobs:
           lib/knowledge/__integration__/search-source-progress.integration.ts
           lib/knowledge/__integration__/search-source-pagination.integration.ts
           lib/knowledge/__integration__/search-reference-batching.integration.ts
+          lib/knowledge/__integration__/embedding-insert-batches.integration.ts
           lib/knowledge/__integration__/kb-block-search.integration.ts
           lib/core/outbox/service.integration.ts
           lib/knowledge/__integration__/connector-upload.integration.ts

apps/docs/content/docs/search/gmail.mdx

@@ -133,7 +133,7 @@ Search schedules syncs hourly. The first sync lists every thread in scope and ca
 
 **Member accounts:** later syncs use each mailbox's Gmail change history, unless the configuration has a search filter. A full relisting runs about weekly, or sooner if Gmail no longer retains the saved history.
 
-**Service account:** each sync revisits the selected active mailboxes and resumes unfinished listings. A failed mailbox read leaves the crawl incomplete; it does not cause existing indexed mail to be deleted from Search.
+**Service account:** each sync revisits the selected active mailboxes and resumes unfinished listings. If Google reports that a user's mailbox is not set up or returns a mailbox `failedPrecondition`, Sim records a warning and continues with the remaining users. The crawl stays incomplete and retries affected users on the next scheduled crawl; existing indexed mail is not deleted because a mailbox could not be read. Credential, delegation, and Directory failures still stop the crawl.
 
 Updates, removals, and access refresh in the background. Empty mailboxes and filters with no matches complete normally with zero documents. Threads exceeding indexing size limits are skipped and reconsidered when they change.
 
@@ -149,7 +149,7 @@ An individual thread failure does not mean the whole mailbox failed. Sim retries
 | Reconnect | Click **Reconnect** and authorize the same account again. |
 | Unavailable or needs admin attention | Ask your Sim admin to check source status and the deployment's Google OAuth configuration. |
 | Directory or delegation error | Check both central crawl scopes, the service-account key, and the Directory administrator's user-read privileges. A normal OAuth account cannot replace the central service account. |
-| Gmail access fails for a selected user | Verify delegation is authorized and [Gmail is enabled](https://knowledge.workspace.google.com/admin/gmail/control-gmail-access-for-your-organizations-users) for that primary Workspace account. Set **Users** to accounts with Gmail enabled; leaving it blank includes all active users and can stop sync on a service-access error. Aliases and external accounts cannot be selected. |
+| Gmail access fails for a selected user | Verify delegation is authorized and [Gmail is enabled](https://knowledge.workspace.google.com/admin/gmail/control-gmail-access-for-your-organizations-users) for that primary Workspace account. Check the affected users in **Sync history**. Set **Users** to accounts with Gmail enabled if some active users should not be crawled. Aliases and external accounts cannot be selected. |
 | A central source indexes mail but a teammate sees no results | Confirm their verified Sim email is the mailbox's primary email and they belong to the Sim organization. Administrators do not receive other people's mailbox access. |
 
 ## Self-hosted operator setup

apps/docs/content/docs/search/google-calendar.mdx

@@ -131,7 +131,7 @@ Sim indexes event titles, descriptions, times, locations, and the selected atten
 
 Cancelled events, attachment contents, meeting recordings, and transcripts are not indexed. Status entries such as working location, out of office, focus time, and birthdays, and automatically generated reservation events from Gmail are not indexed. Events Google returns only as free/busy blocks, without searchable details, are not indexed. Events outside the selected date window are excluded. Private event details that Google withholds are not available in Search; see [Google's calendar sharing rules](https://developers.google.com/workspace/calendar/api/concepts/sharing).
 
-Search schedules syncs hourly. Event edits, cancellations, access changes, inactive or removed users, and events moving outside the date window are reconciled during completed background syncs. Central crawls page through each selected user and resume unfinished work before removing documents no longer listed. Authorization, quota, and provider failures stop the sync rather than treating unread calendars as empty. The first sync may take longer, and results appear as indexing progresses; Search is not a live Calendar read.
+Search schedules syncs hourly. Event edits, cancellations, access changes, inactive or removed users, and events moving outside the date window are reconciled during completed background syncs. Central crawls page through each selected user and resume unfinished work before removing documents no longer listed. If an individual user's event listing returns a `403` with no reason or only `forbidden`, Sim records a warning and continues with the remaining users. The crawl stays incomplete and retries affected users on the next scheduled crawl; unread calendars are not treated as empty. Credential, delegation, Directory, and other provider failures still stop the crawl. The first sync may take longer, and results appear as indexing progresses; Search is not a live Calendar read.
 
 ## Troubleshooting
 
@@ -146,7 +146,7 @@ When a sync fails, **Sync history** includes the Google API operation, HTTP stat
 | Reconnect | Click **Reconnect** and complete Google authorization again. Allow pop-ups if the connection tab does not open. |
 | Unavailable or needs admin attention | Ask your Sim admin to check source status and the deployment's Google OAuth configuration. |
 | Service-account authorization or Directory error | Confirm both delegated scopes, enabled APIs, and the Directory administrator's user-read privilege. Check whether delegation still awaits approval or propagation. |
-| Calendar is disabled for a selected user | An active Workspace user may have Calendar turned off. [Enable Calendar](https://knowledge.workspace.google.com/admin/users/access/turn-calendar-on-or-off-for-users) for them, or set **Users** to accounts with Calendar enabled. Leaving **Users** blank includes all active users and can stop sync on a service-access error. |
+| Calendar access fails for a selected user | Check the affected users in **Sync history**. An active Workspace user may have Calendar turned off, but a `403` alone does not prove this. Check their access to the selected calendars and [Calendar service settings](https://knowledge.workspace.google.com/admin/users/access/turn-calendar-on-or-off-for-users), or set **Users** to accounts that should be crawled. |
 | User not found or inactive | Use an active primary email in the same Workspace customer. Aliases, external or guest accounts, suspended users, and archived users cannot be selected. |
 | A central source has no results for a teammate | Confirm their primary Workspace email matches their verified Sim email, they belong to the Sim organization, and they are included in **Users**. Check calendar IDs and **Sync history**. |
 

apps/docs/content/docs/search/google-drive.mdx

@@ -155,13 +155,14 @@ Search schedules syncs hourly. Central crawls revisit the selected users' files
 
 ## Troubleshooting
 
-**Directory permission sync failed** means Sim could not fully verify group membership. Check the Directory administrator’s access to the affected group and any nested groups; this is separate from file-download access. An incomplete membership read does not replace the last verified membership, which remains subject to freshness checks.
+Directory permission warnings mean Sim could not fully verify group membership. When other groups refresh successfully and only nested groups outside your Workspace customer deny access or cannot be found, Sim continues syncing content and reports a partial permission sync. An incomplete membership read does not replace or refresh the last verified membership, which remains subject to freshness checks. Directory-wide failures still stop the sync. Check the Directory administrator’s access to the affected group and nested groups; this is separate from file-download access.
 
 | Problem | Next step |
 | --- | --- |
 | Google rejects authorization (`unauthorized_client`) | In **Manage Domain Wide Delegation**, verify the numeric **Client ID** matches `client_id` in the JSON key uploaded to Sim and all required scopes appear under **View details**. Check pending approval and allow time for recent changes to propagate. Changing the OAuth consent screen alone does not authorize delegation. |
 | Directory access failed | Check all four delegated scopes and the **Directory administrator email** user's administrator privileges. A normal Google OAuth credential cannot supply this central Search path. |
 | Missing files in a central crawl | Check **Users**, folder and file-type filters, and whether selected active Workspace users can download the file and read its permissions. Opening a file alone does not prove either. Check Sync history for errors. Files reachable only by excluded or inactive accounts are not crawled; files with unverified permissions stay hidden. |
+| No text could be extracted | Images and scanned PDFs use OCR, but files with no extractable text cannot be indexed. Check whether the original contains readable text; a successful download does not guarantee searchable content. |
 | User not found or inactive | Use a primary email in the same Google Workspace customer. Aliases, external or guest accounts, suspended users, and archived users cannot be selected for crawling. |
 | A teammate sees no results | Confirm they have joined the Sim organization and their verified Sim email matches the Drive permission or group membership. For member accounts, finish their personal Drive connection too. |
 | A public or shared-link file is missing | Check **Openly shared files**. Link-only sharing does not grant Search access. A named user or group permission can still make the file searchable. |

apps/sim/connectors/gmail/company-crawl.test.ts

@@ -409,6 +409,69 @@ describe('company-wide Gmail indexing', () => {
     ).rejects.toThrow('403')
   })
 
+  it('continues after Gmail failedPrecondition with the later mailbox owner ACL intact', async () => {
+    fetchProvider.mockImplementation(async (url: string, init?: RequestInit) => {
+      if (
+        new URL(url).pathname.endsWith('/threads') &&
+        new Headers(init?.headers).get('Authorization')?.includes(ALICE.email)
+      ) {
+        return Response.json(
+          {
+            error: {
+              message: 'private provider response',
+              errors: [{ reason: 'failedPrecondition' }],
+            },
+          },
+          { status: 400 }
+        )
+      }
+      return providerResponse(url, init)
+    })
+    const first = await gmailConnector.listDocuments(
+      'directory-token',
+      CONFIG,
+      undefined,
+      centralContext()
+    )
+    expect(first).toMatchObject({
+      documents: [],
+      reconciliationSafe: false,
+      listingFailures: {
+        count: 1,
+        samples: [
+          {
+            scope: ALICE.email,
+            operation: 'gmail.threads.list',
+            status: 400,
+            reasons: ['failedPrecondition'],
+          },
+        ],
+      },
+    })
+    expect(JSON.stringify(first)).not.toContain('private provider response')
+    const ctx = centralContext()
+    const second = await gmailConnector.listDocuments(
+      'directory-token',
+      CONFIG,
+      first.nextCursor,
+      ctx
+    )
+    expect(second).toMatchObject({
+      hasMore: false,
+      reconciliationSafe: false,
+      listingFailures: first.listingFailures,
+    })
+    expect(second.documents[0].acl).toEqual([`u:${BOB.email}`])
+    const hydrated = await gmailConnector.getDocument(
+      'directory-token',
+      CONFIG,
+      second.documents[0].externalId,
+      ctx
+    )
+    expect(hydrated?.acl).toEqual([`u:${BOB.email}`])
+    expect(hydrated?.content).toContain('Bob private body')
+  })
+
   it('invalidates previous hydration authority when the next mailbox fails', async () => {
     const context = centralContext()
     const first = await gmailConnector.listDocuments('directory-token', CONFIG, undefined, context)

apps/sim/connectors/google-calendar/company-crawl.test.ts

@@ -340,16 +340,53 @@ describe('Google Calendar company crawl', () => {
     expect(next.documents[0].acl).toEqual([`u:${BOB.email}`])
   })
 
-  it.each([401, 403])(
-    'propagates provider HTTP %s without completing a user’s listing',
-    async (status) => {
-      fetchMock.mockResolvedValue(response({ error: { code: status } }, status))
+  it.each([
+    { status: 401, reason: 'authError' },
+    { status: 403, reason: 'insufficientPermissions' },
+    { status: 403, reason: 'SERVICE_DISABLED' },
+  ])(
+    'propagates provider authorization failures without completing a listing: $reason',
+    async ({ status, reason }) => {
+      fetchMock.mockResolvedValue(response({ error: { errors: [{ reason }] } }, status))
       await expect(
         googleCalendarConnector.listDocuments('directory-token', {}, undefined, context())
       ).rejects.toThrow()
     }
   )
 
+  it('continues another user after an unclassified list access denial and retains its diagnostic', async () => {
+    fetchMock.mockResolvedValueOnce(response({ error: { code: 403 } }, 403))
+    const first = await googleCalendarConnector.listDocuments(
+      'directory-token',
+      {},
+      undefined,
+      context()
+    )
+    expect(first).toMatchObject({
+      documents: [],
+      hasMore: true,
+      reconciliationSafe: false,
+      listingFailures: {
+        count: 1,
+        samples: [
+          { scope: ALICE.email, operation: 'calendar.events.list', status: 403, reasons: [] },
+        ],
+      },
+    })
+    const second = await googleCalendarConnector.listDocuments(
+      'directory-token',
+      {},
+      first.nextCursor,
+      context()
+    )
+    expect(second.documents[0].acl).toEqual([`u:${BOB.email}`])
+    expect(second).toMatchObject({
+      hasMore: false,
+      reconciliationSafe: false,
+      listingFailures: first.listingFailures,
+    })
+  })
+
   it('skips a user who became inactive before their page and never delegates to them', async () => {
     mockGetUser.mockResolvedValueOnce({ ...ALICE, active: false })
     const syncContext = context()

apps/sim/connectors/google-drive/directory.test.ts

@@ -3,6 +3,7 @@
  */
 import { beforeEach, describe, expect, it, vi } from 'vitest'
 import { listDomainGroups, openGoogleDirectory } from '@/connectors/google-drive/directory'
+import { ConnectorDirectoryGroupAccessError } from '@/connectors/source-error'
 
 const mockFetch = vi.fn()
 
@@ -234,6 +235,61 @@ describe('the membership a directory reports', () => {
     })
   })
 
+  it.each([
+    { status: 403, reason: 'forbidden' },
+    { status: 404, reason: 'notFound' },
+  ])(
+    'classifies inaccessible external nested groups explicitly: $status $reason',
+    async ({ status, reason }) => {
+      directory({ 'eng@corp.com': [USER('alice@corp.com'), NESTED('restricted@external.com')] })
+      const healthy = mockFetch.getMockImplementation()!
+      mockFetch.mockImplementation(async (url: string) => {
+        if (
+          decodeURIComponent(new URL(url).pathname).includes('/restricted@external.com/members')
+        ) {
+          return jsonResponse(
+            { error: { errors: [{ reason }], message: 'private detail' } },
+            status
+          )
+        }
+        return healthy(url)
+      })
+      const failure = await membersOf(GROUP).catch((error: unknown) => error)
+      expect(failure).toBeInstanceOf(ConnectorDirectoryGroupAccessError)
+      expect(failure).toMatchObject({
+        cause: { status, diagnostic: { operation: 'directory.members.list', reasons: [reason] } },
+      })
+      expect(String(failure)).not.toContain('private detail')
+    }
+  )
+
+  it.each([
+    { email: 'restricted@corp.io', status: 403, reasons: ['forbidden'] },
+    { email: 'restricted@external.com', status: 403, reasons: [] },
+    { email: 'restricted@external.com', status: 403, reasons: ['forbidden', 'unknownReason'] },
+    {
+      email: 'restricted@external.com',
+      status: 403,
+      reasons: ['forbidden', 'insufficientPermissions'],
+    },
+    { email: 'restricted@external.com', status: 401, reasons: ['authError'] },
+  ])(
+    'does not classify uncertain or customer-owned failures as external access failures: $email $status $reasons',
+    async ({ email, status, reasons }) => {
+      directory({ 'eng@corp.com': [NESTED(email)] })
+      const healthy = mockFetch.getMockImplementation()!
+      mockFetch.mockImplementation(async (url: string) => {
+        if (decodeURIComponent(new URL(url).pathname).includes(`/${email}/members`)) {
+          return jsonResponse({ error: { errors: reasons.map((reason) => ({ reason })) } }, status)
+        }
+        return healthy(url)
+      })
+      const failure = await membersOf(GROUP).catch((error: unknown) => error)
+      expect(failure).not.toBeInstanceOf(ConnectorDirectoryGroupAccessError)
+      expect(failure).toMatchObject({ status })
+    }
+  )
+
   /** A directory that hiccups must not cost a group its membership; transient errors are retried. */
   it('retries a transient directory error before giving up', async () => {
     directory({ 'eng@corp.com': [USER('alice@corp.com')] })

apps/sim/connectors/google-drive/directory.ts

@@ -16,6 +16,7 @@ import {
   fetchGoogleDriveWithRetry,
   GoogleDriveApiError,
 } from '@/connectors/google-drive/google-drive-errors'
+import { ConnectorDirectoryGroupAccessError } from '@/connectors/source-error'
 import type {
   ConnectorDirectory,
   ConnectorDirectoryGroup,
@@ -249,7 +250,28 @@ async function listGroupMembers(
       return
     }
 
-    for (const member of await membersOf(groupId)) {
+    let members: RawMember[]
+    try {
+      members = await membersOf(groupId)
+    } catch (error) {
+      const groupDomain = emailDomain(groupId)
+      if (
+        depth > 0 &&
+        groupDomain &&
+        !customerDomains.includes(groupDomain) &&
+        error instanceof GoogleDriveApiError &&
+        error.reasonsComplete &&
+        ((error.status === 403 && error.reasons.length === 1 && error.reasons[0] === 'forbidden') ||
+          (error.status === 404 && error.reasons.length === 1 && error.reasons[0] === 'notFound'))
+      ) {
+        throw new ConnectorDirectoryGroupAccessError('An external nested group cannot be read', {
+          cause: error,
+        })
+      }
+      throw error
+    }
+
+    for (const member of members) {
       if (member.status && member.status.toUpperCase() !== 'ACTIVE') continue
       const type = member.type?.toUpperCase()