Skip to content

Handle buffer allocation failures - #3152

Open
vinioliveiras wants to merge 2 commits into
WayfireWM:masterfrom
vinioliveiras:handle-buffer-allocation-failures
Open

vinioliveiras wants to merge 2 commits into
WayfireWM:masterfrom
vinioliveiras:handle-buffer-allocation-failures

Conversation

@vinioliveiras

Copy link
Copy Markdown

Handle failed GPU buffer allocations instead of crashing

Refs #3018.

When the GPU runs out of memory, Wayfire currently crashes: several places
ignore buffer_reallocation_result_t::FAILED (or a NULL texture / render
pass from wlroots) and render into nothing. This is easy to hit on NVIDIA,
whose Linux driver does not spill VRAM into system memory: a game using
~6.5 of 8 GB was enough to take the whole session down within minutes.

Following the suggestion in #3018 (disable the effect when there isn't
enough memory), this skips the effect or the frame instead:

core

  • render_pass_t::clear/add_rect/add_texture: no-op without a pass
    (wlr_renderer_begin_buffer_pass() returned NULL, "Failed to create FBO")
    or without a wlr_texture.
  • texture_t::get_width/height, gles_texture_t: tolerate a NULL texture
    (texture id 0, draws nothing).
  • transformer_base_node_t::get_updated_contents() returns nullptr when
    the buffer or its texture can't be made; 2D/3D transformers skip the frame.
  • View snapshots (close animation) are skipped when the buffer fails.

plugins

  • blur: no saved pixels / blur passes → the window is drawn without blur.
  • animate (unmapped-view snapshot, squeezimize), workspace-wall, grid
    crossfade: skip.

Tested on wlroots 0.20.2 with an RTX 4060 Laptop (8 GB): a game kept the card
at 7.8–7.9 GB for long sessions; the log showed thousands of
Failed to allocate auxilliary buffer / gbm_bo_create failed lines and no
crash, where before the session died (backtraces below). Some crashes this
fixes:

Fatal error: Segmentation fault
#3  wlr_render_pass_add_rect
#4  wf::render_pass_t::clear(wf::regionf_t const&, wf::color_t const&)
#5  wf::render_pass_t::run_partial()
#6  wf::render_pass_t::run(wf::render_pass_params_t const&)
#7  wf::scene::transformer_base_node_t::get_updated_contents(...)
#8  wf::scene::blur_render_instance_t::render(...)::{lambda()#1}::operator()() const

(preceded by [render/gles2/renderer.c:114] Failed to create FBO)

and Cannot run a render pass without a valid target aborts from
view_interface_t::take_snapshot() during the close animation.

Happy to split this further or change the approach.

When GPU memory is exhausted (e.g. a game using most of the VRAM on
NVIDIA, where the driver does not fall back to system memory),
auxilliary_buffer_t::allocate() returns FAILED and wlroots may also fail
to create a texture or an FBO for a buffer. Several callers ignored that
and rendered into a NULL buffer/pass, taking the compositor down.

- render_pass_t: clear/add_rect/add_texture skip when there is no pass
  (wlr_renderer_begin_buffer_pass() returned NULL, "Failed to create FBO")
  or the texture has no wlr_texture.
- texture_t::get_width/height and gles_texture_t tolerate a NULL texture
  (texture id 0, draws nothing) instead of asserting/crashing.
- transformer_base_node_t::get_updated_contents() returns nullptr when its
  buffer or texture can't be allocated; the 2D/3D transformers skip that
  frame.
- view snapshots (close animation) are skipped when the buffer fails.
- write_to_file() checks the texture.

Refs WayfireWM#3018
Same as the previous commit, for the plugins that allocate their own
auxiliary buffers: blur (saved pixels, blur passes; the window is drawn
without blur that frame), animate (unmapped view snapshot, squeezimize),
workspace-wall and grid's crossfade.

Refs WayfireWM#3018
@ammen99

ammen99 commented Oct 3, 2026

Copy link
Copy Markdown
Member

I wonder whether it won't be simpler to preallocate a small 1x1 buffer and return that in case of failed allocations. Visually it won't work, asviews with transformers will be reduced to 1x1 buffers so likely will look broken. On the upside, plugins don't have to handle the failure case explicitly.

The alternative would be a bigger refactor of most plugins. For example, a view transformer might detect that an allocation has failed, and if so, then defer rendering to its child nodes directly and skip applying the transformation. The effect is not visible, but at least the view does not disappear. For plugins like Cube and Expo, we'd have to pre-allocate all buffers we need and not start the effect if they are not allocated.

Maybe for now we can try the first approach - it will 'fix' crashes in all plugins. Then, we can update plugins one by one to handle this in the 'correct' way.

vinioliveiras added a commit to vinioliveiras/Sonata2 that referenced this pull request Oct 4, 2026
Upstream review (ammen99, WayfireWM/wayfire#3152): a failed allocation
holds a 1x1 buffer made at startup, so any code that ignores the failure
(plugins not covered yet: cube, wobbly...) renders into it instead of
crashing. The effects already handled keep skipping (better looking).
Run tools/build-wayfire.sh to rebuild.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013JHgujc28JdHH6TW3AtgCP
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants