Handle buffer allocation failures - #3152
vinioliveiras wants to merge 2 commits into
Conversation
When GPU memory is exhausted (e.g. a game using most of the VRAM on NVIDIA, where the driver does not fall back to system memory), auxilliary_buffer_t::allocate() returns FAILED and wlroots may also fail to create a texture or an FBO for a buffer. Several callers ignored that and rendered into a NULL buffer/pass, taking the compositor down. - render_pass_t: clear/add_rect/add_texture skip when there is no pass (wlr_renderer_begin_buffer_pass() returned NULL, "Failed to create FBO") or the texture has no wlr_texture. - texture_t::get_width/height and gles_texture_t tolerate a NULL texture (texture id 0, draws nothing) instead of asserting/crashing. - transformer_base_node_t::get_updated_contents() returns nullptr when its buffer or texture can't be allocated; the 2D/3D transformers skip that frame. - view snapshots (close animation) are skipped when the buffer fails. - write_to_file() checks the texture. Refs WayfireWM#3018
Same as the previous commit, for the plugins that allocate their own auxiliary buffers: blur (saved pixels, blur passes; the window is drawn without blur that frame), animate (unmapped view snapshot, squeezimize), workspace-wall and grid's crossfade. Refs WayfireWM#3018
|
I wonder whether it won't be simpler to preallocate a small 1x1 buffer and return that in case of failed allocations. Visually it won't work, asviews with transformers will be reduced to 1x1 buffers so likely will look broken. On the upside, plugins don't have to handle the failure case explicitly. The alternative would be a bigger refactor of most plugins. For example, a view transformer might detect that an allocation has failed, and if so, then defer rendering to its child nodes directly and skip applying the transformation. The effect is not visible, but at least the view does not disappear. For plugins like Cube and Expo, we'd have to pre-allocate all buffers we need and not start the effect if they are not allocated. Maybe for now we can try the first approach - it will 'fix' crashes in all plugins. Then, we can update plugins one by one to handle this in the 'correct' way. |
Upstream review (ammen99, WayfireWM/wayfire#3152): a failed allocation holds a 1x1 buffer made at startup, so any code that ignores the failure (plugins not covered yet: cube, wobbly...) renders into it instead of crashing. The effects already handled keep skipping (better looking). Run tools/build-wayfire.sh to rebuild. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013JHgujc28JdHH6TW3AtgCP
Handle failed GPU buffer allocations instead of crashing
Refs #3018.
When the GPU runs out of memory, Wayfire currently crashes: several places
ignore
buffer_reallocation_result_t::FAILED(or a NULL texture / renderpass from wlroots) and render into nothing. This is easy to hit on NVIDIA,
whose Linux driver does not spill VRAM into system memory: a game using
~6.5 of 8 GB was enough to take the whole session down within minutes.
Following the suggestion in #3018 (disable the effect when there isn't
enough memory), this skips the effect or the frame instead:
core
render_pass_t::clear/add_rect/add_texture: no-op without a pass(
wlr_renderer_begin_buffer_pass()returned NULL, "Failed to create FBO")or without a
wlr_texture.texture_t::get_width/height,gles_texture_t: tolerate a NULL texture(texture id 0, draws nothing).
transformer_base_node_t::get_updated_contents()returnsnullptrwhenthe buffer or its texture can't be made; 2D/3D transformers skip the frame.
plugins
crossfade: skip.
Tested on wlroots 0.20.2 with an RTX 4060 Laptop (8 GB): a game kept the card
at 7.8–7.9 GB for long sessions; the log showed thousands of
Failed to allocate auxilliary buffer/gbm_bo_create failedlines and nocrash, where before the session died (backtraces below). Some crashes this
fixes:
(preceded by
[render/gles2/renderer.c:114] Failed to create FBO)and
Cannot run a render pass without a valid targetaborts fromview_interface_t::take_snapshot()during the close animation.Happy to split this further or change the approach.