Display / 1080p Video Output on iOS

[evanextreme]

On iOS there is a capability to display a video output stream when connected to a device with USB-C capabilities or via AirPlay. Some apps utilize this functionality to create a separate UX on a display besides just mirroring the device itself. Was wondering if Rootshell could implement this functionality to project a larger display when connected versus the existing mirror. This is especially useful on modern iphones, as it would allow for one to pair the phone with a keyboard and connect it to a computer monitor (or in my case, AR glasses like XREAL) to access their shells.

[kitknox]

It looks like the APIs for this all just changed in iOS 27, so I will need to decide if I wait and only implement this against the updated APIs. Seems worth doing some initial testing. Docs show that up to 40K 60FPS is supported. (Obviously not if connected to XREAL at 1080p).

Should be possible for our Screen Sharing feature to also render on this which itself supports up to 4K 60FPS as well.

[kitknox]

I have an initial proof of concept up and running on these APIs. It turns out that this is also an interesting set of constraints to deliver a good experience with. Apple describes this display mode as non interactive. The challenge is that normal UIKit and SwiftUI chrome doesn’t have a good way to fully inject relayed interactions that I have found so far.

So far the best experience I’ve found is by adding a focus toggle between local and external, and while in external mode flip the controlling iPhone or iPad into a scaled down mirror of the external display and allow for full touch and keyboard/mouse/trackpad interaction with that as well. I experimented with just a rectangle control surface kind of like how AppleTV remotes work, but that was not a good experience and was basically impossible for things like text selection.

There is no system ability to render a mouse pointer etc on these surfaces. I did get Screen Sharing to work and it was kind of nuts to have a full 4K resolution render of my Mac knowing that it had to decode on my iPhone, re-encode, and then stream back to the AppleTV.

I’ve been testing with AirPlay so far and it is interesting how much dramatically better the quality is when AirPlaying to AppleTV instead of directly the LG OLED TV that I have.

I’m still a bit on the fence about shipping this as it requires layering in complexity in quite a few places of the app for a feature I am unsure will have widespread use.

[kitknox]

I pushed up my experimental feature/external-display branch that has the currently buggy implementation if anyone wants to work on this to get it into shape for shipping.