feat: support multimodal chat#602
Conversation
3625eac to
7a83ef5
Compare
01cf41b to
9918ad3
Compare
julien-nc
left a comment
julien-nc
left a comment
There was a problem hiding this comment.
- It works very well 👍
- The message attachments could be in a horizontal layout, right now they take a lot of space
- When there are multiple attachments, the viewer does not open when clicking one attachment. It works when there is only one attachment.
I'll make code reviews next week.
The viewer didn't open for input attachments only output attachments. I've switched it to now work for all of them and also made the attachments work horizontally |
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz> Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
Signed-off-by: Lukas Schaefer <lukas@lschaefer.xyz>
019e266 to
dd7156b
Compare
kyteinsky
left a comment
kyteinsky
left a comment
There was a problem hiding this comment.
multimodal, let's gooo!
a few nits that could be ironed out:
nit: creating a new chat with the attachments selected does not clear them up, not sure if it's intentional (since they are already uploaded so would save that for the new chat) but was something unexpected.
maybe a "Clear All" button for the attachments can act as a middle ground keeping the above behaviour.
nit: maybe the filename could be truncated here, or hidden even and can be seen when hovered
| picker: (callback) => getFilePickerBuilder(t('assistant', 'Choose a file')) | ||
| .setMultiSelect(true) | ||
| .allowDirectories(false) | ||
| .addButton({ | ||
| id: 'choose-input-file', | ||
| label: t('assistant', 'Choose'), | ||
| variant: 'primary', | ||
| callback: callback(), | ||
| }) | ||
| .build(), |
There was a problem hiding this comment.
nit: not sure if it's possible to limit the mimetypes here since it cannot be judged for every provider, but we may still limit it to images, videos, audio, and text (including office and pdf) files so files like .zip are not shown as selectable, but as said it may be too much work for little gain since it would fail eventually anyway.
| uploadFiles(files) { | ||
| if (!files || files.length === 0) { | ||
| return Promise.resolve() | ||
| } |
There was a problem hiding this comment.
the this.attachedFileIds.length > MAX_ATTACHED_FILES check may be better here (or in pickerSubmitted) so we don't upload the files at all if the limit exceeds, and a warning/error message can be sent to the user stating what failed instead of just disabling the send button.
a file size limit for all files might also make sense, although generous, it will prevent upload of very large files weighing 2 GB for example.
| onPaste(e) { | ||
| if (!this.multimodalChatAvailable || e.clipboardData.files.length === 0) { | ||
| return | ||
| } | ||
| e.preventDefault() | ||
| this.uploadFiles(e.clipboardData.files) | ||
| }, |
| $content[] = [ | ||
| 'type' => 'text', | ||
| 'text' => $message->getContent(), | ||
| ]; |
There was a problem hiding this comment.
'role' => 'user' maybe since this is the user message
3 participants
Adds support for multimodal chat. Note this does not support ContextAgent right now, so should probably wait to merge it until that is done.
Needs these prs:
You can also now use llm2 instead
For context agent you can use this pr:
🤖 AI (if applicable)