UX of GenAI

how do we design UX for AI?

ambient intelligence, inferring intent

• we need to move beyond the constraints of the chat box
• “ambient intelligence” is key - designers must consider where “intelligence” should show up to feel natural
  • system should infer intent to augment the human, like auto-complete
  • there should be a collaborative experience
  • the system should react to signals, like a thermostat reacts to temperature
• example: granola.ai - records conversations ambiently for summarization and recap…operates in the background
  • of course, there is tension between ambience and privacy
    • one must ask: “if this thing i’m building succeeds, what are the downstream consequences?”
• we are again in “early days of UX” when it comes to AI experiences - the rules for what is good or bad have not been fleshed out yet
  • we should be thinking, “what does the next generation of this technology look like?”
• there is a shift to designing reactive probabilistic experiences

hyper-personalization. dynamic, on-the-fly UI

• a vision of the future of HCI :
  • AI is involved in every interface interaction.
  • AI is in every widget
  • there will be intelligent ‘mode’ switching, where one ‘gesture’ could have infinite meanings, implicitly determined by AI through context
  • any pixel can be UI - no more CSS, every single aspect of UI can be dynamically altered, including content structure, mental models, etc.
• proposed scenarios
  • write without ever typing - interactions like ‘rotating’ a paragraph to automatically reorganize sentence structure
  • everything is dynamic - properties are dynamically determined, text boxes have generative autocomplete
  • See: AI-Instruments Paper
  • drawing canvases that autocomplete
    • defining new art brushes with text
  • one gesture, infinite meanings
    • given some lineart, a scribble over the sky could mean color-fill the sky, while a scribble over a lamp could mean to remove that lamp from the drawing
• AI predicts your intention
  • Sequence of interactions and system state inform context
  • questions:
    • how do we embrace ambiguity for delivering magical experiences?
    • what are novel feedforward and negotiation mechanisms?
    • what are robust affordances and semantic anchors?
      • e.g. a book page intuitively indicates a swiping interaction
• “Your interface, your dance partner”
  • co-adaptive, hyper-personalized experiences
  • “the perfect app for you”
  • ^ a point brought up: how is collaboration changed or inhibited when everybody sees a view tailored only for them?

leveraging AI for business

• it is most important to understand the business goals, in order to express opinions and have real influence
• digital spaces should have goal-driven, dynamic navigation - high priority tasks should be dynamically surfaced and easily navigable
• system should understand what task the user is doing, based on where they click, etc.
• products have been designed around personas, structured around “jobs to be done”, defining typical tasks and pain points
  • AI is changing what jobs should be done - personas may no longer be future-proof
• shift away from persona to “goals to be done”
  • classify tasks as tedious, fulfilling, etc. and delegate agents

AI for creativity

• to use AI as a creative, thinking partner, it needs enough context on you - it needs human-scale, long memory → “what deserves to be remembered?”

• one experimental method: routine dual journaling, where user and AI both journal their reflections on their collaborations - builds long-term memory - an imitation of consciousness, cognitive roleplay, simulation → simulacra

  • reflection instead of summary

• LyricStudio - a tool assisting musicians with writing lyrics - next-line generative complete - but not intended to grow a dependency or be addictive

  • users report having lasting creativity boosts even after ceasing to use the app

• this should be a goal of AI systems: amplify human creativity instead of replacing it - non-addictive AI systems

• There is no objective truth, and LLMs should not try to become a “100% accurate”, infallible source - we should not deprive people of the search for truth

• Hallucinations = willingness to be wrong, which is arguably something that should be encouraged in creative contexts - we should be taking advantage of LLMs in exploring the whole probabilistic space, rather than focusing on ‘convergence’ and steering it down the most probabilistic path

AI for ‘companionship’ - supporting a journey, not one task

• human journeys are continuous but,
• “today, we design AI moments in product silos”
  • but users care about navigating towards goals
  • they are seeking (exploring options, information gathering) → striving (goal-setting, prioritizing) → shifting (reflecting, reframing)
  • AI should be able to proactively help with, and adjust to, the context of the person’s journey
• Scene recognition
• Sentiment awareness
• Signals should be ‘linked across these silos’ - frame AI interactions by journey instead of by app
• again, all about inferring from context

considering consequences, trust and mistakes

• LLMs must own mistakes and understand the root cause

• Important to understand:

  • what types of mistakes does AI make?
  • how easy is it to spot those mistakes?
    • it’s hard to fact-check LLM outputs!
  • what are the consequences of those mistakes?

• See: When combinations of humans and AI are useful - A systematic review and meta-analysis

  • Interestingly, studies find that human+AI combos can be less productive than human or AI acting alone

• We should be designing for error - mitigate overreliance, help people recognize errors

  • example: one study highlighted tokens with low generation probability
  • see: HAX toolkit design library
  • need visibility into workings of AI agents

• when LLMs provide ‘reasons’ for their answers this can actually increase overreliance because humans are more inclined to trust it when it could still be wrong

• need multimodal validation

• it all goes back to the question: “is the user able to accomplish their goal?”

• “pathetic fallacy” - projecting human-ness onto inanimate objects - humans will form emotional bonds with AI regardless of their perceived sentience

• “AI personification” = giving AI human attributes - this should not be pursued unless truly necessary for a specific intent, clearly differentiable from real humans, and bias resistant

• currently, AI agents can be difficult to differentiate from people when inserted into workflows, which causes problems or controversy