AI Lens - Revolutionizing User Interaction with Contextual AI
AI “Lens” is a breakthrough AI feature designed to augment user interactions with any UI object visible on the screen. It provides an AI assistant that comprehends and interacts based on the context of the screen elements, significantly enhancing user workflows and task completion rates.
In the same way humans interact by pointing and discussing, AI Lens works in the same way, giving a new relationship to how humans interact with AI assistants to discuss complex data.
Lens operates on three foundational principles:
Co-Work with Context: The AI should act like a colleague, understanding context across platforms.
Trust but Verify: Users can challenge the AI’s responses, promoting trustworthiness.
Point and Ask: Users can point to any object on the screen to interact with the AI, similar to asking a colleague questions about specific items.
Creating Lens involved overcoming several complex hurdles. The primary challenge was to design an AI system that could accurately understand and interact with diverse UI objects across different platforms and applications.
The AI needed to provide context-aware assistance in real-time, maintain an intuitive user experience, and build user trust through verifiable interactions. Additionally, multi-selection and relational understanding between different UI objects added layers of complexity to the project.
Point. Click. Context.
Lens Select
The Lens Select tool activates a context focused AI Assistant
Contextual Selection
Using the select tool the user drags over any visual area of the software product to focus the AI Assistant on the object
Contextual Prompt
Once an area is selected the user can ask the AI anything in reference to that object and get a contextual answer.
Contextual Response
The response is directly related to the visual prompt.
The Process:
Discovery and Hypothesis: Collaborated across teams to understand unique business reasons, user landscapes, and industry challenges. Defined the hypothesis to test.
Prototyping: Created varying fidelity prototypes to test the hypothesis with users, illustrating the intended narrative.
Validation: Conducted user interviews and collected feedback continuously to refine the hypothesis and the prototype.
User Story Mapping: Used workshops to incorporate user feedback and validate the story map with engineering teams, guiding development priorities.
Implementation and Testing: Integrated prototypes into the real environment, gathering immediate feedback and iterating rapidly.
Final Release: Launched the feature based on rigorous testing and validation to ensure it met the user’s needs and the business objectives.
Outcome:
The Lens project resulted in a fully functional feature that enhances user interactions with UI objects through an AI assistant. The benefits of Lens include:
- Improved user workflows and task completion.
- Enhanced user trust through verifiable and trustworthy AI responses.
- Efficient multi-select functionality to handle complex queries involving multiple UI objects.
- Intuitive communication between users and the AI, similar to human collaboration.
Patent
The design and technology behind AI Lens has been submitted for patent protection.
Systems and Methods for Expanding the Security Context of AI
US Patent Application #18428774