After failing to successfully crack the digital assistant market with Cortana a decade ago, Microsoft is re-entering the voice command market with Copilot, the AI assistant, in Windows 11. Starting today, Microsoft is rolling out updates to Copilot's voice and visual features, allowing users to simply say "Hey, Copilot" and ask questions based on what's onscreen, aiming to create a user-friendly experience similar to the ship's computer in "Star Trek."
From understanding screen content to actual action
This new feature allows Copilot to actually "see" what's on your screen. For example, if you're browsing photos of Hawaii, you can directly ask Windows 11 where those photos were taken, ask it to plan a flight itinerary, and even get budget suggestions.
In Microsoft's promotional video, users only need to ask Copilot to show them how to stream music at the "best quality," and the AI will immediately point out the relevant settings in Spotify and recommend the lossless quality option.
Unlike previous voice assistants, Copilot can not only perform web searches or generate AI artwork, but also help users easily complete practical tasks. Microsoft's goal is to enable Copilot to perform Windows tasks, which is the original intention of developing the experimental feature "Copilot Actions".
Privacy Concerns and Security Measures
However, the "Hey Copilot" feature and all Copilot visual features are cloud-based, meaning that the user's desktop image data will be transmitted to Microsoft servers. This requires a certain level of trust, and Microsoft has already lost some of this trust in the hearts of many users after the previous chaos with the Recall feature.
To reassure users, Microsoft emphasized that "Hey, Copilot" is a completely opt-in feature, hidden deep within the Copilot app settings. Similarly, Copilot Actions is an opt-in feature, allowing users to step through all actions in the Copilot app and take over tasks at any time.
Copilot Actions: The Evolution of AI Assistants
Initially launched as a tool for performing tasks on websites, Copilot Actions has been expanded to handle manual tasks, such as resizing and correcting photos in a folder. When questions arise, it prompts users for answers within the Copilot app. More importantly, while Copilot Actions works in the background, users are free to perform other computer tasks.
Microsoft said it has been conducting "extensive" internal testing of Copilot Actions and will gradually roll out the feature to collect feedback. All tasks are performed in a controlled environment, allowing for more granular permission control and runtime isolation to ensure that Copilot does not affect other parts of the system beyond the scope of a specific task.
Fully integrated Copilot experience
In addition to the voice feature, Microsoft also plans to add an "Ask Copilot" search function to the Windows 11 taskbar, touting this feature as part of its plan to turn the taskbar into a "dynamic hub" for getting things done. Like other new features, the Ask Copilot toolbar will be completely user-selectable.
Meanwhile, Microsoft is expanding the AI capabilities built into Windows 11, including new integrations with Manus, an AI agent that can convert multiple files into websites, and Filmora, which lets users create AI-powered movies directly from File Explorer.
For those who have been skeptical of Microsoft's Copilot program so far, if "Hey, Copilot" works as advertised, it will indeed have practical value.
The new "Hey, Copilot" and Copilot visual features are available starting today on all Windows 11 PCs with access to Copilot. According to Microsoft, Copilot Actions and the Ask Copilot taskbar feature will be rolled out "gradually" to Windows 11 Insiders program members.
