Microsoft wants to make Windows an AI operating system, launches Copilot+ PCs

Windows 11’s forthcoming Recall feature can “remember” apps and content a user accessed on their PC weeks or even months ago, for example helping them to find a Discord chat where they were discussing clothes they were considering buying. Users can use Recall’s timeline to “scroll back” to see what they were working on in the recent past and drill down in files like PowerPoint presentations to surface info potentially relevant to their searches.

Microsoft says that Recall can create associations between colors, images and more to let users search for practically anything on their PCs in natural language (not dissimilar to startup Rewind’s tech); developers will be able to improve Recall by adding contextual information to their apps. And Microsoft claims all user data associated with Recall is kept private and on-device — and not used to train AI models, importantly.

Here’s more from Microsoft: “Your snapshots are yours; they stay locally on your PC. You can delete individual snapshots, adjust and delete ranges of time in Settings, or pause at any point right from the icon in the System Tray on your Taskbar. You can also filter apps and websites from ever being saved.”

Image editing and live translations

There’s now more AI in Windows than ever and some of it exclusively on the new Copilot+ PCs.

A new feature called Super Resolution can restore old photos by automatically upscaling them. And Copilot can now analyze images to give users ideas for creative compositions. Through a feature called Cocreator, users can generate images and also ask the AI model to follow what they’re drawing to change or restyle the image.

Elsewhere, Live Captions with live translations translates any audio that passes through a PC — whether from YouTube or a local file — into the language of the user’s choosing. Live translations will initially support around 40 languages including English, Spanish, Mandarin and Russian.

A separate but related new feature in Microsoft Edge offers real-time video translation on sites like LinkedIn, YouTube, Coursera, Reuters, CNBC, Bloomberg and more. Set to become available in the near future, the feature — which supports the translation of Spanish into English and English to German, Hindi, Italian, Russian and Spanish — translates spoken content through both dubbing and subtitles live.

Team Copilot and extensions

Team Copilot is the latest expansion of Microsoft’s growing Copilot suite of generative AI tech. It integrates with Teams, the company’s videoconferencing app, to help manage meeting agendas and take notes that anyone in a meeting can co-author. And it extends to Loop and Planner, Microsoft’s collaboration and planning platforms, to create and assign tasks, track deadlines and notify team members when their input’s needed.

In somewhat related Copilot news, Microsoft has launched (in private preview) Copilot Extensions, which allow developers to extend GitHub’s code-generating tool GitHub Copilot with third-party apps and skills. Launch partners include DataStax, Docker and LambdaTest; extensions will live in the GitHub Marketplace, but developers will also be able to create their own private extensions to integrate with their internal systems and APIs.

Windows Copilot Runtime

Powering capabilities such as Recall and Super Resolution is the Windows Copilot Runtime, a collection of ~40 generative AI models make up what Microsoft describes as “a new layer” of Windows. In tandem with the semantic index, a vector-based system local to an individual Copilot+ PC, the Windows Copilot Runtime allows generative AI-powered apps — including third-party apps — to run without necessarily needing an internet connection.

“[The runtime] consists of ready-to-use AI APIs like Studio Effects, Live Captions translations, OCR, Recall with user activity and [more], which will be available to developers in June,” Davuluri said on Tuesday.

Microsoft says that CapCut, the popular video editor from TikTok owner ByteDance, will use the Windows Copilot Runtime and accompanying new Windows Copilot Library, a set of APIs and AI dev tools, to speed up its AI features. And Meta will add the aforementioned Studio Effects to WhatsApp to deliver features such as background blur and eye contact during video calls.

Upgraded bot builders

Azure AI Studio, the toolset within Microsoft’s Azure OpenAI Service that lets customers combine an AI model and build an app that “reasons over” that data, will soon allow developers to create apps using pay-as-you-go inference APIs — the APIs through which developers can access and fine-tune generative AI models hosted on Azure infrastructure. Microsoft calls this “model-as-a-service,” and it’s launching with models from Nixtla and Core42 to start.

In the adjacent Copilot Studio product suite, Microsoft is launching Copilot agents, which the company describes as AI bots that can “independently orchestrate tasks tailored to specific roles and functions.” (Copilot Studio provides tools to connect Copilot for Microsoft 365, the AI-powered “copilot” in apps like Excel and Word, to third-party data.) Leveraging memory and knowledge of context, Copilot agents can navigate various types of business workflows, learning from user feedback and asking for help when they encounter situations they don’t know how to handle.

Snapdragon Dev Kit

There’s a new dev kit from Qualcomm aimed at developers building apps for Arm-chip-packing Copilot+ PCs

The $899.99 Snapdragon Dev Kit for Windows — which measures around the same width, height and length as Apple’s Mac Mini, incidentally — houses Qualcomm’s Snapdragon X Elite chip paired with 32GB of RAM, 512GB of storage and lots of I/O. The Dev Kit supports Wi-Fi 7 and Bluetooth 5.4, and, via its various USB-C and HDMI ports, it can drive up to three 4K monitors at once.

Phi-3

Microsoft announced a new addition to its generative AI model family Phi, Phi-3-vision, that can do general visual analysis and reasoning tasks, such as answering questions about charts and images. The model can read both text and images and is efficient enough to run on a mobile device.

Phi-3-vision is available in preview, while the model’s previously-announced text-only counterparts — Phi-3-mini, Phi-3-small, and Phi-3-medium — are now generally available.

Partnership with Khan Academy

Microsoft is teaming up with Khan Academy to donate access to cloud compute infrastructure — allowing Khan Academy to offer educators in the U.S. free access to Khan Academy’s AI-powered tools. The two companies will also collaborate to explore opportunities to improve AI apps for math tutoring through generative AI, Microsoft on Tuesday.

We’re launching an AI newsletter! Sign up here to start receiving it in your inboxes on June 5.