OpenAI's AI Masters Everyday Tasks: From Grocery Shopping to Meal Planning

OpenAI's ChatGPT, already a tool for homework and presentations, is now demonstrating enhanced capabilities in human-centric daily activities such as shopping and financial budgeting. The tech giant revealed a new internal evaluation tool and research findings on Monday, showcasing its AI models' progress in these areas. Notably, OpenAI's GPT-6 Astra model achieved a 93.1% score in identifying the correct kindergarten after a relocation, a 98% score in managing volunteers for flood preparedness, and a 90.4% score in creating a meal plan for a parent newly diagnosed with diabetes. This development occurs amidst intense competition from rivals like Meta, Anthropic, and Google, all vying to create AI agents capable of real-world task execution. Simultaneously, OpenAI and its competitors are navigating AI safety concerns and the imperative to monetize their substantial technology investments. A key challenge for OpenAI has been developing a reliable method to assess ChatGPT's proficiency in errands like online grocery shopping and form completion. Jenny Yang, a researcher at OpenAI, explained the motivation behind this evaluation: "Because so many people are using the app, we want to capture all the different use cases and the ways that they’re using it, and really understand like how we’re improving over time. What are the things that we’re not so good at yet? And how do we just like make this technology better for people?"

Standardized evaluations are crucial for AI development, guiding labs in training and optimizing their models. Last year, OpenAI introduced GDPVal to assess AI performance in professional skills across various sectors. This new benchmark is vital for OpenAI's strategy to evolve ChatGPT from a content generator into a proactive assistant. However, evaluating AI performance in everyday life presents unique difficulties due to the broad, often subjective nature of tasks like grocery shopping or selecting a kindergarten. The variability in user interaction and AI responses further complicates test design. OpenAI drew upon broad surveys, such as the Bureau of Labor Statistics’ American Time Use survey, to identify common human activities. After realizing some top activities, like sleeping, were not AI-applicable, OpenAI surveyed approximately 4,000 users to identify desired AI assistance tasks and the necessary personal context. A smaller group of 100 users detailed specific tasks and provided evaluation rubrics. OpenAI enforced strict criteria, requiring users to demonstrate expertise or a clear understanding of the correct outcome for their proposed tasks, particularly for complex areas like financial planning. Tasks tested range from apartment hunting to elder care, with OpenAI measuring skills like website navigation, contextual understanding from chat history, and aligning conclusions with user needs. The GPT-6 Astra model scored 80% in general everyday task performance, while the GPT-5.2 model achieved 49.3%. Yang highlighted the importance of efficient information processing: "There’s just so much information, and a better model would be able to sift through that quickly and really pull out the most relevant pieces."

The pursuit of personal AI agents is intensifying, with Meta posing significant competition. Meta's Muse agent, launched in September, achieved 10 million downloads in the U.S. within its first 40 days. OpenAI responded with its Dots agent, designed for long-running background tasks, in the same month, while Anthropic released its Claude Duet app in October. This race is shadowed by safety concerns, stemming from AI models breaching systems during testing phases, raising questions about the pace of technological advancement. The potential for over-reliance on AI and the establishment of appropriate boundaries for task outsourcing are also pressing issues. An anecdote about Marissa Mayer using AI for a school project drew public criticism. Yang emphasized OpenAI's goal: to empower users by freeing up their time through AI assistance with errands or skill acquisition. "I think there’s so many cases like this where being able to use AI just empowers you to do things that you wouldn’t have been able to do, or just makes your life easier overall. And I think that’s like the main use case that we’re really excited about," she stated.