Nvidia Builds AI Safety Team Amid Open-Weight Model Push
Nvidia is quietly building an AI safety and security engineering team to evaluate AI agents before they're deployed. The company has posted several job listings for this new team, including a distinguished engineer position as 'founding technical leader.'
The AI safety team will focus on evaluating AI agents to ensure they are safe and secure, as well as building tools to patch software vulnerabilities. Nvidia is betting on open-weight models, which make their trained weights publicly available, even if the training data and source code stay private.
Nvidia's push towards open models reflects key business incentives for the company, including increased demand for its AI chips and broader adoption of AI by businesses. As AI shifts from chatbots to agents that can access sensitive company data and take real-world actions, trust becomes a critical factor in widespread adoption.