Adobe teases new AI-generative video capabilities rolling out soon

After bringing generative AI tools to Photoshop, software giant Adobe will soon be adding similar capabilities to video editing tools like Premiere Pro. On Wednesday, Sept. 11, Adobe teased AI video advancements to the Firefly Video Model that are estimated to hit before the end of the year. Tools like text-to video and photo-to-video generation are expected to arrive directly on Firefly’s browser-based software, while tools like Generative Expand will be integrated into the editing software Premiere Pro.

The announcement isn’t the first time Adobe has teased AI-generated video features, following a beta earlier this year that helped remove objects from a video, as well as existing machine-learned audio capabilities. The upcoming tools feel much like the video version of Photoshop’s existing options for using AI to remove an object, expand the edges of a photo, or add objects with text-to-image.

And while this is undeniably a leap forward for the technology, it’s also a little scary too. Let’s break it down:

Adobe is adding generative AI to Firefly and Premiere Pro

Text-to-video generation looks like it could be a game changer

The upcoming AI features will be integrated into two different places. The first is Firefly.Adobe.com, Adobe’s browser-based software. By the end of the year, the Firefly software will include text-to-video generation, which allows users to type in a description into a prompt box and have the AI create a video based on what is written.

The second feature headed to Firefly is image-to-video generation. This tool mixes an existing image and a text prompt to generate a video. In a sample of the technology, Adobe took an image of the Milky Way and created a video zooming out from that photo to show the stars reflected in a close-up of a person’s eye.

The third feature is heading directly into Premiere Pro is generative expand. Unlike the tool in Photoshop with the same name, this generative expand isn’t used for “uncropping” an image, but rather for extending a video’s length to make it better fit the edit. For example, Adobe says the tool can be used to fill gaps if the original footage was too short, or to help create smoother transitions.

Adobe’s new Firefly AI-generated videos are not trained on copyrighted content

The company says their models are trained on public domain and licensed content and therefore safe for commercial use

One of Firefly’s key features is that Adobe says the generated content is safe for commercial use. Many generative AI programs like ChatGPT, Google Gemini, and Grok were built on data scraped from the internet. That data sourcing has already led to a number of AI companies landing in legal hot water.

But, according to Adobe, data used to train Firefly is made up entirely of licensed content and public domain works. Adobe’s licensed content typically means Adobe Stock, where creatives submit and sell their work in an online marketplace. Because the data is sourced this way, Adobe says that the Firefly Video Model is safe to use for commercial work, which could mean that the ads and movies of the future could have bits of AI, or even a majority of AI.

While the technology brings more ways to use AI in creative filmmaking, copyright laws will likely mean that Firefly is used to enhance an existing creative vision, rather than to create one entirely from scratch. While Adobe’s demonstrations suggest that a generated video is entirely possible in the future, “art” needs to be created by a human to be eligible for copyright protection. That means films generated entirely by AI could, theoretically, be easily “pirated” with few legal ramifications.

Adobe’s upcoming generative video AI looks scarily realistic

Their technology handily creates textures for skin, fabric and more

As generative AI technology advances, determining if a video is real or fictional with a quick glance is becoming increasingly hard to do. One of the signs of AI-generated imagery, for example, is a lack of texture. But, in the videos Adobe shared demonstrating the new Firefly capabilities, a close-up of a man’s face in the rain showed distinct texture in the beard and skin. Another video of dancing yarn monsters was clearly intended as an animation rather than a realistic video but also showed texture in the yarn.

Like other generative AI, the impact of Firefly’s new video technology will likely depend on the ethics of whoever is behind the keyboard. Extending an existing video or adding creative effects and B-Roll with generative AI seems innocent enough. In fact, I use similar tools to do things like remove a trashcan from the background of a portrait in Photoshop.

The tool could also be used to create graphics that would otherwise be impossible, or at least very expensive. One of the unique tools that the Adobe Firefly Video Model offers is options to control things like camera angle and movement. In another demonstration, the company created a drone shot flying over a volcano, with hot lava landing on the lens itself. With Firefly, that sort of content could be created without investing thousands of dollars in drone equipment.

But, in the hands of a more nefarious user, the technology will likely further the ease with which deep fakes, political propaganda, and misleading ads can be created. The realism demonstrated in the examples suggests Firefly could generate videos that are difficult to distinguish from the real thing. Of course, software makers are notorious for only sharing the best examples and leaving out the cringe-worthy results, so we’ll have to wait and see just how realistic the software actually is.

Adobe

While Adobe’s participation in the Content Authenticity Initiative and licensed training data makes the company one of the more ethical brands building AI, Adobe did not share whether the content created by Firefly will be properly labeled as such. Currently, images generated at firefly.AI.com use content credentials inside the metadata that indicate the image is AI-generated. Someone with the skills to read metadata can easily determine that the image is AI-generated. In addition, Photoshop users can opt out of the young Content Credentials program and leave those AI labels out. It’s unclear what labels the upcoming video features will have.