Text-to-video AI inches closer as startup Runway announces new model

As with any technology, the use of AI in creative projects raises questions about ethics and privacy. One concern is the potential for bias in the algorithms used by AI tools, which can perpetuate and amplify existing societal biases. Additionally, the use of AI to create deep fakes or manipulate images and videos raises concerns about the authenticity of media and the potential for misinformation. Runway Gen-2 is a revolutionary tool that uses advanced artificial intelligence to synthesize new videos from nothing but text.

With the goal of refining bugs and smoothing out their machine-learned processes, this is usually how it goes for all new products and features. With text-to-video, users can describe things or places and the technology will generate a short video based on the description it receives. Text + Image to Video allows users to generate a video using a driving image and a text prompt while the image to video lets you generate a video using only a driving image. The startup is offering a web-based video editor that specialises in AI tools such as pose detection and background removal. Unlike ChatGPT which is primarily a language model, Runway’s text-to-video technology will allow users to type in a description and it’ll generate a corresponding video in the user’s preferred style. The company has found commercial success with its video editing technology, which allows users to remove objects from a video, change the background and track the motion of objects.

Automatic1111 has finally released Stable Diffusion WebUI v1.6.0

Whether you’re an amateur TikToker or a professional movie studio, Runway AI’s tools can help you create content that is innovative, engaging, and inspiring. With its user-friendly browser-based tool, anyone can get started with Runway AI and unleash their creative potential. Runway closed its Series C round a few months after the release of Stable Diffusion, a popular image generation model it had helped develop. This past February, the startup released a generative AI system for creating videos. The system can take an existing clip as input and customize it based on a text prompt or reference image provide by the user.

It provides filmmakers with a powerful tool to engage and captivate audiences. Many great stories, whether told through the written word or on the silver screen, share a common thread—a hero’s journey. The Hero’s Journey is a narrative structure that has been a fundamental part of storytelling for centuries, and it continues to be a powerful tool in filmmaking. And while Runway does offer a free plan, it has a hard credit limit as well as limitations on video length as well as other upscale limits and watermarks.

Artificial Intelligence: Runway AI Lands $141M For Its Generative AI Tools

An article that collected the highest number of picks is dubbed „People’s Choice”. Read some of our other articles before you decide and click this botton; you can only select one article every month. Within Runway’s platform there are over 30 so-called “AI Magic Tools” such as motion tracking, audio clean-up, colorize, subtitle creation, frame interpolation, instant rotoscoping, and background removal with Green Screen. Please note this pages serves informational purposes only and does not constitute an endorsement of any AI tool. Some company descriptions are assisted by our GPT-4 research assistant and are provided without any expressed or implied warranties.

That means the startup is now worth three times more than last December, when it closed a $50 million Series C round led by Felicis Ventures. Use of this site constitutes acceptance of our User Agreement and Privacy Policy and Cookie Statement and Your California Privacy Rights. PA may earn a portion of sales from products that are purchased through our site as part of our Affiliate Partnerships with retailers. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of PA Ad Choices. VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact.

Yakov Livshits
Founder of the DevEducation project
A prolific businessman and investor, and the founder of several large companies in Israel, the USA and the UAE, Yakov’s corporation comprises over 2,000 employees all over the world. He graduated from the University of Oxford in the UK and Technion in Israel, before moving on to study complex systems science at NECSI in the USA. Yakov has a Masters in Software Development.

Runway’s AI tools are also highly customizable, allowing users to tailor them to their specific needs and preferences. Gen-1 focused on transforming existing video footage, letting users input a rough 3D animation or shaky smartphone clip and apply an AI-generated overlay. In the clip below, for example, footage of cardboard packaging is paired with an image of an industrial factory to produce a clip that could be used for storyboarding or pitching a more polished feature. Or it can combine disparate concepts to generate videos that are strangely amusing, like a cow at a birthday party. Runway Text-To-Video Gen-2 AI is a cloud-based AI that can generate videos from text prompts. You can use it to create videos for any purpose, from marketing videos to educational videos to personal videos.

So if you’re at all serious about using this image-to-video tech you’re going to need to pay. According to Runway’s viewpoint, there should be no delay between a concept and its actualization. A tool should be easy to use, free up your mind, and let you focus on your original ideas. With Runway, the business aspires to set the pace for the transformation occurring in all the creative industries. Runway co-founder and CEO Cristóbal Valenzuela told Built In in an email that the company plans to launch a text-to-video tool next year. Remember that Runway is a browser-based tool, so every shot you want to run through it needs to be uploaded.

Runway’s Multi-Modal AI Generates Video From Text And Images

The videos are obviously not photorealistic at this point but you can imagine the creative implications for anyone who wants to make a video but does not, and does not care to, know how to do so traditionally. The same shift is already occurring on the photo side with tools like DALL-E and Midjourney, where people are creating Yakov Livshits images and photographs without ever taking a picture or painting themselves. The software doubles as a video-ready production tool, making it possible to create new, realistic videos without the need for heavy equipment or complex settings. Privacy is another concern surrounding the use of AI in creative projects.

• The characters, scores, b-roll, backgrounds, everything will be able to be generated with AI, and that will open the door for new creators who have important stories to tell.
• Runway’s focus has gradually shifted over the years to generative AI, particularly on the video side.
• In any case, after the release of Stable Diffusion, both companies have become benefactors of the same hype in generative AI.
• I have long been curious about the text-to-video solution’s outcome in the generative AI field, where we generate video based on text instructions (prompts) with the help of artificial intelligence.

Technologies and software programs to create and edit videos that are often called deepfake videos. A few users on Twitter have expressed their excitement about Runway’s Gen-2 and toys with the idea of how text-to-video programs might change the video-making game. Using Runway, I combined the images and prompts to generate a total of 121 video units, out of which 39 made it into the final material. The most significant challenge arose when highlighting close-up body parts, such as tying shoelaces, grasping a doorknob, or closing eyes.

Amazon News

Runway’s tools were also utilized in award-winning blockbusters, like Everything, Everywhere, All at Once. Runway was founded in 2018 by Cristóbal Valenzuela, Anastasis Germanidis, and Alejandro Matamala-Ortiz, who met at NYU’s Interactive Telecommunications Program, a grad program at the intersection of technology and the arts. There, the three had become interested in how to leverage neural networks outside of their normal applications, like self-driving cars, specifically in tools for artists and filmmakers.

They collaborated on machine learning research that soon evolved into Runway, which was founded in 2018. Like early versions of tools such as DALL-E and Midjourney, the technology sometimes combines concepts and images in curious ways. If you ask for a teddy bear playing basketball, it might give a kind of mutant stuffed animal with a basketball for a hand.