TOP SECRET META AIs Mind Blowing ROBOT Workers Revealed
Full tutorial link > https://www.youtube.com/watch?v=hYcnA9E1xw8
LSC: Language-guided Skill Coordination for Open-Vocabulary Mobile Pick-and-Place. Get ready to witness a groundbreaking revolution that will redefine our everyday lives. Brace yourself for the arrival of an extraordinary breed of worker robots, equipped with the ability to multitask and comprehend natural language. These remarkable machines are poised to enter our households, similar to the way robot vacuum cleaners have become a staple in modern homes. Prepare to be amazed as we delve into the incredible advancements in robotics and explore how these intelligent beings are set to revolutionize our daily routines. Stay tuned for an awe-inspiring journey into the future of automation and the limitless possibilities that lie ahead.
Article Link
https://languageguidedskillcoordination.github.io/
Our Discord server
https://bit.ly/SECoursesDiscord
If I have been of assistance to you and you would like to show your support for my work, please consider becoming a patron on 🥰
https://www.patreon.com/SECourses
Technology & Science: News, Tips, Tutorials, Tricks, Best Applications, Guides, Reviews
https://www.youtube.com/playlist?list=PL_pbwdIyffsnkay6X91BWb9rrfLATUMr3
Playlist of StableDiffusion Tutorials, Automatic1111 and Google Colab Guides, DreamBooth, Textual Inversion / Embedding, LoRA, AI Upscaling, Pix2Pix, Img2Img
https://www.youtube.com/playlist?list=PL_pbwdIyffsmclLl0O144nQRnezKlNdx3
From the official page
CVPR Expo Meta AI: Location: Meta AI Booth/Time: June 20, 21, 22, 00:10:00 AM to 6:00PM
CVPR Demo Track: Location: Booth ID: 3, Demo ID: 44/Time: June 22, 00:10:00 AM to 6:00PM
We demonstrate Language-guided Skill Coordination (LSC) for open-vocabulary mobile pick-and-place on Spot in partially pre-mapped environments (known receptacle locations, unknown object or other obstacle locations). A user provides a natural language instruction: "Bring me the chocolates box, cereal box, and pill bottle, and put them on the bedroom table". And the robot navigates to the location of the target objects and places them on the room table.
Approach
Natural language provides a simple way for humans to specify a goal for a robot, such as "go to the kitchen and bring the sugar to me." However, current state-of-the-art language-based embodied AI systems do not have such capability since they rely on a fixed-set vocabulary that cannot generalize to diverse instructions. In this demo, we propose a method that uses large language models (LLMs) to receive a free-form natural language instruction for object rearrangement.
The proposed method "Language-guided Skill Coordination" (LSC) consists of three parts: (1) an LLM that takes natural language instruction as input and makes calls to a library of skills, along with their corresponding skills' input parameters such as the target object name, (2) a perception module that provides ground-truth locations of receptacles on that map and open-vocabulary object detection, and (3) a voice-to-text model that processes the audio into text.
For example, the top-most video shows a user saying "Bring me the chocolates box, cereal box, and pill bottle, and put them on the bedroom table". The LLM takes the text input and makes calls to low-level skills: Nav(counter), Pick(cereal), Nav(room table), and Place(). When executing Nav(counter), a perception module is queried to get the location of the counter. Then this location is fed into a reinforcement-learning-trained navigation policy to let Spot navigate to the target. After Spot finds the counter, Pick(cereal) is used to pick up the target object. The pick policy is trained to move the arm to the target grasping location of the object with the input of the object bounding box returned by the perception module. Finally, Spot uses Nav(room table) to find the room table and then calls Place() to place the object there. The entire process repeats for rearranging chocolates and pills.
In terms of robustness of LSC, the navigation policy is able to take an alternative route to avoid collision with humans. In addition, the pick policy is able to change the grasping location if the object moves or the base is not closed enough to the receptacle. These show that LSC is robust to disturbance.
This demo pieces many efforts together within the Embodied AI team at FAIR. For example, one critical component of this demo is the library of skills. They are from Adaptive Skill Coordination for Robotic Mobile Manipulation paper.
Limitations and Future Work
There are several limitations and areas for future work. First, while our demo is a reasonably general approach, we do not integrate Spot with the skills such as opening doors or drawers. This will improve generalization to other mobile manipulation tasks. Second, we run our demo in static environments where the location of the object is approximately fixed and the robot is the only agent.
-
00:00:00 Meta AI released amazing demo videos for their upcoming event.
-
00:00:04 Let's watch them together.
-
00:00:10 Okay so the first command is taking chocolate box and cereal box to the bedroom table.
-
00:00:25 Okay all three of them go to the same place.
-
00:00:33 You notice that there are also other objects in the scene so it has to differentiate between
-
00:00:38 different objects and translate the command and understand those objects.
-
00:00:47 Very nice.
-
00:00:58 Okay first one is done.
-
00:01:08 I think in this example there should have been a bottle on the table as well
-
00:01:13 along with plushies so we would see how it performs good or not.
-
00:01:46 Okay it dropped the cup and didn't take it back.
-
00:01:56 But this is amazing.
-
00:01:57 From natural command it is understanding the object and the commands
-
00:02:03 and executing them.
-
00:02:04 You see it's forgotten that ball and returned back to get it done.
-
00:02:15 This is amazing.
-
00:02:16 Okay dropped again but not taking back.
-
00:02:21 Of course this is just demo therefore we can't expect it to be fully functional.
-
00:02:28 Everything is shared in this page.
-
00:02:30 I will put the link of this page into the description of
-
00:02:33 the video.
-
00:02:34 I think these robots will make the next revolution in our robotic usage in our homes
-
00:02:41 like robot vacuum cleaners.
-
00:02:43 They will get into our houses.
-
00:02:46 They will make our job.
-
00:02:47 However probably they will also make us more lazy therefore
-
00:02:53 people will likely to get even more fat.
-
00:02:56 Unfortunately this will be a side effect but I think they are coming very very strong
-
00:03:02 and very very fast.
-
00:03:04 Hopefully see you in another amazing video.