Workshop on Machine Learning for Automation @ ICRA 2024

Type and Duration

Full-day workshop; preferred day May 13, 2024


Professor (Samuel) Qing-Shan Jia
Tsinghua University, Beijing, China
Email for Qing-Shan Jia | Website for Qing-Shan Jia

Assistant Professor Bing Yan
Rochester Institute of Technology, NY, USA
Email for Bing Yan | Website for Bing Yan

Associate Professor Aparna Varde
Montclair State University, NJ, USA
Email for Aparna Varde | Website for Aparna Varde

Professor Bengt Lennartson
Chalmers University of Technology, Gothenburg, Sweden
Email for Bengt Lennartson | Website for Bengt Lennartson

Professor Maria Pia Fanti
Polytechnic of Bari, Bari, Italy
Email for Maria Pia Fanti | Website for Maria Pia Fanti


The workshop on Machine Learning for Automation intends to provide a great opportunity for researchers and practitioners from machine learning and automation communities to exchange ideas on research challenges and opportunities. Using semiconductor manufacturing as an example, we invite keynote speakers to discuss the scheduling, optimization, energy management, logistics, control of robotic arms, and deep machine learning. We also invite posters and demos to show case the successful case studies around the theme. A one hour panel discussion on the Future of Machine Learning for Automation will provide in-depth interaction between the participants and the leading experts in the field. This workshop fits nicely into the main theme of the ICRA 2024 “CONNECT+”. Supported by x Technical Committees of IEEE RAS, this workshop will promote the interdisciplinary research of machine learning for automation, and will help these TCs to develop continued efforts to promote research in this field.

Content of the Workshop

Machine learning has been changing significantly not only how we live but also how to create. As a key application field of automation, manufacturing has been reinvented by machine learning in the past decades. Hundreds of thousands of sensors have been embedded into the machines and workshops, monitoring every detailed aspects of the process, and collecting huge amount of data for decision making. Logistic networks have been reinvented to not just respond to demand but foresee the requirement and intelligently use various delivery methods to satisfy the manufacturing process, making it smooth, flexible, and resilient. Energy consumption and CO2 emission has been an important new consideration to make the manufacturing sustainable. Optimization of the movement of the robotic arms has also been improved significantly by latest sensing and learning algorithms. In this workshop we share with the audience these exciting latest development in machine learning for automation (6 invited keynote speakers), to show case successful stories (2 invited demos), and to discuss current research challenges and opportunities (1 panel discussion, and 1 poster session). This workshop has received support from various IEEE RAS TCs such as the TC on Machine Learning for Automation, TC on logistics, TC on semiconductor manufacturing, TC on smart building, and TC on Digital Manufacturing and Human Machine Interaction. This workshop provides a unique opportunity for the machine learning experts to understand the state of art in manufacturing, a typical example of automation system, and for experts in automation to know the latest development in machine learning. This fits nicely into the main theme of ICRA 2024 “CONNECT+”.

This workshop has 8 invited keynote speakers covering the various important aspects in manufacturing including scheduling (Mengchu), logistics (Maria Pia), energy management (Yang), robotics (Xiang), learning (Aparna), intelligence (Fan-Tien), optimization (Bengt) and commercial package (Matlab). The speakers are composed of researchers at different stage of their academic career, namely two assistant professors (Yu Yang and Xiang Li), one associate professors (Aparna Varda), and four full professors (Maria Pia, Mengchu, Fan-Tien, and Bengt), as well as an industrial speaker (Matlab). This diversified composition of speakers will allow the speakers and the audience to have rich feedback and in-depth conversation throughout the one-day program.

All the invited speakers plan to attend the workshop, give the presentation, and participate the panel discussion in person. For the convenience of our speakers, we will provide hybrid mode for registered participants, including the regular onsite participation and online zoom meetings. For on-site participants, we are working on sponsored free lunches with leading industry partners in the field. For the zoom meeting up to 100 online participants, it will provided by the organizers’ personal account and therefore free of charge. When the number of registered participants exceed 100, we will stream the talks and collect questions through emails. The questions will be filtered and fed to the speakers in real time. Depending on the time, questions will be addressed according to their presence.

The organizing team of this workshop has rich experience in organizing similar one-day workshops previously. One example is that this team has successfully organized a one-day workshop on Machine Learning for Automation during the 2023 IEEE International Conference on Automation Science and Engineering in Auckland, New Zealand. That workshop was well attended by more than 200 registered participants from industry and academia as shown below.

Group shot of attendees of a previous conference.
Conference room with people sitting at circular tables.

Invited Keynote Speakers

Portrait of Mengchu Zhou.Category: Machine learning for semiconductor manufacturing - challenges and opportunities (Semiconductor TC)

Title: Machine Learning Approaches to Transforming Semiconductor Manufacturing Industry from Automation to Intelligenization

Abstract: Machine Learning Approaches and Industry 4.0/5.0 intends to address a fast-changing and challenging manufacturing environment with diverse demands, short order lead-time and product life cycle, limited capacities, and highly complex process technologies. A manufacturing system integrated with AI, machine learning, big data analytics, digital twin, Internet of Things, and Internet of Behaviors is capable of performing real-time monitoring and optimization of manufacturing processes in various aspects from high level strategic resource and production planning down to real-time equipment-level smart dispatching and predictive maintenance. By fully using real-time data and machine learning approaches, the system is able to help manufacturers shorten production and R&D processes, increase production capacity, reduce production cost, guarantee product quality, and improve product yield. It is suitable to help not only high-tech industries such as semiconductor wafer fabrication, but also conventional labor-intensive sectors. This talk illustrates the transformation of semiconductor manufacturing activities from automation to intelligenization by using Machine Learning and Industry 4.0/5.0 technologies through real-life wafer fabrication applications.

Bio Sketch: (Fellow, IEEE) received his B.S. degree in Control Engineering from Nanjing University of Science and Technology, Nanjing, China in 1983, M.S. degree in Automatic Control from Beijing Institute of Technology, Beijing, China in 1986, and Ph. D. degree in Computer and Systems Engineering from Rensselaer Polytechnic Institute, Troy, NY in 1990. He joined New Jersey Institute of Technology (NJIT), Newark, NJ in 1990, and has been Distinguished Professor in Electrical and Computer Engineering since 2013. His research interests are in Petri nets, semiconductor manufacturing automation, Internet of Things, Internet of Behaviors, and AI. He has over 1100 publications including 14 books, 750+ journal papers (600+ in IEEE transactions), 31 patents and 32 book-chapters. He is the founding Editor of IEEE Press Book Series on Systems Science and Engineering, and Associate Editor of IEEE Internet of Things Journal, IEEE Transactions on Intelligent Transportation Systems, and IEEE Transactions on Systems, Man, and Cybernetics: Systems. He was Editor-in-Chief of IEEE/CAA Journal of Automatica Sinica from 2018-2022. He is a recipient of Humboldt Research Award for US Senior Scientists from Alexander von Humboldt Foundation, Franklin V. Taylor Memorial Award and the Norbert Wiener Award from IEEE Systems, Man and Cybernetics Society, Computer-Integrated Manufacturing UNIVERSITY-LEAD Award from Society of Manufacturing Engineers, and Excellence in Research Prize and Medal from NJIT. He is a highly cited scholar and ranked top one in the field of engineering worldwide in 2012 by Web of Science. His Google citation count is over 62100 with h-index being 124. He was ranked #99 in the world among the 2023 Top 1000 Scientists in Computer Science in the World, He is a life member of Chinese Association for Science and Technology-USA and served as its President in 1999. He is a Fellow of International Federation of Automatic Control (IFAC), American Association for the Advancement of Science (AAAS), Chinese Association of Automation (CAA) and National Academy of Inventors (NAI).

Portrait of Xiang Li.Category: robots in manufacturing

Title: Robotic Dexterous Manipulation of Deformable Objects: Theory and Experiments

Abstract: The task involving deformable objects (DOs), such as cables, tissues, and flour, can be found in many applications, such as manufacturing, medical care, and home service. Controlling the robot to autonomously and intelligently manipulate DOs can automate such tasks and hence provide better service for human society. However, compared with the rigid object, it is challenging for the robot to manipulate DOs, mainly because of its deformable nature. As a result, the deformation model is usually unknown and varies significantly; the feature may be occluded and unobservable; the planned path needs to be shaped online according to the unforeseen deformation; the object’s infinite degrees of freedom are difficult to control. This talk discusses the robotic dexterous manipulation method for DOs, and it covers all the aspects of modeling, perception, planning, and control. Real-world experiments and applications of the proposed method are presented.

Bio Sketch: Xiang Li is an Associate Professor with the Department of Automation, Tsinghua University. He has been the Associate Editor of IEEE Robotics and Automation Letters since 2022 and the Associate Editor of IEEE Transactions on Automation Science and Engineering since 2023. He was the Associate Editor of IEEE Robotics & Automation Magazine from 2019 to 2021 and the Associate Editor of ICRA from 2019 to 2021. He received the Highly Commended Paper Award in 2013 IFToMM, the Best Paper in Robotic Control in 2017 ICAR, the Best Application Paper Finalists in 2017 IROS, the T. J. Tarn Best Paper in Robotics in 2018 IEEE ROBIO, and the Best Paper Award in 2023 ICRA DOM Workshop. He is the Program Chair of the 2023 IEEE International Conference on Real-time Computing and Robotics. His research interests include robotic manipulation, visual servoing, micro/nano robots, and human-robot interaction.

Portrait of Yu Yang.Category: Energy management for green automation (Smart Building TC)

Title: Distributed Nonconvex Optimization with Guaranteed Convergence for Smart and Automated Buildings

Abstract: By coordinating the nodes or agents to solve small-sized subproblems instead of relying on a central computer to solve large-scale optimization problems, distributed optimization is in growing demand by diverse networked systems for efficient, scalable, and robust computation. Examples include but not limited to smart buildings, smart grids and machine learning tasks which are commonly faced the difficulty to handle nonconvex objectives or constraints. While for convex problems, substantial distributed methods are available, the results for the more general nonconvex counterparts are extremely lacking. The direct extensions of distributed convex optimization to nonconvex settings often fail to achieve convergence. In this talk, we would like to introduce a new distributed nonconvex method with guaranteed convergence which is expected to find broad applications both in the area of machine learning and engineering systems. Specifically, we proposed the idea of updating dual variables with the general framework of method of multipliers (MM) for constrained optimization in a discounted manner which is the key to ensure convergence in nonconvex settings. A showcase of the method to the control of multi-zone heating, ventilation, and air-conditioning (HVAC) with improved energy efficiency and human comfort for smart and automated buildings will be discussed.

Bio Sketch: Dr. Yu Yang is a faculty with School of Automation Science and Engineering, Xi’an Jiaotong University, Shaanxi, China. Prior to that, she received her B.E. degree from Huazhong University of Science and Technology, Wuhan, China, in 2013, and Ph.D. degree from Tsinghua University, Beijing, China, in 2018. Between 2018 to 2021, she worked as a postdoctoral scholar with Berkeley Education Alliance for Research in Singapore (BEARS), affiliated with University of California, Berkeley. Her research interests lie in the control, optimization and decision-making of Cyber-Physical Systems (CPS) including smart buildings and smart grids. She has special interest towards developing new theories and smoothing the applications of distributed optimization and learning. Part of her researches are already available from Automatica, IEEE Transactions on Smart Grids, IEEE Transactions on Control Systems Technology, etc. Since 2021, she has served as the associate editor of Results in Control and Optimization.

Portrait of Maria Pia FantiCategory: Machine learning in logistics for manufacturing

Title: Machine learning and deep reinforcement learning applied to cooperative, connected and automated vehicles

Abstract: Machine learning (ML) and deep reinforcement learning (DRL) have the potential to bring about significant impacts in automation across various industries and domains. The number of cooperative, connected and automated vehicles (CCAVs) in urban areas will gradually increase in the near future. As a consequence, mixed traffic made of both regular human driven and CCAVs will likely be a typical scenario over the next few years. Connected and automated vehicles can benefit the whole traffic experience by preventing collisions and optimizing traffic waves, by developing and implementing innovative services. The talk will explain how ML and DRL techniques can be applied for a full integration of CCAVs in the real traffic for transportation of both passengers and goods. The goal is providing benefits to all citizens and positive impacts for society are: i) safety (i.e., reducing the number of road accidents caused by human error; ii) environment (i.e., reducing transport emissions and congestion by smoothening traffic flow and avoiding unnecessary trips); iii) inclusiveness (i.e., ensuring inclusive mobility and good access for all).

Bio Sketch: Maria Pia Fanti received the Laurea degree in electronic engineering from the University of Pisa, Pisa, Italy. She was a visiting researcher at the Rensselaer Polytechnic Institute of Troy, New York, in 1999. Since 1983, she has been with the Department of Electrical and Information Engineering of the Polytechnic University of Bari, Italy, where she is currently a Full Professor of system and control engineering and Chair of the Laboratory of Automation and Control. Her research interests include management and modeling of complex systems, such as transportation, logistics and manufacturing systems; discrete event systems; Petri nets; consensus protocols; fault detection. Prof. Fanti is IEEE fellow and has published more than 340 papers and two textbooks on her research topics. She was senior editor of the IEEE Trans. on Automation Science and Engineering and she is Associate Editor of the IEEE Trans. on Systems, Man, and Cybernetics: Systems. She was member at large of the Board of Governors of the IEEE Systems, Man, and Cybernetics Society, and currently she is member of the AdCom of the IEEE Robotics and Automaton Society, and chair of the Technical Committee on Automation in Logistics of the IEEE Robotics and Automation Society. Prof. Fanti was General Chair of the 2011 IEEE Conference on Automation Science and Engineering, the 2017 IEEE International Conference on Service Operations and logistics, and Informatics and the 2019 IEEE Systems, Man, and Cybernetics Conference.

Portrait of Bengt Lennartson.Category: Machine learning for optimization

Title: Efficient Planning and Control based on CP-SAT and Reinforcement Learning

Abstract: Automation solutions for complex systems, including flexible manufacturing, energy distribution, and smart cities, are dependent on efficient planning and control strategies. Generation of such control systems often requires efficient optimization tools, typically based on operation research (OR), such as mixed integer linear programming (MILP) and/or constraint programming (CP). Some satisfiability (SAT) solvers, mainly developed for formal verification, are also able to generate optimal solutions. One example is Z3Opt that is able to generate optimal plans clearly more efficiently than the best MILP solvers. More recently, Google's OR-Tools has combined CP and SAT (CP-SAT), resulting in 100-1000 times faster generation of standard optimal plans. In this talk we show how this extremely efficient tool can be used for generation of optimal schedules based on Petri nets, a generic approach that can be used in many automation applications. The proposed solution is non-trivial, but guidelines are given how to use CP-SAT in an efficient way. Data-driven optimization based on reinforcement learning (RL) is an additional step to achieve robust and flexible planning and control strategies. The best RL solvers are mostly based on deep learning. In this talk we also show how much more data-efficient RL can be achieved based on a recently proposed simplified linear regression approach. Instead of requiring more than 100.000 time steps as in deep RL, an optimal state feedback controller can be achieved in about 500-1000 time steps. This simple strategy opens up new learning-based optimal control strategies in applications where only a limited amount of data is available.

Bio Sketch: Bengt Lennartson received the Ph.D. degree from the Chalmers University of Technology, Gothenburg, Sweden, in 1986. Since 1999, he has been a Professor of the Chair of Automation at the Department of Electrical Engineering, Chalmers University of Technology, where he was the Dean of Education from 2004 to 2007. Since 2005, he has also been a part time Professor at University West, Trollhättan, Sweden. He has (co)authored three books and more than 300 peer reviewed articles in international journals and conferences. His main areas of interest include discrete-event systems, AI planning and learning, as well as sustainable production. Prof. Lennartson was the General Chair of the 11th IEEE Conference on Automation Science and Engineering, CASE 2015, and the 9th International Workshop on Discrete Event Systems, WODES’08. He was an Associate Editor of Automatica from 2002 to 2005 and IEEE Transactions on Automation Science and Engineering from 2012 to 2015, and he is a Fellow of the IEEE for his contributions to hybrid and discrete event systems for automation and sustainable production.

Portrait of Aparna Varde.Category: Latest development in machine learning for manufacturing

Title: Deep Learning with Explainable AI Models in Robot Training, Autonomous Driving & Drone Data Analytics

Abstract: Deep learning models, thriving on CNN (convolutional neural networks) and other architectures, are useful in automation when extensive datasets are available for knowledge discovery. These models provide high accuracy and are preferred due to mimicking the human brain, yet they are “opaque-box” paradigms lacking explainability. Explainable AI models, including decision trees and knowledge bases, offer “clear-box” paradigms enhancing comprehension and interpretability. Both paradigms are advantageous, and can possibly be used with mathematical modeling for optimization as needed. They can be combined for enhanced results: higher accuracy through deep learning, and enhanced interpretability via explainable AI. The latter is vital when co-learning occurs, e.g. robots with humans, robots learning from each other, and next-generation multipurpose robots. It is important to understand how concepts are learned, which can also aid failure-analysis and error-rectification. Moreover, if huge datasets are not available for deep learning, the processes can be complemented by explainable AI, e.g. knowledge bases to infuse semantic and pragmatic information. This can be crucial in autonomous decision-making for first-time scenarios or relatively new encounters by robots. Similar claims apply to fully autonomous / semi-autonomous driving and smart mobility contexts. For instance, vehicles lack commonsense reasoning by default (which human drivers have), hence training them solely based on deep learning may not suffice in unfamiliar situations. Pragmatics via commonsense knowledge can augment decision-making in autonomous vehicles, especially to prevent unforeseen risks. Likewise, drones automate data collection to assist humans; yet the data analysis typically requires domain expert judgement to derive meaningful inferences. Thus, while deep learning approaches are excellent to analyze drone imagery, further analysis using explainable AI techniques such as decision trees, can be beneficial to enhance interpretability and reasoning, e.g. fathoming causes behind ../images classified in a certain category and relating them to occurrences in a terrain. Hence, this research focuses on areas where deep learning and explainable AI are beneficial, citing approaches from robot training, autonomous driving, and drone data analytics; mentioning where mathematical optimization is deployed as well. When human-guided analysis is combined with automated analysis, explainable AI models complement deep learning models, offering the best of both worlds.

Bio Sketch: Dr. Aparna Varde is an Associate Professor, and Associate Director for Graduate Studies & Research, in the School of Computing at Montclair State University, NJ, USA. She is an Associate Director of CESAC (Clean Energy & Sustainability Analytics Center), Montclair. Dr. Varde has been a visiting researcher at Max Planck Institute for Informatics, Saarbrucken, Germany. Her main research areas are Artificial Intelligence, Machine Learning and Data Mining. Her honors include 4 best paper awards at IEEE conferences. She is Doctoral Faculty in the PhD Program in Environmental Science & Management at Montclair. Dr. Varde has 150+ publications (journals, conferences, book chapters, edited volumes) by IEEE, ACM, AAAI, Springer & more. Her research spans Commonsense Knowledge, Explainable AI, Smart Cities, Geo-informatics, Robotics, and Text Mining. She has been a dissertation advisor / committee member / mentor for 9 PhD students at Montclair (including a Fulbright Scholar), and external committee member for 4 PhD students worldwide (e.g. Queensland University of Technology, Australia). Dr. Varde has been a panelist for NSF, PC member / organizer at conferences, and reviewer / editorial board member for journals: IEEE, ACM, Elsevier etc. Her research funding is from PSE&G, NSF, and NOAA, in the United States.

Portrait of Fan-Tien Cheng.Category: Intelligent Manufacturing

Title: Evolution of Automatic Virtual Metrology (AVM) Generations

Abstract: The technology of virtual metrology can convert sampling inspection with metrology delay into real-time and online total inspection. Considering the requirements of rigorous manufacturing environment for realizing virtual metrology, the framework of Automatic Virtual Metrology (AVM) was firstly developed. The first generation of AVM, denoted as AVMI, is implemented by applying the Back Propagation Neural Networks (BPNN) of machine learning. After that, the second generation of AVM, denoted as AVMII, replaces the original BPNN in AVMI with the Convolutional Neural Networks (CNN) of deep learning, to not only shorten the time for selecting the model-creation parameters, but also enhance the AVM prediction accuracy. Meanwhile, since both AVMI and AVMII do not have the capability to recognize periodic data trends in sequential manners, either AVMI or AVMII cannot be applied to perform energy demand forecasting. To resolve this issue, the third generation AVM, denoted as AVMIII, is developed to substitute AVMI/AVMII in the conjecture model of the AVM framework for energy demand forecasting by adopting the technology of Long Short-Term Memory (LSTM).

Bio Sketch: Fan-Tien Cheng received the B.S. degree from the National Cheng Kung University (NCKU) in 1976, and the master and Ph.D. degrees from The Ohio State University, Columbus, OH, USA, in 1982 and 1989, respectively. After the graduation of the B.S. degree in 1976, he started to work at the Chung-Shan Institute of Science and Technology (CSIST), Ministry of National Defense, Taoyuan, Taiwan, until 1995. He then began his academic career at the Institute of Manufacturing Engineering and the Department of Computer Science, NCKU. Since 2009, he has become a Chair Professor with NCKU. Fan-Tien Cheng founded the Intelligent Manufacturing Research Center (iMRC) at NCKU in 2018, and he has been the director of the iMRC since then. he is an IEEE Fellow since January 2008 and an IEEE Life Fellow since 2021, and he won 2013 IEEE Inaba Technical Award for Innovation Leading to Production (for contributions to the development of the AVM System). He is currently the Chair of IEEE CASE Steering Committee since August 2020. He received major honors and awards domestically and internationally: the 2011 Award for Outstanding Contributions in science and technology from the Executive Yuan; Outstanding Research Awards from National Science Council (NSC) for three times (2006, 2009, and 2013); National Invention and Creation Award from Ministry of Economic Affairs (MoEA) for three times (2011, 2012, 2018); IEEE ICRA Best Automation Paper Award for two times (1999 and 2013); CASE 2017 Best Application Paper Award; the Chinese Institute of Automation Engineers (CIAE) Award in 2022; and the Merit National Science and Technology Council (NSTC) Research Fellow in 2023.

Category: Commercial package for machine learning (Matlab)

Title: Machine learning with MATLAB and Simulink for next generation manufacturing automation

Abstract: As industries increasingly embrace automation, the fusion of robotics and machine learning within manufacturing has become a powerful catalyst for efficiency and innovation. In this talk, we delve into the dynamic realm of automated manufacturing, where machine learning takes center stage as a versatile tool for optimizing processes and decision-making. We will explore how MATLAB and Simulink, industry-leading platforms, empower engineers and researchers to apply machine learning effectively within manufacturing workflows. By seamlessly integrating these tools, we can develop predictive models, automate quality control, optimize production schedules, and enhance overall system performance.

Bio Sketch: Roberto G. Valenti is currently a Senior Robotics Research Scientist at MathWorks. His research interests include robotics, robotics sensing for navigation, sensor fusion, mobile autonomous robots (self-driving cars, unmanned aerial vehicles), inertial navigation and orientation estimation, control, computer vision, and deep learning. Previously, he worked as a Research and Development Engineer within the Autonomous Driving team at Nvidia. He obtained a Ph.D. in Electrical Engineering at the City University of New York, The City College, NY, USA where he focused his research on state estimation and control for autonomous navigation of micro aerial vehicles. Dr. Valenti received his M.Sc. in Electronics Engineering from the University of Catania, Italy. He is a member of IEEE and RAS

Panel Discussion

Future Directions in Machine Learning for Automation

A 60-minute panel discussion will occur at the workshop's end. The invited speakers (Bengt Lennartson, Maria Pia Fanti, Fan-Tien Cheng, Mengchu Zhou, Xiang Li, Yu Yang, Aparna Varde) will serve as panelists. The early-career researchers and attendees will participate in the discussions in an interactive manner. The workshop organizers will moderate the panel.

Poster Session

The posters will be displayed and presented in an interactive manner within the workshop premises. A dedicated 30-minute poster session will take place after the morning tea/coffee break.

Demo Session

A 30-minute demo session will occur after the invited talks. The participants get the opportunity to showcase practical applications or prototypes of their research or projects. It allows attendees to see real-world examples and interact with the technology or concepts discussed in the invited talks or presentations. Two teams will present their demos.

Yuqian Lu (University of Auckland, Auckland, New Zealand)

Abstract: In this demo, we will show the assembly quality tracking and assistance system developed at the University of Auckland. The system shows recent advances on computer vision and machine learning techniques in understanding fine-grained human assembly activities in real-time. The tracking results, together with prior engineering knowledge provides assembly workers with some useful assistance, including such as detailed assembly instructions, next-step suggestion and assembly operation accuracy. These assistances are communicated with the assembly worker via augmented reality-based projector overlay. Such an assistive system is critical to improving efficiency for high-value product assembly in advanced manufacturing. The demo will be run lively from The University of Auckland.

Screenshot of the Assistive Assembly System app.

Figure: Screenshot of the system

Top down view of assembly bench.

Figure: Projector overlay showing dynamic next-step suggestion to assembly worker

Person standing at an assembly bench with cameras and projector.

Figure: System hardware setup with cameras, projector and the assembly bench

Rafael Hidalgo, Jesse Parron, Aparna S. Varde, Weitian Wang (Montclair State University, NJ, USA)

Abstract: The prevalent need for robotics across multiple sectors in automation motivates smarter systems with greater autonomy. Machine learning can be vital here. Collaborative robots are adept at assembly due to training on extensive datasets. Yet they can falter in object recognition and context understanding during task organization, especially in unfamiliar situations without rigorous training. Motivated by these challenges, this demo presents a novel system “Robo-CSK-Organizer”. This system instills Explainable AI (XAI) via Commonsense Knowledge (CSK) into machine learning. The goal is to optimize decision-making in robotic arms, especially during first encounters with new tasks. It enhances autonomous object recognition, grasping, and sorting under different task contexts. Machine learning in Robo-CSK-Organizer occurs via algorithms in the Detectron2 library using Masked R-CNN (Region-based Convolutional Neural Network) for object identification; and another approach BLIP (Bootstrapping Language-Image Pre-training) for context discernment. CSK is harnessed by extracting information from the ConceptNet repository, thereby offering a knowledge base of concepts with meaningful mathematical values to enhance semantic comprehension in Robo-CSK-Organizer. This system offers transparent object organization based on context such that it is easy to fathom how robots learn; hence convenient to correct mistakes by tracing causes due to XAI via CSK. It can help in human-robot collaboration and robots learning from each other. Real-world experiments demonstrate the effectiveness of Robo-CSK-Organizer in grasping everyday objects, e.g. fruits, snacks, computer / TV equipment, and organizing them in contextually relevant bins corresponding to household / office settings: kitchen, living room, garden etc. Hence, Robo-CSK-Organizer makes a step towards more transparent AI in robotics. It can be useful in next-generation multipurpose robots performing various automated tasks. It offers enhanced autonomy with explainability, guiding automation.

ConceptNet diagram.

Figure: Robo-CSK-Organizer uses knowledge from ConceptNet (top picture) to identify the most relevant context for the object “pear” as “kitchen”.

A photo of objects on a table with an overlay identifying it as a pear, and next to that another photo with a robotic arm placing the pear in a bin labeled kitchen.

Figure: Robo-CSK-Organizer grasps the pear to make the autonomous decision of placing it in the bin corresponding to “kitchen”. Likewise, many other objects are correctly grasped and organized.


Abstract: This video demonstrates that Meituan autonomous delivery vehicles can safely and flexibly cope with complex traffic scenarios such as the morning and evening rush hours in the city, the busy streets of Shenzhen at night, crowded roads on campuses, and narrow roads in residential areas. They can drive smoothly and flexibly while ensuring safe driving.

Graphic showing what the autonomous delivery vehicle sees.

Figure: Meituan autonomous delivery vehicle.

Additional Info

Plan to solicit participation (500 words max)

The organizers intend to implement the following strategies to promote the workshop and generate interest among a diverse range of participants.

  1. Crafting a Call for Contributions (flyer) with the following key elements:
    • Outline the workshop's objectives and the advantages of attending.
    • Showcase the roster of distinguished speakers, including experts from academia and industry, along with the titles of their presentations.
    • Display the logos of endorsing RAS TCs.
    • Emphasize the dynamic and interactive atmosphere, with a focus on soliciting technical and visionary contributions through posters and demonstrations.
    • Highlight the sponsored best paper awards by RAS TCs and its cash prizes.
    • Notify attendees about the sponsored lunch to all workshop participants.
  2. We will leverage the automation-worldwide mailing list to publicize our workshop and distribute our Call for Participants and Papers (flyer).
  3. The endorsing TCs will actively endorse our workshop and encourage their members to contribute by sharing information about it through their blogs and social media channels.
  4. The organizers intend to submit a proposal for a Special/Focused Issue/Section, comprising the finest journal versions of workshop presentations and posters, to esteemed journals such as IEEE Transactions on Automation Science and Engineering.

Providing a direct link to our workshop website (IROS 2024 website) for hassle-free registration

To be added once the workshop is approved.

Plan to encourage interaction among participants. (500 words max)

The organizers intend to implement the following strategies to encourage interaction among participants.

  1. Diverse Session Formats: Leverage the diversity of session formats. Invited talks provide a platform for experts to share insights, while poster and demo sessions allow participants to engage directly with presenters. Panel discussions offer a forum for in-depth debates. Ensure a balanced mix of these formats.
  2. Interactive Q&A: After each invited talk and panel discussion, allocate time for interactive Q&A sessions. Encourage participants to ask questions and engage in discussions with speakers and panelists, fostering a direct exchange of ideas.
  3. Poster/Demo Presentations: Create opportunities for presenters during the poster and demo sessions to actively interact with attendees.
  4. Panel Audience Participation: Allow the audience to pose questions and provide input during panel discussions. This promotes a dynamic dialogue among panelists and participants.
  5. Networking Breaks: Schedule networking breaks in between sessions, providing participants with a chance to connect, discuss topics of interest, and build relationships.
  6. Online Engagement: Use online platforms to facilitate real-time interactions among virtual and in-person attendees.
  7. Feedback Loops: Create channels for feedback, both formal (surveys) and informal (open discussions). Use participant input to adapt the workshop dynamics to their preferences.
  8. Moderators: Appoint moderators for each session to encourage active participation, guide discussions, and ensure a respectful and inclusive environment.

Dissemination (500 words max)

Our strategy involves extensive dissemination and accessibility of the workshop's valuable content.

  1. We intend to create a dedicated workshop website where all presentation materials and videos, spanning invited talks, posters, demos, and panel discussions, will be made available. It offers attendees a comprehensive resource hub to revisit and study workshop content at their own pace. Moreover, it opens up the workshop's insights to a global audience, enabling knowledge sharing and fostering collaboration beyond the immediate event.
  2. Collaboration with endorsing TCs is another key facet of our approach. By encouraging these TCs to post workshop-related poster and paper materials on their own platforms, we extend the reach of the content even further, tapping into the existing networks and communities associated with these TCs.
  3. For the smooth execution of the workshop, we will work closely with the IROS 2024 organization, ensuring seamless logistics and personnel coordination. This includes setting up the workshop room with necessary audiovisual equipment and staffing experienced professionals for video recording. Permission will be asked of authors and participants well in advance.

Compliance with RAS Guidelines for Workshop and Tutorial Organizers

This is confirm that the organizers will be present at the workshop and that the workshop will comply with the RAS Guidelines.

Alignment with the ICRA 2024 “CONNECT+”.

The workshop aligns perfectly with the ICRA 2024 "CONNECT+" theme by promoting interdisciplinary collaboration, addressing global challenges, facilitating knowledge exchange, creating collaborative opportunities, and engaging with industry partners—all of which contribute to building connections and collaborations within the robotics and automation community.

  1. Interdisciplinary Collaboration: ICRA 2024 aims to foster connections and collaborations across various fields. This workshop, by delving into the application of machine learning in manufacturing, bridges the gap between robotics and automation and the manufacturing industry. It encourages experts from these distinct domains to come together, share insights, and explore opportunities for collaboration.
  2. Addressing Global Challenges: ICRA emphasizes the role of robotics and automation in addressing global challenges. Machine learning has the potential to revolutionize manufacturing processes, making them more efficient, cost-effective, and sustainable. By showcasing the application of machine learning in manufacturing, this workshop contributes directly to addressing the challenges of modern manufacturing.
  3. Knowledge Exchange: This workshop provides a platform for researchers, practitioners, and industry professionals to exchange knowledge and ideas. This aligns with ICRA's goal of facilitating knowledge sharing and networking among participants. Attendees can learn about the latest advancements in machine learning for automation and explore how these technologies can be applied in manufacturing.
  4. Collaborative Opportunities: ICRA features a variety of sessions, including plenary talks, contributed papers, workshops, and forums. This workshop can leverage these opportunities to present research findings, case studies, and practical applications of machine learning for automation. This not only contributes to the workshop's theme but also allows participants to connect with a broader audience interested in robotics and automation.
  5. Industry Engagement: ICRA typically attracts industrial partners and sponsors. This workshop on machine learning for automation may draw the attention of industry stakeholders who are interested in adopting these technologies. This alignment provides a platform for industry-academic collaboration, promoting the translation of research into real-world applications.


We expect around 50-60 in-person attendees (including speakers, organizers and participants) and 100 online participants, as well as 10+ posters for the workshop. To accommodate these participants, we will require a workshop room with a maximum capacity of 80 seats, complete with projectors and speakers. In addition, we kindly request two additional microphones to facilitate participant interaction

Support of an IEEE RAS Technical Committee.

  • TC on Machine Learning for Automation
  • TC on Logistics
  • TC on Semiconductor Manufacturing
  • TC on Smart Building
  • TC on Digital Manufacturing and Human Machine Interaction