Rechercher des publications

The behavioral analysis of cyber-physical systems in safety-critical scenarios is a challenging task. In this context, the endogenous and exogenous aspects of resilience are of a cornerstone importance in system design and verification. Endogenous resilience is the inherent ability of the system to detect and process internal faults and malicious attacks. Exogenous resilience is the permanent capability of the system to maintain a safe operation within its ambient environment. In this article, we present a predictive dual-sided contract-based formal methodology to address both aspects of resilience on top of a distributed object-oriented component-based software model. It is illustrated by a case study of urban drone rescue systems. We exploit the formalism of timed automata and the toolbox UPPAAL to predict by abstraction and analyze (simulate and verify) endogenous resilience. Instead of presenting the final models of the case study, we reflect our experience with UPPAAL in generic patterns of system design and contract specification, reusable in other contexts with adaptations. The analysis of exogenous resilience is specific to the considered drone rescue system. It consists of synthesizing by iterative model-checking safe flight paths for the drones within a 3D virtual model of urban surroundings true to modern cities.

Intelligent Transportation System (ITS) is the focus of many researchers due to the growing of traffic, which ensures complexity, especially in urban scenario. In this regard, Ultra-Reliable Low Latency Communication (URLLC) achievement is the main purpose for ITS. Latency represents a relevant metric for real time exchange of messages and improve the network Quality of Service (QoS). In this contribution, the purpose is delay investigation into Long Term Evolution Vehicle (LTE-V), which is an enabling cellular technology for vehicular connectivity. The idea is RB (Resource Block) allocation for delay minimization through a direct interaction enabled by LTE-V features. We start by a mathematical analysis of a cross-layer structure based on the joint association between physical layer together with a Medium Access Control (MAC) sub layer. Hence, latency minimization is transformed into an optimization problem in which data rate is maximized under the constraint of queue length. Resource allocation resolution is based on Lyapunov policy. Numerical results are obtained according to a Manhattan scenario towards an urban model where the performance criterion is Complementary Cumulative Distribution Function (CCDF) of queue length. The obtained results highlight the efficiency of the proposed resource allocation algorithm.

The rising cost of healthcare, the exponential growth of vulnerable population such as the sick and the aged, and the shortage of qualified professionals in recent times has led to logical alternatives of providing healthcare to patients. One of the major means of providing alternative care is the use of social robots. Several robotic examples have already been developed to interact and assist with autistic kids. This work is a contribution in the same domain. In this paper, we present our proposed human-robotic interaction system to assist autistic children. Our methodology makes use of multimodal fusion and fission as the basis for the interaction. The proposed solution is different from existing ones in the sense that the interaction between the autistic child and the robot is based on the context of the child. The robot is intended to be intelligent to adapt to the context of the autistic child.

In an intelligent vehicle (autonomous or semi-autonomous), detection and recognition of road obstacle is very important for it is the failure to recognize an obstacle on time which is the primary reason for road vehicular accidents that very often leads to human fatalities. In the intelligent vehicle of the future, safe driving is a primary consideration. This is accomplished by integrating features what will assist drivers in times of needs, one of which is avoidance of obstacle. In this paper, our knowledge engineering is focused on the detection, classification and avoidance of road obstacles. Ontology and formal specifications are used to describe such mechanism. Different supervised learning algorithms are used to recognize and classify obstacles. The avoidance of obstacles is implemented using reinforcement learning. This work is a contribution to the ongoing research in safe driving, and a specific application of the use of machine learning to prevent road accidents.

What machine learning (ML) technique is used for system intelligence implementation in ADAS (advanced driving assistance system)? This paper tries to answer this question. This paper analyzes ADAS and ML independently and then relate which ML technique is applicable to what ADAS component and why. The paper gives a good grasp of the current state-of-the-art. Sample works in supervised, unsupervised, deep and reinforcement learnings are presented; their strengths and rooms for improvements are also discussed. This forms part of the basics in understanding autonomous vehicle. This work is a contribution to the ongoing research in ML aimed at reducing road traffic accidents and fatalities, and the invocation of safe driving.

Nowadays mobile phones are gaining popularity all around the world. They become the most ubiquitous communication device. In order to improve the usability and interaction with users, mobile phones should provide their services according to the current context in an unobtrusive manner with a minimum explicit intervention from users. Context-awareness is a key feature for mobile phones. In this paper, we propose an approach for context modeling using logic predicates and a mobile phone services adaptation using rule-based system. Our context model is based on five logic predicates which cover almost all context aspects for a mobile phone which considerably improve the expressiveness of the model.

In this paper, we present the functionalities of an intelligent connected vehicle. It is equipped with various sensors and connected objects that enable communica-tion between the driver and its environment. This system provides assistance to-wards safe and green driving. The driving assistance may be directed towards the driver (semi-autonomous vehicle) or completely towards the vehicle (self-driving, autonomous vehicle). The assistance is based on the driving context which is the fusion of parameters representing the context of the driver, the vehicle and the environment. This cyber-physical vehicle has three main components: the embed-ded system, the networking and real-time system and the intelligent system. The architecture for data transfer within the connected vehicle is implemented through publish-subscribed infrastructure in which services are transferred and controlled in an orderly manner. These functionalities are tested both in the laboratory and on the road with satisfactory results. This is the fruit of labor of a consortium composed of five industrial and two academic partners.

In this paper, an alternative driving assistance is presented. It is alternative in the since that it is not closed to proprietary constraints which is the case for those associated with car manufacturers. Machine learning is used to identify a driving event and optimization is done to identify the optimal actions that should be performed following the identified driving event. Ontology is used to represent knowledge and driving situations. For the learning part, a training set is created, storing driving situation patterns and from which an intelligent system can used to determine the driving situation. For the optimization part, algorithms are developed that maximizes scores for an action that corre-sponds to safe driving, green driving and comfortable driving. This work is a contribution to make driving assistance systems affordable and available to regular vehicles that common people possess.

In this work, we propose a demonstration of Urban Traffic Light Control based on an IoT network (IoTUTLC) for smart cities. We mocked up a real crossroad by integrating a 6LoWPAN Wireless Sensor Network (WSN) to control mini traffic light panels. The network’s nodes are wireless sensors and actuators interacting with an IoT Cloud Platform. MQTT Quality of Service (QoS) protocol has been implemented to manage the priority levels of exchanged data between the Cloud and WSN. Our IoT-UTLC has been found functional after verification and validation using the UPPAAL model checker.

Channel capacity and software definition of radio functions are important issues for Vehicular Ad-hoc NETworks (VANETs). In our paper, we propose to integrate a 5G radio access technology to software defined PHY layer of IEEE 802.11p for frequency reuse. We superpose two signals with Non-orthogonal Multiple Access (NOMA), and we extract them through Successive Interference Cancellation (SIC). NOMA and SIC solutions have been designed using Software Defined Radio (SDR) chains offering PHY reconfigurability. We prove that this solution allows us to superpose two users sharing the same subcarrier frequency at the same time. Simulations have been performed with two quadrature modulators sharing the same subcarrier frequency. NOMA/SIC would improve the network throughput, since after cancellation, the obtained Bit Error Rate (BER) has been reduced.

Software-Defined Radio (SDR) platforms are useful tools to design new wireless technologies or to improve specifications of existing ones. The IEEE 802.11p is the de-facto standard for Wireless Vehicular Ad-hoc NETworks (VANETs). It has been implemented on GNU Radio SDR, which experiences frames decoding/encoding latency. In this paper an FPGA based SDR is proposed as a solution to accelerate frame decoding of IEEE 802.11p waveforms. We share our experience in designing and validating an SDR on the top of real FPGA based embedded hardware architecture. On the top of an FPGA Zynq platform, we port an SDR GNU Radio following hardware/software (HW/SW) codesign approach. A real time profiling of the SDR system shows that the OFDM equalizer with FFT software function is the most time consuming task of the whole SDR receiver’s functions (34.74%). We suggest hardware acceleration of the FFT processing function using the free logic gates of the FPGA Zynq as a Hardware (HW) accelerator. The HW OFDM Equalizer with FFT sub-function could speed-up the SDR chain processing in comparison with a pure software implementation. Our FPGA based SDR design is a key step toward proposing an embedded SDR. It would be an alternative for not only the IEEE 802.11p standard but for any OFDM based SDR.

In this paper, we presented our innovative way of determining the driving context for a driving assistance system. We invoke the fusion of all parameters that describe the context of the environment, the vehicle and the driver to obtain the driving context. We created a training set that stores driving situation patterns and from which the system consults to determine the driving situation. A machine-learning algorithm predicts the driving situation. The driving situation is an input to the fission process that yields the action that must be implemented when the driver needs to be informed or assisted from the given the driving situation. The action may be directed towards the driver, the vehicle or both. This is an ongoing work whose goal is to offer an alternative driving assistance system for safe driving, green driving and comfortable driving. Here, ontologies are used for knowledge representation.

In this paper, we present our approach for the cognition of driving context in a connected and semiautonomous vehicle. This cyber-physical vehicle has three main components: the embedded system, the networking and real-time system, and the intelligent system that communicate with each other via Cloud computing infrastructure. This paper is about the functionalities of the intelligent system in the cognition of driving context and the actions that it invokes for each of these driving situations. The driving context is determined as a result of the fusion of various parameters representing the context of the environment, the vehicle and the driver. The fission process yields the action that must be implemented with regards to the driving situation in consideration. Ontology is used as a tool for knowledge representation. This work is a contribution to safe driving in a connected and semi-autonomous vehicle.

Project CASA (Car Safety Apps) is a project of a consortium of industrial and academic partners in France, with an aim of building an Android app that promotes safe and green driving. In this paper, an innovative way of capturing real-time signals, combining them to determine the current driving situation, associate the required action and generate the corresponding message towards the driver and the vehicle will be presented. Every message is suitable to the real-time driving context and event. The contributions of this work are as follows: (i) generation of assistive messages that are suitable for safe and green driving, (ii) dynamic and real-time consideration of driver status, vehicle status, and of the environment (iii) providing priority level to traffic messages based on the danger involved in the scenario.

This work is an undertaking intended to contribute to the promotion of autonomy and good health for the elderly using smart, context-aware vehicular system. The project’s principal users are the old people, normally in their retirement age. Serious Gaming is integrated into this system, providing physical and mental activity challenges to the actors concerned, including an option to participate in events with their peers. The secondary users are medical practitioners who provide the challenges that are suitable to the health status of the participants. Results of the activities are also evaluated by medical personnel. The vehicular system is smart and context-aware, allowing it to invoke actions directed towards the driver, the vehicle or both during driving situations where safety driving intervention is needed. The results of preliminary evaluation based on one-month test with participants indicated that their mobility, autonomy and health conditions improve with participation in the project.

Project CASA is a collaborative work of five industrial partners and one educational institution. Its goal is to contribute to the advancement of safe and green driving by proposing ADAS (advanced driving assistance system) based on smartphone. It is implemented by capturing and combining all the parameters related to the context of the driver, the vehicle, the environment and the infrastructure altogether to deduce the current driving situation. Based upon such driving situation, we may invoke driving assistance scheme in a form of an audio and visual message to notify or alert the driver. In the area of human-computer interaction, we have developed an interface that alerts or notifies a driver. We also work on the detection of level of stress of the driver. It is necessary on our part that a stressed driver should not be further stressed by our driving assistance messages. This work has been presented to the public and had been received warmly.

Given the context of the driver, of the vehicle and of the environment, this safe driving assistance system deduces the traffic context and provide the driver with the necessary assistance by providing message notification or alert about the situation or of the infraction that was committed, or trigger an action directly affecting the vehicle. To do so, the system performs signal processing cognition related to these context parameters. This paper illustrates the system’s knowledge representation using ontology, the rules related to the fusion of context parameters and the deduction of driving context using the Semantic Web Rule Language. A fission component deals with the steps needed to implement the desired safe driving action that is directed towards the driver, to the vehicle, or both. The paper shows how ontology is used to represent driving model, the road environment and the set of actions related to invoke safe driving assistance. This work is a contribution to the ongoing research for the prevention of vehicular traffic accident.

In this paper, we present our approach for the cognition of driving context in a connected and semi-autonomous vehicle. This cyber-physical vehicle has three main components: the embedded system, the networking and real-time system and the intelligent system that communicate with each other via Cloud computing infrastructure. This paper is about the functionalities of the intelligent system in the cognition of driving context and the actions that it invokes for each of these driving situations. The driving context is determined as a result of the fusion of various parameters representing the context of the environment, the vehicle and the driver. The fission process yields the action that must be implemented with regards to the driving situation in consideration. Ontology is used as a tool for knowledge representation. This work is a contribution to safe driving in a connected and semi-autonomous vehicle.

In this paper, a novel approach of managing driving context information in smart transportation is presented. The driving context refers to the ensemble of parameters that make up the contexts of the environment, the vehicle and the driver. To manage this rich information, knowledge representation using ontology is used and through it, such information becomes a source of knowledge. When this context information (i.e. basically a template or model) is instantiated with actual instances of objects, we can describe any kind of driving situation. Furthermore, through ontological knowledge management, we can find the answers related to various queries of the given driving situation. A smart vehicle is equipped with machine learning functionalities that are capable of classifying any driving situation, and accord assistance to the driver or the vehicle or both to avoid accident, when necessary. This work is a contribution to the ongoing research in safe driving, and a specific application of using data from the internet of things.

The contribution of this paper is articulated around a new software design approach of autonomous control systems for connected vehicle platoons. Our control system is distributed and real-time based on object-oriented component-based method of design that brakes with the industrial traditions subject to cyclic OS-free approaches. We illustrate our design by relevant case studies of the longitudinal speed control widely studied in industrial and academic research around automotive platooning. Our software is mainly implemented using the Ada standard of programming (in particular the annexes D and E of real-time and distributed systems). The distribution in our software is managed by the versatile middleware PolyORB. The control scenarios and communication aspects covered by the case studies are animated by wheeled robot prototypes commanded by single-board ARM Cortex computers under real-time Linux kernels.

“Serious Gaming for the Elderly” is an undertaking that is intended to contribute to the promotion of autonomy and good health for the elderly. Its principal users are the old people, normally in their retirement age. They are given physical activity challenges as well as mental challenge and an option to participate in events with their peers. The secondary users are medical practitioners who provide exercises suitable to the health status of the participants. Results of the activities are also evaluated by medical personnel. New challenges are designed based on results of previous ones. The beta version of the system is initially tested with five specimen participants. The results of preliminary evaluation based on one-month test with participants indicated that their autonomy and health conditions improve with participation in serious gaming.

In this work, given the context of the driver, of the vehicle and of the environment, our objective is to correctly recognize the traffic situation and provide the driver with the corresponding assistance by providing notification or alert about the situation or the infraction that was committed, or acting directly on the vehicle. To do so, we need to consider the signal processing related to these context parameters. We built knowledge representation using ontology, built rules related to the fusion of context parameters and the deduction corresponding to the traffic situation using Semantic Web Rule Language. We built fission component that deals with traffic situation and the corresponding action directed towards the driver or the vehicle. Ontology is used to represent driving model and road environment. This work is our contribution in the ongoing research for the prevention of vehicular traffic accident.

In this paper, we present a synopsis of Project ASSIA aimed at delivering safe driving environment. We also present our multimodal signal processing techniques in fusioning various signals denoting context of the driver, the vehicle and the environment in order to obtain the actual driving situation. For each driving situation, an action might be needed. The fission component determines what action needs to be done for such situation. The recipient of the action may be the driver, the vehicle or both. The paper contains significant contributions with regards to techniques in multimodal fusion, fission, knowledge representation using ontology, machine learning and formal verification. A scenario is presented to simulate the system’s functionalities.

Project CASA (Car Safety Apps) is a joint project of a consortium composed of 5 industrial and one academic partners in France with an aim of building an Android app that promotes safe and green driving, and secured data transmission. In this paper, we focus on the security aspects and features of our project. We present that the multimodal data processing of our work is safe and secured. The driver notification and alert messages are intended to keep the driver safe from road accidents. The data transmission using LiFi, instead of WiFi, makes the communication secured due to the absence of radio wave’ interference with other road users and the absence of electromagnetic interference makes it a green technology. The contributions of our work are as follows: (i) generation of assistive messages that are suitable for safe and green driving, and (ii) secured and green data transmission.

In this paper, we present an architecture that demonstrates multimodal fusion and fission using semantic agents and web services that interacts with worldwide-web computers, services and users. This solution extracts the meaning of a situation using the semantic memory of agents to manage the interaction process involved. The fusion of values from different sensors produces an event that needs implementation. The fission process suggests a detailed set of actions that are for implementation. Before such actions are implemented by actuators, these actions are first evaluated in a virtual environment which mimics the real-world environment. If no danger arises from such virtual evaluation, then implementation is feasible. Otherwise, there might be a need to add one or more smaller actions to render the action safe and free from danger. Our work presents the following contributions: (i) a design of agent memory and a model of the world environment using a knowledge representation language that is compatible with the current standards, (ii) creation of a pervasive architecture with several scenarios of composition and adaptation, (iii) presentation of how agents and services interact to provide support in a real-world, and (iv) simulation of an event in a virtual environment to assess the feasibility of the event’s implementation.

Various proprietary vehicular technologies have been integrated on commercial vehicles. Most of these assistance systems, however, are systems of perception, of alert or of very simple commands which are attached to one or two sensors only. There is not even a support with regards to information involving environmental context. There is not even use of the user profile and actual state of the driver. There is not even sharing of modalities (such as voice synthesis for example) because the products and software of each OEM (original equipment manufacturer) or designer are often closed to proprietary constraints. These are the weaknesses that our work would like to address. We take a traffic situation, consider the traffic rules, and consider the context of the driver, the car and the environment and provide guidance to the driver. Our aim is to provide assistance to prevent traffic accident. We begin with “Turn Left in an intersection” situation and progress forward. An architectural design has already been developed to cover all kinds of traffic conditions. Ontology is used to represent driving modelling and road environment. Our aim is to contribute to the body of knowledge in the domain of prevention of vehicular traffic accident.

In this paper, we present our techniques of multimodal fusion and fission for an intelligent robotic application. The fusion of data from different sensors produces an event that needs implementation. The fission process produces a detailed set of actions that are to be implemented. Before such set of actions is implemented, this set of actions is evaluated first in a virtual environment to determine if the actions can be carried out in a real-world environment. If an obstacle is noted during the virtual evaluation, this implies that the action cannot be carried out in real environment. To mitigate such situation, one or more smaller actions may be added in order to get over the obstacle. We consider a sample robotic application scenario to demonstrate this situation. The contributions of our work are as follows: (i) a design of agent memory and a model of environment using a knowledge representation language that is compatible with current standards, (ii) creation of a pervasive architecture with several scenarios of composition and adaptation, (iii) presentation of how agents and services interact to provide support in real-world, and (iv) simulation of event on a virtual environment to assess the feasibility of an action’s implementation.

We propose a formal approach for the design of object-oriented component-based systems by means of behavioral contracts. This formalism merges interface automata describing communication protocols of components with the semantics of their operations. On grounds of consistency with the object-oriented paradigms, we revisit the notions of incremental design and independent implementability of interface automata by defining otherwise components compatibility, composition, and refinement. The optimistic composition approach of interface automata is adapted to fulfill the interaction aspects of object-oriented components. The composition of two interface automata is computed by removing from their synchronized product all states from which the environment cannot prevent deadlock states (caused by semantic and protocol mismatches) by enabling autonomous (output and local) actions. We define the concept of autonomous actions differently by reclassifying them into method, return, and exception actions. Refinement is intended to ensure an independent implementability of components at the semantic and protocol levels. We present refinement as an expanding simulation relation between interface automata, based on two requirements: (1) a component refinement contains more details about common provided services with the abstraction, and (2) may provide more services than the abstraction. These features lead to define refinement as covariance on input and output events of a component. The alternating simulation originally proposed to refine interface automata, requires contravariance on input and output events, not quite consistent, from our perspective, with the object-oriented context. Our work is illustrated by a design use case of CBTC railway systems to argue their relevance in the safety-critical context.

In pervasive computing, our aim for the end user is continuous computing anytime, anywhere whenever possible, in a way that the computing system adapts to the user's context. An integral component of pervasive computing is pervasive task, a computing task that follows wherever the user is, adapting to the computing needs of the user and self-optimizing on user's behalf. In this paper, a task is represented by a set of computing files which then are mapped to their corresponding applications and suppliers, and optimal quality of service dimensions for each application. Here, we present how pervasive task can be realized, implementing selfconfiguration and self-optimization to adapt to the user's preferences. This work is a contribution in the ongoing research towards adaptation of computing system to suit the user's context and preferences in a pervasive computing system.

La réutilisation d’un composant logiciel consiste à l’utiliser dans des contextes différents sans affecter son code et en s’appuyant uniquement sur ses interfaces. En ce sens, il est nécessaire de proposer des techniques d’adaptation des composants permettant d’assurer leur communication malgré les disparités qui peuvent exister au niveau de leurs interfaces. Dans cet article, nous proposons une approche formelle d’adaptation des composants dont les protocoles comportementaux sont décrits par les automates d’interface. Ce formalisme est basé sur une approche optimiste qui considère deux composants comme compatibles s’il existe un environnement convenable avec lequel ils peuvent interagir correctement. L’approche proposée permet, d’une part, d’éliminer les disparités entre deux composants au niveau de la signature, de la sémantique et du protocole, et, d’autre part, de garantir leur compatibilité.

This paper presents a design of an architecture that facilitates the work of a fusion engine. The logical combination or merging of input streams invoked by the fusion engine is based upon the definition of a set of patterns and its similarity with previously collected data from various modalities. Previous fusion engine designs had many weaknesses, among them their being specialized on a specific domain of application. The proposed architecture addresses such weakness and provides additional features, namely its ability to handle large number of modalities and due to its using knowledge base and standardization characteristics; it becomes suitable to various types of multimodal systems. The techniques used to achieve these features are discussed in this paper.

Communication is an important aspect of human life; it is with communication that helps human beings connect with each other as individuals and as independent groups. In informatics, the very purpose of the existence of computer is information dissemination – to be able to send and receive information. Humans are quite successful in conveying ideas with one another and reacting appropriately because we share the richness of our language, have a common understanding of how things work and have an implicit understanding of everyday situations. When human communicate with human, they comprehend the information that is apparent to the current situation, or context, hence increasing the conversational bandwidth. This ability to convey ideas, however, does not transfer when human interacts with computer. On its own, computers do not understand our language, do not understand how the world works and cannot sense information about the current situation. In a typical impoverished computing set-up where providing computer with information is through the use of mouse, keyboard and screen, the result is we explicitly provide information to computers, producing an effect that is contrary to the promise of transparency and calm technology in Marc Weiser’s vision of ubiquitous computing. To reverse this, it is imperative that methodologies are developed that will enable computers to have access to context. It is through context-awareness that we can increase the richness of communication in human-computer interaction, through which we can reap the most likely benefit of more useful computational services.

Le contexte évolue dans le temps et l'acquisition de l'information contextuelle est essentielle. Nous tenons que le contenu de l'information contextuelle ne devrait être défini que par l'utilisateur. Ceci nous mène au concept de l'acquisition incrémental du contexte où des paramètres de contexte sont ajoutés, modifiés ou supprimés, un paramètre de contexte à la fois. Dans ce même ordre d'idée, nous élargissons la notion du contexte au contexte de l'interaction (CI). Le CI qui est employé pour se rapporter au contexte collectif de l'utilisateur, de son milieu de travail et de son système de calcul. Afin de maximiser les bienfaits de la richesse du CI dans la communication personne-machine, la multimodalité tient compte d'un éventail de modes et de formes de communication, choisis et adaptés au contexte de l'utilisateur. La multimodalité ubiquitaire et adaptative enrichit la communication dans l'interaction personne-machine et fournit les modes les plus appropriés pour l'entrée / la sortie de données par rapport à l'évolution du CI. Cette thèse - Le paradigme du système multimodal multimédia ubiquitaire sensible au contexte de l'itération - est une conception architecturale qui montre l'adaptibilité au CI. Il est intelligent et diffus. Il est conçu avec deux buts à l'esprit. D'abord, étant donné une instance de CI qui évolue avec le temps, notre système détermine les modalités optimales qui s'adaptent à un tel CI. En second lieu, nous avons conçu un mécanisme qui permet le choix automatique des applications de l'utilisateur, les fournisseurs préférés à ces applications et les configurations préférées de la qualité du service de ces fournisseurs.

In our days, the technology allows us to produce extended and totally human-controlled multimodal systems. These systems are equipped with multimodal interfaces allowing a more natural and more efficient interaction between man and machine. End users could take advantage of natural modalities (e.g. eye gaze, speech, gesture, etc.) to communicate or exchange information with applications. The use of multimodal applications in web services, integrated with natural modalities, is an effective solution for users who cannot use a keyboard or a mouse, on users who have visual handicap, on mobile users equipped with wireless telephone/mobile devices, on weakened users, etc. Our work presents an approach in which various input modalities (speech, touch screen, keyboard, eye gaze, etc.) are available at user’s disposition in order to access web services. While current state-of-the-art uses two (on rare cases, three) pre-defined modalities, our approach allows an unlimited number of concurrent modalities using semantic level called “multimodal fusion”. Such approach gives user the flexibility to use the modalities as he sees fit for his situation. The detailed description of the proposed approach as well as the application that has been developed that uses these multimodalities is presented in this paper.

These days, a human-controlled multimodal system equipped with multimodal interfaces is possible, allowing for a more natural and more efficient interaction between man and machine. In such a system, users can take advantage of the modalities to communicate or exchange information with applications. The use of multimodal applications, integrated with natural modalities, is an effective solution for users who would like to access ubiquitous applications such as web services. The novelty of this work is that all modalities that are made available to the user to access web services are already found to be suitable to the user’s current situation. By suitablity, we mean these are optimal modalities – found to be suitable to the user’s interaction context (i.e. the combined context of the user, his environment and his computing system) and media devices are available to support them. These modalities can be invoked for the data input/output by the user to access web service using a semantic combination of modalities, called “multimodal fusion”. While current state-of-the-art uses two (on rare cases, three) predefined modalities, our approach allows an unlimited number of concurrent modalities. This approach gives user some flexibility to use the modalities that he sees fit for his situation and comfortable with it. The description of the detection of optimal modality as well as the fusion process, together with sample application are presented in this paper.

Autonomic communication in a computing system analyzes the individual system element as it is affected by and affects other elements. In the human-machine communication aspect, the autonomic system performs its services autonomously, adjusting the behavior of its services to suit what the user might request implicitly or explicitly. The pervasive multimodal multimedia computing system aims at realizing anytime, anywhere computing. Its autonomic communication includes the protocols that selects, on behalf of the user, the modalities and media devices that are appropriate to the given interaction context. The modalities are the modes of interaction (i.e. for data input and output) between the user and the computer while the media devices are the physical devices that are used to support the chosen modalities. The interaction context itself is the combined user, environment, and system contexts. In this paper, we present the autonomic communication protocols involved in the detection of interaction context and the multimodal computing system’s corresponding adaptation. The heart of this paradigm’s design is the machine learning’s knowledge acquisition and the use of the layered virtual machine for definition and detection of interaction context.

An interesting formal approach to specify component interfaces is interface automata based approach, which is proposed by L. Alfaro and T. Henzinger. These formalisms have the ability to model both the input and output requirements of components system. In this paper, we propose a method to enrich interface automata by the semantics of actions in order to verify components interoperability at the levels of signatures, semantics, and protocol interactions of actions. These interfaces consist of a set of required and offered actions specified by Pre and Post conditions. The verification of the compatibility between interface automata reuse the L.Alfaro and T.Henzinger proposed algorithm and adapt it by taking into account the action semantics. Our approach is illustrated by a case study of the vehicle CyCab.

Interface automata are light-weight models that capture the temporal interface behavior of software components. They have the ability to model both the input requirements and the output behavior of a component. They support the compatibility check between interface models to ensure a correct interaction between components and they adopt an alternating simulation approach to design refinement. In this paper, we extend our previous works on checking interface automata interoperability by adapting their alternating refinement relation to the action semantics. We show the relation between pre and post-conditions of transitions in the abstract version of an interface and their corresponding ones in its concrete version. We illustrate our extensions by a case study of the CyCab car component-based system.