Creating scalable machine learning (ML) systems for analyzing real-time data in Python — Part-1

To be edited

You need to set up servers, database, socket programming, and plenty of algorithms. Hence, instead of reinventing wheel, we will use mesibo, which not only a powerful real-time communication platform but offers open-source python library for Linux, Mac and Raspberry Pi, and also allows you to download the entire platform and run in your own premise. Mesibo also offers APIs for Android, iOS, Javascript, and C++ so that also makes it easier to interface your python apps with those apps. there is still the real-time communication which has it's own challenges and algorithms -

and Python libraries and frameworks like TensorFlow, PyTorch, NumPy. Today, more and more developers andcompanies are processing their data using these tools to estimate trends, decision making and and hence having real-time capabilty, both for input as well as processed data, has become indispensable and most soughted requirement.

However that's the one part, there is stiull a challenge on making entire system real-time. . however we have mesibo python library

real-time weather information to informing passenger for flight delay, domains, different approaches and techniques were used, such as association, clustering, classification, prediction, sequential patterns identification, decision trees or, what is more usual, a hybrid approach resulting from a combination of these approaches. However, when a big data framework

weather -> flight delay -> group of passenger

lockdown ->dating

Each day, the amount of data and the number of changing data sources continue to grow. As companies are collecting vast amounts of data from the Internet, their own web sites, social media channels, customer information, call center reports or financial transactions; the need for analytical tools able to leverage knowledge behind all these data is imperative.

Even more important than this is the fact that this growth is going to increase exponentially in the future, as there are other emerging areas which are about to come such as Smart Cities and Internet of Things, where the number of potential devices capable of generating high volumes of data is going to be multiplied as a result of what is known as M2M (machine-to-machine interaction). All these incoming changes will require an adequate scaled infrastructure which allows storing, processing and responding to an increasing number of batch and stream requests. This vast amount of information, if conveniently processed, can reveal relevant insights about each business. For instance, analysis of the former data may serve to predict the upcoming friendships or interests of a social network user, suggest related products in which a customer may be interested to purchase or adapt the content and structure of an online course to better fit the students’ needs. At this point it is where machine learning techniques can help to analyze big data sources and extract the important trends, links, rules...

In this article, we will describe how to add real-time communication cababilities to your Python apps which can open up numerous possibilities; processing real-time stock market or geographic sensors are just some random examples. If you are a Python developer, you will find this article an invaluable starting point.

Python’s popularity is on the rise not only because it is easy to learn, but also because of rise of machine learning and scientific computing and Python libraries and frameworks like TensorFlow, PyTorch, NumPy. Today, more and more developers andcompanies are processing their data using these tools to estimate trends, decision making and and hence having real-time capabilty, both for input as well as processed data, has become indispensable and most soughted requirement.

https://www.infoworld.com/article/3204016/what-is-python-powerful-intuitive-programming.htmlopen_in_new

https://blog.learningpeople.com/the-rise-of-pythonopen_in_new

https://www.statista.com/chart/21017/most-popular-programming-languages/open_in_new

Data Processing - Convential vs Real-time

to Python makes Python apps incredibly powerful and

Some example would be, monitoring stock quotes inputs and delivering analysis to a group of people, or to analyze incoming customer request in a chatbot, find a reponse and reply in real-time. The possibilties are endless.

However, building real-time apps and achieving near-zero latency is not trivial. There are more parameters like sequencing, delivery guarantees and ackknowledgement, network delays, offline delivery, queueing, etc. You need to set up servers, database, socket programming, and plenty of algorithms. Hence, instead of reinventing wheel, we will use mesibo, which not only a powerful real-time communication platform but offers open-source python library for Linux, Mac and Raspberry Pi, and also allows you to download the entire platform and run in your own premise. Mesibo also offers APIs for Android, iOS, Javascript, and C++ so that also makes it easier to interface your python apps with those apps.

In this article, we will start by exploring how to add Real-Time communication to your Python app by building a simple real-time chat app. Before we get there, let us first understand the basics of real-time messaging.

mesibo python APIs are available for Linux, Mac and Raspberry Pi. You will need to wait if you are using Windows

Processing Real-time Vs Convential Data

Real-time messaging is the near-simultaneous exchange of information/message from the sender to the receiver in a connection with negligible latency.

Some popular examples of real-time messaging include:

You sending 'Hi' to your friends over WhatsApp is a real-time message.
You sending a request to Uber for a cab is a real-time message - it's invisible to you but underneath what goes is a real-time message.
Subsequently, Uber sending messages to a group of drivers to get you a cab is a message (group message).

One of the distinguishing characteristics of real-time messaging is that messages are delivered as it happens, almost instantaneously and you get to know the status of the sent message immediately, like in Whatsapp.

If you have ever used real-time, you likely to be familiar with mesibo.

What is mesibo?

mesibo is a high-performance real-time communication platform that provides messaging, calls, and conferencing APIs for Android, iOS, Javascript, Python, C++, etc. In this article, we will be using the mesibo Python SDKopen_in_new for Real-Time messaging. mesibo APIs are powerful but easy to use. mesibo makes it simple to enable real-time messaging. You only need to follow two steps:

Step-1: Create User Tokens

You need to create an authentication token (or access token) for each of your users so that they can log in to the mesibo servers and send messages to each other. We will learn more about creating users and getting tokens later.

Step-2: Use Real-Time APIs to send messages

Your apps need to use the access token created in Step-1 to initialize mesibo Real-Time APIs (Here, we will be using mesibo for Python) for 1-to-1 messaging, group messaging, etc.

mesibo Real-Time APIs have two components:

Real-Time Functions
Real-Time Callbacks

Real-Time Functions

Based on the type of data (message) you need to send, you can use a wide range of API Functions. Additionally, Mesibo provides numerous APIs for other functionalities like storing and retrieving messages in the database, etc.

For example,

to send a text message, call sendMessage() with parameters destination and the message
to send a group message, call sendMessage() with parameters group ID and the message
to send a file, such as an image you can use sendFile()
to read messages from the database, you can use readDBSessions APIs

Real-time Callbacks

Since your app can receive messages anytime, you instruct mesibo to let you know as soon as it happens. You can do it by passing a Callback object to mesibo (using the addListener API). When a particular event occurs, mesibo will invoke an appropriate callback to let you know about the event. For example,

when a message is received, Mesibo_OnMessage will be called
when a message is sent, delivered, or read Mesibo_OnMessageStatus will be called
when a call is received MesiboCall_OnIncoming will be called
when your app goes online or offline Mesibo_OnConnectionStatus will be called
..., etc.

You now have a basic understanding of real-time messaging and how mesibo APIs work. Let us now build a basic chat app.

Building a simple chat app

We will now build a simple chat app, using which two users can chat with each other. For this, first, we will create two mesibo users and generate tokens for each of them. Then, we will use these tokens in our python app which uses the mesibo Python SDK to send messages.

Creating mesibo users

Sign-up to get a free mesibo account. This is a one-time activity, and you can use this account to build unlimited apps. Go to https://console.mesibo.comopen_in_new to create your account.
Create a mesibo application.
Once you have created your application, it's time to create users. Typically, you will add users and groups dynamically on a need basis using mesibo Backend APIs, with the App token. However, for this basic chat app, we will directly create two users manually in the mesibo console so that you can quickly test messaging between them.
Click on the Settings icon of your app
Click on the Users section in your app. Click on the + NEW USER button.
To create a user, Enter a User Address and App ID. The address can be any non-numeric string. For example, the address can be 123, and the APP Id can be something like com.mesibo.firstapp
Once a user is created, if you click on the edit button for the user, you will get the access token for that user.
Similarly, create another user with the address 456.

We have now created two users with addresses 123 & 456, with appid com.mesibo.firstapp, and generated their tokens. We will now use this in our python app to initialize mesibo. Refer to mesibo documentation Get Started: Create Users and generate Tokens if you face any issues creating users.

Next, we will create our Python App.

Creating the python app

Install the mesibo Python Package by following the instructions here.
Import the function class Mesibo and the callback class MesiboListener

Initialize mesibo for user 123 like below

mesibo invokes a class of Listeners for various events as we discussed before. Derive from the MesiboListener class to implement listeners as shown below

Testing your Python app

Run your Python script

python user1.py

Mesibo_OnConnectionStatus should cycle through various status information. Finally, you should receive status=1 which indicates that your app is successfully connected to the mesibo real-time server and ready to send and receive real-time messages. If you are not getting connection status 1, and getting 4 or 5, you are not passing the correct appid while creating a token. Ensure that you pass the same app id to the setAppName API call.
Copy the same script into user2.py. Modify ACCESS_TOKEN to use the token of user 456. Run your Python script

python user2.py

Now, you have two python scripts acting as two python users. The two users will now be able to send messages to each other.

Sending and Receiving Messages

To send messages, we will use sendMessage real-time API for which we will need the destination user, message-id, and the message itself. Invoke the following function anywhere from your code to send a text message.

When you send a message, you will immediately receive the status - sent, delivered, or read, of the sent message through the listener Mesibo_OnMessageStatus. The recipient will receive the message through the callback Mesibo_OnMessage

That's it! Your python chat app is ready. The two users can now send messages to each other.

Reading Messages

When you send or receive messages, they will be stored in a local database. You can read messages from this database, for example, to load chat history.

mesibo provides the ReadDbSession API to read stored messages in the database and sending read receipts. A read receipt is an acknowledgment that the user has read the message.

To read stored messages from the database, you need to create a read session and set the criteria to read messages; for example,

read all messages
read messages from a sender
read messages for a particular group
read messages matching a search query etc
The read session you create filters the messages in the database based on the criteria provided and gives the resulting list of messages to you.

To read messages for a user,

To read the last message from each user (summary),

On calling read, the listener Mesibo_OnMessage will be called for each message read in the database in this read session.

In this article we have learnt the basic concepts of real-time communication using mesibo. In the next article, we will use the concepts from this article to build real-time machine learning apps.

Stay Tuned!

Useful links: