Creating scalable machine learning (ML) systems for analyzing real-time data in Python — Part-1
and Python libraries and frameworks like TensorFlow, PyTorch, NumPy. Today, more and more developers andcompanies are processing their data using these tools to estimate trends, decision making and and hence having real-time capabilty, both for input as well as processed data, has become indispensable and most soughted requirement.
However that's the one part, there is stiull a challenge on making entire system real-time.
real-time weather information to informing passenger for flight delay, domains, different approaches and techniques were used, such as association, clustering, classification, prediction, sequential patterns identification, decision trees or, what is more usual, a hybrid approach resulting from a combination of these approaches. However, when a big data framework
weather -> flight delay -> group of passenger
Each day, the amount of data and the number of changing data sources continue to grow. As companies are collecting vast amounts of data from the Internet, their own web sites, social media channels, customer information, call center reports or financial transactions; the need for analytical tools able to leverage knowledge behind all these data is imperative.
Even more important than this is the fact that this growth is going to increase exponentially in the future, as there are other emerging areas which are about to come such as Smart Cities and Internet of Things, where the number of potential devices capable of generating high volumes of data is going to be multiplied as a result of what is known as M2M (machine-to-machine interaction). All these incoming changes will require an adequate scaled infrastructure which allows storing, processing and responding to an increasing number of batch and stream requests. This vast amount of information, if conveniently processed, can reveal relevant insights about each business. For instance, analysis of the former data may serve to predict the upcoming friendships or interests of a social network user, suggest related products in which a customer may be interested to purchase or adapt the content and structure of an online course to better fit the students’ needs. At this point it is where machine learning techniques can help to analyze big data sources and extract the important trends, links, rules...
In this article, we will describe how to add real-time communication cababilities to your Python apps which can open up numerous possibilities; processing real-time stock market or geographic sensors are just some random examples. If you are a Python developer, you will find this article an invaluable starting point.
Python’s popularity is on the rise not only because it is easy to learn, but also because of rise of machine learning and scientific computing and Python libraries and frameworks like TensorFlow, PyTorch, NumPy. Today, more and more developers andcompanies are processing their data using these tools to estimate trends, decision making and and hence having real-time capabilty, both for input as well as processed data, has become indispensable and most soughted requirement.
to Python makes Python apps incredibly powerful and
Some example would be, monitoring stock quotes inputs and delivering analysis to a group of people, or to analyze incoming customer request in a chatbot, find a reponse and reply in real-time. The possibilties are endless.
In this article, we will start by exploring how to add Real-Time communication to your Python app by building a simple real-time chat app. Before we get there, let us first understand the basics of real-time messaging.
mesibo python APIs are available for Linux, Mac and Raspberry Pi. You will need to wait if you are using Windows
Real-time messaging is the near-simultaneous exchange of information/message from the sender to the receiver in a connection with negligible latency.
Some popular examples of real-time messaging include:
- You sending 'Hi' to your friends over WhatsApp is a real-time message.
- You sending a request to Uber for a cab is a real-time message - it's invisible to you but underneath what goes is a real-time message.
- Subsequently, Uber sending messages to a group of drivers to get you a cab is a message (group message).
One of the distinguishing characteristics of real-time messaging is that messages are delivered as it happens, almost instantaneously and you get to know the status of the sent message immediately, like in Whatsapp.
If you have ever used real-time, you likely to be familiar with mesibo.
You need to create an authentication token (or access token) for each of your users so that they can log in to the mesibo servers and send messages to each other. We will learn more about creating users and getting tokens later.
Your apps need to use the access token created in Step-1 to initialize mesibo Real-Time APIs (Here, we will be using mesibo for Python) for 1-to-1 messaging, group messaging, etc.
mesibo Real-Time APIs have two components:
- Real-Time Functions
- Real-Time Callbacks
Based on the type of data (message) you need to send, you can use a wide range of API Functions. Additionally, Mesibo provides numerous APIs for other functionalities like storing and retrieving messages in the database, etc.
- to send a text message, call
- to send a group message, call
group IDand the
- to send a file, such as an image you can use
- to read messages from the database, you can use
Since your app can receive messages anytime, you instruct mesibo to let you know as soon as it happens. You can do it by passing a Callback object to mesibo (using the addListener API). When a particular event occurs, mesibo will invoke an appropriate callback to let you know about the event. For example,
- when a message is received,
Mesibo_OnMessagewill be called
- when a message is sent, delivered, or read
Mesibo_OnMessageStatuswill be called
- when a call is received
MesiboCall_OnIncomingwill be called
- when your app goes online or offline
Mesibo_OnConnectionStatuswill be called
- ..., etc.
You now have a basic understanding of real-time messaging and how mesibo APIs work. Let us now build a basic chat app.
We will now build a simple chat app, using which two users can chat with each other. For this, first, we will create two mesibo users and generate tokens for each of them. Then, we will use these tokens in our python app which uses the mesibo Python SDK to send messages.
Sign-up to get a free mesibo account. This is a one-time activity, and you can use this account to build unlimited apps. Go to https://console.mesibo.comopen_in_new to create your account.
Create a mesibo application.
Once you have created your application, it's time to create users. Typically, you will add users and groups dynamically on a need basis using mesibo Backend APIs, with the App token. However, for this basic chat app, we will directly create two users manually in the mesibo console so that you can quickly test messaging between them.
Click on the Settings icon of your app
Click on the Users section in your app. Click on the + NEW USER button.
To create a user, Enter a User Address and App ID. The address can be any non-numeric string. For example, the address can be
123, and the APP Id can be something like
Once a user is created, if you click on the edit button for the user, you will get the access token for that user.
Similarly, create another user with the address 456.
We have now created two users with addresses
456, with appid
com.mesibo.firstapp, and generated their tokens. We will now use this in our python app to initialize mesibo. Refer to mesibo documentation
Get Started: Create Users and generate Tokens if you face any issues creating users.
Next, we will create our Python App.
Install the mesibo Python Package by following the instructions here.
Import the function class
Mesiboand the callback class
- Initialize mesibo for user
- mesibo invokes a class of Listeners for various events as we discussed before. Derive from the
MesiboListenerclass to implement listeners as shown below
- Run your Python script
Mesibo_OnConnectionStatusshould cycle through various status information. Finally, you should receive status=1 which indicates that your app is successfully connected to the mesibo real-time server and ready to send and receive real-time messages. If you are not getting connection status 1, and getting 4 or 5, you are not passing the correct appid while creating a token. Ensure that you pass the same app id to the
Copy the same script into user2.py. Modify
ACCESS_TOKENto use the token of user
456. Run your Python script
Now, you have two python scripts acting as two python users. The two users will now be able to send messages to each other.
To send messages, we will use
sendMessage real-time API for which we will need the destination user, message-id, and the message itself.
Invoke the following function anywhere from your code to send a text message.
When you send a message, you will immediately receive the status - sent, delivered, or read, of the sent message through the listener
Mesibo_OnMessageStatus. The recipient will receive the message through the callback
That's it! Your python chat app is ready. The two users can now send messages to each other.
When you send or receive messages, they will be stored in a local database. You can read messages from this database, for example, to load chat history.
mesibo provides the
ReadDbSession API to read stored messages in the database and sending read receipts. A read receipt is an acknowledgment that the user has read the message.
To read stored messages from the database, you need to create a read session and set the criteria to read messages; for example,
- read all messages
- read messages from a sender
- read messages for a particular group
- read messages matching a search query etc
- The read session you create filters the messages in the database based on the criteria provided and gives the resulting list of messages to you.
To read messages for a user,
To read the last message from each user (summary),
On calling read, the listener
Mesibo_OnMessage will be called for each message read in the database in this read session.
In this article we have learnt the basic concepts of real-time communication using mesibo. In the next article, we will use the concepts from this article to build real-time machine learning apps.