Keywords

1 Introduction

The 4G mobile network provides higher bandwidths and supports Multimedia services, with about 100 Mbps of bandwidth achievable. Based entirely on packet switched networks, the 4G network differs from 3G networks which use both circuit and packet switching. On the network security front, 4G has an enhanced security mechanism compared to that of 3G networks. 4G networks are also better scalable with their architecture built in such a way and it achieves global mobility as well [1].

With the onset of Big Data and Cloud-based services, there arose new requirements for Network performance. It has become important for service providers to provide a minimum level of performance, with additional challenges arising from the evolution of new network technologies towards voice and data services. Operators therefore need to have a thorough understanding to manage quality in terms of technical operations (QoS-Quality of Service) and the Quality of Experience at the user level (user perceived QoS or the QoE) [2].

2 Literature Review

2.1 Video Experience

The Quality of Service (QoS) from the Service Provider’s network plays a vital role in the Quality of Experience (QoE) of the end user. It is no exception in the case of Video streaming and video conferencing. There are many real-time video creating apps like YouTube that require a very high bandwidth to work well. This adds a push to the Service Provider that must see to it that the network provides the necessary quality. Different types of streaming (Person to Person- share video in real time) or (Content to Person) need different service provisioning capabilities and quality. For real-time video sharing, a low latency and a high bandwidth for both uplink and downlink is essential, when compared to Content to Person streaming. The QoE expectation for video streaming is high with the need of low jitter and extremely low packet loss. Also, video conferencing needs a high service quality and video, and audio streaming requires median service quality [2]. For multimedia streaming, the QoE is measured by the streaming bit rate, playback smoothness, Peak to signal noise ratio, among other satisfaction factors [3]. To provide better QoS to the users, many new devices and streaming codec have been introduced. However, network traffic is still unpredictable and still causes issues [4]. Since network traffic passes through routers, gateways, and firewalls, it goes through packet loss, delay, jitter and packet reordering many a time. The traffic from routers sometimes waits in queue or network path for a long time that is unsteady due to load balancing [5, 6] and all this causes network disturbances. This has an impact on the streaming of videos and hence the video quality at the end user may not be as expected. Sometimes, the broadcaster sends the data through the network which passes the data packets in a particular sequence, but they arrive at the destination out of sequence. This is called packet reordering and leads to missing data packets. This packet reordering has a huge impact on quality of the video and the perceived QoS is not attained at the end user level [7, 8]. When the user does not experience results as on the Service Level agreements (SLA), they tend to leave the network thus leading to a drop in business revenue to the service provider [9]. QoE as a parameter of network satisfaction is better than network QoS in the sense that feedback received helps improve the network parameters, for a better User QoS (QoE) which is the user perception about the service [10].

2.2 Games Experience

Gaming has become popular in today’s context with people playing online games through varied platforms like PC, mobile applications, online servers and cloud services. A QoE assessment was made on user experience of gaming through mobile applications and online. The research measured the satisfaction level of playing through these platforms, also avoiding the usage of game CDS or kiosks. Results showed that application based gaming and big screen displays had better visual quality and was more sought after by players than online platform where the graphics quality was low. Another QoE assessment experiment that was carried out involved the network speed measurement using speedtest.net and upload speed was recorded to be 3.78 MB for PC online gaming over a 4G network, and it was found that at instances where compression was done to save the delay/buffer time, the quality of gaming reduced and so did the QoE of the user [11]. Multiplayer games require very low latencies and there is a need to develop new standards in data transmission and routing to enjoy least latency experience [12].

2.3 Voice App Experience

To carry voice over the internet, the channel used should have a low bandwidth. Low latency packets must be transmitted, so that it gives out low jitter. While determining the channel bandwidth, it must be emphasized that packet transmission is done at the right time period. Voice over the internet would require that the network has a low latency and jitter combination and low bandwidth of 21–320 kbps per call and one way latency of less than 150 ms and a one-way jitter less than 30 ms, to satisfy QoE of the end user by providing optimal network quality as mentioned [2]. QoE of voice applications depends on packet loss rate, delay and encoding rate [13]. There are many approaches that have been proposed to improve user perceived speech quality and bandwidth utilization. A few of these are mentioned here.

Adaptive rate control is a technique that provides optimized voice quality and manages congestion problems by matching the transmission rate to the network capacity. New hybrid algorithms that adjust the rate of transmission based on network parameters and perceptual quality show better quality of speech and improve bandwidth utilization. Rate control mechanisms that are based on only network impairments may fail to provide optimum QoS in terms of user-perceived quality. Another study shows the dynamic adjustment of encoding bit rate with feedback about network congestion through RTCP (Real-time Transport Control Protocol) packets. Other works include rate adaptation using Adaptive Multi-rate Codec and voice packets being marked on priority basis. All these are aimed at minimizing packet loss and delay. One of the concerns on Voice over IP (Internet Protocol) is that it does not use TCP (Transport Control Protocol). Hence the huge volume of VOIP flows that have increased over the past few years may cause network instability and congestion collapse. VOIP uses UDP (User Datagram Protocol) and does not provide any congestion control. It is understood that for network stability, any new protocols and services should be TCP-friendly, consuming comparable bandwidth as with TCP, under the same network conditions. Thus, any adaptive networked multimedia applications like voice and video must also be TCP-friendly [14].

The network’s perceived quality is not measured by taking it at any one moment in time, rather over a period, according to a prior work. Hence, regression analysis was done on a voice service for a period of 120 days, and it was found that the user perceived QoS became slightly lower as the user got used to utilizing it more often with more familiarity [15].

2.4 Download Speed Experience

Download speed is defined as the rate at which data is received from the internet. It is a good indicator of browsing experience. As part of a study, this was measured by downloading a large file over a minute to deal with temporary speed changes [16]. File transfer and web browsing are elastic services that are considered to have a utility function that is increasing, concave and continuously differentiable function of throughput. Previous work based on this assumption shows that this kind of data transfer utility function is logarithmic with respect to rate R [17]. The MOS (Mean Opinion Score) that was proposed in another work [18] was used in this context to measure the quality of user experience [19].

$$ {\text{MOS}} = {\text{a}} \cdot {\text{log}} 10\left( {{\text{b}} \cdot {\text{R}}} \right) $$

Here the maximum and minimum user perceived quality determined the parameters a and b. If the subscriber had signed up for a particular rate service and got the same rate R with no packet loss, then the MOS was set to a maximum, say 4.5. A minimum transmission rate (say, 10 kbps) was also defined and MOS set to 1. MOS as a function of data transmission rate was plotted by varying the data transmission rates [16]. Many studies use a MOS to measure the user perceived QoS or the QoE.

The key to the success of video applications like Ultra HD and high frame rate technology that require higher bandwidth lies in high download rates. ISPs therefore have to continuously monitor the quality of the network at the user level, so that they know the quality a customer using a particular bandwidth will experience. In this respect, a study was done to measure the impact of upload and download speeds on the QoE parameter. The test was done on different upload and download speeds. Multiple speed tests were done to alleviate any influence network connectivity issues may have on the results. The maximum upload and download speeds were measured. There were test cases that were done on single video browsing, multiple video browsing and more interactive behavior-based browsing. Each case was coupled with a unique combination of upload and download speeds. Different KPIs were studied – Page Load time, Initial loading delay, Total stalling time. All these as known affected the QoE measured. Moreover, ISPs are also evaluated publicly which results in competition with other providers on network quality [20].

2.5 Upload Speed Experience

Upload speed is the rate at which data can be sent to the internet. This helps in studying the responsiveness of real-time applications [16]. Upload of large files to servers over the internet through mobile communications has increased a lot. This has its own limitations – bottlenecks in server processing ability or insufficient available bandwidth. There are many measures taken to better the user perceived quality. For example, there was a study that was done by using network resources at the edges flexibly and uploading parallelly, divided files. This improved the upload throughput 10 times more [21].

A proposal was made for an intelligent media distribution architecture of IP Multimedia Subsystem (IMS) for video streaming purposes which involved uploading a multimedia file to a server in the IMS which could later be downloaded as well. This study took into consideration bandwidth, jitter, delay and packet loss that impact the QoE of the user. In addition, it also considered CPU, RAM temperature and the number of users connected to the network, since these also affect the QoE of the end users from an energy efficiency standpoint. Per the article, it was successful in terms of ensuring the upload speed of the multimedia file and guaranteed QoE of the end user. One of the areas of focus was to optimize the upload time of multimedia users. For this, an upload client was defined as an active device that uploaded the videos in to the proposed IMS. It connected the system via SIP (Session Initiation Protocol) and opened the Application Programming Interface (API) and allowed to record videos and to give permissions for sharing them. A manager controlled the upload of the videos and also selected the server to which the uploaded video had to be sent to be downloaded at the other end. For the entire process, the system used optimal transcoding that includes parameters like codec, bitrate and resolution. Evaluating the described multimedia system, with these parameters and also determining the MOS (Mean Opinion Score) showed the impact of the system on the QoE of end users [22].

Apart from prior work, it is to be noted that the download usage of a user is more than the upload usage. In fact, for a median European user, this download-upload share is 88%–12% [23]. Other features of the LTE (Long Term Evolution) technology to be understood is that it uses Single-Carrier Frequency Division Multiple Access (SC-FDMA) to increase coverage and reduce user equipment cost and for energy efficiency on the upload communication side. In this technology, uplink data transmission rates up to 75 Mbps can be achieved. LTE technology thus provides a very high rate of data transmission [24].

3 Methodology

3.1 Data Source

The latest publicly available data for India’s 4G Mobile network providers is published by Open Signal at https://www.opensignal.com/reports/2020/09/india/mobile-network-experience [25]. The public reports and insights are made available to Network providers to understand and improve their service and is also available for use by regulators and analysts.

3.2 Time Period

The latest available time period as of now is September 2020, including data over 90 days starting from May 1, 2020.

3.3 Sample Representativeness

The sample size is large with over 100 million devices used for collecting data, running into daily measurements in billions globally. No restrictions were made in terms of apps, type of user or age of device. Measurements were made on smart phones both indoors and outdoors. The end-to-end consumer network experience from the user device to the Content Delivery Networks was measured.

3.4 Technique

Data Envelopment Analysis (DEA).

DEA could be seen as a multiple-criteria evaluation methodology, that minimizes inputs and maximizes output. The DEA score gives the overall performance of an organization (Decision Making Units or DMUs). This ‘overall performance’ is a composite measure out of aggregating individual indicators [42].

Various studies have been made on determining the efficiency of services provided by the Telecommunications and other sectors as follows: Evaluated the efficiency of the wireless communications sector in around 42 countries [26]; proposed a benchmark to the wireless communications sector in the USA [27]; evaluated the performance of the telecommunications companies in Korea using the Data Envelopment Analysis [28]; assessed the performance and rank the telecommunications industry based on the International Telecommunications Union standard [29]; performed an evaluation of mobile subscribers using the DEA technique and the Principal Components Analysis (PCA) techniques. Data sample was taken out of 27 countries and nine indicators were used to show the significance of the approach. It identified strong and weak points of the telecommunication companies and gave the most efficient output and input of each sector [30]; compared the efficiencies of 30 OECD (Organization of Economic Co-operation and Development) members using the Analytical Hierarchy process (AHP) and DEA tools. This study showed that 8 countries proved to be efficient in terms of productivity and revenue [31]; evaluated the efficiency of Telecommunication companies using DEA for mobile operators in Tanzania. Two approaches - a parametric approach using an econometric model and a non-parametric approach using a mathematical model were used in evaluating the efficiency. The DEA technique was used for measuring the efficiency and the best Decision-Making Unit (DMU) was used for comparison with the others. 7 DMUs were used, with a Constant Return to Scale (CRS), non-radial, oriented model on a Slack Based Measure to determine the efficiency. For ranking the DMUs by their efficiency, the Super SBM- oriented model was used to produce a score [32]. A recent study was done on evaluating the efficiency of public sector banks in India during the financial year 2018–2019 using the Data Envelopment Analysis. The mix efficiency, CCR efficiency/SBM efficiency was calculated to reduce the error rate [32]. Another study reviewed the application of different DEA models on airline efficiency. It concluded that in radial models, the standard CCR, BCC models, a combination of standard and other approaches, and extended and modified models were used for airline studies. For non-radial models, the airline studies were made using Slack Based Model (SBM), RAM and EBM models were used [33]. A robot selection problem for a Taiwanese manufacturing company is used for illustrating the use of radial DEA and the SBM based models. The results showed that the efficiency from the radial DEA model overstated the robot’s performance while the SBM based model was more reliable in ranking the efficiency and for the robot selection [34]. A new risk analysis model for Failure Mode and effect analysis (FMEA) was proposed based on DEA, for ranking the failure modes per risk priority [35]. DEA was used to evaluate the efficiency of 109 sampled small towns in the Jiangsu province of China. Different types of towns showed differences in efficiency characteristics [36]. A novel, modified SBM based model and least absolute shrinkage and selection operator was used to evaluate the effect of entropy-based variable on the efficiency score for hotel performance [37].

Slack Based Measure (SBM).

For an optimal solution that is better than the efficiency scores produced by the DEA technique, a slack based measure (SBM) in DEA was proposed which directly deals with output shortfalls and input excesses of the Decision-making unit. The CCR (Charnes-Cooper-Rhodes) model that was used in earlier studies provided ratio maximization, while this proposed Slack Based Measure could be interpreted as profit maximization. An efficiency measurement tool, this measure is also compatible with other measures of efficiency. This technique considers the ratio efficiency and the slacks. The SBM can be understood as the product of input and output efficiencies [38]. Although the results of this analysis are discussed later, target setting using the SBM approach for a single output measure is shown in Fig. 1.

Fig. 1.
figure 1

Target setting using SBM

Full size image

Advantages of the SBM Based Approach [39]

  • It divides the set of observations into two – efficient and inefficient

  • SBM model evaluates input excesses and output shortfalls (slacks) and identifies all inefficiencies in the concerned Decision-Making Units (DMUs). A DMU is SBM efficient if the slack is zero.

  • Has features like indication of efficiency, monotonicity and unit invariance

Limitations of the SBM Model [40, 41]

  • While evaluating efficiency change over time, the non-zero slacks tend to vary during different time periods. This causes the problem of finding which pattern is reasonable

  • When these slacks are used as sources of inefficiency, and further statistical analyses are made over these, distortion of results is possible.

  • Lack of discrimination in the efficient set of observations

3.5 Software Used

DEA Frontier Solver [42] was used to carry out the analysis and included both DEA efficiency output and the output targets for each metric.

4 Analysis and Results

Five variables for four Service Providers were taken and these are detailed in Table 1.

Table 1. User experience on 4G mobile networks
Full size table

From the data, we see that Airtel is performing best for Video experience, with Vodafone-Idea very close. For all the other metrics, we see that Vodafone-Idea (Vi) is performing best. BSNL appears to lag the other service providers. The below question though remains – is Vodafone-Idea the undisputed leader; is BSNL performing way below the others?

To answer these questions, we perform the Data Envelopment Analysis that will provide relative efficiencies of the four Service providers. Table 2 shows the results of the analysis.

Table 2. Data envelopment analysis – overall efficiency
Full size table

The results show why such a mathematical technique is needed, as only by looking at Table 1, we would have interpreted results differently. From Table 1, we would have expected to see Vodafone-Idea to be the undisputed leader. The results from the analytical technique, though, show that both Airtel and Vodafone-Idea fall on the Efficiency frontier. Jio is close behind with 96.33%. Similarly, from Table 1, we would have expected BSNL to be far behind. The results from Table 2, though, shows that BSNL is close to 90% of the efficiency mark.

We are next interested to see what the best possible path for Jio and BSNL are to reach the efficiency frontier. In simple terms, the lines from the origin passing through the current positions of Jio and BSNL and touching the envelope, are the recommended paths for them. A slack based measure in DEA achieves the same.

The results from SBM gives the below results as shown in Fig. 2.

Fig. 2.
figure 2

Recommendations for BSNL and Jio

Full size image

Out of the five Experience metrics, we see both BSNL and Jio do not require any improvement in their Voice App Experience. On download and upload speed experiences, BSNL requires improvement of more than 100%, while Jio requires more than 50%. In terms of Video and Games Experience, the improvement required is lower but still significant for BSNL and Jio.

To summarize, if the Service Providers have the following metrics as in Table 3, they would all be performing equally.

Table 3. SBM recommendations for all providers to achieve 100% efficiency
Full size table

One strong point of the technique is that recommendations are not the same as the best provider, rather, the closest path to the efficiency frontier. Let us take BSNL as an example; all recommended targets are still below their competitors, yet still falls on the efficiency frontier.

For Jio too, all targets are below what Vi currently achieves but still falls on the efficiency frontier.

5 Managerial Contributions

At least four managerial contributions are made in this paper.

  1. 1.

    Realistic, and scientific targets for Service Providers that fall short against their competitors are provided.

  2. 2.

    These recommended targets are at a very granular level, relating to each performance metric.

  3. 3.

    While organizations can have specific stretch targets, this analysis provides the most optimal path to attain 100% efficiency and be on par with their competitors.

  4. 4.

    The analysis provided is also quick to conduct and replicate, even for larger volumes of data, like state level drill downs.

6 Limitations and Directions for Future Research

This research assumed all input parameters to be equal. The reason for this is two-fold. First, a separate study needs to be conducted to identify all significant variables that impact the studied output metrics. Second, even post such a study, organizations may be reluctant to post such material in the public domain. It was therefore decided that few and arbitrary inputs should not be considered while making recommendations for Experience metrics. Future studies though can attempt to analyse data if all the information is made available. The study presents results for India as a whole. What will be more useful to Service Providers could be state or region wise breakdown of such targets. This study has taken into consideration five Experience metrics. Future studies can attempt combining other performance metrics such as Quality of Service (QoS) like bandwidth, latency, packet loss and other related KPIs (Key Performance Indicators).

7 Conclusion

This paper has attempted to scientifically measure performance of Quality of Experience metrics for the top four 4G Service Providers. This is vital in providing recommendations to the Service Providers by analyzing the User perceived QoS. Such performance analysis not only helps the Service Providers to improve their QoS provision but could also help them increase their active customer base.