Provision of end-to-end QoS in heterogeneous multi-domain networks

Abstract

In this paper, we present the framework to provision end-to-end QoS in heterogeneous multi-domain networks that was implemented in EuQoS system and tested in Pan-European research network. It assumes that a pair of end users, possibly attached to different access networks as xDSL, UMTS, LAN/Ethernet, WiFi, MPLS or Satellite, may choose for its connection an appropriate end-to-end Class of Service, depending on the application they use, e.g. VoIP, VoD, FTP, etc. Each end-to-end Class of Service has its own traffic control mechanisms and algorithms and, as a result, it has the ability to handle traffic streams with assumed guarantees for packet transfer characteristics expressed in the form of loss ratio, mean delay and delay variation. The end-to-end Classes of Service are supported in all the domains (including inter-domain links) independently using specialised inter-domain Class of Service-aware QoS routing protocol which establishes the end-to-end QoS paths. This paper describes the solution and includes exemplary simulation and experimental results.

1 Introduction

Any solution intended to provide end-to-end Quality of Service (QoS) in the Internet has to consider heterogeneity of network technologies and multi-domain aspects.

This paper describes the framework assumed for the provision of end-to-end QoS in the EuQoS system designed to be implemented over heterogeneous multi-domain network. The objective of the framework is to provide an effective and comprehensive solution to ensure the requested QoS at the network level when the end users, who require the QoS connection, are attached to possibly different access networks. Furthermore, the access networks can be built on different technologies as xDSL, UMTS, LAN, WiFi, MPLS or Satellite and connected by many IP-based transit domains. Finally, by implementing the framework, we expect to transfer the packet streams with adequate quality expressed in terms of target values of IP Throughput (IPT), mean IP Packet Transfer Delay (IPTD), IP Packet Delay Variation (IPDV) and IP Packet Loss Ratio (IPLR) [51].

The investigated approach for the framework assumes that a number of so-called Classes of Service (CoSs) are established in the network. The CoS refers to a service offered by the network to provide appropriate transfer of submitted packet streams. By implementing a given CoS, we expect that the packets handled by this CoS will be transferred by the network according to the guarantees specified between the user and the provider of this service. For instance, when considering best effort service, one can expect that the packets submitted to this service may suffer unpredictable transfer delay and may even be lost. Therefore, in the EuQoS system, the implemented CoSs will guarantee for the packet streams the specific QoS expressed as target values of IPT, mean IPTD, IPDV and IPLR parameters.

We believe that the proposed solution can be an important step forward in the standardisation activity on the Internet towards the multi-domain, heterogeneous networks that are based on QoS architectures. There are several standardisation documents, e.g. provided by IETF, ITU-T, 3GPP organisations, referring to these issues [1]. There are also plenty of papers evaluating QoS taking into account different aspects as architectures including traffic control mechanisms, QoS provision in core IP networks or in access networks, see [2, 3] or [4]. Other related works in the area of QoS architectures, mainly referring to IP-based DiffServ networks [5], are the results of EU projects, e.g. AQUILA [6], TEQUILA [7] and CADENUS [8].

Even though there are many recommendations about architectures for Next Generation Networks (NGN) [9] and definitions of CoSs for specific technologies, e.g. DiffServ IP networks [10], UMTS [11], WiFi [12], LAN/Ethernet [13], etc., the reality is that today’s Internet offers only best effort service and it still lacks in sufficient mechanisms to assure end-to-end QoS. In particular, currently available QoS control mechanisms are specific for each network technology and, in addition, they differ in their objectives. For example, the WiFi technology has a different set of CoSs when comparing to the UMTS technology. On the other hand, for assuring end-to-end QoS, we need a coherent set of end-to-end CoSs (e2e CoSs) that is supported by all technologies as well as the signalling system for handling the requests and performing appropriate resource reservations. In this paper, we introduce a comprehensive approach for providing end-to-end QoS as developed in the EuQoS project [14, 15, 49]. This approach covers the above-discussed issues.

The EuQoS architecture fulfils the requirements of the ITU-T QoS standards for IP-based Networks [16], for NGN [9] and for absolute end-to-end QoS guarantees [17]. We are also in line with the IETF recommendations for QoS architectures and CoS concept as proposed in [10]. It also fulfils signalling requirements for IP QoS [18] and Resource and Admission Control Functions in NGN [19]. Moreover, in the EuQoS system, we propose several specific admission control and resource reservation algorithms, which allow us to assure end-to-end QoS, and usually, they are not under standardisation. In this paper, we discuss some of them.

The proposed CoSs are supported independently in particular domains and the solutions are specific for a given technology. In addition, the policy of the network providers may have an impact on the selection of the supported CoSs. With regards to the end-to-end QoS level results from QoS levels offered by all domains along the path, we propose a specialised CoS-aware inter-domain QoS routing protocol, called Enhanced QoS Border Gateway Protocol (EQ-BGP). The EQ-BGP builds the end-to-end paths for each e2e CoS and, for this purpose, it takes into account QoS level of particular domain and objectives of e2e CoSs.

The rest of the paper is organised as follows. Section 2 describes in detail the concept of implemented e2e CoSs in the EuQoS system and QoS objectives. In Section 3, we present the EQ-BGP, which builds end-to-end path across multi-domain network. Next, we explain the main assumptions used to design the QoS mechanisms and algorithms required for supporting e2e CoSs in particular technologies. Especially, we focus on the specification of generic Connection Admission Control (CAC) algorithm that is the key element for providing QoS guarantees at the network level. The applied solutions for providing e2e CoSs in each network technology as in core IP, xDSL, LAN/Ethernet and WiFi access networks are described in Section 5 and exemplary results for e2e CoSs in IP links are presented in Section 6. Finally, Section 7 summarises the paper.

2 End-to-end classes of service in heterogeneous networks

We assume six e2e CoSs that differ in QoS objectives and traffic characteristics. These e2e CoSs are visible to the end users (user applications) and are deployed across the multi-domain network, which may be built on the basis of different network technologies. Table 1 shows in “italics” the EuQoS e2e CoSs within the complete set of the CoSs as proposed for DiffServ architecture [10] jointly with the requirements for QoS objectives provided in the target values of IPLR, mean IPTD and IPDV parameters. An e2e CoS in the EuQoS system is fully dedicated to handle the packet streams generated by one or more applications, as shown in Table 1. Moreover, according to [52] we can group e2e CoSs into four treatment aggregate classes’ type: Control (CTRL), Real Time (RT), Non-Real Time/Assured Elastic and Elastic.

Table 1 Mapping of EuQoS applications to classes of service

The EuQoS e2e CoSs are the following.

• Real Time type:

  • Telephony e2e CoS: this class is mainly dedicated to handling Constant Bit Rate (CBR) or Variable Bit Rate (VBR) traffic produced by Voice over IP (VoIP) applications.

  • RT Interactive e2e CoS: this class handles CBR or VBR traffic coming from applications as Video-Tele Conference (VTC) and NEXUIZ interactive game [20]. Let us remark that the traffic submitted to RT Interactive e2e CoS and Telephony e2e CoS differs in the packet lengths (VoIP packets are rather small compared to VTC packets) and in required bandwidth (again, VoIP requires rather small bandwidth compared to the bandwidth required by VTC), whereas the required QoS level is similar.

  • Signalling e2e CoS (S-CoS): this class is designated to handle signalling traffic. For the traffic submitted to this CoS, we require guarantees of target mean IPTD and IPLR, whilst the value of IPDV is not critical. Thanks to this e2e CoS, we may guarantee connection set-up delays at the predefined value as presented in [21].

• Non-Real Time/Assured Elastic type:

  • Multi-Media (MM) Streaming e2e CoS: this class is dedicated to handling streaming traffic (CBR or VBR) generated by VoD applications. This e2e CoS should provide guarantees of target mean IPTD and IPLR, whilst the value of IPDV is not critical.

  • High Throughput Data (HTD) e2e CoS: this class is dedicated to handling TCP-controlled elastic traffic as, e.g. the traffic generated by the data transfer component of the composed medical application Medigraf [22]. Similarly to MM Streaming e2e CoS, the HTD should provide guarantees of target mean IPTD and IPLR, whilst the value of IPDV is not critical.

• Elastic type:

  • Standard e2e CoS (STD): this class handles best effort traffic and it means that no guarantees about IPTD, IPDV and IPLR are provided. Anyway, for this e2e CoS, the network allocates some resources as a given volume of bandwidth and buffer capacities (let us remark that in the case of Low Priority Data Class, network does not allocate any bandwidth).

The IP network recognises the affiliation of an IP packet to a given e2e CoS by analysing the Differentiated Services Code Point (DSCP) field in IPv4 or Traffic Class (TC) field in IPv6. The appropriate code in the IP packet is assigned by the user equipment and, once again, by the first network element that the packet crosses. Table 2 shows the DSCP codes corresponding to the different e2e CoSs in EuQoS system (as proposed in [10]).

Table 2 DSCP codes for e2e CoSs in EuQoS

For the implementation of the discussed set of e2e CoSs, we assume that they are globally well-known and are visible by the user QoS-aware applications. An end user activates the desired application (VoD, VoIP, etc.) and the application submits its QoS request to the predefined e2e CoS, in accordance with Table 2. This request is handled by the EuQoS system. The network resources necessary for the new incoming connection are reserved and allocated in selected nodes along the end-to-end path, led by the traffic engineering rules. Inside the EuQoS system, the possible paths are determined by using the EQ-BGP protocol. The EQ-BGP takes into account the QoS level offered by particular domains and inter-domain links and chooses the most relevant path in terms of e2e QoS objectives. Once the path is fixed, the QoS request is sent to the Resource Managers (RMs) situated on the path from the source access domain, along transit domains, until the destination access domain. When one RM receives the QoS request, it communicates with the associated Resource Allocator (RA) to check whether the requested resources are available in the underlying network. For this purpose, we use appropriate CAC functions associated with given e2e CoS and with underlying network technology.

Figure 1 shows the concept to implement the above-specified set of e2e CoSs in the EuQoS system. The implementation requires that each network technology supports the specified set of e2e CoSs. When a given underlying network technology supports by itself the same CoSs as e2e CoSs (in terms of both handled traffic profiles and QoS guarantees), then, it is sufficient to perform a one to one mapping between e2e CoSs and CoSs offered by this technology. However, in the standards for the underlying network technologies (WiFi, xDSL, UMTS, LAN/Ethernet, MPLS, Satellite and IP), generally, the CoSs are not compatible with the discussed e2e CoSs. In this case, some new solutions as e2e CoSs aggregation have been proposed and implemented to provide packet transfer capabilities as requested by the e2e CoSs. These solutions are mainly focused on proposing adequate CAC functions to limit the volume of traffic carried by each e2e CoS and setting parameters of available QoS mechanisms (as schedulers, shapers, policers, etc.) in the network elements. In the EuQoS system, these functions are enforced by the RA elements.

Fig. 1

Concept of e2e CoSs for implementation in EuQoS system

The proposed approaches are in many cases EuQoS-specific solutions and, as a consequence, they are neither commercially available nor standardised.

3 Enhanced QoS border gateway protocol

The EQ-BGP [14, 23] is the CoS-aware inter-domain QoS routing protocol that was designed, developed and tested within the EuQoS project. The objective of this protocol is to establish inter-domain routing paths for particular e2e CoSs. In accordance with the indications for QoS routing (see [24]), the EQ-BGP extends the currently used BGP-4 inter-domain routing protocol [14] in several ways: (1) it defines new path attribute, called as QoS Network Layer Reachability Information (QoS NLRI), which conveys information about e2e CoSs offered on advertised paths, (2) it uses the QoS assembling function for computing the final guaranteed QoS level on the basis of the QoS level guaranteed by each segment of advertised path, (3) it applies the QoS-aware decision algorithm for selecting the best path for each e2e CoS and (4) it maintains a number of separate routing tables associated to each particular e2e CoS that allow router for independent routing and forwarding of packets belonging to given e2e CoS. Summarising, EQ-BGP may select more feasible routing paths for e2e CoSs comparing to the shortest path routing algorithm used by BGP-4. This takes place because not all the shortest paths may satisfy assumed QoS objectives for e2e CoSs.

The concept of extending BGP with capabilities to support QoS has been considered in the literature as in [25] and further investigated, e.g. in [26–28] and [29]. The proposals assumed an exchange of information about the bandwidth available on the paths, the congestion alarms or values of QoS parameters characterising particular domains, such as packet delay, packet losses, etc. This information aided to support path selection process mainly from the point of view of traffic engineering objectives, e.g. focusing on optimisation of resource utilisation or packet delay. On the other hand, in [30], the authors considered QoS enhancements of BGP as a tool for supporting meta-QoS classes. The meta-QoS class represents a set of packet streams that requires only qualitative QoS assurances as, e.g. “low delay”, “any losses”. Moreover, the meta-QoS classes are loosely coordinated between domains. Opposite to the meta-QoS class concept, the EQ-BGP proposal is directly oriented for supporting e2e CoSs providing absolute QoS guarantees as presented in Table 1. Summarising, the main innovation of the EQ-BGP proposal is to take together the e2e CoSs with their QoS objectives and the inter-domain QoS routing performed by the EQ-BGP protocol.

In [23] and [14], we introduced the EQ-BGP protocol. The solution assumes that the EQ-BGP routers advertise information about the reachable destinations and the cumulated values of QoS parameters guaranteed by each e2e CoS on the path. These cumulated values are calculated considering the impact of all domains and inter-domain links on the path towards advertised destination. Then, the neighbouring EQ-BGP routers update the received values of QoS parameters considering the contribution of their domains and they take a decision about routing. In case of routing changes, the router advertises them to the neighbours immediately. At last, the EQ-BGP sets the roadmap of paths that are available for each e2e CoS in the network. The roadmap also provides the values of the QoS parameters guaranteed between each pair of source and destination prefixes.

In this paper, we focus on: (1) the implementation of the EQ-BGP protocol covering detailed description of its main components, (2) the approaches for the deployment of EQ-BGP in the network and (3) the trials performed in the Pan-Europeran EuQoS testbed.

3.1 Implementation of EQ-BGP

The main components implemented in the EQ-BGP protocol are: QoS NLRI attribute and the CoS-aware decision algorithm. The role of the QoS NLRI is to convey the values of QoS parameters corresponding to e2e CoSs. Figure 2 shows the assumed format of the QoS NLRI path attribute. In fact, [25] and [28] proposed other formats, which do not follow the EQ-BGP requirements. However, all the proposals have attribute headers containing flags, type indicator and attribute length. The flags are used to inform the routers that the QoS NLRI attribute is optional, non-transitive and complete. The main part of the attribute contains a number of structures describing particular e2e CoSs. Each structure covers the e2e CoS identifier (1 byte) and three fields with the values of mean IPTD, IPDV and IPLR (12 bytes). Mean IPTD and IPDV are expressed in microseconds whilst IPLR is expressed in the exponent form of packet losses: −1,000 × log₁₀(IPLR).

Fig. 2

The assumed format for the QoS NLRI path attribute

The CoS-aware decision algorithm allows the EQ-BGP routers to compare the possible paths towards destination domain and associated to given CoS and selects the best one. For this purpose, the algorithm adds a new step to the routing decision process that evaluates the Degree of Preference (DoP) factor taking into account the values of QoS parameters carried in the QoS NLRI attribute. In this decision algorithm, the DoP criterion is preferential over the path length criterion (see Fig. 3). As a consequence, EQ-BGP considers the QoS level offered by the available paths as the primary criterion and, only if more paths have the same QoS level, then the shortest path will be selected. The rest of decision steps remain the same as in BGP-4 protocol.

Fig. 3

EQ-BGP decision algorithm

The key element of the proposed algorithm is a DoP function. It calculates value of the DoP factor based on a set of QoS parameters corresponding to a given e2e CoS. In this way, we transform the multiple constrained path selection problem that, in the general case, is NP-complete [50] into the shortest path selection problem, which we can easily solve. Therefore, the proposed approach is a heuristic solution, of which effectiveness may depend on the applied DoP function. In our experiments, we assume a weighted sum of normalised distance inverses as the DoP [23]. The analysis of the effectiveness of different DoP functions appropriate for a particular e2e CoS is one of the directions for further studies.

3.2 EQ-BGP deployment

Taking into account that the EQ-BGP protocol is currently not standardised, we consider two options for its deployment in the network. The first option assumes that EQ-BGP is implemented directly on the border routers (see Stub domain 2, AS2 or AS4 in Fig. 4). Such deployment is possible on open source routers as, e.g. currently available Quagga [31] or XORP [32]. The second option assumes that the EQ-BGP protocol is implemented on an additional host located inside a given domain, called the EQ-BGP route controller (see Stub domain 1, AS1 and AS3 in Fig. 4).

Fig. 4

The options for EQ-BGP deployment

The route controller exchanges information with neighbouring EQ-BGP routers and, on that basis, it selects one path for a particular e2e CoS. Then, the route controller provides routing information to all border routers using the BGP-4 protocol. This option is proposed for the domains, of which border routers do not support the EQ-BGP protocol. Moreover, the route controller option is especially attractive for domains with a large number of border routers as it allows avoiding the full mesh internal connectivity between border routers. In this sense, the route controller option is similar to the commonly used route reflection approach as recommended in [33].

3.3 Performance evaluation

In this section, we present the results of exemplary EQ-BGP tests that were performed in the Pan-European EuQoS testbed presented in Fig. 5. The objectives of these tests were twofold: (1) to validate the behaviour of EQ-BGP operating directly on Linux border routers as well as the route controller managing CISCO and Juniper border routers and (2) to evaluate the convergence of EQ-BGP prototype implementation in “live” network. For the evaluation of the EQ-BGP performance, we considered the following convergence metrics:

• The Network Convergence Time (NCT) is the time duration between the moment of a new prefix advertisement (or the withdrawal of known one) and the moment when the processing of the last routing message caused by this event is finished.

• The total Number of Messages (NUM) exchanged during the NCT.

The NCT and NUM values were measured only for the basic routing protocol events, which are prefix advertisement and withdrawal. The testbed consists of ten domains/Autonomous Systems (ASs) across Europe connected by virtual links (GRE tunnels) provided by GÉANT and NREN research networks (see Fig. 5). Although the EuQoS testbed is limited to ten ASs, the results may be a guideline to understand the impact of network characteristics such as packet delays or Minimum Route Advertise Interval (MRAI) on the EQ-BGP convergence.

Fig. 5

Configuration of EuQoS testbed

Taking into account that such parameters as the type of network topology and the number of domains have strong impact on the routing protocol performance, we assumed for tests three different configurations of EuQoS testbed that constituted full mesh, ring and chain networks. For each configuration, we analysed three test cases with four, seven and ten ASs in the core. In addition, we assumed homogeneous ASs supporting only e2e Telephony CoS provisioned to guarantee values of mean IPTD = 6 ms, IPDV = 2 ms and IPLR = 10⁻⁶ in both intra-domain area and inter-domain links. Moreover, for all routers, we set the default value of the MRAI timer equal to 30 s.

Figure 6 presents the comparison of the NCT values measured for the EQ-BGP and simulated for the BGP-4 that corresponds to a prefix advertisement (withdrawal) in the ring, full mesh and chain networks. Taking into account that trials were performed in “live” network where other prefixes could be advertised or withdrawn during the test, we simulated two limit cases: (1) Sim1 corresponds to “the best case” situation when only one prefix is advertised (withdrawn) within the network; in this case, the routers immediately sent updates without waiting for the MRAI timer, on the contrary, (2) Sim2 assumes “the worst case” situation when two prefixes were advertised (withdrawn) one just after the other. In this case, the routers could send updates for the second event only when the MRAI timer for the first event expired.

Fig. 6

The Network Convergence Time for the EQ-BGP (measured) and the BGP-4 (simulated) running on a ring, b full mesh and c chain networks

The results indicate that the NCT time of the EQ-BGP is close to the NCT time of BGP-4 simulated in Sim2. So, we may conclude that the convergence of the EQ-BGP is strongly influenced by the advertisement of other prefixes. However, the most important conclusion is that the EQ-BGP NCT time is not greater than in the case of BGP-4.

The NUM results presented in Fig. 7 indicate that, in the case of prefix advertisement, EQ-BGP exchanges the same number of messages as BGP-4. Whereas, in the case of prefix withdrawal, the EQ-BGP uses even fewer messages than BGP-4. This phenomenon we can especially note in the case of full mesh network. More detailed results are presented in [34].

Fig. 7

Number of update messages proceeded by the EQ-BGP (measured) and the BGP-4 (simulated) running in a ring, b full mesh and c chain networks

4 QoS mechanisms and algorithms for specification of end-to-end classes of service

The communicating hosts, depending on the type of application used, have a possibility for asking the network to establish the connection inside one of the four e2e CoSs. Choosing the i-th e2e CoS (i = 1, 2, 3, 4), the end user may expect that the submitted packets experience network degradation expressed in values of mean IPTD, IPDV and IPLR not greater than the assumed target values of IPTD_e2e,i , IPDV_e2e,i and IPLR_e2e,i , respectively. Furthermore, for given end-to-end path crossing N domains, we should have a CoS compatible with the i-th e2e CoS in each of these domains, say CoS _{j ,i} (j = 1, 2, …, N). The capabilities of each CoS _{j ,i} are also expressed by the maximum allowed values of three QoS parameters, i.e. IPTD _{j ,i} , IPDV _{j ,i} and IPLR _{j ,i} . Thanks to the additive properties of the mean IPTD, IPDV and IPLR, we can write the relations 1:

$$\begin{array}{*{20}l} {{{\text{IPTD}}_{{{\text{e}}2{\text{e}},i}} = {\sum\limits_{j = 1}^N {{\text{IPTD}}_{{j,i}} } }} \hfill} \\ {{{\text{IPDV}}_{{{\text{e}}2{\text{e}},i}} = {\sum\limits_{j = 1}^N {{\text{IPDV}}_{{j,i}} } }} \hfill} \\ {{{\text{IPLR}}_{{{\text{e}}2{\text{e}},i}} = {\sum\limits_{j = 1}^N {{\text{IPLR}}_{{j,i}} } }} \hfill} \\ \end{array} .$$

(1)

Note, that IPLR is additive for low values of IPLR _{j ,i} , (IPLR _{j ,i}  < 10⁻²) and IPDV is additive only if IPTD _{j ,i} is not a heavy-tailed distribution, whereas IPTD is always additive since IPTD _{j ,i} are mean values, as referred to in [35].

Finally, we come to the conclusion that the specification of the i-th e2e CoS leads to design associated CoS _{k ,i} (k = 1, 2, …, K) in all the domains, which offer this i-th e2e CoS. The values of the parameters IPTD _{k ,i} , IPDV _{k ,i} and IPLR _{k ,i} should be fixed during the provisioning process of the EuQoS system on the basis of the values of IPTD_e2e,i , IPDV_e2e,i and IPLR_e2e,i and the multi-domain network topology. In the further part of this section, we assume that the provisioning process was already performed.

The starting point to design the requested CoSs in particular domains is the analysis of QoS mechanisms and algorithms for considered underlying technologies. This point will be treated in the next section. The general principles to design a CoS (e2e CoS or particular CoS) are as follows: (1) to allocate the resources for the considered class, i.e. link capacity and buffer size, (2) to apply available QoS mechanisms in network devices to enforce required packet transfer characteristics, and (3) to limit the traffic submitted to these resources by applying appropriate CAC function. Next, we illustrate these rules by considering an example of CoS that handles traffic streams described only by the Peak Rate (PR) value whilst the mean IPTD, IPDV and IPLR values should not be greater than the predefined target values.

Example: designing CoS with predefined maximum values of parameters mean IPTD, IPDV and IPLR. The CoS handles the traffic streams with declared PRs.

4.1 Allocation of resources for the considered class

The required resources for a given CoS are usually represented by the link capacity (C) and associated buffer size (B). The CoS is designed for handling packet streams emitted by applications with similar traffic characteristics. For the sake of simplicity, we assume that the applications generate the packets with constant length, say L. For this case, we can control IPDV by setting B and C, since:

$${\text{IPDV}} = \frac{{L \times B}}{C}.$$

(2)

Furthermore, the commonly known condition in the case when a number of packet streams are multiplexed in a single link is that the link utilisation should be less than 1. The condition for maximum link utilisation, say ρ _max, comes from the constraints for IPLR or mean IPTD. The relations for IPLR and mean IPTD, derived from the analysis of the M/D/1/B [36] and M/D/1 (e.g. [37]) systems, are 3 and 4, respectively:

$$\rho _{{\text{IPLR}}} = \frac{{2B}}{{2B - \ln \left( {{\text{IPLR}}} \right)}}$$

(3)

$$\rho _{{\text{IPTD}}} = \frac{{2\left( {{\text{IPTD}} - T_{{\text{prop}}} - \tfrac{L}{C}} \right)}}{{2{\text{IPTD}} - 2T_{{\text{prop}}} - \tfrac{L}{C}}}$$

(4)

where T _prop denotes the propagation delay of the link. Finally, we calculate ρ _max from:

$$\rho _{\max } = {\text{Min}}\left[ {\rho _{{\text{IPLR}}} ,\rho _{{\text{IPTD}}} } \right].$$

(5)

In fact, constraint 3 takes place in the most practical cases. Constraint (4) occurs only when the links have large propagation delays and provisioned capacity C is rather low, e.g. for the case when T _prop = 90 ms, C < 4.4 Mbps, B = 10 packets, IPLR = 10⁻³, L = 1,500 bytes and mean IPTD = 100 ms.

4.2 Applying available QoS mechanisms in network devices to enforce required packet transfer characteristics

The set of QoS mechanisms, which are available in the network devices, differs depending on the underlying technology. Anyway, the reference QoS mechanisms are specified as Per Hop Behaviour (PHB) mechanisms for IP network in the context of DiffServ architecture. From the point of view of assuring requested packet transfer characteristics, the most important is the type of available scheduler. The preferred schedulers are Priority Queuing–Weighted Fair Queuing (PQ-WFQ) or WFQ since they assure isolation in handling the traffic streams submitted to different CoSs. It means that for each CoS, we guarantee a part of total link capacity and separate buffer space.

Unfortunately, the above scheme cannot always be achieved in a straightforward way, e.g. in the case of wireless access with the Medium Access Control (MAC) based on contention.

4.3 Limit of the traffic submitted to the CoS resources by applying appropriate CAC function

To limit the traffic submitted to a given CoS, we may apply the following well-known formula for Peak Rate allocation scheme [36]:

$${\text{PR}}_{{\text{new}}} + \sum\limits_{i = 1}^M {{\text{PR}}_i \leqslant \rho _{\max } C} $$

(6)

where PR_new is the Peak Rate of the new incoming connection and M is the number of running connections, each one described by its PR _i (i = 1, 2, …, M). The CAC function is invoked during the call set-up procedure. From formulas 2, 3, 4, 5, and 6, we could calculate the necessary buffer and link resources for incoming flow and check whether these resources are available in the system at this moment.

5 Implementation of end-to-end classes of service in underlying technologies

In the previous section, we presented the requirements for implementing a CoS jointly with generic formulas for CAC function and necessary QoS mechanisms. When we face up to the implementation of a CoS in a particular domain, we must carefully pay attention to the possibilities of the particular underlying technology. This section provides a brief description of the solutions for providing CoSs, associated to the considered e2e CoSs, in chosen underlying technologies, i.e. IP links, xDSL, LAN/Ethernet and WiFi. The approaches have been implemented in the EuQoS system and tested in the Pan-European testbed environment.

5.1 Inter-domain links

The inter-domain links connect two peering ASs and have a form of two unidirectional links, one for each direction. More precisely, the inter-domain link for one direction begins at the output port of the egress Border Router (BR) in one domain and it terminates at ingress BR of the peering domain, as it is depicted on Fig. 8.

Fig. 8

Intra-domain and inter-domain CoSs (forward direction)

Thanks to IP technology, we have available the PHB mechanisms that are implemented in the egress BR, including, e.g. schedulers as PQ-WFQ or/and WFQ. This allows us to set CoSs in a straightforward way as it was described for the general case of e2e CoSs.

For the inter-domain area, in EuQoS, we define four inter-domain CoSs, which are: (1) S-CoS, (2) RT CoS, (3) NRT CoS and (4) STD CoS. Table 3 shows the mapping of EuQoS e2e CoSs to the inter-domain CoSs.

Table 3 Mapping between EuQoS e2e CoSs and the inter-domain CoSs

The RT and NRT inter-domain CoSs are the so-called aggregated CoSs, i.e. they are designed to merge in one class traffic corresponding to a number of different e2e CoSs but with similar traffic profiles and QoS objectives. For example, we have RT CoS that is dedicated to handle streaming traffic belonging to both Telephony and RT Interactive e2e CoSs. In this case, to assure that the QoS guarantees for the mentioned e2e CoSs, the aggregated RT CoS is designed to assure the most critical QoS requirements (mean IPTD, IPDV and IPLR) of both e2e CoSs, Telephony and RT Interactive. However, to keep QoS guarantees for e2e CoSs inside an aggregate CoS, we need to specify some additional conditions. These conditions are crucial when performing the CAC algorithm and they mainly focus on controlling traffic profile and on specifying the limits of traffic volume for each e2e CoS. One of these conditions occurs in the case of NRT CoS, for which we require the same traffic profile for both HTD and MM Streaming traffic. Therefore, since the HTD traffic is originally an elastic TCP-controlled traffic, we must shape it at the network entry point to change it into a VBR streaming traffic.

5.1.1 CAC algorithms

In the inter-domain area, the CAC function is performed within the egress BR (see Fig. 8) at the output port. The provisioning process is responsible for partitioning the resources (link capacities with associated buffer sizes) between supported CoSs. The configuration of the output port is depicted in Fig. 9.

Fig. 9

Configuration of PHB mechanisms in the egress border router output port for inter-domain CoSs

In the output port, we have the following traffic control mechanisms: DiffServ classifier, queue selection, scheduler: PQ-WFQ or PQ–Weighted Round Robin (PQ-WRR) and queue management (drop tail). The CAC function is specified only for RT and NRT CoSs, whereas, for S-CoS, the approach is to maintain over-provisioning [21] and there is no CAC control for STD. The generic applied CAC algorithms for both RT and NRT CoSs follow Eq. 6. Anyway, in the case of RT CoS, the detailed system analysis is required to determine the maximum value of link utilisation ρ _max caused by the differences in packet lengths between VoIP and VTC streams. The details of the system analysis are shown in [38]. In the case of NRT CoS, the complete approach is described in [39] and it is focused on the rules to set the shaping rates for TCP-controlled traffic.

5.2 xDSL

In the xDSL network (see Fig. 10), four network points may be the bottlenecks and need to be considered for controlling traffic by CAC. These points are the user gateway/CPE (xDSL modem), DSL Aggregation Module (DSLAM), aggregation switch(es) and IP edge node. In the same way, we should take into account all the above-mentioned nodes for complete QoS provision in xDSL. However, in practice, some simplifications can be justified, depending on the specific characteristics of the network technologies and capabilities of particular elements.

Fig. 10

Exemplary xDSL network in EuQoS

In order to achieve CAC applicable for any variant of DSL access network, the proposed CAC algorithms should be used for every IP-aware port with implemented QoS mechanisms. In the links where there is no IP QoS or CoS, differentiating is limited, the whole QoS traffic should be merged and handled as traffic submitted to the most demanding CoS. In this paper, we consider only the access and aggregation segments and we focus on two elements: the DSLAM (more precisely, IP DSLAM, which has implemented the QoS mechanisms for IP traffic) and the IP edge node (BRAS).

The proposed CAC algorithms for these two elements differ in the assumed QoS provision since there are some essential differences between them; in the aggregation segment, we may apply a static partitioning of the link capacity between maintained CoSs, similarly as we performed in the inter-domain links, whilst, for the access segment, we will focus on sharing the link capacity that is rather low comparing to the application demands.

5.2.1 CAC for BRAS in aggregation segment

Since in the aggregation segment we have implemented PHBs as in border routers, we can apply the same methodology for CAC as in the case of inter-domain links. The difference is that now the mapping between e2e CoSs and CoSs supported by the aggregation segment can be “one to one”. So, the formulas for CAC follow Eq. 6.

The same approach should be applicable in both upstream and downstream directions. Let us remark that we can expect some problems with setting the values of required buffer sizes for CoSs since, in some cases, there is no access to internal configuration for setting these values or there are some additional barriers as, e.g. the buffer size cannot be less (or greater) than given predefined threshold.

5.2.2 CAC for access segment

The access segment is in fact a part of the end-user equipment and the question is whether the network operator can deal with it. Anyway, the operator can suggest some solutions for the user equipment configuration for uplink traffic (downlink traffic to the user is controlled by DSLAM). We assume that both CPE and DSLAM have implemented QoS mechanisms at the IP layer. Furthermore, upstream and downstream traffic is handled in a different way and the reasons are that the link capacities in both directions are different (for upstream, about 256–512 kbps, for downstream, about 1–2 Mbps) as well as traffic emitted by majority of the applications is also asymmetric with higher values for downstream. For the access segment, one can find different xDSL solutions depending on the vendors. The solutions also differ in QoS support level. Nevertheless, if it is feasible that the QoS mechanisms are implemented for both directions, respectively, for downstream on DSLAM and upstream on CPE.

The access segment has a relatively low capacity and is used mainly by a single customer with a very limited number of flows running in parallel. As a consequence, in this case, the multiplexing effect inside a given CoS is negligible as well as static resource partitioning between CoSs is not sensible.

The CAC algorithm is as follows: PR_new,i denotes the requested Peak Rate value by new call submitted to the i-th e2e CoS (i = 1, 2, 3, 4). Note that QoS request are not used for S-CoS and STD CoS. This call can be accepted if the following conditions are satisfied:

$$\begin{array}{*{20}l} {{{\text{PR}}_{{{\text{new}},i}} \leqslant C - {\sum\limits_{j = 1:j \ne i}^4 {{\text{Max}}{\left[ {{\sum\limits_{k = 1}^{N_{j} } {PR_{{k,j}} } },C_{{\min ,j}} } \right]}} } - C_{{{\text{STD}}}} - C_{{{\text{S - CoS}}}} ,} \hfill} \\ {{{\text{PR}}_{{{\text{new}},i}} \leqslant C_{{\max ,i}} - {\sum\limits_{k = 1}^{N_{i} } {PR_{{k,i}} } }} \hfill} \\ \end{array} $$

(7)

where C represents the link capacity, N _j the number of running connections in the j-th e2e CoS, PR _{k ,i} represents the Peak Rate of running k-th connection belonging to the i-th e2e CoS, C _min,k and C _max,k represent the minimum and maximum guaranteed capacity for the connections belonging to the k-th e2e CoS, respectively, and finally, C _S-CoS and C _STD represent the link capacity dedicated to the e2e S-CoS and e2e STD CoS (usually C _STD is a small part of C, say 0.1C).

5.3 LAN/Ethernet

In Ethernet switches, the basic mechanism to differentiate traffic is Priority Queuing (PQ) scheduler. According to the IEEE 802.1Q [40] and 802.1p (part of the IEEE 802.1D [13]) standards, for MAC layer, eight priority levels are specified, each for different Ethernet CoS. The priority level of an Ethernet frame is marked in the 3-bit priority field. Let us remark that the eight priority levels are not currently available in all the equipment and one can find equipment with capabilities of handling only four or even two priority levels. Table 4 shows the proposal for mapping between e2e CoSs and Ethernet CoSs in the case when eight priority levels are available.

Table 4 Mapping between e2e CoSs and Ethernet CoSs

The implementation of e2e CoSs in LAN/Ethernet is not a trivial issue even when we have PQ scheduler for traffic differentiating and links of high capacity (usually 10, 100 and even 1,000 Mbps). The main barrier is caused by the organisation of buffer management based on shared buffer architecture (see Fig. 11), which takes place in majority of Ethernet switches currently available on the market. In such architecture, the packets belonging to different CoSs share common buffer space and, in addition, this space is common for all output ports. This limits the capabilities to reach a clear separation between packets belonging to different CoSs and between traffic submitted to different output ports. An special undesired situation can happen when uncontrolled best effort traffic (carried inside e2e STD CoS) fills the whole buffer space and, in this way, it limits access to the buffer for the rest of incoming packets, even packets of higher priority that are simply lost. As a consequence, the IPLR value for a given CoS traffic is out of control. Notice that thanks to the PQ scheduler, we can control the mean IPTD and IPDV values but only for the packets, which enter the buffer. Anyway, we need to control the IPLR value since it is a very important parameter for all the e2e CoSs except for STD.

Fig. 11

Considered joint system for QoS provision: Ethernet access network and Edge Router

For providing isolation between e2e CoSs and to control IPLR, we propose to explore the following additional features, which are available in the Ethernet switch:

• the ability to identify traffic flows based on the information at layer 3 and 4, namely, source and destination IP addresses, ports and transport protocol. This can be useful for EuQoS flow identification [41];

• the ability to perform data bit rate control on a per flow basis [42, 43];

• the ability to apply Weighted Random Early Detection (WRED) mechanism at the Ethernet output port.

So, from the point of view of the QoS provision, we focus on a joint system consisting of the Ethernet Switch (ES) and Edge Router (ER), as depicted in Fig. 11.

This system is composed of an ES with n full duplex ports, each of capacity 100 Mbps. One of the Ethernet links connects ES to the ER for access to the Internet (uplink of C₁ link is 100 Mbps). The rest n − 1 Ethernet ports are used for connecting local terminals. ER supports four CoSs, similarly as in the case of inter-domain links.

To support the isolation between CoSs, we explore the WRED QoS mechanism and rate-limiting mechanism to limit STD CoS traffic. The implemented WRED mechanism [44, 45] allows us to set the queue threshold and the dropping probability for each output port and each Ethernet CoS independently. In this case, the role of WRED is to limit the volume of traffic of STD CoS in the shared buffer. Notice that such approach is effective only for TCP flows, which is anyway the dominating traffic in STD CoS. For limiting UDP traffic submitted to STD CoS, we use the rate-limiting mechanism that is simply per flow policing.

5.3.1 CAC algorithm

The CAC function is performed in two elements, ES and ER. The CAC algorithm follows Eq. 6. Let us assume that CAC is performed on i-th (i = 1, 2, …, n) ES output port for the j-th CoS (j = 1, 2, 3, 4) and it works on the network resources (i.e. buffer B _{i ,j} and capacity C _{i ,j} ) which are not physically separated but only allocated in the common pool in the provisioning process (see Fig. 12).

Fig. 12

Resource Allocator (RA) module controls the volume of the traffic by performing CAC algorithm on the allocated resources: buffer size B _{i ,j} and capacity C _{i ,j}

In [46], another approach for QoS provision in the LAN/Ethernet that applies a shaper in ER to limit the TCP-controlled traffic for STD CoS is presented.

5.4 WiFi

The solution for WiFi exploits the Enhanced Distributed Coordination Access (EDCA) protocol as defined in the WiFi Multi-Media (WMM) extension [12]. The EDCA protocol allows for differentiation of traffic using four so-called Access Categories (AC). However, the EDCA by itself does not provide absolute QoS guarantees that is required for EuQoS CoSs. For this purpose, we specify CoSs in WiFi that use enhanced ACs with additional QoS mechanisms that allow (1) to provision the network resources, which are in WiFi bandwidth, buffer size and MAC protocol parameters, (2) to perform CAC, (3) to condition the traffic generated by users (policing, shaping, marking) and (4) to apply packet scheduling mechanism in Access Point (AP).

5.4.1 WiFi CoSs

Table 5 shows the mapping between e2e CoSs and WiFi CoSs. The WiFi CoSs are: RT, NRT, signalling (SIG) and Best Effort (BE). This mapping is similar to that assumed for inter-domain links (see Table 3).

Table 5 Mapping between e2e CoSs and WiFi CoSs

5.4.2 QoS mechanisms for WiFi CoSs

The solution in WiFi WMM assumes that a single Access Point handles traffic belonging to all WiFi CoSs (including best effort traffic) as it is presented on Fig. 13.

Fig. 13

WiFi WMM network (single Access Point)

The EDCA algorithm allows us to provide traffic isolation between the CoSs. More specifically, we may emulate PQ scheduling between WiFi CoSs by setting appropriate values of MAC protocol parameters (associated to each AC) as Arbitrary Inter-Frame Space (AIFS) and Contention Window (CW). The setting rule is such that the AIFS of AC with assigned lower priority should be greater than the sum of AIFS and maximum CW (CW^max) of AC with assigned higher priority:

$$\begin{array}{*{20}l} {{{\text{AIFS}}_{{{\text{NRT\& SIG}}}} = {\text{AIFS}}_{{{\text{RT}}}} + {\text{CW}}^{{\max }}_{{{\text{RT}}}} } \hfill} \\ {{{\text{AIFS}}_{{{\text{BE}}}} = {\text{AIFS}}_{{{\text{NRT\& SIG}}}} + {\text{CW}}^{{\max }}_{{{\text{NRT\& SIG}}}} } \hfill} \\ \end{array} .$$

(8)

The values of AIFS_RT, $\operatorname{CW} _{\operatorname{RT} }^{\max }$ and $\operatorname{CW} _{NRT\& SIG}^{max}$ should be fixed during the provisioning process. In addition, to appropriate setting of MAC protocol parameters, we require additional QoS mechanisms working at the IP layer of the AP (see Fig. 14).

Fig. 14

QoS mechanisms for handling WiFi CoSs

Applied traffic conditioning mechanisms differ in uplink or downlink traffic as well as in the type of WiFi CoS:

• In uplink, we police the RT traffic taking into account the possible violation of traffic profile after passing the WiFi network.

• In uplink, we shape the rate of TCP flows submitted to e2e HTD CoS as well as the aggregated SIG traffic.

• In downlink, we shape the traffic related to particular NRT flows and aggregated SIG traffic in order to protect the MAC layer from overload conditions.

• We mark WiFi frames to appropriate ACs in all terminals and AP.

• We mark the packets by appropriate DSCP codes related with e2e CoS in AP for all flows going in uplink direction.

• We can investigate the option of using the scheduling mechanism working in AP at the IP layer with capabilities of handling three priority queues (for RT, NRT and BE traffic).

5.4.3 CAC algorithms

The CAC algorithms for WiFi network should take into account the characteristics of radio transmission medium as well as the MAC protocol features corresponding to both collision avoidance algorithm and random exponential back off procedure. As a consequence, the WiFi network capacity is described by not simple function of the following elements: (1) the MAC protocol parameters such as: AIFS _i , ${\text{CW}}_i^{\max }$, i ∈ {RT, NRT&SIG, BE} and MAC protocol retransmission limit, (2) Frame Error Rate, FER _i , i ∈ {RT, NRT&SIG} that depends on the radio channel characteristics, (3) the values of QoS parameters (mean IPTD, IPDV and IPLR) for each WiFi CoSs, (4) the number of stations handling RT and NRT connections and (5) the traffic characteristics of running RT and NRT connections.

The proposed CAC algorithm assumes no a priori resource reservation for particular CoSs (as it was made, e.g. in the inter-domain case); therefore, the WiFi bandwidth is fully available for any incoming connection. For checking whether we have available resources for a new connection, we estimate the WiFi network capacity when the system is loaded by currently running and new incoming connection. For this purpose, we use the values of traffic descriptors (Peak Rate and Packet Length) associated to each considered connection. The new connection is accepted only if the following conditions are satisfied:

$$\begin{array}{*{20}l} {{P_{{{\text{RT}}}} {\left( . \right)} + P_{{{\text{NRT}}}} {\left( . \right)} \leqslant \rho } \hfill} \\ {{L_{{{\text{RT}}}} {\left( . \right)} \leqslant {\text{IPLR}}_{{{\text{RT}}}} } \hfill} \\ {{L_{{{\text{NRT}}}} {\left( . \right)} \leqslant {\text{IPLR}}_{{{\text{NRT}}}} } \hfill} \\ \end{array} $$

(9)

where P _RT(.)/P _NRT(.) denotes the traffic load from all RT/NRT connections (including traffic load of running and new connection), ρ is the target system load (ρ < 1), L _RT(.)/L _NRT(.) is the value of IPLR for RT/NRT connections and IPLR_RT/IPLR_NRT is the target IPLR value for RT/NRT connections. More details of the presented approach are provided in [47].

6 Exemplary simulation results of e2e Telephony CoS

In this section, we present exemplary simulation results showing the effectiveness of the presented approach to assure end-to-end QoS for e2e Telephony CoS. Furthermore, the simulations cover the limited multi-domain scenario in which the CAC algorithms are performed only in the inter-domain peering links as depicted on Fig. 15. In this case, we investigate in more detail the problem of providing QoS guarantees for a single flow, which crosses many domains and, as a consequence, many inter-domain links with different levels of traffic aggregation. Another analysed problem is that the traffic profile of a flow may change when the flow crosses the network whilst the CAC algorithms in each network part were performed on the basis of the unique traffic profile declared by the user during its set-up phase. In fact, the above-mentioned problems, related with crossing multi-CAC points and loosing original traffic profiles, are similar for any CoS.

Fig. 15

Traffic scenario for verifying the QoS obtained for single flows

In the simulation experiments, we centre on e2e Telephony CoS. We can do that since, in the investigated framework, we assure separation between CoSs, and this allows us to study the effectiveness of each CoS independently. The aim of simulation studies was to prove that the target QoS level for e2e CoS is kept within the specified ranges not only for the whole aggregated traffic (i.e. the whole CoS which is dimensioned) but also for each of the submitted flows separately.

In the simulation studies, we assume that the network topology consists of a cascade of IP routers (see Fig. 15). This cascade represents the three aggregation points. The first aggregation point (AGP1) models the source access network where relatively small number of flows is aggregated. The second aggregation point (AGP2) models the transit networks were usually a huge number of traffic flows is mixed together. Finally, the third aggregation point (AGP3) models the destination access network where again only few flows are aggregated. The simulations were performed in ns-2 simulation tool updated by new modules developed within the EuQoS project [48].

To verify the QoS level experienced by a single flow in each aggregation point, we calculate appropriate traffic conditions for the assumed inter-domain CAC. First, we fixed values of e2e QoS parameters and the splitting values among the three aggregation points (AGP) as shown in Table 6.

Table 6 Assumed values of target e2e QoS parameters and their splitting among three AGPs

We assumed that the flows submitted to the e2e Telephony CoS have traffic profiles as G.711 VoIP codec, i.e. they are 80 kbps CBR streams containing packets of constant length equal to 200 bytes. Furthermore, assuming that, for this e2e CoS, we allocate link capacities equal to 2, 200 and 2 Mbps in AGP1, AGP2 and AGP3, respectively, and by applying formulas 2, 3, 4 and 5, we calculate the values of buffer size and AC limit in each AGP as summarised in Table 7.

Table 7 Network resources provisioned for e2e Telephony CoS

The proposed simulation scenario depicted in Fig. 15 takes into account the following features. For the foreground VoIP flows, we select two flows, say flow 1 and flow 2. Each of these flows crosses different paths of the same aggregation level and they meet in AGP3.

The background traffic in AGP1 and AGP3 is generated by 16 and 15 independent VoIP sources, respectively, since the AC limit assumes in these cases maximum number of admissible flows equal to 17. Note that, in AGP1, we have only flow 1 or flow 2 whilst, in AGP3, we have flow 1 and flow 2. In AGP2, the background traffic is generated by a Poissonian source.

Table 8 presents the simulation results of IPLR, mean IPTD and IPDV experienced by flow 1, flow 2 in each AGP and end-to-end. The table also shows the simulation results for aggregated traffic in each AGP.

Table 8 Simulation results with 95% confidence intervals for flow 1, flow 2, aggregated traffic of Telephony CoS

Simulation tests have proved that the QoS parameters are kept within the specified ranges not only for the whole aggregated traffic (i.e. the whole class of service which is dimensioned) but also for each of the flows separately. In this way, we prove admission control algorithms for inter-domain links. The results confirm our expectations. However, more extensive simulation results are still needed for scenarios with more domains and different underlying network technologies.

7 Summary

In this paper, we described the approach to provision end-to-end QoS in heterogeneous multi-domain networks that investigates the concept of deploying e2e CoSs, which are visible to the end users (applications). The proposal bases on set of e2e CoSs, recommended by IETF for DiffServ architecture, which take into account the currently used types of applications and its QoS requirements on packet transfer characteristics at the network level, i.e. packet delays (mean and variation) and packet losses.

In order to implement the e2e CoSs over heterogeneous multi-domain networks, we dealt with two main problems, which are not solved in the current Internet: one related to the inter-domain QoS routing and the second one related to the design of the appropriate CoSs for each network technology with associated QoS mechanisms and algorithms, which allow us to meet the requirements about traffic handling as assumed for e2e CoSs.

To build end-to-end path with regard to specified e2e CoS, we propose specialised CoS-aware inter-domain QoS routing protocol, called EQ-BGP. EQ-BGP extends commonly used BGP-4 and it is relatively easy to be implemented even in domains with commercial routers with only BGP-4 functionalities. The effectiveness of EQ-BGP was validated in the Pan-European EuQoS testbed.

The different state of the QoS capabilities offered by existing network technologies makes the implementation of e2e CoSs in heterogeneous environment difficult. As a consequence, we proposed specific solutions for each network technology that are, in most cases, out of standardisation. In the paper, we described our solutions for WiFi WMM, LAN/Ethernet, xDSL and inter-domain IP links focusing on CAC functions and appropriate QoS mechanisms. We included exemplary simulation results confirming the effectiveness of the approach for e2e Telephony CoS in three-domain network and CAC performed only in inter-domain IP links.

The conclusion we took after our studies is the existing lack in common vision of desired QoS capabilities provided by particular network technologies. These results are crucial for providing end-to-end QoS in heterogeneous multi-domain networks. However, as it was shown in this paper, we may reach such a complete vision by investigating the concept of e2e CoSs.