Skip to main content

Assessment of smartphones using multimethodology


Recent releases of mobile phones, announced by the media as “smartphones” have ever more high technology embedded and they are true handheld computers, where the characteristics of information processing, physical and virtual memory capabilities have incremented with each new release. It is also remarkable that other objective and subjective critical aspects to the total satisfaction of the user are still present when making the decision to buy or not one of these devices. Therefore, we present, using the approach of data envelopment analysis (DEA), an exploratory study considering the main requirements to examine different Smartphone alternatives with respect to characteristics related to the user and the product.


From its appearance in the mid-70s to its spread in the 1990s (Ferguson 2007), cell phone has become an integral part in everyday life of a common person, to the point that its absence seems almost unimaginable today (Plant 2000). These devices, which served exclusively as a way of communication, in the course of time have acquired new features, such as text messaging, access to media content, colorful touch screen, camera, etc.

Recent releases of mobile phones, announced by the manufacturers and the media as “smartphones”, have ever more high technology embedded, and are true portable computers, where capabilities such as data processing and virtual memory increment with every new release. It is notorious also that other objective and subjective aspects critical to the full user satisfaction, studied in detail by Han et al. (2004), as weight, shape, finish, elegance, simplicity, among others, are still present in making the decision to purchase or not one of these devices.

As a thermometer of smartphones representativeness in the global mobile phone market, smartphone world sales would achieve 918.6 million units, exceeding the conventional mobile phones in 2013 (Mlot 2013). The Brazilian market, fourth largest in number of smartphones in the world, would reach the milestone of 75 million units sold in 2013, and may even surpass Japan, which currently ranks third, behind China and the United States (Savitz 2012).

As far as we try to understand decision making regarding smartphone acquisition, we need tools that help us to provide measures, which can consider factors and variables that allow differentiating devices. The importance of each of these variables can change significantly according to the profile of the consumer. It is also necessary that the information is both readily available and reliable.

Performance assessment through best practice frontier is an attractive methodology, since it minimizes the subjective assumptions regarding weights assigned either to devices or to factors/variables. One linear programming based method that allows comparison across different units is Data Envelopment Analysis (DEA). However, technological homogeneity is one important assumption in DEA, since it allows direct comparison between operational units. By complying with this concept, we propose a multimethodology approach (Pessoa et al. 2015) that first applies cluster analysis to search for homogeneous clusters of smartphones, which still stand for consumer preference profiles. After that, we use data envelopment analysis (DEA) to assess the performance of units inside each cluster, thus providing smartphones assessment model that analyzes different alternatives of smartphones, appropriate to specific profiles to support smartphone consumers in decision-making. All the necessary information to feed the model is available through web recommendations systems.

In late 1990s and early 2000s, the available devices on the market were traditional mobile phones, the first and second generation (1G and 2G). Preliminary studies aimed at determining critical aspects regarding user satisfaction, which is defined according to Han et al. (2004), as subjective feelings that users conceive in their mind about the image or impression on the product. They analyze the expressions of users in the form of adjectives and verbal phrases, representing subjective feelings, and from them, extracted 10 dimensions applied to evaluate user satisfaction with corresponding aspects in mobile phones.

On the other hand, drawing a parallel between the use of smartphones and computers, some theories originated from Fishbein and Ajzen (1975) and Davis et al. (1989), namely the Theory of Reasoned Action (TRA) and the Technology Acceptance Model (TAM) respectively. They focus on the use of subjective measures to predict, explain and increase user acceptance when using computers, which afforded several studies in the area of software selection and development of new information technologies.

Işıklar and Büyüközkan (2007) propose product-related and user-related variables corresponding to aspects that influence the experience gained by the user in devices of the generation 2G, which already had color display, WAP access and camera, besides objective aspects quantified (specified) by the manufacturers. Two recent papers by Chen et al. (2010) and Akyene (2012) present assessments based solely on product-related variables. The latter included 10 different models of mobile phones that were assessed using 12 objective variables.

In this study, the variables were grouped into categories, which are related to both the product and the user.

Table 1 summarizes the variables used in each aforementioned study.

Table 1 Summary table of the variables in the literature review

The present study considers objective aspects previously quantified (specified) by the manufacturers through variables obtained from (Plant 2000) for the devices of the 2G generation, which had already a colorful screen, WAP access and camera. Besides literature review, the analysis of available data basis will support variable selection for the present assessment, however taking into account that we dropped the primary focus on the problem structuring in favor of data availability. The study also comprises the experience obtained by the user This section displays the information available in specialized mobile phone recommendation systems, preferentially existing in the web, in order to select the variables that we want to investigate.

We will consider the following assumptions to distinguish between smartphones and traditional mobile phones. The devices included in the study are those that have touch screen with size between 2.5 and 6.9 inches, with a CPU and RAM memory capacities of at least 400 MHz and 140 MB, respectively, and must have advanced operating system, similar iOS, Android, Windows Phone, Symbian, or equivalent functional variety.

These characteristics distinguish the smartphone from the conventional devices, regarding the considerable amount of existing mobile phones. We consider as a premise that the complete information for a particular device should be available in data basis. The following data basis were used:

Buscapé Presents website prices for a number of products and online stores, which ranges from small traders to popular retail chains. This portal is present in over 20 countries in Latin America, receives30-million page views/month and has 11 million products (Naspers 1999).

TestFreaks Recommendation system based on web, which aims to recommend products supported by subjective character assessment provided by consumers and experts, collected from websites or specialized magazines. Currently this system has 15 million reviews to 450,000 products (TestFreaks AB 2007).

PassMark Hardware and software web-based benchmark system for performance of computers, and more recently, cell phones (PassMark Software 2013). It presents performance tests for smartphones comprising17 different tests grouped in five sets (CPU, Storage, RAM, 2D and 3D graphics).

GSMArena Web-based system, which among other features, stores a fairly complete database of mobile phones, containing detailed specifications of handsets launched by most diverse manufacturers ( 2000).

The operational task of organizing the dataset accordingly would require a great effort, since there are many manufacturers, each of them produce a large number of devices, and each device contains a web page with a lot of specifications. Therefore, we overcome this restriction by using software resources that performs this task automatically.

We started the extraction of the specifications based on information for 982 devices from 17 manufacturers. After several crossings of information from different queried recommendation systems, we obtained complete information for122 smartphones from ten manufacturers: Acer (2) Apple (5) ASUS (1) HTC (23), Huawei (7), LG (19) Motorola (8) Pantech (1) Samsung (37) and Sony (19).

Table 2 summarizes the available technical product-oriented variables and those that are easily perceived by the consumer. It should be noted that this study did not address product quality, technical support, respect to their consumers and socio-environmental variables, which can be used, as new data become available.

Table 2 Resume data of the variables


The model for assessing smartphones will connect the dual approach: device benchmarking and weighting characteristics, provided by the Data Envelopment Analysis (DEA) methodology. Thus, the mathematical model will evaluate the efficiency of a given device compared to the others, highlighting the strengths (variables) of the object under evaluation, that cause it to be efficient, in face of competitors. In particular, we will start from classic DEA models: the CCR (Charnes et al. 1978) and BCC (Banker et al. 1984), and then proceed to cross-evaluation models (Lins and Meza 2000), in order to overcome some drawbacks.

However, due to inherent characteristics to the application in question, it is necessary to observe some basic issues. The first one concerns the different projects of smartphones (corresponding to user profiles), which are envisioned by the manufacturers, so that a particular smartphone can achieve a specific niche and get a market share. Actually, when searching for a smartphone, a consumer finds different configurations and varying price ranges and sizes, many of them are discarded a priori since they are far from reaching the consumer’ needs. Therefore, a preliminary clustering of similar devices is useful both for describing users profile and for enhancing the homogeneity required by DEA models in order to ensure useful comparisons and benchmarking.

The concept (profile) behind the operating strategy adopted by manufacturers is characterized by different regions in the multivariate space, which is captured through statistical methods that ensure similarity between the components of the sample. To this end, we adopt a multimethodological approach that consists of the combination of two methodologies: the first is the Principal Component Analysis (PCA), a multivariate method (Rencher and Christensen 2012), which enables dimensionality reduction and ensures the explanatory power of the chosen original variables. The second is the k-means algorithm, a cluster analysis technique that is based on centroids to calculate the pertinence of points to groups. The formal quantitative modeling was preceded by a problem structuring qualitative method that used cognitive mapping, integrated in the multimethodology (Pessoa et al. 2015), however outside the scope of the present paper.

After clustering smartphones and determining smartphone profiles, we perform smartphone assessment through DEA modeling in each cluster, thus ensuring to evaluate similar profiles, avoiding, e.g. to compare a smartphone which has been designed to match large scales of price and portability to another which was designed to match the computational performance and design.

Furthermore there is no need to include arbitrary assumptions regarding weights assigned to each of the variables (characteristics), since DEA will be in charge of this task, revealing the optimal multipliers set for each device and all variables. However, classic DEA presents some drawbacks, as non-Pareto efficiency, which implies in poor differentiation between units. To overcome this hindrance we perform cross-evaluation assessments, thus evaluating each smartphones in a particular profile using the optimal weighting scheme of the remaining devices in that cluster. For this task we use DEA Cross-Efficiency (Doyle and Green 1994), which improves differentiation and robustness of results, since it considers the “point of view” of the other smartphones (Lins and Meza 2000). Thus, we avoid biased efficiencies resulting from inappropriate weighting scheme determined for each handset, with no need to incorporate information obtained a priori (Angulo-Meza and Lins 2002).

Since we are interested in the central structure of a sample with 122 observations (smartphones), we adopt a principal component analysis to reduce the number of dimensions and cluster analysis to group objects. After using these techniques, we were able to segregate the smartphones by similarities in six following groups, whose similar values determine these categories:

Low cost smartphones

This category highlights in the following variables: cost (lower value with an average cost of 95 euros); weight (lower average compared to the other groups), computational performance, RAM memory, camera resolution, screen size, battery capacity, friendly usability.

Basic smartphones

They have low cost, outperforms “Smartphones Low Cost” regarding computational performance, capacity of RAM, screen size, battery capacity, design and ease of use. Other interesting features are the low weight and reduced volume (minimum amongst all categories).

Intermediate smartphones

They have marked improvement in computing performance, RAM, screen density and camera resolution compared to low cost and basic smartphones. We should note, however, that these devices are poorly evaluated by users in design and autonomy (worst rating among groups), and are bulkier, excluding the phablets.

High-end smartphones

It excels (except for phablets) in computational performance, RAM, camera resolution, screen size and density and battery. In contrast, the difference between the average prices of high-end smartphones and intermediaries is more than double (342–156 euros).

Intermediate phablets

This group has as strength the screen size, whose average size is 5.60″. The battery capacity, is also the largest amongst groups. RAM memory is the second largest and computational performance is close to high-end smartphones. With respect to price, it equates to the high-end smartphones.

High-end phablets

They are far superior with respect to any other when evaluating the computational performance, RAM, screen density and resolution camera. They also excel in subjective aspects, such as the assessment of experts and ease of use. They are the second best group regarding battery capacity, screen size, design and autonomy variables. On the other hand, they present the highest prices amongst all the devices.

Cross-evaluation DEA

We now advance to the assessment of smartphones within their respective profiles, and for such a task we use cross-evaluation DEA models, for both discriminating between the various efficient DMUs, and to avoid problems due to disregarding some relevant inputs/outputs. From this point we considered the efficiencies obtained by CCR and BCC models as simple efficiencies because they are obtained from a self-assessment, which is made by considering the weights that are favorable to their own performance.

In short, after obtaining the weights that maximize the simple efficiency for each DMU, cross-efficiency consists of assessing a DMU through the perspectives of (weights chosen by) other DMUs. For a given assessed DMU, the averaged cross-efficiency is the average of cross-efficiencies that result from the perspectives (weighting schemes) of all assessing DMUs. Average cross-efficiencies can be extracted from cross-efficiency matrix (Table 3), when we average the columns of the array. The average is performed normally without the elements of the main diagonal according to Doyle and Green (1994).

Table 3 Cross-efficiency matrix for DEA analysis with n DMUs

One important drawback of cross-efficiency analysis is that extreme-efficient DMUs allow for multiple optimal weighting schemes. A method to overcome this problem is the use of the aggressive formulation Bk (Lins and Meza 2000), which gets a unique weighting scheme, taking as criteria the minimization of the cross-efficiencies for all the other DMUs.

The next step is to use the value of the maverick index calculated as the relative difference between the simple efficiency and the average cross-efficiency. The highest maverick indices characterize the Maverick DMUs. Usually this corresponds to many zeroed weights assigned to outputs/inputs to achieve efficiency (Doyle and Green 1994).

Results and discussion

Table 4 shows the low-cost smartphones group efficiencies and cross-efficiencies sorted by maverick indices. We observe that the LG Opt-Hub and LG-OpNet smartphones are more efficient when considering both simple and cross-efficiencies, that is, its efficiency is not greatly affected, when being evaluated according to the weighting schemes, or “viewpoints” of other smartphones that belong to their group. Actually, one smartphone that is best evaluated using cross-efficiency stands out even when assigning higher weights to the variables that are strengths of its competitors.

Table 4 Performance asessment of low-costsmartphones

In Table 4, The DMU HTC-Wilds, presents a maverick index in the third position, indicating that its simple efficiency has a difference of only 0.318, or 31.8 %, compared with the average cross-efficiency, however it stands in the fifth position regarding both simple and cross efficiencies.

Table 5 shows the basic smartphones group efficiencies and cross-efficiencies sorted by maverick indices. We observe that the first six devices present the higher simple and cross efficiencies, and lower mavericks. One interesting aspect is that five, amongst the six best evaluated smartphones were produced by five different manufacturers, which suggests a remarkable competitiveness in its field of electronic devices.

Table 5 Performance asessment of basic smartphones

The analysis for intermediate smartphones, contained in Table 6, reveals that the first smartphone from the list, HTC-Evo4G presents a very low maverick. This indicates that it is robust regarding different weighting schemes chosen by its competitors.

Table 6 Results list of intermediate smartphones

In Table 7, we observe that the first three smartphones LG-OptG, HTC-SensXE and HTC-Firs are both efficient with its chosen weights and robust to the weighting schemes adopted by other manufacturers. However, this group displays interesting “false positives” i.e. devices that obtains the simple efficiency by placing weights in just a few variables. This results, as in the case for Son-XperV, in vulnerability to alternative weighting schemes, despite 100 % efficient. It also allows for certain “intrusion” within the top five group apparatus, which is the Ace-LiqE2, which has less simple and by pairs efficiency than the next device of list.

Table 7 Results list of high-end smartphones

This aspect can also be seen in the Table 4, as to the third device in the list, whose discontinuity in relation to Maverick ratio is even greater.

Table 8 presents the results for the intermediate phablets, where the highlight is for the phablet LG-OptVu, which features a big difference in average cross-efficiency (above 10 %) compared with the second place, which is the unit Sam-GalNot2. The lower Maverick indices in the list correspond to the higher efficiencies, with no “intruders” between them.

Table 8 Results list of intermediate phablets

This group presents another interesting result, since the simple and cross efficiencies rankings do not correspond in the case of DMUs Sam-GalNot and Sam-GalMeg5. The former presents higher average cross-efficiency, while the latter performs better in simple efficiency assessment.

The sixth and final list of high-end smartphone/phablets efficiencies is shown in Table 9, where all the phablet perform 100 % efficient regarding simple efficiencies. The tie is broken by cross-eficiency assessments, revealing that the better rated is Son-XperZ1, followed by LG-G2 and Sam-GalNot3. We should note, however, that high Maverick indices arise, due to big difference between simple and cross efficiencies, ranging from 37 to 56 %. Similar results occur for a subgroup of intermediate phablets in Table 8, suggesting that phablets manufacturers aim at specific niches of market, where their performance excels. Outside this niche they perform badly, revealing vulnerability to alternative preferences.

Table 9 Results list of high-end phablets


Most papers on Data Envelopment Analysis devote to performance assessment of service sectors, where management issues are complex and results are difficult to materialize. The current paper applied DEA modeling as one methodology to explore a usual decision making where the DMU is a common personal appliance. Data base is large enough to allow for clustering as a tool for improving homogeneity, thus constituting a multimethodological approach.

The model assessed the performance of 122 smartphones using 18 variables—1 input and 17 outputs—disclosing ranks for the 6 clusters of profiles according to simple and cross efficiencies. Results can support the consumer in the choice of a device that best meets their needs according to a particular profile.

The objective was reached, since the characteristics of each device, strengths and weaknesses have been identified and exploited. It was not necessary to define weights for the calculation of efficiency, on the contrary, we used one of the most important features of DEA models is the model’s ability to assign weight the DMUs according to their favorable characteristics when compared to the other DMUs.


  • Akyene T (2012) Cell phone evaluation base on entropy and TOPSIS. Interdiscip J Res Bus 1:09–15

    Google Scholar 

  • Angulo-Meza L, Lins MP (2002) Review of methods for increasing discrimination in data envelopment analysis. Ann Oper Res 116(1–4):225–242

    Article  Google Scholar 

  • Banker RD, Charnes A, Cooper WW (1984) Some models for estimating technical and scale inefficiencies in data envelopment analysis. Manage Sci 30:1078–1092

    Article  Google Scholar 

  • Charnes A, Cooper W, Rhodes E (1978) Measuring the efficiency of decision-making units. Eur J Oper Res 2:429–444

    Article  Google Scholar 

  • Chen D-N, Hu PJ-H, Kuo Y-R, Liang T-P (2010) A Web-based personalized recommendation system for mobile phone selection: design, implementation, and evaluation. Expert Syst Appl 37:8201–8210

    Article  Google Scholar 

  • Davis FD, Bagozzi RP, Warshaw PR (1989) User acceptance of computer technology: a comparison of two theoretical models. Manage Sci 35:982–1003

    Article  Google Scholar 

  • Doyle J, Green R (1994) Efficiency and cross-efficiency in DEA: derivations, meanings and uses. The Journal of the Operational Research Society 45(5):567–578

    Article  Google Scholar 

  • Ferguson S (2007) Can you hear me now? It may be hard to remember, but before everyone and their dog owned a mobile (the PetsCell hit shelves late last year), phones had to be transported in cars, bags and bricks. Here’s a look back. This Magazine, 40, 44(1)

  • Fishbein M, Ajzen I (1975) Belief, attitudes, intention, and behavior. Addison-Wesley, Reading

    Google Scholar 

  • (2000) GSMArena. Acesso em 4 de abr de 2013, disponível em GSMArena:

  • Han S, Kim K, Yun M, Hong S, Kim J (2004) Identifying mobile phone design features critical to user satisfaction. Human Factors and Ergonomics in Manufacturing 14:15–29

    Article  Google Scholar 

  • Işıklar G, Büyüközkan G (2007) Using a multi-criteria decision making approach to evaluate mobile phone alternatives. Computer Standards & Interfaces 29:265–274

    Article  Google Scholar 

  • Lins MP, Meza LA (2000) Análise Envoltória de Dados e Perpectivas de Integração no Ambiente de Apoio à Decisão. COPPE/UFRJ, Rio de Janeiro

    Google Scholar 

  • Mlot S (2013) Smartphone shipments to top features phones this year. Acesso em 2 de Abr de 2013, disponível em Academic OneFile:

  • Naspers (1999) Buscapé Brasil. Acesso em 12 de Abr de 2013, disponível em Buscapé:

  • PassMark Software. (2013). Android Benchmarks. Acesso em 4 de abr de 2013, disponível em PassMark Software:

  • Pessoa LM, Estellita Lins MP, Moreira da Siulva AC, Fiszman R (2015) Integrating soft and hard operational research to improve surgical centre management at a University Hospital. Eur J Oper Res 245:851–861

    Article  Google Scholar 

  • Plant S (2000) On the mobile—the effects of mobile telephones on social and individual life. (M. report, Ed.) Acesso em 15 de 12 de 2012, disponível em

  • Rencher AC, Christensen WF (2012) Methods of multivariate analysis (3ª, Edição edn. Wiley, Utah

    Book  Google Scholar 

  • Savitz E (2012) 4 Predictions for the brazilian mobile phone market for 2013. Acesso em 2 de Abr de 2013, disponível em Forbes:

  • TestFreaks AB (2007) Questions & answers. Acesso em 4 de Abr de 2013, disponível em TestFreaks:

Download references

Authors’ contributions

MK performed model specification, data collection and computational work. Also wrote the original draft. MPEL provided the first conception of the research combining the different methodologies and overall guidance as a DSc supervisor. Also performed analysis of results and the final writing. Both authors read and approved the final manuscript.


The second author acknowledges support from the federal agency CNPq for supporting the present research and also from the state foundation FAPERJ in the Rio de Janeiro for a research grant.

Competing interests

The authors declare that they have no competing interests.

Author information

Authors and Affiliations


Corresponding authors

Correspondence to Musbah Koleilat or Marcos Pereira Estellita Lins.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Koleilat, M., Estellita Lins, M.P. Assessment of smartphones using multimethodology. Braz J Sci Technol 3, 22 (2016).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: