Today, Heartfelt Community published an article "The Past of Huawei’s Mobile Phone Operating System" yesterday, in which Wang Chenglu, President of CBG Software Department, described the mental journey of making software a Huawei mobile phone, especially the "Huawei Ark Compiler" they just released.
Simply put, a compiler is software that converts a high-level language used for program development into machine instructions, which can be understood as a bridge between software and hardware. The Ark Compiler will solve the problem that the existing bridge of Android is not smooth, and it is a real deep optimization and innovation of Android, which can be said to inject new vitality into the Android industry.
In Wang Chenglu’s view, from the feedback of the market and users, more and more consumers recognize that Huawei mobile phones are still smooth after long-term use, and there are fewer and fewer complaints about the poor experience of playing games with Huawei mobile phones. This is not only due to the rapid improvement of Huawei’s mobile phone chip and hardware capabilities, but also inseparable from the deep optimization and technological innovation of the software stack.
Wang Chenglu believes that consumer-oriented software is very different. The popular smartphone operating system finally left only iOS and Android. Unlike the closed iOS, which frames everything to ensure the experience, Android is an open source in the AOSP community led by Google, but the native Android is actually a Design Reference. There is no detailed and unified standard to follow in terms of how to choose and expand each function and how to interact with a large number of third-party APPs. All mobile phone manufacturers make a lot of adaptations and optimization modifications according to their own hardware platforms, product positioning and target user groups, and the ecological fragmentation is serious. Everything can only be based on the consumer experience.
After a lot of investigations and interviews, the team quickly found that the most criticized by Android users is the card after long-term use, which is also the first obstacle affecting the consumer experience.
In 2019, with the extension of the replacement cycle and the slowdown of the growth rate, the mobile phone industry has entered a state of saturated competition. On the basis that the hardware has been extremely competitive and gradually opened up its competitors, Huawei software has begun to continue to exert its strength. Wang Chenglu emphasized that only by always focusing on the user experience, consolidating the underlying core capabilities, constantly improving the software engineering capabilities, and building an open ecology can we achieve sustained business leadership.
On April 11th, Yu Chengdong, president of CBG, released P30 series mobile phones in Shanghai. After the press conference, a new term that sounds a bit obscure — — "Ark Compiler" quickly boarded the hot search list of the community, which triggered great concern and extensive discussion among IT practitioners, digital bloggers, senior code farmers and even ordinary consumers. "What is a compiler?" "What is the difference between Huawei Ark Compiler and Android native AOT/JIT(Ahead of Time/Just in Time)?" "What does the Ark compiler mean to the Android ecosystem?" And so on, netizens are very happy to discuss various issues of different shades, and many comments on the posts are brilliant. It must be said that the master is in the folk.
One of the netizens’ comments is very representative: "Yesterday’s Huawei conference surprised me that in this era, a domestic consumer electronics company will introduce the compiler as a highlight. Huawei’s continuous high investment in basic research and development, coupled with the fact that the market share of mobile terminals is the first in the world, makes ‘ Self-built ecology ’ The dream is possible for a China company for the first time. "
Compiler is the software that converts the high-level language used for program development into machine instructions, which can be understood as a bridge between software and hardware. What value will the Ark compiler bring? To put it simply, it is to solve the problem that the existing bridge of Android is not smooth, and it is a real deep optimization and innovation of Android, which can be said to inject new vitality into the Android industry.
This is just the latest achievement of a series of software innovations of Huawei. Today, from the feedback of the market and users, more and more consumers recognize that Huawei mobile phones are still smooth after long-term use, and there are fewer and fewer complaints about the poor experience of playing games with Huawei mobile phones. This is not only due to the rapid improvement of Huawei’s mobile phone chip and hardware capabilities, but also inseparable from the deep optimization and technological innovation of the software stack.
We are well aware that any technical optimization and innovation exploration of software and hardware must aim at the user experience. How does Huawei’s mobile phone operating system EMUI, which serves nearly 500 million online users, continuously innovate and introduce hard-core technology around the user experience?
The story has to start from a few years ago.
Three years ago, I was in charge of the central software institute of the laboratory in 2012. In July, a phone call from General Manager Yu gave me the opportunity to come to CBG to be responsible for the research and development of terminal software. The mood at that time was both excited and uneasy. The excitement is that I have been engaged in work closely related to software since I joined the company. In my mind, to complete large-scale industrialized software research and development is tantamount to designing and building a magnificent palace with 0 and 1 in the virtual world as building materials. How to make our construction achievements and processes reach world-class and bring value to our users and partners has always been my pursuit and dream.
The uneasiness also followed. Although they are all software research and development, the smartphone operating system is a brand-new field for me. First of all, the degree of standardization is very different. Telecom equipment software has very complete industry standards, such as ITU, 3GPP, IETF, etc., which actually standardize the network architecture and network element architecture. What R&D needs to do is to comply with certain development process specifications, such as IPD (Integrated Product Development), and reasonable project management to achieve the corresponding delivery goals. At the same time, the demand for telecom equipment and software is relatively clear and convergent, and the demand of hundreds of operators around the world is almost the same. Moreover, the main body of daily maintenance of the equipment is the operation and maintenance personnel with professional training and considerable technical level, who can feed back various problems in the use of the equipment relatively clearly and accurately.
Consumer-oriented software is quite different. The popular smartphone operating system finally left only iOS and Android. Unlike the closed iOS, which frames everything to ensure the experience, Android is an open source in the AOSP community led by Google, but the native Android is actually a Design Reference. There is no detailed and unified standard to follow in terms of how to choose and expand each function and how to interact with a large number of third-party APPs. All mobile phone manufacturers make a lot of adaptations and optimization modifications according to their own hardware platforms, product positioning and target user groups, and the ecological fragmentation is serious. Everything can only be based on the consumer experience.
What can we do to achieve the best user experience in the open environment of Android? When iOS has established a solid leading image through years of accumulation, how to make EMUI recognized by more and more consumers is the biggest challenge for me and the whole software team.
We decided to take "the best user experience" as the goal pursued by the terminal software. The first problem to be solved is, what affects the user experience the most? After a large number of investigations and interviews, the team quickly found that the most criticized by Android users is Caton after long-term use, which is also the first obstacle affecting the consumer experience.
A colleague who loves electronic products told me about his experience in using smart phones. After Android phones entered the era of big screen, in 2013, he bought the best Android phones S and H at that time, using top-level configuration, but after using them for more than a year, even if he didn’t do anything every night, he would consume more than ten percent of electricity; What he can’t stand even more is that it is almost commonplace to get stuck and crash, and it is necessary to constantly restart and clean up the memory before he can barely use it. In his words, it’s not that I’m using my mobile phone, but that it’s using me.
Why even with high hardware configuration, even the best Android phone at that time will face more stability and fluency problems? What factors caused the jam and crash after long-term use? We decided to "chew" Caton, the biggest hard bone. The EMUI team set up a special "special warfare team" to focus on tackling key problems. The special warfare team analyzed more than 400 mobile phones with dozens of models under Android system in detail. After repeated problem reappearance and full-stack analysis of software and hardware, the following four main reasons were found:
The loose management of Android system, the lack of rules in the use of resources by applications, is like building many roads without traffic rules, which eventually leads to congestion and chaos;
The hardware resources of the system are lack of coordination, so it is not possible to solve the problem only by increasing RAM resources, but to schedule and use resources more effectively.
The cumulative effect of storage fragmentation, with the increase of use time, storage fragmentation is becoming more and more serious, the speed of data reading and writing is obviously reduced, and the system jam is frequent and aggravated;
APP is expanding seriously, especially the function of head application keeps accumulating, which leads to the sharp increase of data generated, which seriously grabs system resources and drags down the running speed of the system.
These problems are almost related to the whole stack from chips to software. It is impossible to really solve the problem if we just respond passively. Only by comprehensive and systematic analysis can we come up with a comprehensive and thorough solution. However, the Android system itself is extremely huge, with hundreds of millions of lines of code. The challenge of "operating" on such a huge system can be imagined. Will it lead to other compatibility problems while optimizing? Is there enough ability to achieve the predetermined goal?
However, barely flying with a faulty engine will only lead to a bigger accident. It’s better to take a chance than to flinch. Through a lot of careful analysis and judgment, we are determined to burn our bridges — — Fuck! Finally, after repeated diagnosis and discussion by experts, five "prescriptions" were put forward to solve the above problems one by one.
To put it simply, artificial intelligence is the core, which analyzes the intelligence of the application side and judges the resource demand characteristics of the application (calculation-intensive, RAM-intensive, IO-intensive, etc.); On the hardware resource side, we can perceive the state of CPU, memory, IO(Input/Output), storage and other resources, realize the best matching between application requirements and hardware resources through AI algorithm, and timely and reasonably sort out and recycle idle resources, so as to solve the above-mentioned problems of hardware resource coordination and excessive expansion of APP occupied space.
For the problem of storage fragmentation, we have adopted a more radical scheme — — Replaced the Android native file system.
Android has been directly built on Linux since its birth, and its file system did not consider the rapid arrangement of fragmented files at first. For a long time, Android phones have inherited the file system from Linux, which is not friendly to Flash storage on smart phones and is also the main reason for fragmentation. We decided to replace the native file system with the F2FS(Flash Friendly File System) file system, not only because F2FS is tailor-made for Flash, but also because it can greatly avoid file fragmentation in mobile phones.
F2FS is original and open source by Youshang S Company. However, S company itself did not take the lead in using it. Because the file system determines the storage and access methods of all files on the storage, such as organ transplantation, the complexity and risk of replacing the file system are enormous.
Although we have carefully verified it in the laboratory for more than half a year, the high complexity and high risk of replacing the file system are still the biggest concerns in decision-making. After repeated analysis, the software management team and the technical team finally made a decisive decision and signed a military order with CBG, promising to "deal with it by military law" if it failed or went wrong.
Thanks to the team’s courageous and cautious execution, the F2FS file system was finally replaced successfully, which became the most important part of the whole scheme to solve the stubborn disease of Caton.
In 2016, along with the EMUI5.0 released by Mate 9 series, we took an easy-to-understand name for this solution to Android jams, which was called "born fast and life fast", and promised consumers that they would not be stuck for 18 months. This seemingly "bragging" promise at that time has become a reality today. Judging from the market reaction and consumer feedback, in the Android camp, Huawei mobile phones took the lead in getting rid of the "Caton" hat.
In 2018, Google absorbed F2FS into the native version of Android, which benefited all Android manufacturers and made a valuable contribution to the Android ecosystem. So far, Google’s self-developed mobile phone Pixel 3 also uses the F2FS file system.
Based on rational technical judgment and decisive decision, we are one step ahead of all Android manufacturers. What’s more commendable is that in order to achieve safe replacement, our team was forced to carefully analyze the bottom of the native system without retreat, and practiced while fighting, which allowed our team to accumulate very valuable experience and taste the sweetness of deeply optimizing Android. In the first exam, the software team and I initially gained the trust of the CBG management team and had a good starting point.
In the process of solving the stubborn disease of Caton, we found that the looseness of Android ecosystem is a challenge, which cannot be solved by technology, but it is closely related to the user experience.
Compared with iOS’s strict control over ecology, Android ecology mainly relies on players’ self-discipline. For an inappropriate analogy, the Android native system is like a highway repaired by Google. Google imagines that the road is naturally a motor vehicle that abides by traffic rules, but the actual situation is that not only motor vehicles are on the road, but all kinds of ox carts, carriages, tricycles and pedestrians are coming, and there is no traffic police (priority dispatching system) to manage it, so congestion becomes the norm.
This situation is particularly serious in China. The technology continuously invested by mobile phone manufacturers only makes the end-side operating environment more and more efficient and tidy. However, the mixed application market has a great impact on the user experience. Android phone manufacturers not only have their own application market, but also have a wide variety of third-party application markets. In addition, they can download and install Apps through browser links, as well as various Trojan apps and malicious software, which guide each other to install, disguise the background, and apply for mobile phone system resources without restrictions. The problem brought about is that the use of Android system resources by apps is extremely irregular and even malicious.
As a result, the user experience is constantly damaged. Huawei is faced with a choice: follow the crowd, or take the lead in raising the banner, unite with heavyweight eco-players, gradually standardize and purify the Android ecosystem, and return the quality experience to consumers? It is not enough to rely on the continuous technical investment and optimization of mobile phone manufacturers themselves. It is a necessary step for Huawei’s terminal software to move towards the "deep water area" of ecological coordination.
After a lot of preliminary communication and discussion, on November 14th, 2016, at the Mate 9 series conference in Shanghai, Huawei jointly launched the Android Green Alliance with Alibaba, Baidu, Tencent and Netease. As an open non-profit organization, we hope to jointly build a green application environment for Android in China, create a safe, reliable, credible and healthy application ecosystem, and bring the ultimate application experience to consumers.
After the establishment of the Android Green Alliance, the experience and wisdom of the sponsors were concentrated, and the 1.0 and 2.0 versions of the Android Green Alliance Application Experience Standard were successively launched, covering the compatibility, stability, security, power consumption and performance standards of application installation and operation. At the same time, it has developed more than 1,000 member companies, covering 80% of mainstream applications.
On the Android Green Alliance platform, Huawei has provided more than 3,000 test machines, which are open to all Android application developers free of charge. As long as you log in remotely, you can apply for the corresponding test resources and perform tests. At the same time, Huawei shared the changes of the latest version of Android to domestic developers in time, which greatly improved the adaptation efficiency of Android applications in the new version.
Mark the applications that have passed the "Android Green Application Experience Standard" as "Green Application", so that consumers can understand the application quality at a glance when downloading the application and can use it with peace of mind. Through more than two years of cooperation, Lvmeng has helped more than 200,000 developers to conduct free testing and traction of Top 3000 Android applications, reducing high power consumption and abuse of rights. Consumers, application developers, Internet vendors and mobile phone manufacturers all benefit from this. Participants in the entire Android ecosystem are willing to share experiences and innovate together on this platform. In 2018, four innovation working groups were established, namely, AI open ecological cooperation, AI security design scheme, AR and game acceleration.
We will continue to develop the Android Green Alliance with our partners, hoping that it can become a bright light for the whole industry, survive the fittest and bring the ultimate application experience to all Android users.
In June 2018, Mr. Yu released the "scary technology" GPU Turbo in Beijing, which not only caused other manufacturers to follow the trend of "scary" and "Turbo", but also caused a big discussion about the technology behind it in some major science and technology forums.
The rapid development of the game industry on mobile phones is beyond many people’s expectations, and almost half of the users are mobile game users. As early as 2016, domestic mobile games have occupied half of the game market, and the popularity of "hardcore" games, such as the glory of the king and Eat Chicken, has pushed the requirements of games for the graphics processing ability of mobile phones to an unprecedented height. However, due to the limitation of ARM Mali GPU, Huawei mobile phone users’ complaints about the game experience have increased sharply in the PK of GPUs with other top friends.
Can the experience of mobile games only rely on the processing power of GPU, or be similar to some so-called game phone stacking? Can it be optimized by soft and hard cooperation like the performance of the whole machine? These problems have entered our field of vision. Just do it, and the related work will start immediately.
In February 2017, we launched a research on the graphics processing performance of mobile phones. Although the newly joined overseas graphics processing experts have given valuable ideas, the whole process of tackling the problem is not a Ma Pingchuan. It has gone through countless attempts, approaches and reinventions. "If you win, you will raise a glass to celebrate, and if you lose, you will be desperate to save each other." Experts from the United States, Finland, Russia, Shenzhen, Hangzhou, Nanjing and Shanghai, regardless of their departments, worked day and night, and finally achieved the expected goal.
When we finally said that GPU Turbo can improve game performance and reduce power consumption at the same time, the industry began to question it, and everyone felt that this logic was contrary to common sense. However, we have sufficient confidence, from GPU Pipeline (processing queue) and CPU/GPU joint scheduling, to the detailed interpretation and careful adjustment of the instructions of OPEN GL (open graphics library), to the reconstruction of graphics processing engine and the cleaning of application load, and the complete full-stack analysis and solutions, so that we have enough technical confidence and dare to disclose our goals and commitments.
On June 6 th, at the press conference, the famous technology platform broadcasted the same game match between Glory Play equipped with GPU Turbo and other friendly mobile phones. In terms of hard indicators such as frame rate, jitter rate, frame drop and power consumption, glory Play wins! More than an hour of real-time battle, let the industry see the strength of GPU Turbo. After the launch of GPU Turbo, the response exceeded expectations, and various spontaneous evaluations by third parties also confirmed that Huawei mobile phones have stabilized the first camp of game performance.
Behind any success is hard work and sweat. All the R&D brothers who participated in the GPU Turbo project can’t remember how many sleepless nights they have experienced. And we always remember that devoting ourselves to technology and focusing on improving the user experience is the king of continuously changing the user’s concept and enhancing the brand and reputation.
At the beginning of July 2018, the afterheat of the listing of GPU Turbo has not dissipated, and suddenly a user complaint caught the attention of the maintenance and marketing team. A considerable number of users have upgraded GPU Turbo, and with the update of a popular game, many scenes have appeared. Because it involves Huawei and game makers, who is it, and whether there is any story in the middle, it quickly became the content of media speculation, and all kinds of statements came one after another.
Although there are many voices, for us, the user experience is the most important. Only two days later, the EMUI team quickly launched a patch package of about 2MB, which quickly solved the problem. Our quick response ability has won almost unanimous praise from consumers.
This rapid patch upgrade capability is unique to Huawei in the terminal field, and other Android manufacturers do not have it. Behind this is the software engineering capability that the EMUI team has been building for several years: three hard-core capabilities: code common trunk, full-stack architecture decoupling and version automatic generation.
If a worker wants to do a good job, he must sharpen his tools first, and software engineering ability is the "tool" of software research and development. Strong software engineering ability is an important part of software research and development, and it is also a hard-core technology of software research and development.
More than three years ago, EMUI had different backbones based on Hess Kirin, Qualcomm and MTK chips. The backbone is only a physical code warehouse, and the quality is basically not guaranteed. Each product version is a branch pulled from the trunk, and the relevant features are manually selected by product managers and development representatives, which is extremely inefficient and the quality cannot be guaranteed. It seems to be "efficient" from the perspective of a single product, but it is almost difficult to achieve the consistency of functional characteristics and the inheritability of quality of each product. No stock model can upgrade the new version of the whole network. This makes the whole software team have a heavy workload, but it is inefficient and has no inheritance, which makes the team exhausted and unable to cope with the upcoming massive delivery.
Since the project of EMUI5.0 version 5.0 was established, the delivery has adopted a full solution operation mode, which integrates user interaction, OS (operating system), Hisilicon, communication protocol and Android native modules into unified planning, homologous design, homologous development and homologous testing. Solution dependencies are well identified and managed, and plans are aligned. At the same time, we set the target of sharing the backbone of HiSilicon Kirin and Qualcomm (MTK platform will be merged into the backbone in 2018, and at this point, one backbone of Huawei EMUI is compatible with three major hardware platforms). The code merging into the backbone is strictly controlled and the quality is given priority, which ensures the code quality of the backbone.
There is no need to pull branches for each product, and the version can be published directly according to the corresponding product at the release point. The backbone version can achieve a full round of automatic testing every week, with better quality and more efficient delivery, and this ability can be continuously passed down. Although the realization of the common trunk has caused a lot of "torture" to the product line and the software R&D team, today’s gains make all the pain worthwhile.
Architecture is the skeleton of software, the foundation and unified language for software teams to work together. Based on Android native system, EMUI has a total code line of over 100 million, which is almost twice that of wireless product line. Such a large-scale code needs to be upgraded in a big version every year, and the amount of changes and additions of Android native code exceeds tens of millions of lines every year. The ambiguity and coupling of architecture make the development of new functions, the modification and change of existing functions, and even the repair of bugs extremely painful, and the flesh and blood skeleton is a mess, which affects the whole body. In fact, such a system has been difficult to maintain and develop continuously. The choice and decoupling of architecture has become a necessity.
I am especially grateful to Yu Zong and He Gang, president of the mobile phone product line, for their support and trust. At that time, when human resources were limited, I was allowed to take out 40% of my manpower to start the architecture decoupling and reconstruction of the entire EMUI stack. The Architecture Department took on this historical mission, and Hu Zheng, an 8-level expert in software, started the architecture decoupling work for more than three years. After many discussions and fierce debates, this work finally lasted for the past few years and made great contributions to the delivery efficiency and quality of EMUI today.
A clear and decoupleable architecture is the unified work foundation within the software team and the common language for our cooperation with the industry, which makes it possible for us to "draw out" the corresponding Android components. This is also the basis for EMUI to quickly replace file system, intelligent scheduling system and realize full-stack patch in the past few years. The new version of Google is also absorbing our excellent practices. We are at the forefront of the industry in the ability of system-wide decoupling.
Finally, the automatic generation ability of the version. The requirements of mobile phone versions are complex, including open channels, operator channels and other different versions. Plus a combination of high, medium and low-grade machines, there are tens of thousands of versions released every year. Version management and publishing are extremely complicated. CCM (Automatic Version Compilation and Configuration Platform) developed by CBG software team has made great contributions to solving this problem. Open market, operator market, different countries, without configuration, etc. are all abstracted as configurable parameters, and the total number of participants is several thousand. When a version needs to be published, according to the demand of the target market, after selecting and modifying the corresponding configuration, one-click configuration distribution can generate the corresponding version. The whole process is efficient and of high quality. Since this system went online, we have saved 200+ software configuration manpower, and it has been running smoothly with zero errors.
At present, the daily users of EMUI are close to 500 million. With the release of each new version, in addition to supporting the launch of new machines, more and more models can be upgraded at the same time, which has directly benefited consumers, and we have gained more and more support and recognition from consumers.
In recent years, in the field of software development, internetization and DevOps are very hot topics. I want to talk about some experiences in the practice of EMUI software development. The full stack of EMUI includes APPs, middleware and OS. If R&D personnel are all required to be young, and IPD must be abandoned and the iterative method of the Internet is adopted, this practice is not reasonable, and it is easy to mislead the team and leaders about EMUI software R&D..
It will be a disaster to unilaterally carry out rejuvenation and iterative development before software engineering capabilities such as common backbone and architecture decoupling are available. It is right that upper-level APPs need young people and fast iteration, but middleware and OS need experience first and innovation second. Without years of experience, it is impossible to do the underlying software well. Without a good bottom layer, the application experience of the upper layer will not be much better.
Today, when the EMUI architecture is completely decoupled, APPs has developed independently from the EMUI platform, and the idea of architecture decoupling has also been implemented in the development and implementation of APPs, realizing independent planning, development and delivery on the shelves, as well as rapid iteration of the full-featured team, which is very close to the Internet development model. The basic platform of EMUI is still based on IPD process, which is developed and accumulated in a down-to-earth manner as planned to achieve the best quality. It is misleading to think that talking about IPD without talking about DevOps is a representative of backwardness. IPD and DevOps are actually tools. The key is to accurately grasp the business demands and choose the most suitable mode and tool according to the business characteristics.
In 2019, with the extension of the replacement cycle and the slowdown of the growth rate, the mobile phone industry has entered a state of saturated competition. On the basis that the hardware has been extremely competitive and gradually got rid of its competitors, Huawei software has begun to exert its strength continuously, and the close cooperation between software and hardware will make the competitiveness of Huawei terminals continue. We have reason to look forward to the future of Huawei terminals!
Always focus on the user experience, consolidate the underlying core capabilities, continuously improve the software engineering capabilities, and build an open ecology. We are confident that EMUI will become the hard power of Huawei mobile phones. The road to user experience improvement and innovation is endless.
AION UT
Haobo HL
Guangqi Honda Ye P7
Platinum 3X
