What’s needed for thorough, comprehensive testing of E-MTBs? Is it possible to objectively denote a battery’s range and is subjectivity an inherent part of every single lab test? We see it as our responsibility to unveil the myriad of practices and myths in the industry and give you an in-depth look at the discrepancies in testing. After all, knowing what’s behind the sales pitch and the stats is the best way to reach your own decisions.
We’re convinced that group tests fulfill a distinct purpose: to purvey concise, concrete advice and direction to assist in your purchasing decisions. But our honest, no-holds-barred approach to group tests means we’ve got to have the balls to address poor products, flaws in concepts and weaknesses. Painting a pretty picture won’t help anyone. We sacrificed five-figure advertising revenues, encountering irate advertisers who, at times, were initially distraught over honest and, in some instances, clearly critical reviews. However, as discussions ran on, it emerged that our test feedback wasn’t all that unwelcome – in certain cases, it’s even been adopted by the manufacturers and fed back into the next year’s models.
We wouldn’t be satisfied with our work if we simply pigeonhole the whole test fleet with a rating shown in stars, how can a single number answer the fundamental question: who is this product best suited to?
Pseudo-objectivity and bulimia: every lab test is subjective
For too long the bike industry tried to fabricate sober neutrality, making bikes comparable through complex point systems and calculations of various factors to conjure up a final figure – especially in the German magazine landscape. A points system with clear criteria may be quantifiable and able to be rolled out time after time, but does it reflect reality? And do the figures really mean anything to anyone?
Out of an essentially limitless number of parameters, a lab test relies on just a curated selection of factors. These purportedly objective factors only reflect a fraction of reality – the part that those performing the testing have deemed the most important. Someone else might have made a different selection. As such, it can be argued that these constitute opinion rather than fact. Test victories can therefore be bagged purely by optimizing a bike to match the select test criteria, which also puts pressure on the manufacturers who’re forced to ‘prove’ their products in these allegedly objective tests in order to keep pace with the rest of the market. Unsurprisingly many manufacturers now equip bikes to satisfy such designated criteria, optimizing them to meet testing standards and get those much-needed ticks of approval. It’s like those school pupils diligently learning facts off by heart without any regard for context; the tragic thing is that neither the teacher nor the pupil gains anything other than a good mark.
People will cheat if they see a route to a higher score
The likelihood that you didn’t cheat at school on at least one occasion is fairly slim. A learn-by-numbers scoring system comes hand in hand with the incentive to ‘optimise’ certain factors to get a better score – the reward of a high mark is enough to satiate the academic institutions. In the past we’ve noticed how brands have submitted test bikes with special motors specifically for tests that have a higher power than the series model. It puts the competition on a back foot, and generates an advantage that may have lead to a higher mark or even an unjust test victory, further reinforcing the need for real-world testing.
Can predictions made inside labs really work outside?
Back when we set up E-MOUNTAINBIKE Magazine in 2013, we had some pretty vocal discussions about the pros and cons of testing inside labs. We took a closer look at the standards and how data is collected, wondering whether to embark down that path. Readers love stats; they help with credibility and give a confirmation of competence. Unfortunately, many customers let themselves be blinded by graphics, diagrams, tables and lab values in the search for a quick and easy answer. That’s why we decided against them, because we’re not here to impress people with “our intelligence” but want to deliver real value. Our test philosophy – honest instead of simple, differentiated instead of “super”!
Luckily, both you and the industry are becoming more aware that reality is usually significantly more complex, and that a bike, its capabilities and characteristics can’t be expressed with just numbers. In the new world of E-MTBs, categories and old-style testing methods are even less useful. What use is the theoretical knowledge of knowing how stiff the headtube on a mountain bike is if its plus-size tires mean it wallows on the trail? Forget fake news, this is called useless knowledge.
There’s little point in purely measuring the stiffness of a seatpost in order to determine how comfortable a road bike is. There are other factors that contribute to this, such as the extent of damping from the tires, any vibration-reducing design features, plus the choice of material. These frequently can’t be assessed in a lab, but does that mean we shouldn’t consider them as decisive in a bike’s performance?
In short: it’s about the bike as a whole, and how it rides in reality. By all means rely on data, but please weave it into a contextual narrative that can resound with us riders.
Experience, the signature and pseudo-professionalism
A bike always comes bearing the hallmark of the development team behind its construction. Once you know the team at Specialized, you’ll see exactly why the Levo Turbo is how it is – same with Haibike, CUBE, MERIDA and the rest. Each manufacturer has their priorities and focal points. If they’re all constricted to these rigid test parameters, then the individual characters of each bike will be lost and we’ll be faced with a pseudo-professional perspective of comparability.
The mother of all questions: ‘How far will I get after charging the battery?’
Who still believes that fuel consumption figures provided by the car industry are the actual reflection of reality? If you answered ‘yes’, you can move onto a different article. For the rest, take note: they’re significantly more exact than those relating to the range on an E-MTB. Why’s that? Well, when it comes to E-MTBs, humans play a pivotal role.
More than 20 factors influence the range of an E-MTB. And each battery is only as good as its worst cell. Then add in the fact that many system manufacturers obtain battery cells from various suppliers, meaning that even the same line of models may have genuine variations in the battery capacities. On top of the differences in battery capacity, there’s also differences in the motor current uptake – i.e. the amount of power it uses. As such, it’s scientifically and methodically wrong to define range based on one test model. An absolute ranking for ranges would inevitably be based on non-scientific methods and would be misleading. By the way, the same thing goes for quoted ranges where, thankfully, only a few manufacturers continue to claim ranges of “up to 100 km”.
Dreaded in every lab: the human influence
While influences from the bike, tires, the system and the route can be vaguely put into figures, the task becomes significantly more complex when you add in the human influence. Every motor has an optimal cadence when it’ll be at its most efficient. Unlike driving a car, it’s the human who has to do the pedaling on an E-MTB – and no two people ride the same. A circular pedal stroke generates a constant load and therefore has a consistent impact on power and the motor’s use of energy – unlike a more impulsive stroke (i.e. inconsistent load). Lab tests also neglect to test the impact of the motor’s cooling process and heat dissipation, for which different manufacturers have developed special systems. Over a long period, batteries tend to gain heat, and on cold winter days they’re forced to work outside of their optimal temperature zone. If two bikes are subjected to the same load, it might look like they’re comparable on paper but in reality there are fundamental influences that have a decisive impact on a bike’s range. So even if a bike’s motor is denoted to be more efficient in the lab, it may have a much more limited range in reality – this leads to confusion and frustration for the customer.
It’s not just about individual parameters and the overall package has to be right! In the worst case, a larger range can make an ebike worse due to the additional weight and imbalance in weight distribution. The right question to ask is, “Which battery concept fits me?” and not, “Which eMTB has the biggest range?” In our know-how feature, we clear up the different battery concepts on the market and help every ebiker find the one that works for them.
We’re sorry!
Even in real-world testing, we’re loathe to say it but there are still a handful of elements that influence results: how tired the test riders are after working their way through more than 10 test bikes; fluctuations in the air temperatures; ideal tire pressures for each individual tire; the chosen gear; and the cadence. Yet external conditions and the control of the battery and the motor from one and the same model lead to irregularities, which hamper the pursuit of giving a general conclusion.
We won’t proclaim to give you solutions. Instead, we strive to provide answers that are as useful and as reflective of reality as possible. And, sadly, that also means admitting that some questions can’t be met with a simple figure or answer.
Phew, that was a long, drawn-out explanation – but this is sort of information that we perceive to be crucial for buyers as well as invaluable for future R&D. Scientifically measured values aren’t game-changers on their own, and lead to confusion when they’re not viewed in the right context. What gets measured in the lab has to be proven in practice. In the wild, it quickly becomes clear that the interplay between different factors, influences and parameters is significantly more complex than can be emulated in the best labs. Don’t put blind trust into these professed facts and pick a bike based on stats. The question of range and comparable data are on a different tangent to what really counts: the individual character of the bike, its strengths and weaknesses and, of course, the satisfaction that you can get from an E-MTB! And this is what we focus on in our group tests! At the end of the day, it’s not about what the bike is capable of in the lab, but out in nature, being used for real on the trails.
Did you enjoy this article? If so, we would be stoked if you decide to support us with a monthly contribution. By becoming a supporter of E-MOUNTAINBIKE, you will help secure a sustainable future for high-quality cycling journalism. Click here to learn more.
Words: Robin Schmitt Photos: Christoph Bayer, Valentin Rühl
